[afnog] Another Perspective - Kentik's View on the Facebook Outage
Markus Akena Wipfler
markus.wipfler at gmail.com
Thu Oct 7 11:33:09 UTC 2021
ofc A records where removed during outage. either by bad design, sabotage
or whatever . @Paul just sub withdrawn with removed from your statement and
u make the same point I did. either way it's bad design and I doubt that FB
makes that kind of mistake.
Also no one is connecting the dots what is happening to FB as a whole atm..
On technical level such mistake is unacceptable when each 1 hour of
downtime costs about 1 billi USD...
I find it highly unlikely that this was an automation mistake coz some one
didn't doublecheck their config.
Further more the /19 supernet was still in GRT during whole outage and only
the more specific DNS prefixes were missing.
Further more I assume without checking, that the DNS IPs are anycasted so
it would mean that the automation script has access globally to all BGP
If you want I can sell you a nice fridge on the north pole...
On Thu, 7 Oct 2021, 12:38 pm , <sm+afrinic at elandsys.com> wrote:
> Hi Paul,
> At 01:50 AM 07-10-2021, Paul McMaster wrote:
> >No DNS records were removed. Certain prefix's were withdrawn
> >globally by an automation config mistake. By design their BGP for
> >the DNS prefix's were withdrawn after X amount of time as their core
> >network was not visible.
> This is the reply from 220.127.116.11 around the time of the outage:
> fb.com. 172800 IN NS a.ns.facebook.com.
> fb.com. 172800 IN NS b.ns.facebook.com.
> fb.com. 172800 IN NS c.ns.facebook.com.
> fb.com. 172800 IN NS d.ns.facebook.com.
> CK0POJMG874LJREF7EFN8430QVIT8BSM.com. 86400 IN NSEC3 1 1 0
> - CK0Q1GIN43N1ARRC9OSM6QPQR81H5M9A NS SOA RRSIG DNSKEY NSEC3PARAM
> CK0POJMG874LJREF7EFN8430QVIT8BSM.com. 86400 IN RRSIG NSEC3 8 2 86400
> 20211011042403 20211004031403 39343 com.
> EM2SGU4GV6DJ5SAH1A2PLQ611POSJBEE.com. 86400 IN NSEC3 1 1 0 -
> EM2TBOBUUOD4AQ2HAJDUQLOREPPGU6DG NS DS RRSIG
> EM2SGU4GV6DJ5SAH1A2PLQ611POSJBEE.com. 86400 IN RRSIG NSEC3 8 2 86400
> 20211010043214 20211003032214 39343 com.
> couldn't get address for 'c.ns.facebook.com': failure
> It looked like a DNS issue. There is an explanation about what
> happened at
> S. Moonesamy
> afnog mailing list
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the afnog