[afnog] Another Perspective - Kentik's View on the Facebook Outage

Thu Oct 7 11:33:09 UTC 2021

ofc A records where removed during outage. either by bad design, sabotage
or whatever . @Paul just sub withdrawn with removed from your statement and
u make the same point I did. either way it's bad design and I doubt that FB
makes that kind of mistake.

Also no one is connecting the dots what is happening to FB as a whole atm..

On technical level such mistake is unacceptable when each 1 hour of
downtime costs about 1 billi USD...

I find it highly unlikely that this was an automation mistake coz some one
didn't doublecheck their config.

Further more the /19 supernet was still in GRT during whole outage and only
the more specific DNS prefixes were missing.

Further more I assume without checking, that the DNS IPs are anycasted so
it would mean that the automation script  has access globally to all BGP
speakers. lol

If you want I can sell you a nice fridge on the north pole...

On Thu, 7 Oct 2021, 12:38 pm , <sm+afrinic at elandsys.com> wrote:

> Hi Paul,
> At 01:50 AM 07-10-2021, Paul McMaster wrote:
> >No DNS records were removed. Certain prefix's were withdrawn
> >globally by an automation config mistake. By design their BGP for
> >the DNS prefix's were withdrawn after X amount of time as their core
> >network was not visible.
>
> Yes.
>
> This is the reply from 192.42.93.30 around the time of the outage:
>
>    fb.com.                 172800  IN      NS      a.ns.facebook.com.
>    fb.com.                 172800  IN      NS      b.ns.facebook.com.
>    fb.com.                 172800  IN      NS      c.ns.facebook.com.
>    fb.com.                 172800  IN      NS      d.ns.facebook.com.
>    CK0POJMG874LJREF7EFN8430QVIT8BSM.com. 86400 IN NSEC3 1 1 0
> -   CK0Q1GIN43N1ARRC9OSM6QPQR81H5M9A NS SOA RRSIG DNSKEY NSEC3PARAM
> CK0POJMG874LJREF7EFN8430QVIT8BSM.com. 86400 IN RRSIG NSEC3 8 2 86400
> 20211011042403 20211004031403 39343 com.
> Qlm6nJK1+jpkxCOGZKqj121icHf+TcKcPrQgGqA3yYsOiqpB7TjVNnDq
> oZVDtIMuNsUKsAxN2mLlbM8XF8xLf5ZD1Z3H498YrDssqNh1VWeUlJgj
> /9ls1iPe2CsB1PNXzU/cXGdrQiiikkHK//3J0IHSYmsHNAwbC0y4/FsS
> U+ZsZisaa2G5FSDdr+Fqlf6m+jmwoxTEsexS9ivi1SaaDQ==
> EM2SGU4GV6DJ5SAH1A2PLQ611POSJBEE.com. 86400 IN NSEC3 1 1 0 -
> EM2TBOBUUOD4AQ2HAJDUQLOREPPGU6DG NS DS RRSIG
> EM2SGU4GV6DJ5SAH1A2PLQ611POSJBEE.com. 86400 IN RRSIG NSEC3 8 2 86400
> 20211010043214 20211003032214 39343 com.
> rlEjMXkv5n24OweHGYTXgU+RJn2PZeRbJzN+afEukSeYjon8DL5he5wi
> Jj/DskPoR+x1M+tmlyU3vCWoeQF+iFLNoMnpYZ35vulCxqDnqdbc26cP
> ubBZXXQqBywjw+eK0T6a/5E4YXaZBGOhe2C+Ye2BDvYyMEQAoTr6vaf9
> RfBBsFexcRckKaKZuwNFX1rNVNCsmIPn9iYI27U+NAPGhA==
>    couldn't get address for 'c.ns.facebook.com': failure
>
> It looked like a DNS issue.  There is an explanation about what
> happened at
> https://engineering.fb.com/2021/10/05/networking-traffic/outage-details/
>
> Regards,
> S. Moonesamy
>
>
> _______________________________________________
> afnog mailing list
> https://www.afnog.org/mailman/listinfo/afnog
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.afnog.org/pipermail/afnog/attachments/20211007/29a7777e/attachment-0001.html>