Which provider caused the routing leak?

[Xaos]

Just wondering if someone has more information about the global internet outage, I keep seeing people mentioning "a major provider" or "the network provider" but I can't really find anymore information about the source of the issue.

 

[garindean]

CloudFlare

 

[Xaos]

CloudFlare

"This leak is impacting many internet services including Cloudflare. We are continuing to work with the network provider that created this route leak to remove it."

They are referring to an external provider.

 

[garindean]

"This leak is impacting many internet services including Cloudflare. We are continuing to work with the network provider that created this route leak to remove it."

They are referring to an external provider.

According to cloudflare, "BGP sessions with AS396531 caused the leak."

 

[Ghast]

CloudFlare.

 

[Xaos]

According to cloudflare, "BGP sessions with AS396531 caused the leak."

Thanks. Did some research and this is traced back to "Allegheny Technologies" ( https://en.wikipedia.org/wiki/Allegheny_Technologies)

The issue

 

[Xaos]

As said here it was a Chinese ISP

You realize that is from June 6 right?

 

[Xaos]

Looks like everything is back to normal

 

[drizzy.vip]

It was most likely a route optimiser (likely related) leak which was definitely made worse because Verizon are incompetent fucks that don't properly filter their clients lmao

 

[Xaos]

It was most likely a route optimiser (likely related) leak which was definitely made worse because Verizon are incompetent fucks that don't properly filter their clients lmao

More detailed explanation:

On Mon, Jun 24, 2019 at 08:18:27AM -0400, Tom Paseka via NANOG wrote:
> a Verizon downstream BGP customer is leaking the full table, and some more
> specific from us and many other providers.
It appears that one of the implicated ASNs, AS 33154 "DQE Communications
LLC" is listed as customer on Noction's website:
https://www.noction.com/clients/dqe

I suspect AS 33154's customer AS 396531 turned up a new circuit with
Verizon, but didn't have routing policies to prevent sending routes from
33154 to 701 and vice versa, or their router didn't have support for RFC
8212.

I'd like to point everyone to an op-ed I wrote on the topic of "BGP
optimizers": https://seclists.org/nanog/2017/Aug/318

So in summary, I believe the following happened:

- 33154 generated fake more-specifics, which are not visible in the DFZ
- 33154 announces those fake more-specifics to at least one customer (396531)
- this customer (396531) propagated to to another upstream provider (701)
- it appears that 701 did not sufficient prefix filtering, or a maximum-prefix limit

While it is easy to point at the alleged BGP optimizer as the root
cause, I do think we now have observed a cascading catastrophic failure
both in process and technologies. Here are some recommendations that all
of us can apply, that may have helped dampen the negative effects:

- deploy RPKI based BGP Origin validation (with invalid == reject)
- apply maximum prefix limits on all EBGP sessions
- ask your router vendor to comply with RFC 8212 ('default deny')
- turn off your 'BGP optimizers'

I suspect we, collectively, suffered significant financial damage in
this incident.

Source: https://mailman.nanog.org/pipermail/nanog/2019-June/101589.html