ಠ_ಠ

[this was originally written as a facebook comment in a conversation about Dyn and “why is everyone talking about this outage in connection with the internet of things? why is this one company’s bad day news? why do tech people seem scared?”]

As someone who actually runs servers for a business where if the servers go down we don't make money, and someone who was affected by the Dyn outage, I want to impress upon y'all what happened here, why it matters, and why I'm fucking terrified.

[someone had asked “won’t we eventually patch all vulnerabilities?]

There are an infinite number of vulnerabilities, because more new insecure shit is being plugged into the internet than ever before. Every day more devices go online and get popped. The devices in this botnet are thought to be surveillance cameras and their DVRs. They don't even have a "vulnerability", per se: they have a default password that nobody fucking changes, and so other people can log into these things over telnet and at that point it's just another low-powered linux machine that happens to also be a camera or DVR.

The DDoS landscape has changed. Before it was hacktivists trying to make a point or assholes trying desperately to achieve an erection that would last longer than Ozymandias's. But in the past 10 or 15 years, organized crime got into electronic credit card fraud in a big way, and from there into spamming. Spamming requires having a lot of machines to send from to evade blacklists from people who have made it their literal full-time job to hunt down spammers, so they learned how to set up and manage botnets.

So organized crime syndicates control botnets and realize they could do the same 'protection racket' bullshit they already do IRL on the internet. The racket works like this: an email comes in to [email protected] and says "hey, pay us $1500 in bitcoin or we take down your site." The whatever.org admins ignore it. The botnet spews waves and waves of traffic at whatever.org. whatever.org blocks their IPs. Crisis averted.

Then the criminals discover that NTP, a protocol used to keep people's clocks in sync over the internet, will produce a lot of text output if you send it a small input, and it uses a lower-level protocol called UDP, which allows you to pretend to be another site. So instead of having their bots hit whatever.org directly, the bots hit a bunch of NTP servers and tell them "hey, i'm actually whatever.org, so here's a few packets" and the NTP servers send a lot more packets onward to whatever.org. This is called reflection or amplification.

And since you've built this infrastructure, you can monetize it even more. Lizard Squad's booter service, for instance: you pay them some bitcoins and buy seconds of attack time that you can use to target the asshole who's beating you in League of Legends or your business competitor when they're holding their product-launch keynote. DDoS becomes accessible to anyone with a grudge and the willingness to buy BTC and risk jail time.

And people work on ways of protecting from this. There's a movement among network operators to just block shit with forged From IPs. It's called BCP38. They start deploying it. It doesn't work, because not everyone's doing it. They also start companies like Akamai and Cloudflare, where the whole business model is "listen, we don't care _what_ you host, but you'll be fucking sure it'll stay up." These companies run geographically distributed content distribution networks (CDNs) and obscure the actual destination of the traffic. (For instance, if you go to something like krebsonsecurity.com, you're going to one of several [formerly-Akamai-now-Google?] datacenters based on geographic location and network peering agreements; they all talk to the actual servers hosting krebsonsecurity, but the krebsonsecurity servers aren't publicly accessible to everyone and are therefore harder to discover for DDoS attackers -- but all this complexity is hidden, because we've spent decades learning how to architect these things in a way where it's all invisible to users. CDNs started as a way to make pages faster, and now they're one of the things that are making the web able to withstand attacks like this. But only parts of it -- DDoS protection isn't cheap. Companies can afford it. People like Brian Krebs can't, he gets his services because he does good work in the industry and offers a challenge for them.)

But these are all "normal" DDoSes. What happens if you have a hundred thousand DVRs on high speed internet connections that you control, because asshole manufacturers made them insecure pieces of shit and consumers don't know enough to know that this is a thing they have to spend extra money on to get a non-hostile version?

That's what happened to get Brian Krebs kicked off Akamai. Dude published a work naming and shaming the operators of a DDoS-for-hire operation. Turns out, it looks like that group wasn't using a standard shitty-windows-or-linux-boxes-reflecting attack network. Or that group had some friends with a surprise: a botnet made of a bunch of IP cameras and DVRs with default usernames and passwords that were exposed to the internet. They'd find one, log in, run malware on it, and that malware would connect up to a command and control server to listen for targets to attack. The malware would also start scanning the internet for more cameras to attack and add. It turns out that you can scan the entire 4 billion IPs of the internet in a little over an hour on what passes as a "decent connection" at a university or tech company these days. If you have thousands of machines scanning a part of it, you need even less bandwidth for each scanner. This gets turned on krebsonsecurity.com, which was hosted by Akamai as an act of equal parts charity and braggadocio, and Akamai... well, they tried to defend themselves against it. They tried to keep the site up. And then they dropped him because it was affecting actual paid customers in their datacenters. The fight had collateral damage for a company whose literal job is providing DDoS immunity.

So the folks that wrote Mirai (that botnet that kicked Krebs off the internet) realized that the only way they weren't going to get arrested for it would be to open source it, have it on multiple thousands of computers across the world, so that having the source code on their machines wouldn't be de facto evidence of being the creators and original deployers of it.

And now that malware is out there, being modified by people to pick up new shitty-IoT-devices, and someone who may or may not even be the original group attacks Dyn.

Dyn basically does one thing: they do DNS. They started as DynDNS, and I have a soft spot in my heart for them as a kid who wanted to run linux servers out of his house but couldn't convince my mom to spend $10/year on a domain. But they expanded and grew, and it turned out, they're _pretty fucking good_ at DNS. And DNS is hard. It's an old thorny protocol from the days when everyone on the Internet could trust everyone else not to be cocks. Security has been bolted onto it, but everything about it is really easy to mess up, and it's a distributed decentralized system with tons of caching at every layer, and changes move at a snail's pace. Normally, your DNS records are cached on the order of hours or days. Nothing else moves at that slow a speed anymore.

And if you have global customers you really need multiple global DNS servers to be performant, and you need those DNS servers to keep working no matter what so that you can redirect customers to an "oh shit everything went wrong sorry please we'll be back soon" server if your datacenters burn down. And Dyn grows, and gets APIs together that let it plug into Amazon's cloudy stuff and other people's cloudy stuff, and startups come into existence that go "you know, we could take this $30,000 in seed funding and set up datacenters, or we could just use AWS and Dyn." And these companies grow, and eventually they start making some serious money and people think "you know, we should probably have multiple DNS services backing us up." And maybe they try adding Hurricane Electric, and then Hurricane Electric has an outage where they pretend they don't know where your site lives anymore -- it knows it has a record, so everyone's computer caches the result, and the location is "fuck you, dunno". So the companies think "okay, maybe we _don't_ need a second DNS provider."

So Dyn grows, and eventually gets some acquihires and new departments, and one of them is infosec. And a dude gives an attack at a NANOG conference today or yesterday (NANOG being "the greybeards who manage the core routing to interconnect the autonomous systems that are the reason we call it 'the internet'") about some DDoS research they've done at Dyn.

And someone with control of a botnet goes "fuck these guys in particular" and points the full power of their tool at Dyn. And Dyn is full of people who handle this shit for a living. And they take actions, they shift their paths around, doing the BGP equivalent of going serpentine, trying to find a way to make their clients' traffic go through but not get crushed under the weight of the garbage they're hit by. And they don't succeed.

Two companies whose core competency is "we keep this service going on the internet," with bright people, wads of money, and racks of physical machines, and they can't stay up. DDoS has gone from "annoying nuisance" to "companies can work around it" to "nothing is safe."