DynDNS (or just Dyn now) got blasted with #DDoS twice last Friday. Since Dyn is the major DNS provider for Twitter, Github, and Spotify, the knock-on effects have had a global reach. Here’s a rather comical exchange among typical users scratching their heads Friday morning:
@githubstatus: GitHub employee here. We’re monitoring an incident with our upstream DNS provider
reply @alfalfasprout [-] Hahahahaha you do realize Twitter is one of the affected sites, right?
But seriously, Dyn is a big provider, and their being offline has real impact. PagerDuty is one of the affected sites, and many people rely on alerts from their service. No one knows many details about the Dyn attacks yet.
No one has claimed responsibility, and Dyn has been somewhat quiet about the attack vectors, but has said that possibly 100,000 hijacked connected devices could have been used in the attack.
The attacks could be fallout from the Mirai IoT Botnet assault against Brian Krebs earlier this month. As Krebs himself notes, the attacks started within hours of a DynDNS researcher, Doug Madory, presenting a talk (video link here) at NANOG about DDoS attacks. Also, according to Krebs, the 620GB Mirai attack against krebsonsecurity.com came just hours after he and Madory released an article looking into some of the shady dealings in the DDoS-for-hire industry.
Maybe Madory is actually the one being targeted. Total speculation there. He and Krebs seem to be in the crosshairs; it could be that someone is hoping to silence them. If so, it seems to be having the opposite effect. But attribution, especially with DDoS, is very problematic, and it could be that someone is taking advantage of the atmosphere to target an entirely different party (one of Dyn’s clients, perhaps).
Of course everyone is wondering if the IoT botnet, Mirai, is playing a part in the Dyn attack. Even if it is, the attacker could be anyone, as the Mirai source code and helpful readme post were released to the world a week ago, and are still available on Github (if you can get there right now).
The Mirai bot’s author (Anna-senpai is the only handle we have) showed the Infosec community (or “you retards” as he/she calls us) that it is easy to build a powerful DDoS botnet out of random consumer appliances on the Internet.
The Infosec community has warned about an IoT botnet for years. Some, including myself, were a bit skeptical of IoT DDoS, as many of the early IoT devices were not fitted with general-purpose CPUs or high-bandwidth uplinks.
But that all changed with intelligent devices like digital video recorders and closed-circuit TV systems. Over 100,000 of these devices were seen in the now-famous attack on KrebsOnSecurity and French hosting company OVH.
In addition to the usual barrage of dumb UDP floods, the bot participants used more sophisticated attacks, the likes of which we rarely see. Check out this rogue’s gallery of nausea-inducing, hard-to-mitigate vectors in Mirai.
Nastier HTTP GET Floods
HTTP GET floods were already pernicious. For years, attackers have been able to disable web sites by sending a flood of HTTP requests for large objects or slow database queries. Typically, these requests flow right through a standard firewall because hey, they look just like normal HTTP requests to most devices with hardware packet processing. The Mirai attack code takes it a step further by fingerprinting cloud-based DDoS scrubbers and then working around some of their HTTP DDoS mitigation techniques (such as redirection).
DNS Water Torture
The Mirai bot includes a “water torture” attack against a target DNS server. This technique is different from the regular DNS reflection and amplification attacks as it requires significantly less queries to be sent by the bot, letting the ISP’s recursive DNS server perform the attack on the target’s authoritative DNS server. In this attack, the bot sends a well-formed DNS query containing the target domain name to resolve, while appending a randomly generated prefix to the name. The attack is effective when the target DNS server becomes overloaded and fails to respond. The ISP’s DNS servers then automatically retransmit the query to try another authoritative DNS server of the target organization, thus attacking those servers on behalf of the bot.
GRE (Generic Routing Encapsulation) is a tunneling protocol that can encapsulate a wide variety of network-layer protocols inside virtual point-to-point links over an IP network. Ironically, GRE tunnels are often used by DDoS scrubbing providers as part of the mitigation architecture to return clean traffic directly to the protected target.
The Mirai botnet code includes GRE attacks with and without Ethernet encapsulation. Most public routers will pass along the GRE packet because it is also a widely used protocol for generating VPN connections. The use of GRE could allow significant payloads to be launched, adding processing overhead of IP defragmentation to impact the target.
Updated Layer 4 Attacks
According to Mirai’s creator, the so-called “TCP STOMP” attack is a variation of the simple ACK flood intended to bypass mitigation devices. While analyzing the actual implementation of this attack, it seems that the bot opens a full TCP connection and then continues flooding with ACK packets that have legitimate sequence numbers, in order to hold the connection alive.
It actually could have been worse – I can think of one or two attacks that might be harder to mitigate, but all of these are tough.
The current situation is suboptimal, to say the least; the Internet of Things is just consumer-grade security writ large. These devices don’t have firewalls or enterprise management to clean them up. Nearly all of the devices had port 23 open with default passwords. I would get fired if I left port 23 open on anything, but today manufacturers ship hundreds of thousands of high-powered computer devices that do just that. There’s no oversight here, and no clear path to remediation. There probably isn’t even a way to notify the consumer that their DVR is in your base, killing your doodz.
So, Doc, is there a fix?
At a SecureLink conference last week in Brussels, Mikko Hypponen, Chief Risk Officer of F-Secure, was asked how the IoT botnet should be stopped. His answer was, while he himself is not a huge fan of more regulation, regulation will likely be the fix for IoT security. He pointed out that consumer devices are already regulated for safety and efficiency. No one wants their refrigerator exploding on them (or their smartphone, ahem). If only Internet security could be regulated like other manufacturing processes, we could solve this problem.
Best Practices for DNS?
One of the reasons DDoS attacks keep evolving is that defenders keep evolving as well. You can bet that by next week, companies will be doing a better job with DNS redundancy. Keeping DNS up under sustained attack isn’t easy; if it were, you’d see people doing a better job . . . but if it’s important enough, they’ll get it done.
Dyn customer pornhub wasn’t impacted as badly because their DNS looks like this:
[email protected]:~/projects$ dig @220.127.116.11 ns +short pornhub.com
Note their use of multiple DNS providers. Could this be yet another example of “porn leads the Internet”?