Friday, May 1, 2009

Incident report: denial of service attack against ConnectedPhotographer.com

FROM THE EDITOR-IN-CHIEF

By David Gewirtz

You may have noticed that Connected Photographer has been offline for the past few days. It's back, and what happened makes for quite a story.

About two months ago, I noticed increased traffic on our Web sites -- and the traffic was causing a load on the servers that didn't seem to be right. In particular, I noticed that our email-to-a-friend page was getting accessed repeatedly, at a rate disproportionate to what regular traffic would generate. I reasoned that a spammer was using the page to send junk mail out through our email-to-a-friend interface, and promptly turned that page off.


"A million individual computers hit our servers in the space of a day."

Traffic was one or two accesses a minute from different IP addresses, all over the world. I traced IP addresses to Russia, Brazil, UK, Turkey, Korea, Ukraine, Australia, Canada. There were also a bunch of IPs that wouldn't give up their real locations.

Current incident

Beginning Tuesday night, performance of our Web servers began to degrade. It took me until Thursday morning to determine that the performance degradation was due to an increase in traffic to a particular set of Web pages. This was, in part, because the server was performing so slowly that accessing any information took a very long time.

Eventually, I was able to determine that the email-to-a-friend page (which no longer existed) was being requested for our Connected Photographer Web site. Each request caused a server error, slowing the system down. Unfortunately, there was no way to stop the server errors, since the code that generated them was compiled into the server's kernel.

Yes, I have access to the kernel code and have added features in the past, but I didn't want to muck with code at such a low level while trying to sustain our level of quality service. It would have just taken too much time.

Through the use of a software firewall, I was able to determine that requests to the email-to-a-friend URL were comining into the server at the rate of thousands of requests per second. I configured the software firewall to ban requests to this particular page, and then ban the IP addresses that originated the request.

However, within about ten minutes, the software firewall ceased to function. It had banned more than 10,000 individual IP addresses, (about 1,000 per minute), exceeded its available memory, and pushed the server to 100% utilization.

I tried re-routing and even turning off the DNS pointing to the server. The requests still kept coming in. My guess is that the URL they were requesting was cached, and so the spamming system knew the IP address, ignoring the DNS completely.

The solution was to change the IP address of the server for other Web sites and then update the DNS records to point to the new IP address. I did not do this for Connected Photographer, and that site remained intentionally offline through much of the weekend.

Key observations

The key observation was the ferocity of accesses. We were being hit by thousands of separate computers per minute, and I tracked more than 10,000 individual computers before the stage-one firewall died. Overall, I estimate somewhere above a million individual computers hit our servers in the space of a day or so.

The second key observation was that our first-line server was unable to withstand the load of such an attack. The only approach that seemed to work was to change the IP on the other sites, and kill the Web site being attacked. We then spent the next few days creating some new technology that sits between the firewall and our primary Web servers, managing and blocking the flow of these attacks. So far, it's working quite well and (knock on wood), I'm hoping it'll keep the wolves at bay, for at least a while.

Cost to us, of course, is we had a dead site for four days, because we couldn't bring it back online without the possibility of a reoccurance of such an attack until we'd put a barrier system in place. Although the attack was clearly spam oriented, it's still undeniably a distributed denial of service because service was denied.

Since we were able to sustain only about ten minutes of direct flow observation, and the attack lasted hours, I suspect millions of IP addresses are firing. Further, since Connected Photographer is a relatively minor site compared to, say, Google, I suspect each computer that fired on our server also fired on thousands of other servers.

Finally, because of the absolute ferocity of this attack, coming from so many computers, it took us a couple of days to engineer a robust defense. Our sites are merely informational. But were an attack like this to hit a server that was important to infrastructure, the damage could be devastating.

I recently wrote an article in Counterterrorism Magazine about how cyberterrorism can damage infrastructure. This week, we witnessed the power of such an attack.