Friday, May 1, 2009

Incident report: denial of service attack against ConnectedPhotographer.com

The solution was to change the IP address of the server for other Web sites and then update the DNS records to point to the new IP address. I did not do this for Connected Photographer, and that site remained intentionally offline through much of the weekend.

Key observations

The key observation was the ferocity of accesses. We were being hit by thousands of separate computers per minute, and I tracked more than 10,000 individual computers before the stage-one firewall died. Overall, I estimate somewhere above a million individual computers hit our servers in the space of a day or so.

The second key observation was that our first-line server was unable to withstand the load of such an attack. The only approach that seemed to work was to change the IP on the other sites, and kill the Web site being attacked. We then spent the next few days creating some new technology that sits between the firewall and our primary Web servers, managing and blocking the flow of these attacks. So far, it's working quite well and (knock on wood), I'm hoping it'll keep the wolves at bay, for at least a while.

Cost to us, of course, is we had a dead site for four days, because we couldn't bring it back online without the possibility of a reoccurance of such an attack until we'd put a barrier system in place. Although the attack was clearly spam oriented, it's still undeniably a distributed denial of service because service was denied.

Since we were able to sustain only about ten minutes of direct flow observation, and the attack lasted hours, I suspect millions of IP addresses are firing. Further, since Connected Photographer is a relatively minor site compared to, say, Google, I suspect each computer that fired on our server also fired on thousands of other servers.

Finally, because of the absolute ferocity of this attack, coming from so many computers, it took us a couple of days to engineer a robust defense. Our sites are merely informational. But were an attack like this to hit a server that was important to infrastructure, the damage could be devastating.

I recently wrote an article in Counterterrorism Magazine about how cyberterrorism can damage infrastructure. This week, we witnessed the power of such an attack.