Battling the "Slashdot Effect"
Tuesday, January 26. 2010
I've always wondered what exactly one can do to combat high volume traffic from slashdot, digg, reddit and other populate news sites - known commonly as the Slashdot effect. The other day, I got a first hand opportunity to find out. One of our clients sites was posted to multiple news site - so what better way to try out and see what works in a real life slashdotting. Below, is the solution I ended up with that seemed to remove a significant amount of strain from the server without costing any additional money.
As I was doing some research, I came across some creative uses of the Coral CDN - and I have previously used lighttpd for high volume sites - so I put the two together. For those of you that don't know about Coral CDN, its a peer-to-peer content distribution network. To take advantage of the CDN its easily - simply append .nyud.net to the end of any dns name and that's it! Since the site I was dealing with was fully dynamic, I did not have a lot of options that could be quickly implemented.
Once you get lighttpd setup, here are the example config bits to setup and perform the redirect:
# make sure this isn't CoralCache requesting content
$HTTP["useragent"] !~ "^CoralWebPrx" {
# make sure that this wasn't sent back to us from CoralCache
$HTTP["querystring"] !~ "(^|&)coral-no-serve$" {
url.redirect = ( "^/.*" => "http://%1.nyud.net$0" )
}
}
The above will redirect all requests that do not have the Coral CDN user agent. It's important to add this or else you'll get a redirect loop. Also - make sure you enable the "rewrite" and "redirect" server modules.
For those of you that are using apache, you can add rules similiar to:
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} !^CoralWebPrx
RewriteRule ^(.*)$ http://yourdomain.com.nyud.net/$1 [R,L]
In the end, caching, static pages, and load balancing is your best bet. But if your in a bind, lighttpd and Coral CDN will do the trick!
