Varnish is a very powerful caching reverse proxy which features a configuration language (VCL) and tools to analyse traffic. I use it primarily to cache anonymous Drupal page requests so that a site can handle a massive spike. When your application is configured correctly to work with Varnish, you will soon find that the next major bottleneck to deal with is your server’s Internet connection :–).
In the last post we went over how to use GoAccess to analyse Apache logs to find potential malicious clients. Varnish can lend a hand towards this purpose as well, and can even be used to thwart an attack.
The primary tool I use for this is varnishtop, in particular to see incoming request headers.
An example output with me just running curl against my frontpage over and over again:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
So with this information I can see the user agent most prominently displayed. Now, assume there was a really unsophisticated script kiddie that wanted to take my site down. He just wants to generate a bunch of pageloads using some random script he downloaded off a site somewhere:
1 2 3 4
Nagios wakes me up because my uptime is starting to suffer and once I come to the conclusion this is a problem regarding traffic levels and I analyse what’s coming in, I will see this bubbling up to the top of the varnishtop output:
Sweet! Now I can use VCL to block the offending user-agent.
1 2 3 4 5 6 7
I reload varnish, and the attacker’s terrible script starts returning this:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
Of course this is a very hypersimplistic example. However, I have used varnishtop and VCL-driven 404s in this fashion several times to great success against several patterns:
- Accept-Language headers unique to a certain region
- Obviously nonexistent URLs, protecting the application from needlessly running code only to spit a 404
- Videos embedded in a frontpage instead of using a streaming service
- This technique shouldn’t be used for a long-term fix, however can be a lifesaver when in the middle of an outage. Consider moving to a CDN if these issues become common.
- Look for out-of-the-ordinary but prominent headers in incoming requests.
- Be careful to not accidentally block legitimate traffic.
Got a question or an interesting Varnish-related story to tell? Please let me know in the comments!