Boom! At long last the eagerly awaited Part 5 of our High Performance WordPress series sees the light of day!
For those who have not been following along, a quick primer on the other parts in the series:
- Part 1 – VPS Setup
- Part 2 – nginx/mysql setup
- Part 3 – WordPress setup & tuning on nginx
- Part 4 – APC Opcode Cache install and configuration
First off I must apologise to those who have been waiting a couple of months for Part 5 to be published. I’ve been exceptionally busy these past few months building themes and only got around to finishing this part in the past 2 weeks. Better late than never I hope!
I’ve also spent the past week or two tweaking and tuning my own VPS which actually back fired a bit the past few days with some outages and sub par performance – kind of ironic for a blog with a whole series on High Performance 🙂 Never the less, it was a very worthwhile experiment which was designed to stretch and tune my varnish config to ensure it is as tight as possible on a low end VPS to ensure you get as much bang for your buck as possible.
What the hell is Varnish Cache?
Quite simply, Varnish Cache is the secret weapon that many of the worlds busiest websites use to supercharge their performance (including WordPress.com! and top WordPress hosts WPEngine). Varnish typically sits in front of your actual web server (in our case nginx but it works equally well with Apache too). Technically Varnish is known as a pure reverse proxy HTTP accelerator. It’s main job is to take the pressure off your web server by caching pages in memory on initial request by a user and then serving these pages directly from virtual memory upon further requests by users so the user never reaches your web server at all. Varnish does a much better job at the simple task of serving http requests as it has been designed from the ground up for exactly this purpose – serving lots of requests for the same content super quickly.
And let me tell you – it is very very fast!
Today we’re going to get you up and running with Varnish sitting in front of your existing nginx powered VPS that we’ve been building up in Parts 1-4 of the series. By the time we’re done you will have what I like to call the Rolls Royce of high perfomance WordPress performance configurations 🙂
Install Varnish
Alright let’s dive in. Step 1 is the install of Varnish itself. To ensure we’re getting the latest and greatest version of Varnish for Ubuntu 10.10 we’ll grab the Varnish version provided by varnish-cache.org directly as follows.
First, ensure you have curl installed if you don’t already:
[sourcecode]
apt-get install curl
[/sourcecode]
Then let’s get on with the Varnish install
[sourcecode]
curl http://repo.varnish-cache.org/debian/GPG-key.txt | apt-key add –
echo "deb http://repo.varnish-cache.org/ubuntu/ lucid varnish-3.0" >> /etc/apt/sources.list
apt-get update
apt-get install varnish
[/sourcecode]
As of the time of writing this should get you Varnish 2.1.3 on Ubuntu 10.10. (You can check this by running
[sourcecode]
varnishd -V
varnishd (varnish-2.1.3 SVN )
Copyright (c) 2006-2009 Linpro AS / Verdens Gang AS
[/sourcecode]
Switch Varnish to Port 80 and nginx to Port 8080
Seeing as we want Varnish to sit in front of nginx we will need to switch Varnish over to port 80 so it intercepts all http requests first. Right now nginx is serving our WordPress website via the standard http port – port 80. As a result, we’ll need to reconfigure nginx to listen on a different port. We’re going to switch nginx to listen on Port 8080 and Varnish will pass requests to it based on the policies with define within Varnish. Thankfully Varnish comes with a very flexible and easy to configure language called VCL. VCL enables you to write policies about how incoming requests should be handled. Let’s switch ports.
By default, Varnish listens on port 6081, we’ll switch to 80. Find this codeblock in:
[sourcecode]
nano /etc/default/varnish
[/sourcecode]
[sourcecode]
## Alternative 2, Configuration with VCL
#
# Listen on port 6081, administration on localhost:6082, and forward to
# one content server selected by the vcl file, based on the request. Use a 1GB
# fixed-size cache file.
#
DAEMON_OPTS="-a :6081 \
-T localhost:6082 \
-f /etc/varnish/default.vcl \
-S /etc/varnish/secret \
-s file,/var/lib/varnish/$INSTANCE/varnish_storage.bin,1G"
[/sourcecode]
and replace port 6081 with 80
[sourcecode]
## Alternative 2, Configuration with VCL
#
# Listen on port 80, administration on localhost:6082, and forward to
# one content server selected by the vcl file, based on the request. Use a 96M
# fixed-size cache file.
#
DAEMON_OPTS="-a :80 \
-T localhost:6082 \
-f /etc/varnish/default.vcl \
-S /etc/varnish/secret \
-s file,/var/lib/varnish/$INSTANCE/varnish_storage.bin,96M"
[/sourcecode]
There is another really important change we made on the last line above. We adjusted the virtual memory allocation for Varnish from 1GB down to 96MB. Why the change? Well I’m assuming that you are running the a low end VPS with maybe 512MB of RAM. If so, you don’t want Varnish consuming too much memory and pushing you into swap file disk thrashing. Keep it low to start with and adjust upwards if necessary. The “varnishstat” command will provide you with all sorts of useful information about how your cache is performing.
The other change you will want to make in this file is to ensure varnishd is set to start at boot
[sourcecode]
# Should we start varnishd at boot? Set to "yes" to enable.
START=yes
[/sourcecode]
Before starting Varnish, we need to switch nginx to Port 8080. Let’s do that now:
[sourcecode]
nano /etc/nginx/sites-enabled/engine.ex.conf (or whatever your config file is called)
[/sourcecode]
In your server block switch the listen port from 80 to 8080.
[sourcecode]
listen 80; ## listen for ipv4; this line is default and implied
[/sourcecode]
becomes
[sourcecode]
listen 8080; ## listen for ipv4; this line is default and implied
[/sourcecode]
At this point let’s restart nginx and varnish
[sourcecode]
service nginx restart
service varnish restart
[/sourcecode]
Now when you run a simple netstat you should see Varnish on Port 80 and nginx on 8080:
[sourcecode]
netstat -anp –tcp –udp | grep LISTEN
[/sourcecode]
Something like this:
[sourcecode]
tcp6 0 0 :::80 :::* LISTEN 30464/varnishd
tcp 0 0 0.0.0.0:8080 0.0.0.0:* LISTEN 30099/nginx
[/sourcecode]
Now it’s time to optimise Varnish to play nice with WordPress!
Optimise your Varnish Config for WordPress
Now it’s time to tune for default VCL to play nice with WordPress. For now let’s go ahead and do that and afterwards we’ll touch on a few key points about this config.
[sourcecode]
nano /etc/varnish/default.vcl
[/sourcecode]
[sourcecode]
# This is a basic VCL configuration file for varnish. See the vcl(7)
# man page for details on VCL syntax and semantics.
#
# Default backend definition. Set this to point to your content
# server.
#
backend default {
.host = "127.0.0.1";
.port = "8080";
}
acl purge {
"127.0.0.1";
}
sub vcl_recv {
# Add a unique header containing the client address
remove req.http.X-Forwarded-For;
set req.http.X-Forwarded-For = client.ip;
# Let’s make sure we aren’t compressing already compressed formats.
if (req.http.Accept-Encoding) {
if (req.url ~ "\.(jpg|png|gif|gz|tgz|bz2|mp3|mp4|m4v)(\?.*|)$") {
remove req.http.Accept-Encoding;
} elsif (req.http.Accept-Encoding ~ "gzip") {
set req.http.Accept-Encoding = "gzip";
} elsif (req.http.Accept-Encoding ~ "deflate") {
set req.http.Accept-Encoding = "deflate";
} else {
remove req.http.Accept-Encoding;
}
}
if (req.request == "PURGE") {
if (!client.ip ~ purge) {
error 405 "Not allowed.";
}
return(lookup);
}
if (req.url ~ "^/$") {
unset req.http.cookie;
}
}
sub vcl_hit {
if (req.request == "PURGE") {
set obj.ttl = 0s;
error 200 "Purged.";
}
}
sub vcl_miss {
if (req.request == "PURGE") {
error 404 "Not in cache.";
}
if (!(req.url ~ "wp-(login|admin)")) {
unset req.http.cookie;
}
if (req.url ~ "^/[^?]+.(jpeg|jpg|png|gif|ico|js|css|txt|gz|zip|lzma|bz2|tgz|tbz|html|htm)(\?.|)$") {
unset req.http.cookie;
set req.url = regsub(req.url, "\?.$", "");
}
if (req.url ~ "^/$") {
unset req.http.cookie;
}
}
sub vcl_fetch {
if (req.url ~ "^/$") {
unset beresp.http.set-cookie;
}
if (!(req.url ~ "wp-(login|admin)")) {
unset beresp.http.set-cookie;
}
}
[/sourcecode]
Ok so what the frick is this all about?
The VCL file above mainly comes from here where it is explained very well exactly what is going on. I’ve made a few minor adjustments over the past 9 or 10 months that I’ve been running it here on themesforge – but it’s working pretty damn well. A few important points to note:
- By default, Varnish will not cache any requests where cookies are being transmitted. WordPress has quite a lot of cookie action. For most blogs, these are insignificant and we can safely drop these in favour of serving content from Varnish cache rather than directly by the webserver – hence the numerous calls to “unset beresp.http.set-cookie;” above.
- There are some notable exceptions where we absolutely don’t want to drop cookies, including logins and admin access, hence the exceptions provided by the “if (!(req.url ~ “wp-(login|admin)”)) {” call.
- Our config will also bust the Varnish cache when a new blog post is published.
- Another important call is the “set req.http.X-Forwarded-For = client.ip;” call which ensures that the users IP address is passed to nginx – otherwise your nginx web logs will be full of localhost/127.0.0.1 entries – which are useless to us! This is also the reason that we installed the “nginx-extras” package back in Part 2 so we could avail of the nginx Real IP module which is not enabled by default in the default package (I know crazy!)
Save your config file and give Varnish and nginx one last restart to reload their configs:
[sourcecode]
service nginx restart
service varnish restart
[/sourcecode]
And with that your VPS is now super charged and ready for bucketloads of traffic!
How do I know if my Varnish config is performing correctly?
Varnish comes with some really useful diagnostics. First, I’m assuming you ran the netstat earlier to ensure Varnish and nginx are on the correct ports right?
Ok, here’s some useful tips for making sure Varnish is actually working.
[sourcecode]
varnishstat
[/sourcecode]
Varnishstat displays a wealth of information about how Varnish is performing, some key values to look out for:
- Hitrate Ratio and Hitrate avg: These numbers represent Varnish’s pulse. Right now my ratios are 10, 100, 181 which corresponds to numbers of seconds. Under each of these values, you will see a decimal value – something like 0.9843 – multiply this number by 100 and you have the percentage of your http requests which are being served by Varnish – i.e. cache hits. When you first launch Varnish you can expect your hitrate percentages to be very low but should dramatically increase once you start pointing traffic at your site.
- Cache Hits – You will want to see this number climbing rapidly as you point traffic to the frontend.
- Cache Misses – At first launch and as users first request new pages this number will increment.
- N LRU nuked objects – This number should be zero – if it starts incrementing then it’s time to increase the size of virtual memory assigned to varnish. This number is basically a count of the number of cache objects that had to be removed to make room for new objects.
This is only really scratching the surface of what varnishstat can tell you. You can find out more about varnishstat here.
Other useful varnish commands that I use frequently:
varnishtop -i rxurl – will show you what URLs are beeing asked for by the client
varnishtop -i RxHeader -I Accept-Encoding will show the most popular Accept-Encoding header
Conclusion – what now?
So that’s it fot the High Performance WordPress blog post series for now. If you’ve followed along from the beginning you are now in possession of a finely tuned speed optimised platform for hosting your WordPress websites. That said, the pace of innovation in this space is incredible and constantly changing (for the better). We’ve only really scratched the surface of many highly complex subjects and you should continue to learn about how to tune your own specific server environment. We’ll be coming back to this series later in the year. For now, if you don’t have the time to build out your own servers and would like some assistance we plan on offering a server deployment service in the near future where we roll out finely tuned VPS machines based on your specific needs. Drop me a line if you’re interested.
Leave a Reply