Varnish : use multiple backends depending on host / URL

With the lack of public IPv4 address and the growth of virtualization, one of the commons solutions consists in using revese proxies to distribute the traffic of a physical host, having a public IPv4 address, between multiple virtual machines (VM), as shown in the schema below :

Varnish_front

In this example, we will assume that each VM hosts a HTTP server (Apache, lighttpd, nginx, etc.) listening on port 80, and that :

  • www.myhost1.com should use VM1
  • www.myhost2.com should use VM2
  • www.myhost3.com should use VM3

Varnish

Varnish, one of the best known reverse proxy engines (actually a caching HTTP reverse proxy), can be very useful in such a situation, by setting specific backends (ie. a VM) depending on the required hostname.

Varnish

I’m not going to explain in this post how to build the whole Varnish VCL file, but you’ll find very useful resources on Varnish’s wiki :  Varnish Default VCL Example.

Defining multiple backends

First of all, you will need to define the different backends that Varnish will rely on. In the example above, we have 3 VMs with each a private IPv4 in the 10.0.0.0/24 range.

At the beggining of your VCL, set your backends :

backend vm1 {
    .host = "10.0.0.11";
    .port = "80";
    .connect_timeout = 6000s;
    .first_byte_timeout = 6000s;
    .between_bytes_timeout = 6000s;
}

backend vm2 {
    .host = "10.0.0.12";
    .port = "80";
    .connect_timeout = 6000s;
    .first_byte_timeout = 6000s;
    .between_bytes_timeout = 6000s;
}

backend vm3 {
    .host = "10.0.0.13";
    .port = "80";
    .connect_timeout = 6000s;
    .first_byte_timeout = 6000s;
    .between_bytes_timeout = 6000s;
}

Using the appropriate backend

To define which backend (local HTTP server) should be used by Varnish to respond HTTP requests, we will set a few custom rules in the vcl_recv section of our VCL config file :

# Default backend is set to VM1
set req.backend = vm1;

if (req.http.host == "www.myhost2.com") {
    set req.backend = vm2;
}

if (req.http.host == "www.myhost3.com") {
    set req.backend = vm3;
}

Now restart Varnish and try to connect to one of the 3 hostnames : you should be forwarded to the appropriate backend.

NB : this post only covers HTTP reverse proxying, as Varnish does not handle HTTPS. If you need to use HTTPS, there are 2 options : using Apache as HTTPS reverse proxy (in French), or combining Varnish with another server such as Pound, which will handle the SSL communcation establishment.

About 

Freelance PHP Symfony2 & Magento developer, passionate about programming and electronics.

  • googleplus
  • twitter
  • Nuno Zimas

    What if one of the backends runs multiple virtual hosts? So far I haven’t found a way to make it work.
    Varnish will always forward the request to the first virtual host inside the back end.

    • Michael BOUVY

      Nuno,
      The original request HTTP Host is preserved when Varnish calls the backend, so there should be no specific work to get multiple vhosts working. How are your vhosts configured ?

      • Nuno Zimas

        Hi Michael. If you have multiple vhosts in each of the backends, Varnish will get invariably messed up.
        Your approach (and many others I have tried) seems to work only if there is one single domain per backend.
        More bizarrely, only the first domain stated in the sub vcl_recv subroutine gets forwarded to the correct backend.
        All other subsequent domains end up forwarded to the default backend.
        Thanks for replying!

        [EDIT]

        Actually, with your exact settings, all requests get forwarded to the default backend.

        [EDIT2]

        Mystery sorted!
        I’m relatively new to Apache 2.4 and did not realise that it requires all vhost files to specify a .conf extension, therefore the requests ended up handled by 000-default.conf, the only valid file in the lot.
        The backend that was already handling the requests correctly runs Apache 2.2.