Tuning Varnish startup parameters


thread_pool_add_delay=2
thread_pools = (Number of CPU cores)
thread_pool_min = (800 / Number of CPU cores)
thread_pool_max = 4000
timeout_linger = 50
workspace_session=262144
malloc,(yourmemory - 20%)G

Varnish4: Cache all!

Varnish offers great capabilities to manipulate requests coming from the backend servers.

The following example let you cache all request despite from what is coming from the backend servers.

sub vcl_backend_response {  

     # client browser and server cache  
     # Force cache: remove expires, Cache-control & Pragma header coming from the backend  
     if (beresp.http.Cache-Control ~ "(no-cache|private)" || beresp.http.Pragma ~ "no-cache") {  
         unset beresp.http.Expires;  
         unset beresp.http.Cache-Control;  
         unset beresp.http.Pragma;  

         # Marker for vcl_deliver to reset Age: /  
         set beresp.http.magicmarker = "1";  

         # Leveraging browser, cache set the clients TTL on this object /  
         set beresp.http.Cache-Control = "public, max-age=60";  

         # cache set the clients TTL on this object /  
         set beresp.ttl = 1m;  

         # Allow stale content, in case the backend goes down.  
         # make Varnish keep all objects for 6 hours beyond their TTL  
         set beresp.grace = 6h;  
         return (deliver);  
     }  
 }
sub vcl_deliver {  

     # Called before a cached object is delivered to the client.  
     if (resp.http.magicmarker) {  
     unset resp.http.magicmarker;  

     # By definition we have a fresh object  
     set resp.http.Age = "0";  
     }  

     if (obj.hits > 0) { # Add debug header to see if it's a HIT/MISS and the number of hits, disable when not needed  
     set resp.http.X-Cache = "HIT";  
     } else { set resp.http.X-Cache = "MISS"; }  

     # Please note that obj.hits behaviour changed in 4.0, now it counts per objecthead, not per object  
     # and obj.hits may not be reset in some cases where bans are in use. See bug 1492 for details.  
     # So take hits with a grain of salt  
     set resp.http.X-Cache-Hits = obj.hits; 
 
     # Set Varnish server name  
     set resp.http.X-Served-By = server.hostname;  

     # Remove some headers: PHP version  
     unset resp.http.X-Powered-By;  

     # Remove some headers: Apache version & OS  
     unset resp.http.Server;  
     unset resp.http.X-Varnish;  
     unset resp.http.Via;  
     unset resp.http.Link;  
     unset resp.http.X-Generator;  
     return (deliver);  
 }

Varnish Administration Certification Class – Milan, Italy

Partecipa al primo training italiano di ‪‎Varnish‬ Software in collaborazione con ‪FullTechnology‬ ed ottieni la certificazione!

Il corso si terrà a Milano il 2-3-4 Febbraio e prevede teoria, esercitazioni ed esame finalizzato al conseguimento della certificazione ed indirizzato a sysAdmin/devOps.

Partecipa all’evento di Milano

Varnish Administration Certification Class - Milan, Italy
Varnish Administration Certification Class – Milan, Italy

HTTPie: a CLI, cURL-like tool for humans

HTTPie is available on Linux, Mac OS X and Windows. On a Debian or Ubuntu system HTTPie can be installed with apt-get install httpie. For other platforms, see http://httpie.org.

Testing httpie is simple:

http -p Hh http://www.atomictag.com
GET / HTTP/1.1
Accept: */*
Accept-Encoding: gzip, deflate
Connection: keep-alive
Host: www.atomictag.com
User-Agent: HTTPie/0.9.2
   
HTTP/1.1 200 OK
CF-RAY: 250beb7295742666-FRA
Cache-Control: max-age=0, public
Content-Encoding: gzip
Content-Type: text/html; charset=UTF-8
Date: Sun, 06 Dec 2015 23:55:06 GMT
ETag: W/"7524-526422bb3df6c"
Expires: Sun, 06 Dec 2015 23:55:06 GMT
Last-Modified: Sun, 06 Dec 2015 22:22:06 GMT
Pragma: public
Server: cloudflare-nginx
Set-Cookie: __cfduid=dad70ed346cbd17091806e91a67d56c1f1449446106; expires=Mon, 05-Dec-16 23:55:06 GMT; path=/; domain=.atomictag.com; HttpOnly
Transfer-Encoding: chunked
Vary: Accept-Encoding,Cookie

The -p option to http can be used to control output. Specifically:

-p H will print request headers.
-p h will print response headers.
-p B will print request body.
-p b will print response body.

 

Varnish and microservices: Zipnish

Zipnish is a MicroServices Monitor that runs on top of Varnish, currently – the Varnish supported version is 4. Data storage is based on MySQL, therefore a running instance of MySQL server will be required.

Download link: https://github.com/varnish/zipnish

Varnish Cache update 4.1

Improved Security and Proxy Support Complement Streaming Architecture, Which Cuts Down of Delivery Times for Larger Objects and Decreased Latency When Accessing Content Through Cache Hierarchies

Updates to Varnish Cache 4.1 include:

Proactive security features

  • Support for different kinds of privilege separation methods, collectively described as jails.
  • Ability for Varnish parent process on most systems to drop effective privileges to normal user mode when not doing operations needing special access;
  • Varnish worker child is enabled to run as a separate vcache user;
  • varnishlog, varnishncsa and other Varnish shared log utilities now must be run in a context with varnish group membership.

PROXY protocol support

  • Socket support for PROXY protocol connections has been added whereby PROXY defines a short preamble on the TCP connection where (usually) a SSL/TLS terminating proxy can signal the real client address.
  • The -a startup argument syntax has been expanded to allow for this: $ varnishd -f /etc/varnish/default.vcl -a :6081 -a 127.0.0.1:6086,PROXY.
  • Both PROXY1 and PROXY2 protocols are supported on the resulting listening socket.
  • For connections coming in over a PROXY socket, client.ip and server.ip will contain the addresses given to Varnish in the PROXY header/preamble (the “real” IPs).
  • The new VCL variables remote.ip and local.ip contains the local TCP connection endpoints. On non-PROXY connections these will be identical to client.ip and server.ip.
  • An expected pattern following this is if (std.port(local.ip) == 80) { } in vcl_recv to see if traffic came in over the HTTP listening socket (so a client redirect to HTTPS can be served).

Warm and cold VCL configurations — Traditionally Varnish has had the concept of active and inactive loaded VCLs. Any loaded VCL lead to state being kept, and a separate set of health checks (if configured) were being run against the backends. To avoid the extra state and backend polling, a loaded VCL is now either warm or cold. Runtime state (incl. backend counters) and health checks are not present for cold VCLs. A warm VCL will automatically be set to cold after vcl cool down seconds.

VMOD backends – Before Varnish 4.1, backends could only be declared in native VCL. Varnish 4.0 moved directors from VCL to VMODs, and VMODs can now also create backends. It is possible to both create the same backends as VCL but dynamically, or create backends that don’t necessarily speak HTTP/1 over TCP to fetch resources.

Backend connection timeout – Backend connections will now be closed by Varnish after backend idle timeout seconds of inactivity. Previously they were kept around forever and the backend servers would close the connection without Varnish noticing it. On the next traffic spike needing these extra backend connections, the request would fail, perhaps multiple times, before a working backend connection was found/created.

Varnish Cache is open source, available under a two-clause BSD license. Varnish Software also delivers additional software and support for more advanced users via subscriptions for Varnish Plus, its premium product. See here.

Online resources

You can download Varnish Cache 4.1 here.

Further details on Varnish Cache 4.1 can be found here.

Caching requests by HTTP status code

Sometimes you may want to cache request based on the status code with a different TTL.

Adding the following in the vcl_backend_response will help you in achieving the result:


if (beresp.status == 403 || beresp.status == 404 || beresp.status >= 500)
{
   set beresp.ttl = 60s;
}

Partner Event in Milan: all things Varnish and how Conde Nast uses Varnish Plus

The Varnish Software and Fulltechnology Partner Conference on 17th September in Milan, Italy is fast approaching! Here are some highlights we want to share with you.

You will have the opportunity to meet both Lars Larsson, CEO of Varnish Software, and Francesco Meani, CEO of Fulltechnology at the conference.

We are also excited to announce that the leading global mass media giant Conde Nast will speak at the event and share their user experience with Varnish Plus. Conde Nast Italy’s CTO, Giuseppe Serrecchia, will present real-life examples of how Conde Nast has developed and used Varnish Plus to meet their specific needs and what the outcomes were. Read more about in our blog post.

If you’re curious about the more technical aspects of Varnish Plus in action including tips and tricks for using Varnish Plus as an advanced load balancer and administration console (Varnish Administration Console), this event is your chance to learn. Fulltechnology, a leading interactive and technology player that designs, builds and executes software platforms, will augment the real-world Varnish Plus information with its own commercial and technical insights.

The program includes:

  • Tips & tricks for using Varnish as an advanced load-balancer and VAC
  • The benefits of Varnish Plus and its latest addition SSL/TLS
  • Commercial and technical insights from Fulltechnology
  • Condé Nast business case study and their user experience with Varnish
  • After event drinks and networking


Space is limited, so sign up now to participate at this conference.

Add to my calendar

To receive updates about the event and information about guest speakers, you can follow our blog here.

We are looking forward to meeting you in Milan!

Force redirect in Varnish

Sometimes in certain occasion you need to manage a complete redirection to an external site directly from Varnish.

Following you will find how to implement this using Varnish 4.0 and some specific rules in the VCL.


sub force_redirect {
   set req.http.host = "www.atomictag.com";
   return(synth(750, "Force redirection to external site."));
}

sub vcl_recv {
   # Force redirect
   call force_redirect;
   # Everything else will be ignored
}

sub vcl_synth {
   # Managing redirection
   if (resp.status == 750) {
      set resp.status = 301;
      set resp.http.Location = "http://www.atomictag.com/";
      return(deliver);
   }
}

Of course you can easily add more intelligence to the VCL, e.g.: by evaluating the redirection based on the http referrer or on a specific hostname.

Enable VCL variables logging

Varnish Cache offers several ways to log and most of the documentation is related to varnishlog which includes several information related to request, response and backend response. Now what’s happen if we need to log our own variables or message to a file?

The key function is std.syslog enabled via std (standard) Varnish module, which is the only built-in “vmod” and is thus natively available and doesn’t have to be compiled.

 import std; 

then you can easily add into your VCL the following line:


std.syslog(180, "log description");

The first parameter, 180, is the priority value. If you want to add some variables to your log message:


std.syslog(180, "log description" + beresp.ttl);

This will logo the message to general system log, which is often, in a Linux system, is located in /var/log/messages.