This is an old revision of the document!


Introduction

By using a reverse proxy, it is possible to create an HTTP mirror without of the heavy burden of rsync and a large amount of allocated space. The reverse proxy will cache the particular files (with special cases to exclude repository files)

Some practical applications might include:

  • Network installs
  • Primary repository for OS updates
  • Anycast installation HTTP servers for larger deployments
  • Booting test images w/o uploading to remote server

Note: The advantages of this proxy won't be seen with only one to two hosts except in the cases where they are being reinstalled constantly.

Requirements

  • 300 MB of disk space
  • LAN w/internet router :)

Varnish Config

Make a /etc/varnish/public-mirror.vcl and edit the startup settings for varnish (On redhat, it's /etc/sysconfig/varnish, In gentoo it's /etc/conf.d/varnishd, …). This particular configuration uses http://mirrors.kernel.org (from the US) as it is a very reliable and usually a Teir 1 or 2 mirror with most projects. Not to say that it couldn't be used with another source such as http://jailtime.org and it's many mirrors as well.

This configuration has been tested and is currently in use with minor OS tweaks for performance.

backend kernelorg_1 {                                                                                     
        .host = "149.20.20.135";                                                                          
        #.host = "mirrors.kernel.org";                                                                    
        .port = "80";                                                                                     
        .probe = {                                                                                        
                .request =                                                                                
                "HEAD /index.html HTTP/1.1"                                                               
                "Host: mirrors.kernel.org"                                                                
                "Connection: close";                                                                      
                .timeout = 0.3 s;                                                                         
                .window = 8;                                                                              
                .threshold = 3;                                                                           
        }                                                                                                 
}                                                                                                         
 
backend kernelorg_2 { 
        #.host = "mirrors.kernel.org";
        .host = "204.152.191.39";     
        .port = "80";                 
        .probe = {                    
                .request =            
                "HEAD /index.html HTTP/1.1"
                "Host: mirrors.kernel.org" 
                "Connection: close";       
                .timeout = 0.3 s;          
                .window = 8;               
                .threshold = 3;            
        }                                  
}                                          
 
director ubuntu random {
        # { .backend = ubuntu_us_1; .weight = 1; }
        # { .backend = ubuntu_us_2; .weight = 1; }
        # { .backend = ubuntu_us_3; .weight = 1; }
        # { .backend = ubuntu_us_4; .weight = 1; }
        # { .backend = ubuntu_osuosl_1; .weight = 50; }
        # { .backend = ubuntu_osuosl_2; .weight = 50; }
        { .backend = kernelorg_2; .weight = 1; }
        { .backend = kernelorg_1; .weight = 1; }
}
 
sub vcl_recv {
 
        // Skip repository files
        if ( req.url ~ "/(Release|Packages|Sources)(|\.gz|\.bz2|\.gpg|)$") {
                pass;
        }
 
        if ( req.url ~ "/repodata/.*" ) {
                pass;
        }
 
        if ( req.url ~ "^/(ubuntu|debian|centos|opensuse|suse|fedora|gentoo)" ) {
                set req.http.host = "mirrors.kernel.org";
                set req.backend = ubuntu;
        }
 
        if (req.backend.healthy) {
                set req.grace = 30s;
        } else {
                set req.grace = 1h;
        }
}
 
sub vcl_fetch {
        set obj.http.X-Varnish-Url = req.url;
        set obj.ttl = 1d;
        set obj.grace = 15m;
        #set beresp.grace = 15m;
        set obj.prefetch = -30s;
}
 
sub vcl_miss {
        if (req.http.user-agent ~ "spider") {
                error 503 "Not presently in cache, please try again later";
        }
}
 
sub vcl_deliver {
        if (obj.hits > 0 ) {
                set resp.http.X-Cache = "HIT";
        } else {
                set resp.http.X-Cache = "MISS";
        }
}

QR Code
QR Code appnotes:varnishmirror (generated for current page)