If your network has several machines all running similar versions of Debian or Ubuntu you might want to save some bandwidth and improve update speeds by keeping a local cache of updates.

Apt-Cacher-NG is a reasonably simple application that, when a request is made for an update package, it serves it from it's local cache or fetches it if it's not currently cached. It's normal design intention is to operate as a http proxy in that you configure apt on your your network computers to use your Apt-Cache-NG server.

I found this to be slightly less than ideal in that it needs your computers to be manually configured. The ideal situation would be to have the computers use the proxy without needing to be configured thus leaving no oppitunity for incorrectly configured machines or machines that travel needing to be reconfigured depending on the network they are connected to.

In this article we will combine Apt-Cacher-NG, Apache and DNS trickery to make the proxy and cache completely transparent to the target computer.

Installation

Create a LXC container of VM dedicated to the apt-cacher-ng task. This allows us to do specific DNS related things that will only effect this host.

sudo apt-get install dnsmasq apt-cacher-ng apache2 ufw dnsutils

DNS

What we want to do is have all machines on our network do a DNS lookup for archive.ubuntu.com (and others) and get given the IP of this server. That's the easy part. The tricky part is that we need this server to be able to make the same queries but get the real answers. Having it point back to itself would mean that it could never fetch the files from upstream.

There are 2 good ways to do this:

  • If you have a BIND DNS infrastructure that all your client machines point to, you could use views.
  • If you do not have a BIND DNS infrastructure but something like pfSense that uses dnsmasq you can:
    • Install dnsmasq on this server also
    • Have it forward “local” DNS queries (i.e. for the local DNS suffix) to the upstream pfSense instance
    • Have it forward all other DNS queries to a public DNS server (ISP or 8.8.8.8)
    • Configure indervidual DNS overrides in the pfSense dnsmasq to point archive.ubuntu.com, etc. to this this server.
      • This way, all machines on the LAN get given this servers IP for Apt updates but this server gets the real IP address.

For this article, I'm assuming you're using option 2.

In you're upstream dnsmasq instance (as mentioned, I use the dnsmasq built into pfSense), add overrides for:

  • archive.ubuntu.com
  • extras.ubuntu.com
  • security.ubuntu.com
  • Your local ubuntu archive mirror, for me in Australia it's:
    • au.archive.ubuntu.com

These should all point to the IP address of your (soon to be) apt-cacher-ng server.

Test

From one of your local Ubuntu servers or desktops:

[email protected]:~$ nslookup au.archive.ubuntu.com
Server:		127.0.0.1
Address:	127.0.0.1#53

Name:	au.archive.ubuntu.com
Address: 172.32.0.152

As you can see, my laptop now thinks that au.archive.ubuntu.com can be found locally at the IP of my apt-cacher-ng server.

DNSMasq

Skip this section if you're using the BIND Views method

Open /etc/dnsmasq.conf, find and uncomment the following line:

#no-resolv

Create /etc/dnsmasq.d/default and give it the following content:

server=<ip of local DNS resolver>

<ip of local DNS resolver> is one of the normal local DNS resolvers that machines on your network use, possibly those handed out by DHCP. If you have more than one local DNS resolver, just insert multiple lines into this file.

Create /etc/dnsmasq.d/<local domain> (e.g. /etc/dnamasq.d/ubuntu.com) and give it the following content:

server=/ubuntu.com/8.8.8.8
server=/ubuntu.com/8.8.4.4

This causes dnsmasq to forward any requests for *.ubuntu.com to the DNS to a public DNS server so that it doesn't get any local overrides that our local DNS server might have configured for the ubuntu.com domain (such as the one we will be creating later).

Now we need to configure this server to only look to itself for DNS. If you are using Ubuntu 12.04 or newer, open /etc/network/interfaces and append the following:

dns-nameservers 127.0.0.1
dns-search ad.ghanima.net

Give the server a reboot. When it comes back up, run some tests:

[email protected]:~$ nslookup archive.ubuntu.com
Server:		127.0.0.1
Address:	127.0.0.1#53

Non-authoritative answer:
Name:	archive.ubuntu.com
Address: 91.189.92.200
Name:	archive.ubuntu.com
Address: 91.189.92.201
Name:	archive.ubuntu.com
Address: 91.189.92.202
Name:	archive.ubuntu.com
Address: 91.189.92.156

[email protected]:~$ nslookup ad.ghanima.net
Server:		127.0.0.1
Address:	127.0.0.1#53

Non-authoritative answer:
Name:	ad.ghanima.net
Address: 172.32.0.150

In the above, you can see that we are NOT getting the local overrides for archive.ubuntu.com (returned IPs are real Internet IPs which is different to what my laptop saw earlier) but we are getting the overridden IPs for ad.ghanima.net. Exactly what we needed.

As one final step, we need to make sure this server uses itself for Apt updates. Create /etc/apt/apt.conf.d/10proxy and gice it the following content:

Acquire::http::Proxy "http://127.0.0.1:3142";

This is the only server that needs this special setting. All other machines are going to get here using the tricked DNS records.

Apache

The following Apache VirtualHost is created in /etc/apache2/sites-available/apt-cacher-ng.conf:

<VirtualHost *:80>
        ServerName archive.ubuntu.com
        ServerAlias *.archive.ubuntu.com security.ubuntu.com archive.canonical.com extras.ubuntu.com download.virtualbox.org ppa.launchpad.net packages.linuxdeepin.com images.linuxcontainers.org ports.ubuntu.com
 
        #Logging
        ErrorLog /var/log/apache2/apt-cache-ng-error.log
        CustomLog /var/log/apache2/apt-cache-ng-access.log combined
 
        #Enable the rewrite engine
        RewriteEngine On
 
        #Preserve the "host" header for requests being proxied
        #ProxyPreserveHost On
 
        #Proxy all requests to the local apt-cacher-ng daemon
        #RewriteRule ^(.*)$ http://127.0.0.1:3142$1 [L,P]
	RewriteRule ^(.*)$ http://%{HTTP_HOST}$1 [P]
 
	ProxyRemote * http://127.0.0.1:3142
</VirtualHost>

This Virtual Host's components are:

  • ServerName and ServerAlias - These are the host names of the sites that this Virtual Host will service. If a request is made to a host name not in this list it will not be handled by this Virtual Host or Apt-Cache-NG.
  • ErrorLog and CustomLog - Custom log files for this Virtual Host
  • RewriteEngine - We need to enable mod_rewrite for this Virtual Host as we will be using it shortly.
  • ProxyPreserveHost - When we send a request to an external server we need to preserve the current value of the host header, not attempt to translate it.
  • RewriteRule - Direct the request to the proxy.

Enable the required apache modules and vhost:

sudo a2enmod rewrite
sudo a2enmod proxy
sudo a2enmod proxy_http
sudo a2ensite apt-cacher-ng

Testing

We are finally ready to give our new cache a test.

On the apt-cacher-ng server get a tail running on the apt-cacher-ng log:

tail -f /var/log/apt-cacher-ng/apt-cacher.log

Still on the apt-cacher-ng server, ask Apt to update it's local package lists:

sudo apt-get update

If the apt-cacher-ng daemon is working, you should see entries like these coming out of your tail command:

1357057418|I|579|127.0.0.1|security.ubuntu.com/ubuntu/dists/quantal-security/main/i18n/Translation-en_AU
1357057418|O|242|127.0.0.1|security.ubuntu.com/ubuntu/dists/quantal-security/main/i18n/Translation-en_AU
1357057418|I|585|127.0.0.1|security.ubuntu.com/ubuntu/dists/quantal-security/multiverse/i18n/Translation-en_AU
1357057418|O|248|127.0.0.1|security.ubuntu.com/ubuntu/dists/quantal-security/multiverse/i18n/Translation-en_AU
1357057419|I|585|127.0.0.1|security.ubuntu.com/ubuntu/dists/quantal-security/restricted/i18n/Translation-en_AU
1357057419|O|248|127.0.0.1|security.ubuntu.com/ubuntu/dists/quantal-security/restricted/i18n/Translation-en_AU
1357057419|I|583|127.0.0.1|security.ubuntu.com/ubuntu/dists/quantal-security/universe/i18n/Translation-en_AU
1357057419|O|246|127.0.0.1|security.ubuntu.com/ubuntu/dists/quantal-security/universe/i18n/Translation-en_AU
1357057419|I|581|127.0.0.1|uburep/dists/quantal-updates/universe/i18n/Translation-en_AU
1357057419|O|247|127.0.0.1|uburep/dists/quantal-updates/universe/i18n/Translation-en_AU

If the apt-get update succeeded, and you saw the log entries being created, the apt-cacher-ng daemon is working.

Next, from another Ubuntu machine on your network, run the same sudo apt-get update. If this succeeds, and you see more log entries being created, everything is working!

Adding additional repositories

packages.linuxdeepin.com

Create /etc/dnsmasq.d/linuxdeepin.com with the following content:

server=/linuxdeepin.com/8.8.8.8
server=/linuxdeepin.com/8.8.4.4

Create /etc/apt-cacher-ng/backends_deepin with the following content:

packages.linuxdeepin.com

Create /usr/lib/apt-cacher-ng/deepin_mirrors with the following content:

packages.linuxdeepin.com

In the file /etc/apt-cacher-ng/acng.conf find the section that has lines that start with Remap-. For example:

Remap-gentoo: file:gentoo_mirrors.gz /gentoo ; file:backends_gentoo # Gentoo Archives

Add a new Remap- line below all of the existing entries like the following:

Remap-deepin: file:deepin_mirrors /deepin ; file:backends_deepin # Deepin Archives

Open the file /etc/apache2/sites-available/apt-cacher-ng.conf and add the following to the end of the ServerAliases line:

 packages.linuxdeepin.com

There should be a space between the end of the previous entry and the start of the new entry.

Restart all the services we reconfigured:

sudo service dnsmasq restart
sudo service apt-cacher-ng restart
sudo service apache2 restart

Troubleshooting

HTTP 500 Errors when downloading updates

I have only had this issue occur once. Restarting both the Apache2 and apt-cacher-ng daemons resolved the issue.

sudo service apache2 restart
sudo service apt-cacher-ng restart

GPG signature errors when running ''apt-get update''

This is another issue I have only had once and again restarting both the Apache2 and apt-cacher-ng daemons resolved the issue.

sudo service apache2 restart
sudo service apt-cacher-ng restart
ihisewe, 2017/06/03 17:43

Неужели, yandex сказал тепло будет днём!

Enter your comment. Wiki syntax is allowed:
K F G L C