Caching system updates for the home lab
If you’re like me, you’ve got a home lab with a dozen or so virtual machines doing all sorts of things - and each of them are pulling down updates from somewhere on the internet.
What if you could have a single endpoint for all VMs to reference? That way, updates that are common would be distributed to all systems at LAN speeds after the first download.
Introducing - mod_cache for Apache :)
Assuming you’re already running Apache somewhere, you can start mapping part of the local path structure to a remote endpoint.
Drop the following into /etc/httpd/conf.d/mod_cache.conf:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
CacheEnable disk /fedora
CacheRoot /var/cache/httpd/fedora
CacheMaxFileSize 524288000
CacheDefaultExpire 14400
CacheDetailHeader on
# common caching directives
CacheQuickHandler off
CacheLock on
CacheLockPath /tmp/mod_cache-lock
CacheLockMaxAge 5
CacheHeader On
# cache control
#CacheIgnoreNoLastMod On
#CacheIgnoreCacheControl On
# unset headers from upstream server
Header unset Expires
Header unset Cache-Control
Header unset Pragma
ProxyRequests Off
ProxyPass /fedora http://dl.fedoraproject.org/pub/fedora
ProxyPassReverse /fedora http://dl.fedoraproject.org/pub/fedora
UseCanonicalName On
When in use, this will map http://my.apache.host/fedora to the Fedora mirror, and cache all responses and downloaded files.
The cache won’t automatically clean itself though - so we need a systemd service to clean things up over time. Create the file /etc/systemd/system/http-cache-clean.service as follows:
1
2
3
4
5
6
7
8
9
10
[Unit]
Description=Apache cache cleaner
After=network-online.target
[Service]
Type=forking
ExecStart=/usr/sbin/htcacheclean -d 60 -i -l 5G -p /var/cache/httpd/fedora/
[Install]
WantedBy=multi-user.target
This will limit the cache size to 5Gb and remove the oldest files first.
There is one gotcha when using this with Fedoras updates and dnf - zchunk. I believe this is because mod_cache doesn’t work
on partial content requests - which is how zchunk functions.
To get around this, we can disable zchunk in the DNF configuration file /etc/dnf/dnf.conf. I also disable deltarpm - as its quicker to download the file from the LAN cache than it is
to rebuild a drpm update.
1
2
3
4
5
6
7
8
9
10
[main]
gpgcheck=True
installonly_limit=3
clean_requirements_on_remove=True
best=False
skip_if_unavailable=True
max_parallel_downloads=10
fastestmirror=True
zchunk=False
deltarpm=0
We can then point the yum repo file to the local apache server - for example, part of /etc/yum.repos.d/fedora-updates.repo:
1
2
3
4
5
6
[updates]
name=Fedora $releasever - $basearch - Updates
#baseurl=http://download.example/pub/fedora/linux/updates/$releasever/Everything/$basearch/
#metalink=https://mirrors.fedoraproject.org/metalink?repo=updates-released-f$releasever&arch=$basearch
baseurl=http://my.apache.host/fedora/linux/updates/$releasever/Everything/$basearch/
enabled=1