bdisk/docs/manual/faq/LONGTIME.adoc

Why does building take so long?

This typically occurs when you’re building from within a LiveCD/LiveUSB situation, in a VM/container/etc., or on a headless server. If this is the case, you may run into what appears to be "stalling", especially while keys are generating for the chroots. Thankfully, there is an easy fix. You can install haveged and run it (this can be done safely while a build is executing). This will show an immediate and non-negligible improvement for the above contexts. If you have extra processing power to throw at the build process (or are using a dedicated build box) as well, I recommend enabling i_am_a_racecar. BDisk will then be more aggressive with its resource consumption.

Running a local mirror

Keep in mind also that the more packages you opt to install, the longer the build process will take. This process will also use quite a fair bit of bandwidth. If you plan on building regular images (e.g. nightly builds, etc.) or are undergoing some custom change testing, I recommend running a private repository mirror on-site. This will not store AUR packages, as those will still be fetched and built (documentation on working around this is TODO) but setting up a local mirror is quite quick and easy.

First, you’ll need at least 70Gb of free disk space. Let’s say our repository clone will be at /srv/repo/arch/.

You’ll also need to find an Arch mirror, ideally one close to you that is up-to-date. The mirrorlist generator and mirror list will assist you here greatly.

Note	You’ll need to find a mirror that supports rsync.

Tip	You can use ANY distro to run a repository mirror, as long as it has rsync installed!

Set up the sync

Create a script and mark it as executable with the following content:

#!/bin/bash
SOURCE='rsync://your.mirror.here/archlinux'
DEST='/srv/repo/arch'
LCK_FLE='/var/run/repo-sync.lck'
PATH='/usr/bin'
if [ -e "${LCK_FLE}" ] ; then
	OTHER_PID=$(cat ${LCK_FLE})
	echo "Another instance already running: ${OTHER_PID}"
	exit 1
fi
# If e.g. /srv/repo is a mountpoint, uncomment below.
#findmnt /srv/repo > /dev/null 2>&1
#if [[ "${?}" -ne '0' ]];
#then
#	echo "External storage not mounted!"
#	exit 1
#fi
echo $$ > "${LCK_FLE}"
rsync -rvtlH --delete-after --delay-updates --safe-links --max-delete=1000 ${SOURCE}/. ${DEST}/. >> /var/log/arch.repo.sync 2>&1
date +%s > ${DEST}/lastsync
rm -f "${LCK_FLE}"

Assuming you want to run the sync script every 6 hours and it is located at /root/bin/arch.repo.clone.sh, this is the cron entry you would use (crontab -e):

0    */6  *   *   * /root/bin/arch.repo.clone.sh > /dev/null 2>&1

The first sync can take quite a while, but subsequent runs shouldn’t take more than five minutes or so.

Configuring the local mirror

You’ll need a way to serve this local mirror in a way pacman can understand. Luckily, it’s fairly easy. I recommend using nginx as it’s available by default in many operating systems. You can of course use others such as lighttpd, apache/httpd, etc. For the example configuration here, we’re going to use an nginx configuration file.

server {
listen [::]:80;
access_log /var/log/nginx/repo.access.log main;
error_log /var/log/nginx/repo.error.log;
#error_log /var/log/nginx/repo.error.log debug;

autoindex on;

root /srv/repo/arch;

}

The configuration may vary according to your distribution’s provided nginx default configuration, but you’ll want this configuration to be served as the default (or set an appropriate server_name directive which you would then use in <basedir>/extra/pre-build.d/etc/pacman.d/mirrorlist).

Configuring BDisk

You’ll then want to configure BDisk’s chroots to use your local mirror first. However, when doing so you run into an issue — in the built image, install operations will take longer than they need to because the local mirror likely won’t be available! This is a small issue as it’s unexpected that you’ll need to install software within the live environment, but I’ve run into cases where it was a necessity once or twice.

There is an easy workaround if you’re using libvirt — you can simply tell your build VM to resolve to the IP address of the box that is running the mirror for the same FQDN that the "preferred" "real" mirror on the Internet is and set that mirror at the top of <basedir>/extra/pre-build.d/etc/pacman.d/mirrorlist. However, that’s not always feasible- most notably if you’re building on a physical box and it’s the same host as the repository clone. In that case you can set the specific local resolution — e.g. http://127.0.0.1/ — at the top of <basedir>/extra/pre-build.d/etc/pacman.d/mirrorlist and then set a mirrorlist WITHOUT that entry in <basedir>/overlay/etc/pacman.d/mirrorlist. For more information on using these type of overrides, see [advanced_customization].

If you’re using the libvirt workaround, remember to configure nginx (or whatever you’re using) with a virtual host and location block that matches the "real", upstream mirror. In our example below, we use http://mirror.us.leaseweb.net/archlinux as the mirror.

server {
listen [::]:80;
access_log /var/log/nginx/repo.access.log main;
error_log /var/log/nginx/repo.error.log;
#error_log /var/log/nginx/repo.error.log debug;

server_name mirror.us.leaseweb.net;

autoindex on;

root /srv/repo/arch;

location /archlinux {
	autoindex on;
	rewrite ^/archlinux(/.*)$ /$1;
	}

}