We built an Ubuntu mirror and then made it faster

At Depot, we watch the performance of pretty much everything because we want to make pretty much everything run faster in your builds. Some steps, such as apt, are run by many different builds.

apt is a command-line tool for managing software packages on Ubuntu. It is usually run for two reasons. First, it updates the system software for things such as security fixes. Second, it installs the packages for the job. For the typical job that runs apt, the step accounts for about 9% of the entire runtime.

We fixed apt performance in two stages. First, our new Ubuntu mirror eliminated the slowdowns and returned the median apt time to 19 seconds. Then our apt and dpkg changes reduced the median from 19 seconds to 3 seconds. Even for the slowest tenth of jobs, apt dropped from 25% of the total runtime to 14%.

Here's the full story of how we got there.

The official Ubuntu mirrors were slowing down

apt downloads packages from nearby Ubuntu mirrors. A mirror holds every package for each Ubuntu release. Increasingly over the last few months, the official Ubuntu mirrors have been getting slower and slower.

The chart shows the median duration of apt steps for our customers from July 14 to August 21. It clearly tracks the Ubuntu mirror slowdowns they saw. Over time, performance was getting worse and staying degraded longer.

We started paying close attention to apt around the end of July. By August 18, it felt like a crisis. It was taking 19% of a median job's entire runtime. This was up 10 percentage points from the previous month.

So we did two things:

1. We built a mirror hosted on S3. The mirror brought the step back down to the typical 8% of total runtime.
2. We sped up dpkg. This dropped the median apt step across customer jobs to 2% of runtime.

A reliable Ubuntu mirror backed by S3

Our Ubuntu mirror stores its contents in S3. It syncs upstream every six hours, and it has both the x86 archive and the Arm64 ports archive.

A few fun facts about the mirror: It's just under 12 TB and contains 4.4 million objects. A normal rsync mirror is much smaller because it uses hard links, but we make additional copies in S3. It syncs with Ubuntu's recommended two-stage rsync process. Then we use a similar two-stage process to copy the content to S3. This approach protects the mirror from races and keeps the complexity low.

Why S3? It's basically the bedrock of the internet. Its reliability and throughput are incredible. On top of that, S3 does not require us to think about disks or scaling.

To be fair, S3 does have outages once in a blue moon. We wrote about one last October. An S3 outage in us-east-1 would certainly take down the mirror. However, in that case, we automatically fail over to other mirrors.

The mirror went live on August 22, and it has been wonderfully reliable. No more peaks appear on the chart.

We keep S3 access logs. I took a sample from August 26 through 30, and it shows the following:

These are familiar numbers for anyone who uses S3. They confirm that the mirror is in good nick.

Making apt and dpkg faster

When the official mirror was healthy, apt was taking about 19 seconds. The new mirror had about the same performance. Despite that, it still felt really slow. Nineteen seconds is an eternity in the computing world.

When something is that slow, there have to be ways to improve it.

apt and dpkg have a vast number of configuration options. We measured before digging into the specifics. Most of those 19 seconds were spent reading from the local disk rather than downloading. On a fresh runner virtual machine, dpkg must load 384 MB of uncached package list data into memory.

The rest of the time was spent on defaults that assume the machine will boot again.

Depot runners are ephemeral, so they are never used again. A lot of software assumes the opposite. For example, dpkg calls fsync for every file it extracts before renaming the file into place. fsync ensures that no half-written files remain after a crash. However, a Depot runner can never recover from a crash because the machine is only ever used one time.

dpkg has an option for this behavior, --force-unsafe-io, and full credit goes to the dpkg maintainers for providing it. The option sounds scary, but it really means no fsync. Using it helped save about 330 ms on average. --force-unsafe-io was just one of many options we used to reduce the apt time for everyone.

The mirror removed the network slowdowns, but healthy apt runs still took about 19 seconds. The distribution view of apt times uses the same data as our chart, with slowdowns removed. It makes the effect of the apt and dpkg changes clear.

On August 30, the median dropped from 19 seconds to 3 seconds. The entire distribution shifted downward, not only the median.

Conclusion

My intuition is that AI workflows are increasing build volume for all systems. Systems that were fine for years can no longer keep up.

At Depot, we obsess over the reliability and performance of the systems that CI depends on. This work matters more as AI workflows continue to scale.

FAQ