Reducing the cost of CI banner
17 min read
product

Stay in the loop

Get notified when we ship new posts.

Reducing the cost of CI

CI costs have exploded over the past year, largely driven by the sheer volume of code entering CI. Teams are using agents to ship bug fixes, features, and whole products in only hours.

In a recent post about how CI is changing, I argue that the volume of code we are pushing means we need to fundamentally change how we think about CI. It's now the largest bottleneck in most software delivery pipelines.

But it's also emerging as a major cost driver for most teams. Rising costs lead to sticker shock and, rightfully, careful inspection into what is causing the increase.

At Depot, we have a unique insight into this problem. Depot isn't just a CI provider, but we do run a fuck ton of CI workloads through Depot on a weekly basis; our CI-engine-as-a-service runs within our broader software delivery platform. So I'd like to do something a touch out of the ordinary and explain how we try to help customers reduce the cost of their CI bill. Or, in other words, how we help them pay us less money.

The advice I'm about to share is not specific to Depot, by the way. The learnings we have accumulated over the past four years are generally applicable to whatever CI system you're using. Yes, even the kite one and the blinding yellow one as well.

Problem 1: Most people (or agents) set it and forget it

A bit of a meta point before diving into specific types of optimizations you can use with CI to reduce costs.

The most important thing any engineering team (with or without agents) can do as it relates to reducing CI spend: don't forget about it.

Sounds simple, but you'd be amazed at the number of teams that say something like this when they are surprised to find out their CI bill is now higher than they expected:

We set up our workflow back when we were just starting this new project. We just needed to get it working at first and hacked together the basic things we needed. The project is now deployed in production, has a team of 10 engineers working on it, and has grown massively.

In other words, folks often don't audit their CI until they are surprised by it. Sure, they might revisit a workflow if something breaks or they adopt a new tool. But few engineering teams audit their efficiency and spend on a regular basis.

Everybody has some version of a watch or autofix CI skill that they have an agent run when they open a PR. Side note: the ones provided by the actual harness like Claude Code or Codex tend to be spectacularly bad at times. I'd strongly suggest writing your own for your specific setup so that you avoid burning tokens on the existing ones doing dumb things.

But monitoring CI shouldn't be reserved exclusively for your pull requests. You should be monitoring it continuously. The customers building on Depot are doing this with their own agents. They have a dedicated agent that is constantly monitoring the CI usage and costs across all of their workflows. This agent is alerting them to any anomalies or unexpected spikes in usage, allowing them to take action before it becomes a bigger problem.

Here is a prompt we use to monitor our own CI optimization opportunities. Give it to your own agent.

You are watching our CI spend. Every weekday morning:

1. Pull yesterday's CI usage from our CI provider's API or billing export,
   broken down by repo, workflow and job: runs, runner size, minutes and cost.
2. Compare it to the same weekday last week. Flag anything that is up more
   than 20% and more than $100 a day.
3. For every flag, work out why. Is it more runs (volume) or more cost per
   run (efficiency)? Name the commit, branch, PR or config change that
   started it.
4. Flag any job running on a large runner (16+ CPUs) that wasn't running
   on one in the last 7 days.
5. Keep a list of the optimizations we've agreed to make. Check whether each
   one actually landed and whether it saved what we expected.

Post a short summary with one example run link for every claim. If nothing
changed, say so in one line.

Problem 2: CI is often over-provisioned

No matter who or what CI system you're using, go figure out how to get a breakdown of the resource usage during a given job. Most CI systems will provide this information in some form. Every Depot product gives you a live view of CPU, memory, and disk usage for a given job. You can also query this information via the API.

It is critical to understand how many resources your jobs are actually using. Many teams over-provision their CI jobs, allocating more CPU, memory, or disk space than necessary. This is a waste of compute and money.

Oftentimes, folks will increase their runner size because they hit an OOM error or a timeout. They bumped it up to the next size, it worked, and they moved on. But those types of errors often get fixed over time and nobody circles back (see problem 1 above). So, you end up with a job that is over-provisioned.

Of course, there are some jobs that are resource intensive and require a lot of CPU or memory. But, if you see an average CPU that is 10% or less, you should benchmark the job on a smaller runner. If the performance is roughly the same, save the cost of the larger runner.

If you want an agent to monitor for over-provisioned jobs, you can use the following prompt:

Find the CI jobs in our repos that run on bigger runners than they need.

1. List the 10 most expensive jobs over the last 14 days with their runner
   size, runs per day, p50 duration and cost per day.
2. For each one, get CPU and memory usage across the job, average and peak,
   from at least 5 runs. If our CI provider doesn't record it, add a step to
   a few runs that samples `vmstat 5` in the background.
3. Flag any job averaging 10% CPU or less. Recommend the smallest runner that
   still fits its peak memory with headroom. Size to the peak, not the
   average.
4. Flag any job pinned near 100% CPU or near its memory limit as undersized.
5. For each change, estimate the savings per day and what it does to the
   job's duration, with one example run link.

Don't change anything yet. Give me a table and I'll pick which ones to
benchmark.

Problem 3: Container images built on the CI runner

This one is close to my heart because it's the problem Depot started with. It's also consistently the biggest single line item we find when we look at a customer's CI.

The pattern looks like this. A workflow runs docker build right on the CI runner. Every run starts a fresh BuildKit with an empty cache, so every layer gets rebuilt from scratch. The build is slow, so somebody bumps the runner up to 32 or 64 CPUs. But most of that build is pulling base images, downloading dependencies, and exporting layers. The CPUs mostly sit there.

One team we worked with this fall had their image build jobs on 32 and 64 CPU runners averaging about 2% CPU. Another was pulling a multi-gigabyte TensorRT base image from scratch on every run, on a 32 CPU runner that averaged under 1 vCPU.

The fix is to build on something that keeps its layer cache between runs. A persistent BuildKit instance, a remote builder, our container builds, whatever fits. Then drop the runner down to 2 or 4 CPUs, because all it's doing now is waiting on the build.

Two other things show up in these Dockerfiles over and over again:

  1. A chown -R or chmod -R at the end of a stage. It rewrites every file it touches into a brand new layer. One team had a single chown -R that took upwards of 20 minutes of every build and added a 20 GB layer. Use COPY --chown or set ownership in the RUN that creates the files.
  2. Every job building the same image. One team had every e2e job build the same test image. It was roughly 1,650 builds a day. All of them were fully cached, and each one had to start just to load the result back that was fully cached.

An agent can help trace where that build time goes:

Find the container image builds in our CI that waste time and money.

1. Find every step that builds an image (docker build, buildx, bake,
   compose build). For each one, note the runner size, how often it runs,
   and whether its layer cache survives between runs. Logs showing a fresh
   docker-container builder, or every RUN step executing in full, mean
   it doesn't.
2. Measure runner CPU during the build. A low average means the runner is
   oversized for what is mostly waiting.
3. Read each Dockerfile and flag: chown -R or chmod -R on large
   directories, dependency installs after COPY . ., environment-specific
   ARGs above expensive layers, a missing .dockerignore, package installs
   without cache mounts, and no-cache turned on by default.
4. Flag any image that several jobs in the same workflow run each build or
   load separately.
5. For each finding, estimate the minutes and dollars saved per day and
   include one example run link.

At the very least, you should be able to see where your container builds are inefficient even if you keep them on the runner. Your agent will spot anything where caching isn't working, multi-platform image builds are slow, or you're building the same image multiple times when you could just build it once.

Problem 4: Runs that never needed to happen

The cheapest CI run is the one that never starts. And with agents pushing more code than ever, the number of runs that never needed to happen is climbing fast. This has quickly become the number one recommendation we send to folks.

Here are the ones we see most often:

  1. The same commit tested two or three times. The full suite runs on the PR, again in the merge queue, and again on main. If main only ever receives code that already passed, that last run isn't buying you much. The merge queue run likely needs to always run because of the nature of a merge queue. But you can often make your PR runs faster and cheaper by keeping them targeted while also validating that the code is safe to merge.

    For example, run a fast, targeted suite on the PR and a full matrix in the merge queue. Or, run a fast suite on the PR and a full matrix on main only if the PR passes. This is also a good time to mention, don't do silly things that require your merge queue to do a ton of rerunning (an example of silly would be writing a timestamp to a file directly on main that causes the merge queue to refresh).

  2. Fan-out storms. One team had a workflow that dispatched another workflow once per item it was testing. A single commit to main kicked off around 285 runs and 4,000 jobs in 90 minutes, where a normal commit ran 10 to 15. On the worst day, it cost them over $7,000.

  3. Bot branches. Dependency bots like Renovate rebase their branches every time something merges to main, and every rebase re-runs CI. For one team, 36 Renovate branches were about a third of a workflow's cost.

  4. Agent branches. For another team, roughly 78% of all test reports came from coding agent branches. Agents open more PRs and push more often. Run a fast, targeted suite on agent and feature branches and keep the full matrix for the merge queue.

Start by asking your agent which runs account for the spend:

Find the CI runs in our repos that never needed to happen.

1. Group the last 14 days of runs by trigger and branch type: default
   branch, merge queue, human PRs, bot branches (Renovate, Dependabot) and
   coding agent branches (claude/*, codex/*, cursor/* and similar). Show
   runs and cost per day for each.
2. Find commits whose full suite ran more than once (PR, merge queue,
   default branch, re-runs) and what the duplicates cost.
3. Find any commit that triggered far more runs than a normal commit, and
   the workflow or dispatch that caused it.
4. Check that superseded PR runs get cancelled and that docs-only or
   unrelated changes skip suites they can't affect.
5. Recommend a fix for each, with the cost per day it saves and an example
   run link. Never suggest cancelling or skipping deploy, release or
   default-branch runs without asking me first.

Problem 5: Failing slowly

Arguably, a failing run is the most expensive run because it has to be fixed/retried.

There are two flavors of this we run into constantly. The first is jobs with no timeout. GitHub Actions lets a job run for 6 hours before it gets killed. One team we worked with had hung jobs push their cancelled minutes from about 5,000 a month to about 55,000. Set a timeout-minutes value on every job at two or three times its normal duration.

The second is a failure you could have caught in seconds, discovered minutes into an expensive job. One team's package mirror credentials broke. For days, about half of a large e2e suite failed 5 minutes in, on 32 CPU runners. On another day, 97% of runs of a job on 64 CPU runners failed because it couldn't reach the package index. That one day cost roughly $1,900 for nothing. A 10 second preflight check that the mirror is reachable would have caught both.

Ask your agent how much time your CI spends on jobs that fail:

Find the CI spend we're wasting on failed, cancelled and timed out jobs.

1. For the last 14 days, show the share of minutes and cost spent on
   failed, cancelled and timed out jobs, by workflow and job.
2. List jobs with no timeout, and jobs that ran far longer than their p95
   before being cancelled. Suggest a timeout at about 2-3x the p95.
3. Group failures by their first error line. Flag any failure that happens
   on the same step every time, especially credentials, network access,
   missing images or lint, and suggest a cheap check that runs first.
4. Flag tests that fail and then pass on retry for the same commit, and
   jobs that retry the whole job instead of the flaky step.
5. Estimate the cost per day of each and include example run links.

Problem 6: Caches that grow forever or stop working

Caches can save you money by making your jobs more efficient. Pretty simple, don't recompute work that can be reused. But it's surprising how often caches stop working or grow out of control in teams.

One team made a small change to their test matrix. Every shard then compiled the same Go code and wrote its own copy to the cache. Their Go cache went from 20 TB to 61 TB in about a week, and kept climbing to 85 TB. Compiling the test binary once and sharing it with the shards, plus a retention policy, brought it back under 10 TB.

The other failure mode is quieter. You stop hitting cache after some commit changes an input every run, like a timestamp, an absolute path, or an unpinned tool version. Nothing breaks and CI doesn't turn red. But everything gets slower, more expensive, and nobody notices for months (problem 1 again).

Ask your agent to find caches that are growing too large or missing too often:

Audit the caches our CI uses.

1. List every cache: CI cache actions, Docker layer cache, registry cache,
   remote build caches (Bazel, Gradle, Turborepo, Nx, sccache, Go). For
   each, show its size over the last 30 days and its hit rate per run.
2. Flag any cache growing much faster than usual, and find the change that
   started it.
3. Flag any cache whose hit rate dropped. Diff the cache inputs between a
   hit and a miss to find the input that changes every run.
4. Flag matrix jobs that all compile the same code, and two cache layers
   doing the same job.
5. Recommend retention settings for caches, registries and artifacts, and
   estimate the time and storage each fix saves.

Problem 7: Paying for setup on every job adds up at a certain volume

Setup gets paid on every job, every shard, and every retry of failed or flaky jobs. So a few seconds of setup here and there isn't a big cost savings opportunity if we are talking about tens of jobs per day. That math quickly changes when teams start pushing tens of thousands of jobs per day.

Checkout, toolchain installs, dependency installs, and starting services before the tests can run. On one team we worked with, checkout alone was 28% of their job time. Another cut per-job setup from 27 seconds down to 1 by caching the installed environment instead of the downloads.

This matters even more when you shard. Sharding makes the wall clock shorter, not the bill smaller. Every new shard pays the setup again. So cut the setup before you add shards.

Have your agent add up the setup time across runs:

Find the fixed setup time our CI pays on every job.

1. For the 10 most expensive jobs, break down step timings from the logs:
   checkout, toolchain setup, dependency install, cache restore and save,
   service startup, the actual work, and artifact upload.
2. Multiply each setup step by runs per day, including every shard and
   retry, to get the minutes and cost it adds per day.
3. Suggest fixes: shallow or sparse checkout, a prebuilt CI image for
   slow-changing tools, caching the installed environment instead of the
   downloads, readiness checks instead of fixed sleeps, and merging tiny
   jobs.
4. For sharded jobs, compare setup time to test time per shard and say
   whether we have too many shards.
5. Estimate the savings per day for each, with an example run link.

Don't wait for the bill

None of the examples above are exotic. It's also not a comprehensive list. I would consider these things the low-hanging fruit that folks often overlook.

A workflow somebody wrote when the project started with the best intentions. A runner somebody bumped up in size after an OOM. A cache nobody ever put a retention policy on. All of it made sense at the time, and nobody went back to look.

That was fine when a team pushed a handful of PRs a day. It's not fine when agents are pushing hundreds, thousands, and even tens of thousands. Every bit of waste in a workflow gets multiplied by every run, and the number of runs is only going up.

The best method we have found to reduce CI costs is to continuously monitor and optimize your workflows to make them more efficient. Don't wait for the bill to arrive before you start looking for inefficiencies.

Start building with Depot
in minutes