How can I benchmark build execution speed across CI platforms?

The fairest way to benchmark build execution speed across CI platforms is to run the same commit, job commands, CPU architecture, machine size, region, and cache state on every platform. Measure several runs per configuration in parallel on each commit and report both p50 and p95, not the fastest run.

Start with the workflow your team actually waits on. A synthetic CPU benchmark can tell you which processor is faster, but it can't tell you how long your pull request checks will take. Real CI jobs include checkout, dependency restores, disk I/O, compilation, tests, artifacts, and runner startup. The benchmark should include all of them.

GitHub Actions runners managed by Depot make the first comparison simple. Keep GitHub Actions and run the same job side by side by changing only the runner label:

name: Runner benchmark

on:
  workflow_dispatch:

jobs:
  benchmark:
    name: ${{ matrix.name }}
    strategy:
      fail-fast: false
      matrix:
        include:
          - name: Current runner
            runner: ubuntu-24.04
          - name: Depot GitHub Actions runner
            runner: depot-ubuntu-24.04
    runs-on: ${{ matrix.runner }}
    steps:
      - uses: actions/checkout@v4

      - uses: pnpm/action-setup@v4
        with:
          version: 10

      - uses: actions/setup-node@v4
        with:
          node-version: 22
          cache: pnpm

      - run: pnpm install --frozen-lockfile
      - run: pnpm test
      - run: pnpm build

Both matrix jobs receive the same event and commit. The commands are identical. The standard 2 vCPU Depot label keeps the advertised resource count close to a standard private-repository Linux runner, while Depot's faster compute, in-memory disk accelerator, and cache transfers of up to 1000 MiB/s show what changes when the runner infrastructure gets faster.

Our Next.js runner benchmark used this pattern. On equivalent 2 vCPU runners, the Next.js build step took 105.6 seconds on a GitHub-hosted runner and 61.7 seconds on Depot. The important part isn't the headline. The workflow and pinned source are public, and the result separates dependency installation from the build itself.

Keep the benchmark controlled

A runner comparison is only useful when the workload is equivalent. Record these inputs with every result:

Two runners with the same vCPU count can still perform differently. CPU generation, sustained clock speed, storage, memory bandwidth, and noisy neighbors all matter. Match the published resources first, then use the real workload to measure the hardware underneath them.

Don't optimize the workflow for one platform before the first comparison. Establish a like-for-like baseline. Then test platform-specific acceleration, larger machines, custom images, or different cache backends as separate experiments.

Measure the whole path

"Build time" often means whatever number makes a chart look best. Break it into clocks with clear boundaries:

This distinction matters. A runner can compile quickly and still deliver slow feedback because jobs wait for capacity. A fast cache download can still lose time extracting thousands of files onto a slow disk. A platform can start one job quickly but fall over when a 40-job matrix arrives.

Use platform timestamps for queue and startup measurements. Time important steps inside the job so the definition stays consistent across platforms. Keep raw durations for every run instead of copying the rounded number from a UI.

Test cold and warm caches separately

Run two benchmark groups:

1. Cold cache: Use a unique cache key or clear the benchmark cache so every platform records a real miss.
2. Warm cache: Seed each platform's cache once, rerun the identical commit, and verify the logs show a cache hit.

Don't average cold and warm runs together. They answer different questions. Cold runs measure the cost of rebuilding state after invalidation. Warm runs measure the feedback loop developers see when the dependency graph hasn't changed.

Also report restore and save times separately. A cache that restores quickly but takes several minutes to upload can still lengthen the critical path. If two jobs race to populate the same key, discard the run and fix the benchmark.

Depot automatically accelerates standard actions/cache and actions/setup-* usage, so the workflow doesn't need a proprietary cache action. That keeps the comparison close to the workflow you already run.

Use p50 and p95 across enough runs

Run each platform at least 30 times for each cache state. Fifty runs gives a more useful p95. Spread the runs across the hours when your team normally uses CI so the benchmark includes real capacity and network variation.

Report:

• p50 for the typical developer experience
• p95 for the slow runs that break flow state
• minimum and maximum to expose the range
• failure and retry counts
• sample size and the time window when the runs occurred

A mean on its own hides long-tail delays. One screenshot of the fastest Depot run and the slowest run elsewhere proves nothing. Publish the raw results or a link to every run so another engineer can check the calculation.

Don't remove an outlier just because it hurts the result. Remove a run only for a documented reason, such as an application failure unrelated to the platform or a cache state that didn't match the test. Keep platform failures, capacity waits, and infrastructure retries in the data. Those are part of CI performance.

Calculate cost per completed job

The cheapest minute isn't always the cheapest completed job. Faster machines can cost more per minute and less per result. Billing minimums, failed attempts, cache storage, artifact storage, and network charges can change the answer again.

Use this calculation:

cost per completed job =
  total compute + platform + cache + storage + network charges across every attempt
  -------------------------------------------------------------------------------
                         successful completed jobs

Include failed attempts and automatic retries in the numerator. For a matrix, add the cost of every job needed for the workflow to pass. Then compare cost per completed job alongside p50 and p95 end-to-end duration.

Depot usage is tracked per second with no one-minute minimum. The runner types and pricing table lists current machine sizes and rates, so the calculation can use actual elapsed time instead of rounded minutes.

Inspect the bottleneck behind the number

A duration tells you which run won. Resource telemetry tells you why.

• CPU near 100% through compilation or tests means faster cores or more useful parallelism can help.
• Memory near the runner limit can cause garbage collection, swapping, or termination.
• High disk wait during dependency extraction, container builds, or linking points to storage throughput.
• Low CPU during downloads points to network throughput, registry latency, or cache performance.
• Idle resources during a long test step usually mean the test graph needs sharding or more internal concurrency.

The Depot GitHub Actions analytics page shows step timing, CPU, and memory data for jobs across the organization. Use those graphs with job-level disk and network measurements to explain the result. Depot also identifies slow jobs and recommends larger or smaller runner sizes from historical resource use.

Once the like-for-like test is complete, repeat the benchmark on a larger Depot GitHub Actions runner. If an 8 vCPU job finishes in less than half the time of a 4 vCPU job, it can improve both feedback time and completed-job cost. If CPU stays idle, more cores won't fix the bottleneck.

Compare complete CI platforms separately

A runner benchmark isolates compute, disk, cache, and provisioning while keeping the GitHub Actions control plane and workflow syntax constant. Comparing a complete CI platform adds scheduling, workflow parsing, dependency handling, logs, artifacts, retries, and API latency. That's a different experiment.

For a full-platform comparison, port the same job graph without changing its commands or parallelism. Measure from one triggering event to the final required result. Run the same cold and warm groups, preserve every retry, and report any feature that couldn't be represented identically.

Depot CI can run existing GitHub Actions workflows on Depot's own CI engine. That makes it possible to benchmark the full feedback loop after the runner-only comparison, without first rewriting every command in another CI format.

The result worth keeping isn't the benchmark with the biggest multiplier. It's the configuration that shortens p50 and p95 for your real workflow, survives burst traffic, and lowers the cost of a completed job.