How we load tested Depot Metal before scaling up

We've recently been working on moving more GitHub Actions jobs onto Depot Metal so they can take advantage of the improvements in the new platform. Before moving traffic over, we needed to understand how the rest of the system would behave at that scale.

While we would eventually find the limits once enough production traffic arrived, we wanted to find and address issues before they could affect customers.

Existing setup

We have a preview / staging environment, but it's not exactly just a smaller copy of production. Various components use different machine sizes and have different scaling mins and maxes. And some components don't have a corresponding copy in that environment. One of the biggest differences is that it receives drastically less traffic. Testing behavior in preview can tell us whether something generally works, but it cannot tell us how it will behave under production load.

How could we put realistic pressure on the system without reproducing all of production?

Replaying production without recreating it

GitHub is normally the trigger for these jobs in real use cases, but we wanted to avoid using it to generate the amount of traffic we needed.

Real production traffic comes from many different GitHub organizations and GitHub Apps. If we tried to generate the same volume through a single test organization or app, we would run into GitHub's rate limits long before reaching the scale we wanted to test. At that point, we would mostly be testing how quickly GitHub rate limited us rather than how our own system behaved under load.

We were not trying to load test GitHub. We wanted to understand what happened inside the part of the system we operate after GitHub delivered an event. And while these rate limits are something we run up against and need to consider, we don't want them to prevent us from scale testing the rest of our system.

Starting at the webhook boundary

One of the main boundaries between GitHub and our system is the webhook. We built a tool that could dispatch synthetic jobs into our preview environment and mimic the webhooks we normally receive. The jobs were fake, but they still exercised the real orchestration work that happens around each job. We stopped before the part where we would need to start a real machine and execute an actual customer workload. This gave us control over both the amount of traffic and how that traffic arrived.

Initially, we could tell the tool to launch a fixed number of jobs at a given rate. This was useful for putting a lot of immediate pressure on the system, but it was not necessarily representative of production. Real CI traffic is uneven. Jobs arrive in bursts, they use different runner sizes, and some finish in seconds while others run for hours.

Shaping the traffic like production

To get closer to that behavior, we created a production-shaped traffic profile based on a real day of usage. We captured the minute-by-minute arrival curve, the mix of runner sizes, and how long the jobs ran. We could then replay a selected part of that day at a different scale. Scaling changed how many jobs arrived, but it did not compress the time. Replaying one production hour still took one hour. This preserved the shape of the traffic, including the quieter periods and the bursts, rather than turning everything into one large spike.

We ended up with two useful testing modes:

• A production-day replay for sustained, production-shaped traffic
• A launch-ingestion test for controlled bursts

Each run was deterministic for a given seed and produced the same workload, making runs repeatable.

Testing the test

Building the first version of the harness was not the same thing as having a realistic load test. We kept finding places where the synthetic workload behaved differently enough from production that it changed what we were measuring.

Some of these were simple bugs. Other differences were more subtle. At one point, every synthetic run used the same workflow. This created an artificial hotspot where concurrent runs all tried to update the same piece of state. We started hitting transaction pool errors, but the test was creating a much more concentrated access pattern than we would normally see in production, so we had to resolve that. The transaction pool errors were still useful because they showed us something real about how the system behaved under contention, but they were not evidence of the same production capacity limit we originally thought we were measuring.

We also discovered that the early test only represented part of a job's lifecycle. It created the initial queued work but did not send all of the later lifecycle events that normally associate, update, and complete that job. Those later events produce their own work, so omitting them made the workload incomplete. Once we added the full lifecycle, the test was more realistic, but our previous baselines were no longer comparable. We needed to establish them again.

This happened a few times as we built out the harness. We were finding bottlenecks in the system, but we were also gradually learning what we needed to reproduce for the test itself to be meaningful.

Deciding when the test was good enough

We could have kept adding more production behavior indefinitely, but each addition risked invalidating measurements and forcing us to establish another baseline. We did not need the test to reproduce production perfectly. We needed it to exercise the behavior we were trying to measure, produce consistent results across repeated runs, and show a meaningful difference when we changed the system.

We considered the test useful once it let us measure a before and after and see whether a change improved the system. Improving the harness sometimes meant establishing a new baseline, but that did not invalidate what earlier versions had already taught us. Each version helped us find the next place where either the system or the test itself needed to improve.

Watching what happened under load

Generating the traffic is only half the experiment. We also needed to understand how the system responded to it.

Establishing a baseline

Before each run, we captured a quiet ten-minute baseline from a mostly unused preview environment. During the test, we recorded measurements at regular checkpoints and around the expected peak concurrency. We used Honeycomb traces, CloudWatch logs and metrics, and PlanetScale Insights to build a picture of what was happening. We looked at things like queue depths, application resource usage, concurrency, database latency, and the time it took jobs to move through their lifecycle.

No single one of those measurements was enough on its own. High resource usage can be perfectly healthy if work continues moving through the system. A growing queue is more concerning if the oldest message also keeps getting older. The interesting point was where adding more load stopped producing proportional results and started creating growing backlogs, errors, or slower lifecycle times.

Making each run comparable

The hardest part of the observability work was correlating one run across several different tools. We had plenty of telemetry, but no single view that answered, "What changed across the whole system during this experiment?" We ended up creating a runbook for collecting the same measurements each time and keeping the run ID, time window, workload manifest, and relevant versions with the results.

PlanetScale Insights was particularly useful once database pressure started increasing. It showed us which queries were running most frequently, which operations were reading or updating far more than expected, and where inefficient access patterns became much more expensive under load.

That gave us a concrete list of work to address. We could make a change, replay the same workload, and see whether it had the effect we expected. Sometimes the failure point moved. Sometimes another dependency became the new limiting factor. Sometimes a change that looked promising did not make much difference at all.

Without a repeatable workload, it would have been much harder to separate the effect of our change from ordinary differences in production traffic.

What this did not test

This was not an exhaustive test of everything involved in running a customer job.

We exercised the workflow orchestration under realistic synthetic demand, but we did not reproduce every downstream operation end to end. In particular, starting an equivalent number of real machines would have created a substantial infrastructure bill. That was not necessary for the control-plane questions we were trying to answer, so we deliberately excluded it.

This means some of our capacity conclusions came from measurements plus extrapolation rather than direct full-scale testing. The results helped us identify bottlenecks and gave us more confidence in the rollout, but they did not prove one absolute production limit.

There is always a tradeoff between fidelity, cost, and how quickly you can run the next experiment. In our case, reproducing the coordination work surrounding each job gave us useful information without also paying to reproduce all of the compute performed by that job.

Results and what we can reuse

We originally built the harness to help answer questions about scaling Depot Metal, but the lasting value is larger than a single rollout.

We now have a reusable production-traffic profile and a repeatable process for running tests, collecting measurements, and comparing results. We can expand the harness as the system changes and use it to validate future performance work.

The testing itself paid off. As we moved more traffic onto Depot Metal, we did not hit the limits we had found and addressed. We found other issues in areas we had intentionally excluded because they were too expensive to reproduce at scale, and addressed those separately.

We will still learn new things as production usage grows. The difference is that production is no longer the first and only place where we can test the limits.

FAQ