Durable logs for Depot Sandboxes and CI

As Depot grows, so does the volume of logs our customers produce. Our previous log platform was reaching its limits, so we built a new one specifically for our workloads. We wanted it to keep up with that growth and let us keep logs for as long as they're useful.

So what does this change for you? Depot CI logs no longer expire after 14 days, and you can now retrieve Sandbox logs after the sandbox has stopped. For workflows that are still running, you can stream logs as they arrive.

A log platform built for workflows

We spend a lot of time making CI jobs faster. That means a lot of short jobs. Here's the duration distribution for GitHub Actions jobs completed on Depot over a 24-hour period. The largest bucket contains jobs lasting less than 30 seconds, with a long tail of longer jobs.

Our workloads also have a useful property: they finish. A workflow starts, runs its jobs and steps, and produces a finite set of logs. We designed around that lifecycle and its high cardinality from the start.

Four design principles guided the implementation:

• Keep long-term storage cheap. We store all logs in S3 so keeping them for a long time stays affordable. We wanted retention to be a product decision.
• Make logs durable as soon as possible. Getting logs into durable storage is the first priority.
• Keep dependencies limited. S3 and SQS are the only external dependencies. Having no database to operate will make the platform easier to deploy through Depot Managed.
• Optimize the write path. We keep ingestion simple and do a bit more work when you query the logs. For example, retries can cause a log record to be processed more than once. We filter out duplicates when reading the logs.

We'll cover the design and trade-offs in a follow-up post.

Durable logs for the Sandbox SDK

Previously, you could stream output from a sandbox, but once it finished, those logs were lost. You can now come back later and query them through the Sandbox SDK. All you need is the sandbox ID:

import {createClient, Sandbox} from '@depot/sandbox'

const client = createClient({
  token: process.env.DEPOT_TOKEN!,
  orgID: process.env.DEPOT_ORG_ID!,
})

// Revisit a sandbox after it has stopped.
const sandbox = await Sandbox.get(client, process.env.SANDBOX_ID!)

for await (const entry of sandbox.logs.tail({entries: 100})) {
  console.log(`${entry.source}: ${entry.message}`)
}

The example fetches the last 100 records. You can also use sandbox.logs.fetch() to query a time range or sandbox.logs.stream() to follow live output.

If you're integrating directly with the API, see the protobuf service definition and log request and record types.

Built for agents, one query at a time

We wanted to build a log platform that works well for agents. Dumping 100,000 lines into an agent's context leaves less room for the code and the problem it's trying to solve. Instead, an agent should be able to fetch a small part of the output and ask quick follow-up questions: read the last 100 lines, look at one step, then expand the scope if it needs more context.

For recently finished jobs, we keep the last half hour of logs cached. As the duration chart above shows, most of our jobs finish well within that window. The cache therefore covers a typical job's full output when an agent starts investigating immediately after it finishes, while the durable storage pipeline is still processing the logs. We also support Server-Sent Events (SSE), so an agent can follow logs as they arrive while the workflow is running.

Historical queries benefit from caching too. When an agent requests a few lines, our query workers preload the execution's log data onto local NVMe SSDs. Subsequent requests can reuse that data without downloading it from S3 again, making the back-and-forth faster as the agent investigates.

Indefinite log retention for Depot CI

Depot CI previously kept logs for 14 days. That's a short window if you're investigating a problem that only shows up occasionally or want to compare a recent build with one from a few months ago. With the new platform, we're keeping Depot CI logs indefinitely, so you can come back to those older runs.

Have questions or feedback about durable logs? Let us know.