How do I test AI-generated pull requests in CI?

Test AI-generated pull requests with the same required checks as human changes. The difference is the feedback loop. An agent should be able to run the smallest useful CI job against its uncommitted changes, inspect the failure, fix the code, and rerun before it opens or updates a pull request.

Depot CI is built for that loop. It runs GitHub Actions YAML on Depot's own CI engine and exposes runs, status, logs, diagnosis, retries, cancellation, and SSH through the CLI and API.

depot ci run --workflow .depot/workflows/ci.yml --job test
depot ci status <run-id>
depot ci diagnose --run <run-id> --output json
depot ci logs <attempt-id>

The agent can edit locally and run the workflow again without creating a commit. Every invocation uploads a fresh patch from the working tree and applies it after checkout.

Keep pull request validation deterministic

Start with required checks that produce a clear pass or fail:

• Formatting, lint, and type checks
• Unit and integration tests
• Security and policy checks that don't mutate external systems
• Container or application builds
• Generated-code and schema drift checks

Don't let an AI-generated pull request deploy, rotate credentials, modify production data, or publish a release as a side effect of ordinary validation. Put those actions in separate workflows with explicit permissions and approval.

A basic validation workflow can stay in GitHub Actions format:

name: Pull request validation

on:
  pull_request:

concurrency:
  group: pr-validation-${{ github.ref }}
  cancel-in-progress: true

permissions:
  contents: read

jobs:
  checks:
    runs-on: depot-ubuntu-24.04
    steps:
      - uses: actions/checkout@v4
      - uses: pnpm/action-setup@v4
        with:
          version: 10
      - uses: actions/setup-node@v4
        with:
          node-version: 22
          cache: pnpm
      - run: pnpm install --frozen-lockfile
      - run: pnpm lint
      - run: pnpm type-check
      - run: pnpm test

The concurrency group cancels stale validation for the same pull request ref when the agent pushes another revision. That matters when an agent can produce several fixes before an older run has finished.

Depot CI supports pull request workflows created from branches in the connected repository. Pull requests triggered from forks aren't currently supported. Keep GitHub Actions or another trusted validation path for public fork contributions until that changes.

Let the agent close the CI loop

Give the agent a bounded run-diagnose-fix-rerun procedure:

1. Run the smallest job that covers the change.
2. Poll status until the job reaches a terminal state.
3. Read depot ci diagnose --output json for bounded failure context.
4. Verify the suggested cause against the attempt logs.
5. Use SSH when the useful evidence exists only inside the sandbox.
6. Edit the code and rerun.
7. Stop after a fixed number of attempts or when the same blocker repeats.

The full coding-agent loop guide includes a reusable agent command, permission examples, and the exact CLI flow. Three to five attempts is a reasonable default before returning the blocker to a human.

depot ci diagnose is AI-generated. Treat it as a compact starting point, not proof. The agent should check the error lines and full logs before changing code.

Scope work before scaling it

An agent changing one package shouldn't start every test for every service while it's still debugging a type error. Use --job for local iteration, job dependencies for required setup, and a matrix only when the change is ready for the complete validation pass.

Depot CI lets you retry a failed job without rerunning jobs that already passed:

depot ci retry <run-id> --job <job-id>

Use depot ci cancel <run-id> when the patch has been replaced and the remaining result no longer matters. High-volume agent validation gets expensive when stale runs continue after a newer revision makes them irrelevant.

For large suites, split tests by historical timing instead of assigning the same number of test files to every shard. Depot uses reported JUnit durations to keep one slow shard from defining the whole pull request latency.

Keep every agent run isolated

Each Depot CI job runs in its own sandbox. That gives agent-generated code a clean filesystem and prevents one job's files or processes from leaking into the next job.

Isolation doesn't make every secret safe to expose. Apply least privilege:

• Set permissions: contents: read unless a job needs more.
• Limit Depot CI secrets by repository, branch, workflow, and environment.
• Don't provide deployment credentials to the pull request workflow.
• Require approval before the agent changes secrets, variables, release behavior, or infrastructure.
• Keep untrusted fork code out of any workflow that receives credentials.

Depot CI uses a GitHub App token for secrets.GITHUB_TOKEN. GitHub Packages doesn't accept that token for package authentication. Use Depot Registry, another registry, or a separately managed GitHub personal access token when the validation job must pull a private GitHub package.

Preserve evidence from failures

Upload JUnit XML even when tests fail. Keep the attempt logs, CPU and memory metrics, test output, and exact commit or local patch associated with the run.

An agent can fix only the failure it can observe. If a retry replaces the old logs, a sandbox disappears before it can be inspected, or every matrix failure is reduced to one generic status, the loop falls back to guessing.

Depot keeps every retry as a separate attempt. The agent can compare them, retrieve exact logs, or pause a rerun after a specific step:

depot ci run \
  --workflow .depot/workflows/ci.yml \
  --job test \
  --ssh-after-step 3

That's the useful boundary for autonomous validation. The agent can run checks, inspect failures, edit code, and retry. Humans still own changes to trust, credentials, release policy, and production.