Methodology

The public arena runs in simulation mode by default: each provider has a performance profile (provision latency, cold start, CPU / IO / network factors, reliability) seeded from the published third-party benchmarks linked below, and a deterministic engine plays a race from a match seed. The same seed always reproduces the same race — that is what makes a replay link trustworthy.

When ARENA_LIVE=1 and a provider's credentials are present, the matching adapter runs a real sandbox instead (Dedalus via its public exec API; E2B via its SDK). Mode is surfaced in the UI on every page. There is no silent fallback: live runs are real, simulated runs are labelled.

Cloud sandboxes need accounts and API keys — there is no keyless way to call E2B, Modal, Daytona and the rest. For a real run with zero accounts, the bundled local executor runs the workload commands on your own machine (ARENA_LOCAL_EXEC=1 pnpm race:local npm-install): genuine execution, real timings, $0. Your machine is not a cloud sandbox — so it is a demonstration of the pipeline, not a ranked contestant.

Both sides of a match get the identical workload bytes, the same resource class (vCPU / RAM), the same region where possible, and the same warm/cold condition — and those conditions are declared on every match. We never secretly race a warm pool against a cold competitor.

For each workload we take the median run per provider, then normalize across the field:

• Speed — total wall time (provision + cold start + exec), min-max normalized. Weight 0.5.
• Reliability — probability the run verifies, with a long-run stability penalty. Weight 0.3.
• Cost — USD billed over the workload execution window once the sandbox is ready; provisioning and cold start are not billed. From published per-second rates, min-max normalized. Weight 0.2.

The per-workload scores are averaged into the overall composite. It is recomputed from medians, never from human-triggered matches, so votes can never move it.

Votes update a Bradley-Terry / Elo rating (K = 24), tracked separately from the objective number and shown beside it — never blended in. One vote per voter per match, deduped by a hashed anonymous cookie. Pairings are randomized and anonymized as Sandbox A / B until after you vote, to prevent brand bias.

The public race runs a fixed, audited catalog only — never arbitrary anonymous code. There are nineworkloads, balanced across dimensions: cold start; two compute (a pure-CPU SHA-256 hash chain and a numpy/pandas import); package install; two network (a depth-1 git clone and a pinned HTTPS fetch); two filesystem (a metadata storm and a 256 MiB write); and a realistic "build a Next.js app" task. Every workload carries a verifier — a reference digest, a checksum, an exact byte count, a resolved HEAD — so a fast-but-broken run cannot win. The exact command for each is published in its info card, and every reference constant is reproducible from that command.

• Northflank — AI sandbox pricing comparison (2026)
• Ry Walker — AI Agent Sandboxes Compared
• Beam — Best stateful sandboxes for code execution (2026)
• Community sandbox benchmark (nickaggarwal/sandbox-test)
• Tensorlake sandboxes docs
• Declaw — Firecracker architecture

Pricing marked with an asterisk on the compare page is our estimate from peer rates because the vendor does not publish a public per-vCPU figure.