Benchmarks
ModernCI publishes its benchmark method before any results. The plan compares ModernCI with GitHub Actions, Blacksmith and Namespace on Rust builds, cost per merged change, agent pull-request-to-merge time, setup effort, failure handling and security boundaries. It uses public repos at pinned commits, matched 4-vCPU Linux runners, published commands and raw data. No comparative results yet: measurement in progress. Nothing on this site claims a speed or cost advantage until those numbers are published here.
Platforms
| Platform | Runner | Merge queue |
|---|---|---|
| GitHub Actions | GitHub-hosted larger runner, Ubuntu 24.04, 4-core, 16 GB | GitHub merge queue |
| Blacksmith | blacksmith-4vcpu-ubuntu-2404 | GitHub merge queue |
| Namespace | profile namespace-profile-bench-4x16 (4 vCPU, 16 GB, Linux amd64) | GitHub merge queue |
| ModernCI (released version named per run) | Latchkey runner, 4 vCPU | ModernCI's own |
Each runner records its own CPU model, core count and memory, so any mismatch shows in the data. Every platform runs twice: with no cache, and with the caching its vendor documents (Swatinem/rust-cache on Actions and Blacksmith, a cache volume on Namespace, result reuse on ModernCI).
Workloads
| Workload | What is measured |
|---|---|
| W1 Rust clean build and test | ripgrep, tokio and nushell at pinned release tags: cargo build --workspace --all-targets --locked, then cargo nextest run --workspace --locked. Wall time, queue time, billed minutes, cost |
| W2 Rust incremental | Four patches from the same base (a leaf crate, a core crate, a lockfile bump, an added test) after a warm-up |
| W3 Cost per merged change | 50 merged tokio pull requests replayed, plus seeded failures and one conflicting pair, arriving steadily (20 an hour) and as a burst. Cost per merged change, time to drain, pull request to merge p50 and p95, wrong failures |
| W4 Agent pull request to merge | A deterministic bot (no AI model) opens 20 pull requests. 6 fail first, and the bot pushes the prepared fix 120 s after it receives the failure |
| W5 Setup and maintenance | From an empty fork to a green required check with auto-merge, following public docs only; then every manual action during W3 and W4 |
| W6 Failure and retry | A flaky test, a killed build, a hung test, out of memory: whether each is reported as a test failure or an infrastructure fault, time to detect, retries spent |
| W7 Security boundary | What a pull request's own job can see: token presence and permissions, cloud metadata reachability, root, persistence between jobs, secrets on fork pull requests. Observation only, within each vendor's terms |
Metrics
All times are UTC from the Git provider's API timestamps, so every platform is measured by the same clock.
- push to result:
git pushreturned → the last required check on that commit completed. - queue: job created → job started. wall: job started → job completed.
- billed minutes and cost: each vendor's rounding rule × its list price, read on the run date (URL and date recorded).
- cost per merged change: all compute cost in the window, failed and cancelled runs included, ÷ pull requests merged.
- wrong failure: a required check fails on a commit, then passes on the identical commit with no code change; or an injected infrastructure fault is reported as a test failure.
Raw data
One NDJSON line per job, published with a manifest (platform and runner image versions, toolchain, ModernCI version, repo commits, the price table with sources) and sha256 checksums. Medians, p95, interquartile range and run counts are reported per cell. The harness is a public repo, and anyone with the same accounts can rerun it.
Results
| Workload | GitHub Actions | Blacksmith | Namespace | ModernCI |
|---|---|---|---|---|
| W1–W7 | measurement in progress | measurement in progress | measurement in progress | measurement in progress |
Internal baseline (not a comparison). busbar, a private Rust workspace, on ModernCI's own 8-vCPU host, 2026-10-12 to 13 (18.3 h, from the install's usage records): median push to green result 43 min, 37 min of it queued, because engines up to 0.4.28 run one pull request's checks one after another. This is the number parallel checks have to beat.
Vendors are sent this method 7 days before results are published, and any correction is published with them. Questions about the method: [email protected].