GitHub Copilot Benchmarks Site: Harness Comparisons and Terminal-Bench-Science
The GitHub Copilot benchmarks site added a Harnesses view in public preview. It compares Copilot CLI with Claude Code and Codex CLI using the same model at the same reasoning effort. The site also added Terminal-Bench-Science 0.1, which tests agents on research workflows across scientific domains.
Key Takeaways
- The new Harnesses view compares Copilot CLI, Claude Code and Codex CLI on the same model at the same reasoning effort.
- Results are plotted as resolution rate versus mean cost per task, with ellipses showing run-to-run variation.
- Paired differences report Copilot CLI's resolution advantage in percentage points and its cost savings, with approximate 95% bootstrap intervals.
- Only exact model pairs at medium reasoning effort appear, so Copilot Fusion and unmatched results are excluded.
- Terminal-Bench-Science 0.1 tests agents on research workflows, with tasks authored by domain experts across life, physical, earth, mathematical and engineering sciences.
- Both features are in public preview, and the data comes from GitHub's own benchmarking, not an independent party.
Comparing Agent Harnesses Head to Head
GitHub's Copilot benchmarks site gained two public preview features on November 17, 2026. The headline addition is a Harnesses view. It compares GitHub Copilot CLI with Claude Code and Codex CLI while holding the model and the reasoning effort constant. That isolates the harness, the tooling around the model, as the variable being measured.
What the View Shows
The Harnesses view includes a scatter plot of resolution rate against mean cost per task. Lines connect matched results, and shaded ellipses show run-to-run variation. For each matched model, the view also lists paired differences. These show Copilot CLI's resolution advantage in percentage points and its cost savings as a percentage, each with approximate 95% bootstrap intervals.
Scope Limits
The comparison is deliberately narrow. As GitHub puts it, "Only exact model pairs at medium reasoning effort are shown." Copilot-only configurations such as GitHub Copilot Fusion, and results without a match, are left out.
Terminal-Bench-Science 0.1
The second addition is Terminal-Bench-Science 0.1, a benchmark that evaluates AI agents on research workflows. Domain experts in the life, physical, earth, mathematical and engineering sciences wrote and reviewed the tasks. Results are available through the benchmark picker.
Why It Matters
Benchmark scores usually mix model quality with harness quality. Matching model and effort across harnesses gives developers a clearer view of what the surrounding tooling contributes, in both accuracy and cost. Note that GitHub publishes and runs these benchmarks itself, and independent figures for Terminal-Bench-Science were not found.