GitHub Copilot Evals: Repeatable Tests for Skills, Agents and MCP Tools
GitHub released GitHub Copilot Evals in public preview. It lets developers build repeatable tests for Copilot skills, custom agents and MCP tools. Users can rerun the same eval after changing a prompt, model or tool setup and compare results to catch regressions. The tool is available in VS Code, Copilot CLI and the GitHub Copilot app for Individual, Business and Enterprise plans.
Key Takeaways
- Evals are repeatable tests for AI behavior, checking both the final result and the path the agent took to reach it.
- Rerunning the same eval after any change helps catch regressions that run-to-run variation would otherwise hide.
- The preview covers skills, custom agents and MCP tools, the main customization points in Copilot.
- Each run needs explicit user approval before it starts.
- Developers can compare before and after runs to see whether a prompt, model or tool change actually helped.
- Evals ships in VS Code, Copilot CLI and the Copilot app, running locally on the Vally engine for Individual, Business and Enterprise plans.
Testing AI Customizations Like Code
GitHub introduced GitHub Copilot Evals in public preview on November 17, 2026. The tool helps developers check whether their Copilot skills, custom agents and MCP tools keep behaving well as prompts, models and tools change over time.
GitHub defines an eval as "a repeatable test for AI behavior." A developer gives an agent a realistic task and defines what a good result looks like. The eval then checks both the final result and how the agent got there.
Why Repeatable Evals Matter
AI output varies from run to run, so one good answer does not prove that a customization works. A skill that performed well yesterday can regress after a model update or a tweak to an MCP configuration. Rerunning the same eval after each change gives developers a way to spot those regressions before they affect real work.
What Developers Can Do
Copilot Evals covers the full loop of creating and checking a test.
- Create an eval for a skill, a custom agent or an MCP tool.
- Run the eval. Each run requires the user's approval first.
- See which checks passed or failed, and why.
- Compare runs from before and after a change to see whether the change helped.
Engine and Installation
Copilot Evals runs on Vally, an extensible local eval tool. Vally's CLI is published as @microsoft/vally-cli and needs Node.js 22.14.0 or later plus a GitHub account with an active Copilot subscription. Evals can be installed through the VS Code extension, the Copilot CLI plugin, or the GitHub Copilot app via the github/evals marketplace.
Availability and Feedback
The public preview is open to Copilot Individual, Copilot Business and Copilot Enterprise plans. Feedback can go through an issue in the github/evals repository or the discussion in GitHub Community. Full documentation is at aka.ms/evals.