Simon Willison releases smevals testing suite

Simon Willison, in collaboration with Jesse Vincent's Prime Radiant applied AI research lab, has launched smevals, a lightweight evaluation framework designed to test models, prompts, and agent harnesses. The command-line tool, which can be explored by running "uvx smevals docs" to view its documentation, allows developers to define evaluations using simple directories containing YAML files. Users can execute these evaluations across multiple model configurations simultaneously, using commands like "uvx smevals run path-to-eval/ -m gpt-5.5 -m claude-opus-4.6" to compare different systems.

The architecture of smevals decouples the execution of tests from the grading process. An eval represents a collection of tasks—such as generating a specific vector graphic—which are tested against various configs that define the target models, system prompts, or parameters. A runner script executes these tasks to generate runs. Afterward, a grader evaluates the outputs using a series of checks. These checks can range from basic string matching and XML validation to complex custom checker scripts that leverage other language models to judge the results.

To analyze the performance data, developers can run a local web server or use the "smevals build" command to generate a shareable, static HTML report. This workflow addresses a common pain point for AI practitioners by providing a structured, reproducible way to benchmark prompt engineering and model upgrades. Instead of relying on subjective manual testing, developers can now systematically track how changes to system prompts or model versions impact output quality, making it easier to optimize applications before deployment.