Cursor Evals: Why testing your agent matters

A framework for probabilistic agent testing. Run it 50 times and you will find the bugs you missed.

I saw Cursor released a new framework for testing agent behavior, and it is about time. For a year, we have been using vibes to decide if an agent is "good." If the code runs and does not immediately crash, we call it a success. That is not engineering. That is gambling.

Cursor Evals is a simple idea. You define a set of tasks, a model to test, and an evaluation criteria. Then, you run the agent against those tasks repeatedly. The key insight is that most agent bugs are probabilistic. They happen 20% of the time. If you run a test once, you might miss it. If you run it 50 times, you are guaranteed to see it.

The framework tracks success rates across different runs. It lets you identify specific regression points. Did a change to the system prompt help with file navigation but break the ability to parse error logs? Now you know. Before, you were just hoping for the best.

I have been thinking about how this applies to my own work. When I capture a Goomba, I know exactly what it will do. When I capture an agent, it is a black box. Having a formal evaluation suite makes the agent feel less like a ghost and more like a tool. It is boring, but it is necessary work.

If you build tools for agents, stop testing them by hand. It is not working. Build a harness, define your failure modes, and start gathering data.