Evals: The Thing
Nobody Does
Until It's Too Late
You wouldn't ship a backend without tests. But most AI teams push models to production without a single eval. Here's the framework we actually use, and the mistakes we've seen kill products.
February 28, 2026 · AI EngineeringNobody skips unit tests in backend engineering. It's considered basic hygiene. But ask 10 AI teams how they validate model behavior before shipping and 8 of them will say "we tested it manually" or "we had the team try it out."
That works when your model has 5 users. It does not work when you have 5,000, and someone finds the edge case your team never thought to try, and it embarrasses your company, or worse, hurts someone.
The reason teams skip evals isn't laziness. It's that evals feel hard to get right and easy to dismiss. What counts as a good eval? How many examples do you need? How do you score open-ended outputs? These are real questions and most teams never sit down to answer them.
So they ship on vibes and hope for the best. Sometimes it works. A lot of the time, it doesn't.
- ✕ You find regressions in production
- ✕ Model swaps are terrifying
- ✕ Prompt changes break unknown things
- ✕ No objective measure of improvement
- ✕ "Feels good" becomes the spec
- ✓ Regressions caught before they ship
- ✓ Model swaps are a 20-minute check
- ✓ Every change has a before/after score
- ✓ You can actually measure progress
- ✓ Failures are reproducible
Four Types of Evals You Actually Need
Does the model do what it's supposed to do? For a QA bot, does it answer the question correctly? For a classifier, does it classify correctly? This is your baseline. If you have zero evals, start here.
Write 50-100 test cases with expected outputs. Script the calls. Assert on the outputs. This is table stakes.
Anti-Patterns That Kill AI Products
These are the five ways teams screw up their evals. All five come from real post-mortems.
Your whole team plays with the product, it "feels good", you ship. Two weeks later a customer finds the thing that breaks everything. Vibes are not evals.
Write the cases down. Script the runs. Assert on outputs.
Using the same examples you used to prompt-engineer or fine-tune your model to evaluate it. Of course it scores well. Those are the exact cases it was optimized for.
Keep a held-out test set that nobody touches during development.
Your eval set produces 85% accuracy. Great. But what's the distribution? Are there entire categories of inputs that are wrong 100% of the time that average out fine across the rest?
Slice your evals by input type, user segment, and use case. Don't let averages hide systematic failures.
You ask GPT-4 to score your model's outputs and accept whatever it says. The grader model has its own biases, its own blindspots, and no idea what 'good' means in your specific context.
Write a detailed grading rubric. Include examples of good, mediocre, and bad outputs. Make the grader use the rubric explicitly.
You run your eval suite before launch, ship, and never run it again. But your model provider updated their model. Your system prompt drifted. Your context grew. Everything changed.
Run evals on every change. Add them to your CI pipeline. They're not a one-time thing.