Ask ten companies how they review AI features before shipping them to production, and you'll get ten different descriptions of roughly the same thing: someone runs it a couple times, it seems reasonable, and it ships. That process catches some classes of problems. It doesn't catch the ones that actually cause incidents.
What a "safety review" usually consists of
In most organizations, it's informal and single-pass. A developer or designer writes a prompt, runs it through the model a few times, asks "does this look okay," and if nobody raises their hand, it ships. Some companies formalize this slightly: add it to the PR checklist, or require someone from Product to sign off.
None of that catches:
- Consistent failure modes that only show up in 5% of cases
- Behaviors that vary based on input characteristics you didn't test
- Scenarios where the model's output is technically accurate but contextually wrong
- Accumulation of small biases that only show up at scale
What breaks in production, in specific
A healthcare startup tested an AI triage system that was supposed to flag urgent cases for immediate review. In the pre-launch review, it worked great — it caught the urgent cases they tested it on. Once live, it systematically deprioritized cases from patients with certain insurance types, not because anyone programmed that in, but because the training data was unbalanced and the model learned the correlation. Three hundred patients got slower responses before anyone noticed.
A fintech company shipped a fraud detection agent that performed perfectly in review against test cases but failed systematically when the volume of transactions tripled. The model's confidence calibration broke — it started returning high-confidence scores on decisions it had lower signal for, because the distribution of inputs shifted. No one had tested at scale.
A support platform shipped an AI classifier that worked great on English and Spanish but degraded badly on code-mixed messages (customers switching languages mid-sentence), which wasn't part of the review because no one thought to test it. That accounted for 15% of the actual support volume.
What a real safety review actually needs
Start with baseline expectations, written down before you run the model on anything:
- Accuracy floor: what's the minimum accuracy rate acceptable for production?
- Confidence calibration: when the model says it's 90% confident, is it right 90% of the time, or 70%, or 95%?
- Edge cases: what are the things the model should explicitly refuse or flag rather than guess on?
- Drift detection: how will you know if the model's behavior is changing over time?
Then test systematically against those baselines:
- Run the model 500+ times against real data, not synthetic test cases
- Stratify results by input characteristics (language, volume, complexity, user segment)
- Have a human expert review a random sample and rate accuracy independent of the model
- Test the same scenario in different orders and contexts to see if context changes output
- Run it against adversarial cases designed to make it fail
Keep testing:
- Log every prediction, outcome, and confidence score
- Set up monitoring that flags when accuracy or calibration drifts
- Run quarterly spot checks against a fresh sample of real data
- Have a clear escalation path if monitoring flags an issue
Why most companies don't do this
It's expensive. Running 500 real examples through a model, having an expert review them, and setting up monitoring infrastructure takes weeks, not hours. It costs money. It's boring work that doesn't show up in launch announcements.
But the cost of not doing it — a systematic failure that affects thousands of users before anyone notices — is higher. And the companies that have actually shipped AI in production, the ones with production incidents behind them, almost universally wish they'd done this work before launch rather than after.
What this means for hiring
If you're staffing an AI team and you don't have someone whose job is "make sure our AI features don't break in production," you should. That's a specific role, somewhere between engineering and QA and data science, and it's becoming as critical as the person who writes the model code. It's the kind of judgment we screen for when building dedicated AI engineering teams.
