A good demo is the least useful signal you'll get before shipping a model to customers. It tells you the happy path works, once, for the ten examples you picked. It tells you nothing about the other ten thousand.
Here's what we actually check before a model reaches production.
A held-out set the model has never seen
Every project gets an evaluation set built from real, anonymized production data — not from the same examples used to write the prompt or fine-tune the model. If the same person wrote the eval and tuned the system, the eval is measuring their taste, not the model's reliability.
Failure modes, ranked by cost — not frequency
A model that's wrong 1% of the time in a way that's mildly annoying is fine. A model that's wrong 0.1% of the time in a way that damages customer trust is not. We rank failure modes by what it costs the business when they happen, and set the acceptance bar per failure mode, not as one blended accuracy number.
Regression tests, same as code
Every fixed failure becomes a permanent test case. The eval set only grows. This is the part teams skip under deadline pressure, and it's the part that prevents the same embarrassing mistake from shipping twice.