Two LLMs, one support dataset: an evaluation suite with cost per correct answer

Brightline's support team of 11 handles about 6,500 tickets a month. The CTO wants an LLM to draft first replies. Two options are on the table: a large frontier model at roughly 15 times the price per token, and a small, cheap model. The product manager says the cheap one is "obviously good enough". The support lead says the large one is "obviously safer". Neither has tested anything.

You will settle it with evidence. Using a public customer-support dataset as the base, you will build a fixed test set of 300 tickets drawn from Brightline's five main categories (billing, calendar sync, permissions, integrations and bugs), each with a reference answer and a short list of facts the reply must contain. You will run both models with the same prompt and the same help-centre context, grade the replies with a mix of automatic checks and a calibrated LLM judge, and measure latency and token cost.

The decision rule the CTO agreed to: pick the cheaper model unless the larger one is at least 5 points more accurate on the test set, or the cheaper one makes a harmful error (a wrong refund promise, a security claim, or leaked account data) on more than 1% of tickets.

Done looks like a Python package that re-runs the whole comparison with one command, a results table with confidence intervals, a cost-per-correct-answer figure for each model at 6,500 tickets a month, and a two-page decision memo. Use any two models you can access through an API or run locally, as long as one is clearly larger than the other.