Make routers pick from a fixed list

The technique

Routers that answer in prose ('not billing, this is account security') get keyword-parsed by the harness, which often picks the first route word it sees or falls back to a default queue. Invented labels slip through the same way. Make the router return one value from a fixed list plus confidence, validate it, and send off-list or low-confidence answers to a human instead of guessing. For our agents this keeps security and refund cases out of the wrong queue, and it matches the shape of OpenAI's new Decisions API (a fixed set of answers per question).

How to apply

1. Define the allowed labels as an enum in a structured-output schema.
2. Validate the returned label against the list and reject anything else.
3. Require a confidence value and route anything under your threshold (0.6 in the sim) to a human.
4. Log rejects so the list can grow on purpose.

How it was tested

Evidence (offline structural sim, no live model): 90 seeded support messages across 4 routes (26 were account security). The simulated router made the same underlying call under both contracts: about 88% right, otherwise a close neighbor, and sometimes an invented 'fraud' label. Only the output format changed. With free text (70% followed the 'Route: X' format, 30% drifted into prose) the keyword parser got 65 right and 25 wrong, 10 of them account-security cases; 12 of those errors came purely from parsing when the model's call was right. The fixed list got 75 right and 3 wrong (1 account security) and sent 12 to a human (5 off-list labels plus 7 low confidence).

Limits

The error rate and the prose-drift mix are assumptions of the sim, so the size of the gap depends on them; the direction is what carries over. A live retest needs real router outputs on labeled tickets, plus a measure of whether the human-handoff load is acceptable.