Daily Digest
Check confidence before routing on it
A small model's doubt finds its mistakes on some tasks, not all.
The problem
A common way to cut AI costs is to let a small, cheap model answer first and pass a question to a bigger, pricier model only when the small one seems unsure. It rests on one bet: that the small model's doubt lines up with the questions it gets wrong and the big model gets right. When the small model is wrong but sure of itself, the wrong questions go up, and you pay for the big model where it doesn't help.
The technique
Before you route on confidence, check that it picks the right questions on your task, against a random pick of the same size. A study posted Oct 16 (Hossen and others, arXiv:2610.23249) tested this with three pairs of small and large open models (1B to 4B), four languages and three kinds of task: 42,288 answers. Confidence worked for reading comprehension and topic labels: sending up half the questions, it beat random picking by 1.4 to 4.7 and 2.5 to 5.6 points. On knowledge and reasoning questions it barely beat a coin flip at finding the right ones, and gave no steady gain over random, because the small models were often sure and wrong. One shared cutoff sent up to 30 points more of one language's questions than another's. Those are their numbers, and the paper isn't peer-reviewed yet.
- Run both models on a few hundred labeled questions from your workload.
- Mark the cases where the small model is wrong and the big one is right. Those are the only hand-offs that pay.
- Route by the small model's confidence at the share you can afford, and compare with random routing.
- If confidence doesn't beat random, don't route on it for that task. If you serve several languages or task types, give each group its own quota.
What we saw
Our own check was much smaller: Qwen2.5 0.5B as the small model, 1.5B as the big one, on 400 science questions (ARC-Challenge) and 400 news headlines to sort by topic (AG News).
On headlines, confidence worked. Sending the least sure 25% to the big model scored 84.0%, against 80.9% for a random 25%, and within half a point of the big model alone (84.5%), for about two-thirds of the compute time. It caught 84% of the cases the big model fixes.
On science questions, it didn't. At 25%, confidence routing scored 52.8% against 52.6% for random, while the big model alone scored 76.2%. Only the small model's most confident fifth stood out: it was wrong 24% of the time there, and 56% to 68% in every other fifth. Sending up half, confidence pulled 3.2 points ahead of random.
Limits
One model family, two small models, English, multiple choice, one run. Headlines came in blocks across the test set, not a random sample. The 3.2-point science gain isn't solid at this size (likely range about 0 to 6.5 points); the headline gains are (0.9 to 5.3 points at 25%). Like the paper, we used the gap between the top two answer probabilities; free-text answers need another signal. Compute is estimated from run times: our small model took about 40% of the big one's compute time, so savings were modest. With both tasks behind one cutoff, nearly half the science questions went up and under 5% of headlines: fair here, but it shows how one group can take most of the big model.
Try it
Score 300 labeled questions from your workload with both models. Plot confidence routing and random routing from 0% to 100% sent up. If the lines overlap at the share you can afford, don't route that task on confidence.