Don't keep only the shortest answer

The problem

A simple way to get better answers from a model is to ask the same question several times and keep the answer that comes up most often. Recent studies add a twist: the longer a model rambles before answering, the more likely it is wrong. So some teams now keep only the shortest answer, or vote only among the three shortest. That throws away most of what you paid for, and it assumes the length rule holds for your model and your task. It does not always.

The technique

Keep every answer, but give shorter ones a bigger say, and let your own results decide how big. A paper posted Oct 14 (Parascandolo and others, arXiv:2610.19890) calls this length-conditioned majority voting. It needs no labels and no retraining.

1. Ask the question several times and record each answer and how long the reasoning was.
2. Give each answer a weight from a simple curve that starts out favoring shorter reasoning.
3. After each question, nudge that curve: answers that matched the plain vote count as likely right, the rest as likely wrong. If long answers keep winning, the curve flips.
4. Add up the weights per answer and keep the top one.

Across 11 models and 8 math and science test sets, the authors report it had the best average score in each of their four test setups, for example 73.7% against 71.8% for the plain vote with four answers per question. Keeping only the shortest answer often did worse than the plain vote. Those are their numbers.

What we saw

Our own check was much smaller: a tiny open model (Qwen2.5 0.5B, on a plain CPU), 80 grade-school math word problems, 8 answers each. One answer alone was right 30% of the time. The plain vote got 45%. Keeping only the shortest answer fell to 19%, and voting among the three shortest got 26%. A fixed short-wins weight, with no adjusting, got 41%. The adjusting version got 46.5%, on average across 25 question orders.

The reason is in the data. For this model, short answers were not more often right: within the same question, the right answer was the shorter one in only 47% of right-versus-wrong pairs. The adjusting curve noticed and flipped toward longer answers. The hard cutoffs could not.

Limits

Our +1.5 points over the plain vote is not a real gain at this size: in a typical run it changed 3 results against the plain vote (2 fixed, 1 broken), and the likely range runs from 2.5 points worse to 5 points better. We used one tiny model, one task, one sampling setting and one run each. Our version was rebuilt from the paper's equations, not run from the authors' code. The paper's average gains over the plain vote are about 2 to 6 points, with bigger jumps on harder math for some smaller models. It only works where answers can be matched exactly, like numbers or multiple choice.

Try it

If you already ask a model several times and vote, log the length of each answer next to whether it won. If you use a shortest-answer cutoff, test it against the plain vote on your own questions first. Then swap the cutoff for a weight you adjust as you go. The code is short and public at https://github.com/aimagelab/LCMV.

Sources

• Parascandolo et al., Frustratingly Easy Majority Voting Using Length Conditioning, arXiv:2610.19890 (Oct 14, 2026)
• LCMV code (aimagelab)
• GSM8K test set (OpenAI)