Daily Digest
Stop picking tools that are down
Rank tools by recent success, and retry the ones you benched.
The problem
Most agents choose a tool by matching the request against tool descriptions. That works until a tool goes down. The description still matches perfectly, so the agent keeps picking the broken one, the call fails, and the run either retries into the same wall or gives up. Nothing in a text-matching ranker knows the tool is failing right now.
The technique
Keep a running success score for each tool and fold it into the ranking. A paper posted Oct 9 (RATED, arXiv:2610.13889) does this in two stages: a text matcher picks the top few candidate tools, then a small re-ranker reorders them using recent execution results. Older results fade, so a fresh outage counts more than last week's good record. Tools with no history start from a score based on their catalog listing.
You don't need their full setup to use the core idea:
- Log every tool call as a success or a failure. Errors and timeouts count as failures.
- Keep a per-tool success rate where new results weigh more than old ones.
- Rank candidates by text match plus that rate, so a strong match that keeps failing drops below a slightly weaker match that works.
- Every so often, retry a demoted tool so you notice when it comes back.
What we saw
The authors report that when 30% of tools were made to fail, a standard text-matching retriever (ToolLLM's) picked a broken tool 86.2% of the time, while RATED picked one 2.9% of the time and added about 0.17 ms per step. Those are their numbers, on their own benchmark of 5,172 tools.
Our own check was a simplified offline simulation: no model, no real tools, and made-up rates. Healthy tools succeeded 97% of the time and down tools 5%, with three candidate tools per request across 50 request types and 10 seeds. When 30% of the best-matching tools went down, text matching alone picked a broken tool 30.5% of the time and task success fell from 97% to 69%. Adding a success score cut broken picks to between 1.3% and 1.7% and kept success near 95%.
In our simulation, the surprise was recovery. After the outage ended, rankers that never retried demoted tools went back to the best match only about 10% of the time over the next 1,500 calls, and about 26% after 9,000 calls. Adding a 5% retry chance with faster fading raised that to 67% over 9,000 calls. The cost was a few more broken picks during the outage (2.5% instead of 1.3%).
Limits
The paper's outages never end, so it doesn't measure switching back, and our simulation suggests that's the weak spot. Our numbers come from invented rates and a simplified scorer, not the paper's method, so they show how the idea behaves, not real-world gains. Recovery speed depends on traffic: a tool that's called a few times a day takes much longer to earn trust back. A success score also only catches visible failures. A tool that quietly returns wrong answers without an error won't be demoted.
Try it
Pick one agent that calls outside APIs. Log each call's outcome for a week. Then add a recent-success weight to tool selection and a small retry chance for demoted tools. Track two numbers: how often the agent picks a tool that's failing at that moment, and how long it takes to go back to a tool after it recovers.