Test search before saying none

The problem

An agent searches for open incidents, gets an empty list back, and tells you there are none. Usually that's true. Sometimes the token expired, a bad reindex left the index empty, or one partition is down. Plenty of connectors return a clean, successful-looking empty list in those cases instead of an error. The agent can't tell "nothing matches" from "nothing works," so a broken tool turns into a confident answer. That's worse than an error, because nobody goes back to check.

The technique

Keep one record in each data source that you know is there. Call it a canary. When a search comes back empty, look up the canary before trusting the result. If the canary shows up, the tool works and "nothing found" stands. If the canary is missing too, don't claim absence. Say the tool looks broken and you can't confirm either way.

If the data is split across shards or partitions, keep one canary in each. A single canary only proves its own shard is up.

What we saw

We ran this offline as a scripted simulation, with no live model: 120 search calls against a mock index split into two shards. The tool was healthy on 90 calls, fully silent on 15, and had one shard down on 15. 63 of the queries had a real match.

• Trusting empty results gave 11 false "none exist" answers.
• One canary cut that to 5, for 68 extra lookups. All 15 "tool looks broken" flags came during real outages. The 5 misses were all partial outages where the down shard wasn't the canary's shard.
• One canary per shard got it to 0, for 136 extra lookups. All 27 flags came during real outages.

So the check works, but only as far as the canary reaches. A partial outage beats a single canary.

Limits

These are simulation numbers, not production rates. We picked the outage mix and the index, so the counts show how the check behaves, not how often your tools fail. The extra lookup adds latency on every empty result, and the per-shard version doubles that here. Canaries also go stale if someone deletes the record. A real test needs a live connector with injected failures (expired auth, an empty index, one partition down), a measure of the added latency, and a check that the canary records stay put.

Try it

1. For each search tool your agent uses, pick one stable record per source or partition that should always be there.
2. Wrap the tool so an empty result triggers a canary lookup.
3. If any canary is missing, return "search looks unavailable, can't confirm no results" and log it.
4. If every canary is found, pass the empty result through as a real "none found."
5. Alert on canary misses and treat them as incidents, not answers.