Daily Digest
Require held-out lift before shipping harness edits
Only ship a setup tweak if it also helps on cases it wasn't tuned on.
The technique
Harness edits that look great on the training set can collapse on held-out cases. Selecting by train lift alone ships the brittle option. A gate that requires held-out mean lift and non-negative worst-case held-out delta picks the version that transfers.
How it was tested
Offline structural sim (no live model): two mock harnesses A (train overfit) vs B (transfers). Train-only selection shipped A (held lift −0.15). Held-out + worst-Δ gate shipped B (held lift +0.143).