Macro S&P drawdown model: the false positives are the problem for hedging

I run a small research site, and one of the models on it estimates the probability of a 10%+ S&P drawdown over the next 1, 3, 6 and 12 months from macro state variables (the yield curve, high-yield credit spreads, realised and implied vol, money growth, sentiment, labour and housing). It is a ridge-penalised logistic, scored walk-forward out of sample from 2007, so there is no look-ahead. For options this is really a hedge-timing question: if you can say when drawdown risk is genuinely elevated, that is when paying for protection, or stepping off short premium, earns its keep.

Out of sample the ranking works, in a limited way. AUC is roughly 0.60 at 1 month, 0.69 at 3, 0.74 at 6 and 0.68 at 12, so the longer horizons carry most of what signal there is. The 6-month probability sat near 1.0 through late 2008, which is the case everyone wants to see.

The part that actually matters for options is not the hit rate, it is the false positives. Over the full out-of-sample window the 6-month alert (probability above 0.30) fired in 38 months: 17 ahead of a real drawdown, 21 not. A naive "buy protection when it crosses the threshold" rule pays theta on those 21 false alarms, and I have not convinced myself the lead time on the true positives covers that bleed once you price the puts realistically.

The second limit is structural. The inputs are slow macro series, so the model leads slow-building drawdowns (2008, the 2022 grind) and is blind by construction to the fast ones (Feb 2018, the COVID gap, anything overnight and exogenous). Those are the tails an options book cares most about, so I would not sell this as a crash hedge.

So the genuine question, and the reason I am posting here and not in a stats sub: has anyone actually tested a macro drawdown-probability signal against a realistic protection overlay, net of premium, and found the lead time worth the false-positive cost? Every backtest I have seen, mine included, looks fine on the directional call and comes apart once you charge it for the puts it buys when nothing happens.

the write-ups and the live version are in a comment if people want it. It is a research framework, not a trading signal.