Twelve questions a skeptical quant asks about a backtest before believing it. Answer honestly — the scoring is deterministic code, so the only person a generous answer fools is you.
Was the strategy tested on data it never saw during development? 1 / 12 · critical
Was the strategy tested on data it never saw during development? A held-out period the strategy was never tuned against. If every number you have comes from the same window you developed on, you have a description of the past, not a test.
Yes — there is a held-out period the strategy was never fit or tuned on Not sure / the split is fuzzy No — all results come from the development data
At each decision point, did the model only know the past? 2 / 12 · critical
At each decision point, did the model only know the past? Walk-forward or rolling refit: train on old data, decide on the next block, roll forward. One static fit over the whole history does not count.
Yes — walk-forward / rolling refit, training strictly on prior data Partially — one train/test split, no rolling No — the model was fit once on the full history
Were fees AND slippage charged on every position change? 3 / 12 · critical
Were fees AND slippage charged on every position change? Not "fees exist somewhere" — an explicit per-side cost applied each time the strategy changes exposure, on the full traded notional (including leverage).
Yes — realistic per-side costs on every position change Fees only, no slippage / a rough guess No — the backtest is frictionless
Do you know how many times the strategy changes position? 4 / 12
Do you know how many times the strategy changes position? The exact count, over the whole test. Turnover times cost per trade is the toll the strategy has to out-earn before it makes you anything.
Yes — I know the number of position changes Roughly / could work it out No idea
Is every signal delayed so it cannot trade on the bar that produced it? 5 / 12 · critical
Is every signal delayed so it cannot trade on the bar that produced it? Signals computed on bar N execute at bar N+1 (or later). Also covers indicators that quietly use the full series — a mean or z-score computed over all data leaks the future into the past.
Yes — execution is delayed and no indicator sees the future I think so but have not audited for it No / never checked
Does the data include the assets that died? 6 / 12
Does the data include the assets that died? Delisted stocks, rugged tokens, merged tickers. Testing only on things that still exist today means testing only on winners.
Yes — the universe includes delisted/dead assets Single major asset (e.g. BTC, SPY) — less exposed, still unexamined Not sure what the data vendor did No — only currently-listed assets
How many tuned parameters does the strategy have, relative to the number of trades? 7 / 12
How many tuned parameters does the strategy have, relative to the number of trades? Count everything you chose by looking at results: lookbacks, thresholds, number of regimes, filters, the hold period you settled on. Then compare to the number of independent trades in the test.
A handful of parameters, hundreds+ of trades Several parameters, tens of trades per parameter Many parameters, few trades — or I never counted
How many variants did you try before this one? 8 / 12
How many variants did you try before this one? Every rejected version was a draw from the same deck. Test a hundred random strategies and a few will look brilliant by pure chance — reporting only the survivor is the same trick, done to yourself.
A few, and I tracked them all Dozens — the reported one is the best of them Lost count
Does the test window cover more than one kind of market? 9 / 12
Does the test window cover more than one kind of market? Trend up, trend down, chop, at least one genuine drawdown event. Two years of one-directional market is one regime, however many candles it has.
Yes — multiple regimes including at least one bad one Mostly one regime / short window No — the window is short or one-directional
Did it beat doing nothing, after risk? 10 / 12
Did it beat doing nothing, after risk? Compare to buy-and-hold of the same asset over the same out-of-sample window — return AND max drawdown. A bot that makes less than holding, with a deeper drawdown, did not earn its complexity.
Yes — higher return and no worse drawdown than the dumb baseline Beat it on return, but with worse drawdown (or never compared drawdowns) Never compared it to buy-and-hold
Was it tested at the leverage and size you actually intend to trade? 11 / 12
Was it tested at the leverage and size you actually intend to trade? Including whether you could sit through the worst drawdown the test produced, at real dollar size, without pulling the plug at the bottom.
Yes — intended leverage, and the max drawdown is survivable at real size Tested unlevered / at a different size than I will trade No — sizing and leverage were an afterthought
Can you re-run the whole thing from raw data and get the same numbers? 12 / 12
Can you re-run the whole thing from raw data and get the same numbers? One command, raw data in, metrics out. If the results involve a notebook run out of order, hand-edited cells, or numbers copied between files, they are not results — they are recollections.
Yes — one command reproduces every reported number Mostly, with some manual steps No — I could not reproduce them exactly today