Backtest Audit Report

Verdict

DO NOT FUND ON THIS EVIDENCE.

The result is built on a confirmed same-bar look-ahead — the position at bar i earns the return computed from bar i — so the 1.90 Sharpe rests on a move the strategy could not have captured live. Even setting that aside, the deflated Sharpe after 60 variants gives the edge a 30.4% chance of being real (fails the 95% bar), and the observed 1.90 sits below the 2.44 Sharpe that random search on noise would be expected to throw off. This is a clean, honest demonstration of a strategy whose reported numbers the evidence does not support — which, since it is openly labeled synthetic, is exactly what it appears designed to show.

Computed diagnostics

The observed annualized Sharpe is 1.90, but its bootstrap 95% confidence interval is [−0.12, 3.96], which contains zero — the edge is statistically indistinguishable from luck at that confidence. The single-shot Probabilistic Sharpe reads 96.5% (passes 95%), but that figure ignores the 60 variants tried; once selection is priced in, the deflated Sharpe collapses to 30.4% and fails. The selection benchmark E[max Sharpe] is 2.44, so with 60 tries on noise you would expect a best-of-batch Sharpe near 2.44 by luck alone — the observed 1.90 is below that line. The max drawdown of 15.3% is not suspicious: it sits inside the Monte Carlo band [9.8%, 29.0%] with a mean of 16.8%, so the equity curve is realistically rough, not doctored. The lag-1 return autocorrelation of 0.15 is significant (threshold ±0.13) while every later lag is noise — the statistical fingerprint of a same-bar peek, concentrated exactly where look-ahead would put it. The 40-period rolling Sharpe swings from −2.21 to 8.69, so the full-sample 1.90 is an average over wildly different stretches, not a stable property. The longest underwater span is 68 periods. Costs finish the story: net mean return hits zero at 10.2 bps per side, and against a payoff of 1.04 and a breakeven win rate of 49.1% versus observed 56.3%, the margin is thin and cost-fragile.

What holds up

• The summary statistics are internally consistent. Every stated headline matches the computed value: Sharpe 1.90/1.90, drawdown −15.3%/15.3%, win rate 56%/56.3%, payoff 1.04/1.04, trades 240/240. The arithmetic is clean.
• The drawdown is honest. Observed 15.3% sits inside the Monte Carlo expected band with mean 16.8%. No evidence of a too-smooth or doctored equity line.
• The look-ahead is disclosed, not hidden. The prose states "signal and fill use the same bar close," and the code comment says "same-bar look-ahead lives here" outright. The submission tells you where the body is buried.
• The data is labeled synthetic and makes no tradability claim. Seed 53, generator output, stated plainly. It does not pretend to be market data.
• Data-per-parameter is adequate. 240 trades / 6 tuned parameters = 40 per parameter. The problem is not thin data per knob.
• No embedded instructions to the analyst were found in the submission.

What does not hold up

1. Same-bar look-ahead manufactures the Sharpe. The position at bar i earns the return *into* close i, which the signal was computed from. This is not a tradable rule; it is a peek. Confirmed by the code comment and by the significant lag-1 autocorrelation of 0.15 (later lags are noise). This is the failure most likely to flip the sign of the result — remove it and the reported edge may not exist at all. *Fix:* decide the signal on close i, fill on i+1, and re-run every metric.

1. Selection is not priced into the headline. 60 variants were tried; the prose reports raw Sharpe 1.90 with no confidence figure. The deflated Sharpe is 30.4% (fails 95%), and 1.90 sits below the 2.44 luck benchmark. The reported number overstates what the evidence supports. *Fix:* report the deflated Sharpe alongside the raw one, and describe the search space and selection rule so the 60 tries can be audited.

1. The edge is cost-fragile. Net mean return hits zero at 10.2 bps per side. A bot that re-decides position every bar generates many round trips, and roughly 20 bps of round-trip friction erases everything. Zero cost is assumed. *Fix:* add a commission-plus-slippage model and report the round-trip count so the 10.2 bps breakeven can be converted into a realized net figure.

1. One synthetic path proves no regime robustness. Seed 53 is a single generator draw with no crises, gaps, or liquidity events. The rolling Sharpe range of −2.21 to 8.69 shows the edge is unstable even within that one path. *Fix:* real market data, out-of-sample, across multiple regimes — none of which exists here.

1. No out-of-sample or walk-forward split. Every number is in-sample on the same data the parameters were chosen on. *Fix:* a purged, embargoed walk-forward split.

The three numbers that matter most

• Deflated Sharpe probability the edge is real: 30.4% (computed; fails the 95% bar, below the 2.44 luck benchmark).
• Lag-1 autocorrelation: 0.15 (computed; significant at ±0.13 — the look-ahead fingerprint that supports the 1.90 Sharpe).
• Cost breakeven: 10.2 bps per side (computed; the entire edge lives inside a 0-to-10.2 bps window).

What to test next

1. Fix the execution timing first. Signal on close i, fill on i+1. Re-run all metrics. Everything downstream is meaningless until the look-ahead is gone.
2. Re-run on real market data for the same instrument class the generator is meant to stand in for. The synthetic path cannot speak to tradability.
3. Add a cost and slippage model and report net metrics, with the round-trip count made explicit.
4. Document the 60-variant search — the space, the selection rule — and report the deflated Sharpe as the headline, not the raw one.
5. Split out-of-sample with purging and embargo, and report in-sample versus out-of-sample metrics side by side.

Code fixes

The code was provided and cross-checked (lookback=20, entry_z=1.5, ±1 position confirmed). The defects:

• Same-bar look-ahead. *Defect:* the signal is computed from close i and applied to prices[i]/prices[i-1] - 1, the return ending at close i — the code comment admits it. *Fix:* apply the signal decided at close i to the return from i to i+1 (prices[i+1]/prices[i] - 1), i.e. lag the position by one bar. *Why it changes the result:* it removes the return the strategy could not have captured; the 1.90 Sharpe and the 0.15 lag-1 autocorrelation should both fall.
• Zero cost. *Defect:* no commission or slippage anywhere. *Fix:* subtract a per-side cost on every position change and expose it as a parameter. *Why:* the edge is zero at 10.2 bps per side, so any realistic cost materially changes net return.
• No round-trip accounting. *Defect:* the code re-decides every bar but does not count or charge position changes. *Fix:* count each change of position as a round trip and apply cost to it. *Why:* it converts the abstract 10.2 bps breakeven into a realized figure.
• In-sample everything. *Defect:* no train/test separation. *Fix:* add a purged, embargoed walk-forward split and report metrics per fold. *Why:* it stops parameter choice from leaking into the reported result.

Feature and label engineering

• Lag the z-score input. Compute the rolling mean and z-score using data up to and including close i, and act only on i+1. The current construction reads the bar it trades. Removing that peek is the whole point of the next test.
• Check the rolling mean for leakage. Confirm the 20-bar window is strictly backward-looking (no centered window, no inclusion of the current unrealized bar in a way that anticipates the fill).
• Move toward stationary inputs. The z-score is already a normalization, but verify the rolling standard deviation is estimated on a trailing window only, so the scaling itself is not forward-looking.
• Label trades honestly. Positions currently re-decide every bar with no exit rule, which blurs "trade" and "bar." Use volatility-scaled, event-based labels (e.g. triple-barrier: profit-take, stop, and time limit sized to local volatility) so each trade has a defined horizon and outcome rather than an implicit hold-until-flip.
• Split hygiene. Purge observations whose label windows overlap the train/test boundary, and embargo a gap after each test fold. Without this, adjacent overlapping returns leak across the split.

Every item above is about making the measurement trustworthy, not about raising returns.

Fix-it prompt

You wrote a backtest for a z-score mean-reversion bot on a synthetic series
(seed 53). An evidence audit found specific defects. Fix them for measurement
honesty only. Do NOT propose new trade ideas, do NOT tune parameters to raise
returns, and do NOT make any claim about future performance. Report metrics,
not promises.

Fix these defects:

1. SAME-BAR LOOK-AHEAD. The position at bar i currently earns the return
   prices[i]/prices[i-1]-1, computed from close i — the same bar the signal
   reads. Your own comment says "same-bar look-ahead lives here." Change it so
   the signal decided at close i is applied to the return from i to i+1
   (prices[i+1]/prices[i]-1). This is the change most likely to move the result:
   the audit found a significant lag-1 autocorrelation of 0.15 that is the
   fingerprint of this peek.

2. ZERO COST. Add a per-side transaction cost plus slippage on every position
   change, exposed as a parameter. The audit found net mean return hits zero at
   10.2 bps per side, so this is decisive. Count and report the number of round
   trips.

3. SELECTION NOT PRICED IN. 60 variants were tried. Report the deflated Sharpe
   (the audit computed 30.4% probability the edge is real, below the E[max]=2.44
   luck benchmark) as the headline, alongside the raw Sharpe. Document the search
   space and the rule used to pick this variant.

4. NO OUT-OF-SAMPLE SPLIT. Add a purged, embargoed walk-forward split. Purge
   observations whose label windows cross the train/test boundary; embargo a gap
   after each test fold.

Apply this feature and label engineering:
- Ensure the rolling mean and standard deviation use strictly trailing windows.
- Relabel trades with volatility-scaled, event-based labels (triple-barrier:
  profit-take, stop, time limit sized to local volatility) instead of
  hold-until-flip, so each trade has a defined horizon.

Then re-run the backtest and report, side by side, BEFORE and AFTER the fixes:
annualized Sharpe, bootstrap 95% CI, deflated Sharpe, max drawdown, longest
underwater span, win rate, payoff, lag-1 autocorrelation, round-trip count, and
net return at 0, 5, and 10 bps per-side cost. State plainly if the edge does not
survive the execution-timing fix.

Scope of this report

This is an educational analysis of the evidence in the submission, based only on what was provided and the deterministic diagnostics computed from the buyer's own data. Hypothetical and synthetic performance has inherent limitations, including hindsight bias and the absence of real fills, costs, and market regimes. Nothing here is financial advice, a valuation of any strategy, or a prediction of future results; it judges the quality of the evidence and nothing else.