Validating a backtest
A pretty equity curve proves nothing: you get one by trying combinations until one works. What separates an amateur backtest from a professional one is not the result, it is what was done to avoid fooling yourself. This module is the six tools for that second part, and the three screens where they already live.
By the TradingCalculator.Pro team · Updated on · About us
Start your 7-day free trial →What you will learn
The four biases that manufacture false edge
Look-ahead: using information that did not exist at the time — the day's close to decide mid-session. Survivorship: testing only on assets still listed, ignoring the ones that went bust. Data snooping: trying a thousand combinations and keeping the best, which is best by luck. Overfitting: adding rules until the past fits perfectly, noise included. All four produce excellent results and none survives the market.
In-sample and out-of-sample
You hold back a slice of history — the last 30 %, say — and the parameter search never touches it. Once the best rule set is chosen, you evaluate there ONCE. If you look at that slice and re-tune, it is no longer out-of-sample: it has become part of training. A result that only exists in-sample is a description of the past, not a prediction.
Walk-forward: how it would actually have been traded
One parameter set fitted to the whole history uses information from the future: in January you did not know which moving average would work in December. Walk-forward re-optimises in windows — train on the first, trade the next, roll on — which is how it would have gone live. The number that matters is efficiency: how much of the training edge survives being applied forward. Below 0.5 is the classic signature of curve fitting.
Monte Carlo over the ORDER of the trades
Your backtest showed you ONE sequence of trades, and the drawdown you saw depends on the order they came in. Reshuffling those same trades thousands of times gives you the distribution of drawdowns that system could have produced. Look at the 95th percentile, not the mean: that is the one answering "how much must I be willing to sit through?". It is almost always considerably worse than the one sequence you lived.
How many trades it takes
Fewer than 30 trades tell you nothing. A hundred is the minimum to decide on, two hundred convinces, five hundred gives strong significance. And there is a discount almost nobody applies: CORRELATED trades — several in the same trend, the same day, the same sector — count for less than an independent sample. Three hundred highly correlated trades inform you less than eighty that are not. On small samples, a five-point error in the estimated win rate can triple the Kelly fraction you get.
Discounting for how hard you looked
Test forty-seven combinations on completely random data and the best will look good. That is arithmetic, not skill. The deflated Sharpe computes what a search of the same size would have produced by luck alone on a system with NO edge, and demands yours clears it. That is what separates "my backtest made 40 %" from "my backtest made 40 % and that is more than luck would have given me". Below twenty combinations tested the correction is itself imprecise: clearing it there is an indication, not proof.
Why a published edge stops working
The cleanest case is the "pre-FOMC drift": a 2015 study documented that much of the US equity premium was earned in the twenty-four hours before Fed announcements. Later studies extending the sample find it practically vanished after 2015. It is not that the finding was false: publishing it arbitraged it away. That is why a historical edge never guarantees a future one, and why it pays to monitor the one you are trading — the journal analytics computes whether your system's recent drop goes beyond what chance explains.