Over-fitting, explained simply

11 minutesLesson 7 of 9
Femi, the HustlCraft strategy mascot

This is the trap that makes a beautiful backtest worth nothing, and almost nobody walks into it on purpose. It is built out of small, sensible improvements, one at a time, until the rule describes one year of history and nothing else.

A rule fitted to a year that will never happen again

Past prices contain two things mixed together: whatever regular behaviour the market has, and noise. Noise is the part that happened once for no repeatable reason. A rule with enough settings can be bent until it matches both, and the noise half will never appear again.

Last year, the data you tuned onThis yearthe simple rulethe rule tuned to every wiggle

The green line misses several points and still describes something sensible after the divider. The red line passes through every point exactly, then loses its shape the moment it meets a day it was not fitted to. On the left half the red rule has the better backtest, and that is the whole trap.

How it happens by accident

Nobody sits down to over-fit. They sit down to improve. MT5's tester has an Optimization setting that tries every combination of the inputs you tick and hands you the best one, which sounds like what a careful person should do.

What an overnight optimisation really tests

  1. Six inputs: two moving average lengths, a stop, a target, a start hour and an end hour.
  2. Five values tried for each, which is five multiplied by itself six times: 15,625 combinations.
  3. The optimiser runs all night and reports the best one.

Test 15,625 rules against one year of history and the best of them will look excellent by luck alone. The report never separates the luck from the skill, because it cannot see the difference.

It happens in small ways too. A filter added to remove the two worst trades. Tuesdays skipped because Tuesdays were bad last year. A stop moved 3 pips to avoid the ugliest loss on the chart. Each change is reasonable, and together they carve the rule into the shape of one year.

The signs, and Femi's warning

Femi, the HustlCraft strategy mascot
Femi's warning

A rule tuned until last year looks perfect has been fitted to a year that will never happen again. It has not learned the market. It has memorised one stretch of it, and the market does not sit the same exam twice. When somebody shows you a curve with no ugly stretch anywhere in it, ask how many settings were tried before that curve appeared.

The defences, in order of value

  1. Use fewer settings. Every extra input is another chance to fit noise.
  2. Hold data back. Tune on one stretch, then run the finished rule once over a stretch you never touched, not repeatedly until it passes.
  3. Look at the neighbours. If 20 works and 19 and 21 both fail, 20 is luck wearing a number.
  4. Demand a reason for every rule and filter, in a sentence about how the market behaves. "It tested better" is not a reason.
  5. Judge the rule on the forward test, never on the report the tuning itself produced.

A rule you can explain in one sentence to somebody who does not trade is far harder to over-fit than one with six inputs and a story.

The picture to keep

Over-fitting is a tailor who fits the suit to the exact position you were standing in. Perfect while you hold still, useless the moment you walk.

PreviousNext: Why it still needs a human