MyAlgoGuru Help
Education Glossary Open the app
Chapter 2 of 5

Overfitting: how a curve gets beautiful and useless

Nobody sets out to overfit. It happens one reasonable improvement at a time, and the equity curve rewards you at every step.

Here is how it actually happens.

You have an idea. You test it. The result is mediocre — positive, but not worth the screen time. So you look at the losing trades, and you notice something true: most of them happened on days that gapped. You add a filter for gap days. The curve lifts.

Encouraged, you look again. The remaining losses cluster in the last hour. You stop taking entries after 14:30. The curve lifts again. Then you try a 19-period average instead of a 20, because why not, and it lifts a third time.

Every one of those steps was reasonable. Every one was supported by evidence in the data. And the strategy you now hold is worse than the one you started with.

What you were actually doing

A price series contains two things: structure that will recur, and noise that will not. Your strategy's parameters are the dials you use to describe the structure. The trouble is that the dials fit noise just as happily, and noise is more plentiful.

The gap filter may have removed a genuine weakness. Or your sample happened to contain eleven gap days, seven of which went against you, and the filter simply deleted them. From inside the test those two possibilities look identical. They produce the same lift in the same curve.

A spike — do not trust it A plateau — this is worth having period = 19 one value works, its neighbours do not 14 to 22 all work a whole neighbourhood works, so the effect is probably real
The shape of a parameter sweep tells you more than its peak. Both charts show net profit against one parameter. On the left, a single value is spectacular and everything either side of it is flat — that is the signature of a value that happened to catch a handful of lucky trades. On the right, the whole region performs, which is what a real effect looks like: the market does not know the difference between a 17-period average and an 18-period one, and neither should your edge.

This gives you the single most useful habit in strategy testing. Never read a parameter sweep as a search for the best value. Read it as a search for a plateau. If 14 through 22 all make money and 19 makes the most, use 18 — the middle of the region, not its peak. You are deliberately giving up the best backtest number in exchange for a result that does not depend on being right to the decimal.

If there is no plateau — if only one value works — you have not found a parameter. You have found a coincidence.

The ratchet

The reason overfitting is so hard to resist is that the feedback is immediate and always positive. Nothing tells you to stop.

0 the test you keep running the money you will make each step is one reasonable improvement, supported by the data
The two lines you cannot see at the same time. Every tweak lifts the curve you are looking at. The curve you care about — performance on data the strategy has never seen — stays flat for a while and then turns down, because you have started describing this sample's accidents rather than the market's habits. Nothing in the backtest report distinguishes the left half of this picture from the right half. The only way to see the dashed line is to hold data back, which is the next chapter.

A budget you can actually count

There is a crude but genuinely useful discipline here: count your degrees of freedom and compare them with your sample.

Every parameter you tune is a dial. Every filter you add is a dial — and a filter is worse than a parameter, because it usually comes with its own threshold, so it is two. A strategy with an entry average, an exit average, a gap filter with a size threshold, a time-of-day cutoff and a stop multiple has six dials.

Now count the trades. Not the days in your sample — the trades, because trades are the only thing carrying information about whether the rules work.

Illustrative

Six dials, and a test that produced 92 trades over two years.

That is roughly 15 trades per dial. With that little evidence per dial, you can fit nearly anything. The result is not a strategy with a modest edge — it is a description of 92 particular trades, written in the language of rules.

The same six dials against 4,000 trades is a different situation entirely. Still not proof, but the arithmetic is no longer hopeless.

A usable rule of thumb: be suspicious below about 30 trades per degree of freedom, and comfortable above a few hundred. The exact number is not the point. The point is that the ratio exists, that you can compute it in ten seconds, and that almost nobody does.

This is also why "it worked on NIFTY, let me check BANKNIFTY" is such a powerful test and such a poor one when it fails. If the same parameters work on a different but related instrument, you have found evidence that is not from your original sample. If they only work after re-tuning, you have found that you can tune.

What to do instead

Fewer dials beats better dials. A strategy with two parameters and a plain stop, tested over a long sample, is a more trustworthy thing than one with nine parameters and a better curve — and it is far easier to diagnose when it stops working, because there are only two things it could be.

When you do want to improve something, improve it for a stated reason rather than because the curve moved. "Gap days behave differently because the opening auction resolves overnight information, so I will not trade the first fifteen minutes after a gap of more than 0.75%" is a hypothesis. It can be wrong, and you will know why. "Removing Tuesdays helped" is not a hypothesis at all.

And test the improvement the honest way: on the data you held back. That is the subject of the next chapter.

What to take from this chapter

  • Overfitting is not a mistake you make once. It is the accumulated effect of many small, individually reasonable improvements.
  • Read a parameter sweep for a plateau, not a peak. Pick the middle of a working region, not its best value.
  • A single value that works while its neighbours do not is a coincidence, not a parameter.
  • Count trades per degree of freedom. Below roughly 30, you can fit almost anything to almost any sample.
  • Prefer a stated reason over a lifted curve. A hypothesis can be tested; an observation that helped cannot.
  • Every lift you see in a test you have already looked at many times is partly real and partly you. Only held-back data separates them.