MyAlgoGuru Help
Education Glossary Open the app
Chapter 3 of 5

In-sample, out-of-sample, walk-forward

Holding data back is the only tool that separates a real edge from a well-described coincidence. It also only works once, which is the part people get wrong.

The previous chapter ended with a problem. Every improvement you make to a strategy lifts the curve you are looking at, and nothing in that curve tells you whether the lift was structure or noise.

There is exactly one reliable way out: show the strategy data it has never seen, and look at the result once.

The split

Divide your history into two parts. Build on the first. Do not open the second.

The first part is your in-sample data — the workshop. Tune here, add filters here, change your mind here, run as many tests as you like. Nothing you do in the workshop is cheating, because you have not claimed anything yet.

The second part is your out-of-sample data, and it has one job: to be the first honest test of a finished strategy. Its entire value comes from the strategy never having influenced it.

One split: build, then test once in-sample — tune freely out-of-sample open once Walk-forward: many splits, each one honest train test train test train test older today the test blocks never overlap their own training block
Two ways to hold data back. The simple split is easy to run and easy to understand, and gives you one honest number. Walk-forward re-trains on a rolling window and tests on the period immediately after it, repeatedly — so you get several honest numbers, each from a strategy that was fitted only to data before the period it was judged on. The amber blocks are the only results you are allowed to believe.

How much to hold back

The usual answer is somewhere between a quarter and a third of the sample, and that is a reasonable default. The better question is how many trades fall in the held-back part, because a strategy that trades twice a month gives you very little to judge in six months of data.

Aim for enough out-of-sample trades that the result would survive a bad run. Thirty trades is thin; a hundred starts to mean something. If your held-back period contains eleven trades, you have not built a test — you have built a coin flip with extra steps.

One more thing matters about which period you hold back: it should contain a regime you care about. A sample that is all of 2021 and none of 2020 has never shown the strategy a crash. If the only violent period in your history is in the in-sample half, move the split or lengthen the sample.

The rule that is almost always broken

Out-of-sample data works because the strategy has never seen it. The moment you look at the result and go back to adjust something, that has stopped being true.

You are allowed to do it. You are not allowed to keep calling the result out-of-sample.

Illustrative

A trader splits six years: four to build on, two held back. The in-sample result is good. The out-of-sample result is poor — a flat first year and a losing second.

They go back, notice the losses cluster in high-volatility weeks, add a volatility filter, and re-run. Now the out-of-sample period looks fine.

What just happened: those two years became part of the workshop. The filter was chosen because of what those years did. There is now no untouched data anywhere in the sample, and the second result carries no more weight than the first one did.

The honest options at that point are to hold back a third period that has never been opened, to wait for new data to arrive and judge on that, or to forward-test — which is the next chapter.

In practice almost everybody peeks, and the right response is not guilt, it is bookkeeping. Keep a note of how many times you have looked at the held-back period. Once is a test. Three times is a workshop you are describing as a test. By the fifth look you should assume the number means nothing and get more data.

Walk-forward, and what it is really for

Walk-forward answers a different and more practical question than the single split does.

The single split asks: does this strategy work on data it has not seen? Walk-forward asks: if I had been running this, re-fitting it periodically the way I actually would, would it have worked?

That is much closer to how a strategy is lived with. You do not fit a parameter once in 2020 and leave it untouched for six years — you re-estimate it occasionally. Walk-forward tests that whole practice: fit on a window, trade the next period, roll forward, repeat. The result is a chain of out-of-sample periods stitched together, and the equity curve you get from it is the most realistic one a backtest can produce.

It also gives you something the single split cannot: a view of stability. If the fitted parameter jumps from 9 to 34 to 12 between windows, the strategy does not have a parameter — it has a number that gets re-rolled every quarter, and the backtest flattered you by letting it be re-rolled with hindsight. A parameter that stays in the same neighbourhood across windows is a parameter you can trust in live trading, because the next re-fit probably will not move it much either.

Walk-forward costs more compute and more care, and it has its own trap: the more frequently you re-fit, the more chances the procedure has to fit noise. Re-fitting every week with a one-year window is not adaptive, it is twitchy. Longer windows, less frequent re-fits, and a parameter that barely moves — that is the shape of something robust.

What to take from this chapter

  • Holding data back is the only thing that separates a real edge from a well-described coincidence.
  • Hold back a quarter to a third, but judge it by the number of trades in it, not the number of months.
  • Make sure the held-back period contains a regime you care about, including a bad one.
  • Out-of-sample data is single-use. Once a result changes your strategy, that period has joined the workshop — and you must say so.
  • Count your looks. Once is a test; five times is a workshop wearing a test's name.
  • Walk-forward answers the more useful question: would this have worked as something I actually ran and periodically re-fitted?
  • Watch the fitted parameter across windows. A number that jumps between re-fits is not a parameter you can trade.