Sample size
18
The experiment
Below is a strategy with exactly zero edge — a pure coin flip, 50% win rate by construction, wins and losses the same size. We'll "test" it many times on a sample of N trades and record its apparent win rate each time. If small samples were reliable, every test would land near 50%. Watch what actually happens.
Each bar = how many of 2,000 tests landed at that apparent win rate. Teal band = "looks like a real edge" (≥55% or ≤45%).
Looked like a winner (≥55%)
—
Looked "edgeless" (45–55%)
—
Run it at 18. A coin flip — zero edge — will "look like a real edge" a shocking fraction of the time. Now drag the sample size up to 200+ and run again: the illusion collapses toward the truth (50%).
What just happened
At 18 trades, a completely edgeless strategy shows a win rate above 55% (or below 45%) roughly a third of the time, purely by chance. At 200 trades, that almost never happens — the apparent win rate is forced close to the truth. Small samples don't just add noise; they manufacture fake edges.
This is exactly why Tom's Rule of 4 looked good on 18 NFP events and evaporated on 195. It's not that he cheated — it's that 18 events cannot distinguish a real edge from a lucky run. The same math governs every backtest, every guru's screenshot, every "I went 14 for 18 last month."
The SRS contrast: our School Run backtest ran across 4,390 days and 7,000+ trades, and held a profit factor of 1.35 with zero negative years. That's a sample big enough to trust. Rule of 4's 18 was not. The number of samples is the whole difference between a track record and an anecdote.
How to never get fooled again
- Ask "over how many trades?" before anything else. Under ~100, treat the result as a hint, not proof. Under ~30, treat it as noise.
- A high win rate on few trades is the easiest thing in the world to produce by luck. Impressive-sounding, nearly meaningless.
- Demand to see the losers and the drawdowns. A track record with no ugly stretches has been either cherry-picked or run too short.
- Prefer many small samples across regimes (years, conditions) over one big lucky run. Consistency across time beats a single number.
- Your own results are subject to the same trap. Judge yourself over 30+ identical executions, never over one streak — good or bad.
The one-line version: a strategy "proven" on a handful of examples is not proven. Sample size is the difference between knowledge and a story.
See both sides for real.
One strategy that survived 17 years — and one that didn't. Same rigor, opposite verdicts. That's how you know the numbers are real.