Never pick the best setting from the past

Try a hundred settings and keep the best one, and you don't have a rule — you have a description of the past.

Why it's a trap

Every rule has knobs: how many days back to look, how big the move has to be, where to put the exit. There are usually several knobs, and that means hundreds of combinations.

Among a hundred combinations there is always a best one. It will look convincing and you'll even be able to tell a plausible story about why it works. But it was chosen using the same data it was tested on — so it describes that data, not anything about the market.

Easy to confirm: shift the stretch a few months and the best combination turns out to be average, or the worst.

What to do instead

Don't look at the best cell, look at the whole field. If the rule is real, the result changes smoothly: neighbouring settings give similar numbers, and there's a wide region where everything is roughly equally decent.

If only one cell is good and everything next to it is bad, that's noise. A real property of the market is never that narrow.

The second habit: pick the setting up front, on reasoning, and don't change it once you've seen the result. Uncomfortable, but honest.

What usually comes out

You almost never get a smooth field. More often it's blotchy, with the best cell sitting right next to a hole — and that on its own is your answer.

Staircases are a separate danger: the result climbs neatly from one setting to the next and it feels like you've found a law. Test it in a different market mood and the staircase falls apart into random numbers.

What this does not tell you

Even a wide, steady region is no guarantee the rule makes money. It only says what you found isn't an artefact of trying everything.