Scarb Glossary
live
← All sections
How we verify

Test it on a different coin

The main check for everything: same minute, same side, different coin. If the result doesn't change, the rule had nothing to do with it — the market did the work.

Why you need it

Almost any rule shows a profit if the market happened to move the right way while it was running. The rule says "buy", the market went up that month, and out comes a pretty number that has nothing to do with the rule.

You cannot tell the two apart by looking. So you need a check that strips the market out and leaves only the rule.

How it's done

Take every signal the rule fired and record three things: the moment in time, the side (buy or sell) and how long you hold. Then repeat the exact same trade on a randomly picked different coin.

Now compare the two sets. If the rule makes the same money on its own coins as it does on random ones, then what's working isn't the rule — it's the way the market moved at the moments the rule happened to fire.

One detail that matters: the random coin has to come from the same crowd. Comparing a small coin against a large one proves nothing — they move differently, and the gap you see comes from that, not from the rule.

What usually comes out

Usually something unwelcome. The rule turns out not to be a way of picking a coin, but a way of picking a moment. That isn't useless — knowing when the market is about to move is worth something. But you have to use it differently: don't let the rule choose the coin, fire the signal on whatever is most liquid and easiest to trade.

Sometimes the check kills the rule outright: own coins and random coins both come out at the same zero. Then the whole earlier profit was a coincidence.

What this check does not tell you

It doesn't say whether the profit survives fees, whether your money fits in the book, or how the rule behaves when the market is in a different mood. Those are separate questions, and each one can kill a rule on its own.

More in this section

Split the window in half
Cut the stretch in two and count each half on its own — the cheapest way to catch a made-up result.
Never pick the best setting from the past
Try a hundred settings and keep the best one, and you don't have a rule — you have a description of the past.
Check liveness by the data, not by the code
Switched on and actually writing data are two different claims, and only the data can settle the second one.