How we measure whether our own signals work — and why “it worked” is harder to prove than it looks

Most scanners emit signals and never check them. Measuring ours honestly turned out to be the hard part.

Almost every stock scanner shares one quiet habit: it emits signals and never checks whether they worked. The signals go out, the outcomes are never measured, and "it works" rests on vibes. We decided to measure ours — and the surprising part is how hard honest measurement turns out to be. "It worked" is far easier to feel than to prove.

This piece isn't about what we found — those results live in their own articles. It's about the method, because the method is the thing most tools skip, and it's the reason our conclusions are worth more than an untested claim.

Step one: measure forward, never backward

Every signal is logged the day it fires, on the universe that was actually liquid that day, and only later do we fill in what the stock did over the following days. That ordering matters: it makes look-ahead bias impossible. We never get to peek at the future when deciding what fired — the signal is frozen in time before its outcome exists.

The traps that fake confidence

Then the hard part. Naive measurement produces impressive-looking numbers that are statistical illusions. Four traps in particular:

• Everything moves together. On a given day ~1,900 stocks share the same market wind, so they aren't 1,900 independent data points — they behave more like one. Pool them and you manufacture false confidence. The fix is to aggregate to a daily figure first and do the statistics over days, so your real sample size is the number of days, not the number of rows.
• Overlapping windows. Consecutive days' forward returns share most of the same price movement, so they're correlated in a way that inflates significance. We correct for it (a standard technique called Newey–West) rather than pretend each day is independent.
• The textbook threshold over-rejects. The famous "t greater than 2" bar assumes ideal conditions that small, messy samples don't meet, so it flags noise as signal too often. We calibrate the real threshold on simulated random data and use that, not the textbook value.
• The dead stocks vanish. A stock that gets suspended or delisted after a signal — often the worst outcomes — can quietly drop out of the measurement, flattering the result. So we count and report how many outcomes went missing, per signal, instead of ignoring the gap.

Why bother with all that?

Because the alternative is comfortable nonsense. Skip these steps and you can "prove" almost any signal works — the numbers will look great and mean nothing. Every one of these corrections makes our results weaker and less exciting, and that's exactly the point: what survives them is more likely to be real. It's why we can say, for instance, that our own signal-count sort ranks nothing and that a volume surge points the wrong way — unflattering conclusions we'd never have reached by grading our own homework loosely.

This article explains StockLearn's data methodology using illustrative examples. It is educational, not investment advice or a recommendation to buy or sell any security.