Skill, or luck?

Why a few good calls prove nothing

Imagine a fair coin lands heads seven times out of ten. Nobody would call the coin skilled — the run is well within what chance produces, and a single throw tells you nothing about the next. A stock thesis is a noisier version of the same problem. Each call is one decision, the feedback arrives weeks or months later, and the outcome is buffeted by macro, earnings surprises and flows that no amount of reading can anticipate. The variance is enormous, and the sample is tiny.

That is the trap behind almost every "AI beats the market" headline. A seven-of-ten hit rate, quoted with confidence, is statistically indistinguishable from luck. The noise band around a win rate built on a handful of outcomes is wide enough to swallow any edge that might be there — and wide enough to manufacture one that isn't. Before you can say a forecaster is skilled, you have to rule out the explanation that costs nothing to assume: that they got lucky. Doing that needs a base rate to beat and a skill score to measure the beating.

Skill versus luck, made operational

"Skill" sounds unfalsifiable until you make it a number. The operational version is the Brier skill score: does staking conviction add information over always forecasting the base rate? If the high-conviction calls play out no more often than the low-conviction ones, the conviction is decoration — the forecaster would do just as well writing the average probability on every name. A skill score of zero says exactly that. A positive score says the forecasts genuinely discriminate winners from losers; a negative one says conviction actively misleads.

orbyd computes this on its own resolved record and publishes it. The sample is currently 113 resolved theses — meaningful sample by the small-sample convention the engine applies. The skill score sits near zero, which is the honest read of a young record: the conviction tiers do not yet separate outcomes the way a demonstrated edge would. The full track record shows the working, and the decomposition behind it.

How big a sample before a record means anything

There is no magic threshold, but the working convention from forecasting and small-sample statistics is around thirty resolved outcomes before a record is more than directional. Below ten, it is barely a signal at all; between ten and thirty it is indicative but not robust; past thirty it starts to carry weight. Slow-feedback finance reaches that line slowly, because each thesis has to actually resolve — its trigger fires, or its catalysts play out — before it can be scored.

orbyd's resolved count stands at 113 today, which the engine reads as "meaningful sample". That is surfaced as a limit the page owns up to: at this count the record is directional and nothing more, and saying so openly is the whole point of keeping it in public. A score that can sit near zero in plain view is one you can actually audit.

The failure mode: cherry-picking and survivorship

The reason "AI beats the S&P by X%" claims are so common and so hollow is that they are usually unfalsifiable by construction. Winners are kept and quoted; losers quietly drop off the page; and the "performance" is a return marked to a benchmark after the fact, with the entry and exit chosen to flatter it. This is survivorship bias dressed as evidence — you are shown the survivors and asked to infer skill from a sample the failures were deleted from.

The structural answer is an append-only ledger where every call resolves and the losers stay on the board. That is what orbyd is built around: each thesis ships with the exact trigger that would prove it wrong, set in advance, and resolves in public as played-out or invalidated. Nothing is retroactively dropped, because the invalidated theses are the record as much as the played-out ones. You can read how that accountability is enforced rather than promised.

What it would take to move the score

Honesty about a near-zero score is only worth anything if it comes with what would change it. Three things would: more resolved outcomes, past the small-sample threshold, so chance stops being a sufficient explanation; conviction tiers that actually separate, with SUPREME calls outpacing LOW ones on the reliability diagram; and a positive skill score sustained — not a single lucky window — across a sample large enough to trust.

Until then, the claim is deliberately modest. Reading a stock well is not the same as being right, and the public record settles it, not the prose. That is not a hedge; it is the standard the whole project is held to, and the reason the score is published at all.

This is the question the whole record exists to answer. To see how each call is built so it can be scored at all, read what a thesis actually is; to watch the score get kept, browse the resolved ledger on the track record or the full dossier corpus.