Rerun a screen enough times and one version will look great by chance

1. Your trial log

I gave later passes 1 to 3 levels rather than 3. Each one started at last year’s keeper on mostly the same history, so the first level would often find nothing meaningful.

2. Backtests, and what they amount to

A level is 2 runs per weight. The screens had 4 weights, so 8 runs a level. I read “a few main measures” as 3 to 5 weights per ranking, so 6 to 10 runs a level. I didn’t count re-runs of each pass’s unchanged starting point, which would add about 50. If you’ve kept running yearly passes since 2013, add about 64 a year (24–120), which puts the total near 1,700 by now.

Independent tries. By an independent try I mean a system whose backtest luck is unrelated to the others’. The question is how many such systems you’d have to backtest, keeping the best, to end up on average as lucky as the version you kept. Three things bring the raw count down to that number.

First, repeats aren’t new tries. Each yearly pass re-ran mostly the same weight sets on mostly the same history, so what matters is the number of distinct weight sets per ranking. The first pass produced about 25. Each later accepted step added about 7 more, because one of the 8 neighbours is always the set you just left. Depending on how far the weights drifted over the decade, that’s roughly 25 to 100 distinct sets per ranking.

Second, neighbours are near-copies. Weight sets within 15 to 20 points of each other hold mostly the same stocks at the top of the list. I assumed their benchmark-relative returns correlate at 0.75 to 0.95 on average. The correlation has to be measured on benchmark-relative returns: raw returns of any long-only stock portfolios correlate highly just because they all hold stocks, and that would make the count look smaller than it is.

Third, when tries all correlate at ρ, the best one’s expected luck is √(1−ρ) times the luck of the best of the same number of unrelated systems. That’s exact for equally correlated normal results and a fair approximation for a hill climb. Converting back to unrelated systems:

• 25 sets at ρ = 0.95 come to about 2.
• 50 sets at 0.85 come to about 3.
• 100 sets at 0.75 come to about 6.
• If you held few enough stocks that ρ is nearer 0.65, about 9.

So each ranking amounts to about 2 to 8 independent tries, most likely 3 or 4. The screens era comes to about 2 to 5. The screens fed the rankings only through design choices, so I haven’t added them to the rankings’ number.

Two notes on using these numbers:

• Don’t mix them with the DSR shortcut. In the deflated Sharpe ratio, the hurdle rises both with the number of independent trials and with the variance of results across them (Bailey & López de Prado, preprint pp. 7–8). Their shortcut estimates that number by interpolating with the average correlation, N̂ = ρ̂ + (1−ρ̂)M (Appendix A.3, p. 14), which gives larger values than mine. They pair it with the measured spread across your trials, which is small for near-copies. My figures already account for the correlation, so combining them with that spread would count it twice.
• The search could only inflate part of the backtest. Each pass moved weights locally, so the search could only have inflated the gap between your kept weights and where the search started. Whatever selection went into the starting design isn’t in this count.

3. Selections that didn’t look like backtests

On that last row, Harvey, Liu and Zhu argue that after so much data mining across hundreds of factors, a new factor needs a t-ratio above 3.0 rather than 2.0 (RFS 29(1), abstract p. 5). Not charging costs and never testing another rebalance frequency add nothing to the count, because you didn’t search over them. It did leave the search free to favour weights that traded more.

How I added the combination. Novy-Marx covers this case:

• Combining the best k of n candidate signals is biased almost as badly as using the single best of nᵏ candidates (NBER w21329, printed p. 2).
• A combination of similar, individually insignificant signals backtests with a t-statistic close to their average t times the square root of how many are combined (p. 6).
• Weighting each signal in proportion to its in-sample performance (p. 8), which is what you did, produces much higher critical values than equal weighting (p. 12).

His experiments use long-short portfolios built on random signals, and his tables start at 10 candidates, so I worked your case directly using the same mechanism:

• Each kept ranking carries the luck of its own best-of-n.
• Dropping the worst of four adds about a third of a standard error per kept ranking on average, less when they’re correlated.
• Averaging three rankings whose benchmark-relative returns correlate at c multiplies the combination’s luck by √(3/(1+2c)). That is his √k when c = 0.

With 3 or 4 tries per ranking and c around 0.6, the combined list comes to roughly 6 to 9. Weighting by backtest nudges that up, so I’d call it about 10, with 3 to 45 as the plausible range. These numbers match expected luck, which is how the DSR sets its hurdle. Novy-Marx’s equivalents match 5% critical thresholds instead (Table 1, p. 27), so his figures aren’t directly comparable to mine.

4. Where I filled gaps

5. Checks, in order

1. Write down the reason and the pass/fail rule before each test. Do this first because it costs nothing, and it decides whether the checks below count once or become another round of search. Bailey and López de Prado point out that applying a holdout test enough times makes false positives expected rather than unlikely (preprint p. 6). They recommend planning tests in advance, with investment theory rather than computing power deciding which experiments to run (p. 10).

How to run it:

• Keep one row per run: date, ID, what changed, why you expect it to matter, the direction you predict, the settings (universe, dates, rebalance, costs), the result, whether you kept it, and a running count.
• Export each run’s weekly returns.
• Before each check below, write down what result would count as holding up.
• In future yearly passes, log any change whose only reason is “see if it helps” as pure search.

2. Nudge the weights, and include untuned anchors. This comes second because it’s cheap, spends no fresh data, and measures how much of the backtest the search produced at all. Your ±5 neighbours are already known to be slightly worse, since that’s how each pass stopped, so use bigger moves.

How to run it:

• For each ranking, move each weight ±10 and ±20 points using your equal-share offset. That’s 16 runs for a four-weight ranking.
• Add two anchors: equal weights, and your original pre-tuning weights if you still have them.
• Keep the universe, dates, weekly rebalance and gross returns the same as the original runs.
• For the combination, run equal weights across the three, each combination weight ±10, and a version that swaps in the fourth ranking.
• Decide beforehand not to adopt any of these variants, or they become trials.

The gap between your kept version and the untuned anchors is the most the search could have inflated. How quickly results fall off with distance, and whether the gap comes from a few years, shows how that gain is spread. Portfolio123’s ranking-system optimizer can run a batch of weight sets like this in one go.

Then use the exported returns to measure the two numbers behind my ranges: the average benchmark-relative correlation among each ranking’s variants, and the same among the four rankings. Clustering similar trials and counting the clusters is another way to estimate the number of independent trials (summary).

3. Use fewer parts. This comes third because it’s also cheap and in-sample, and it shows what each layer adds. Novy-Marx’s advice is to evaluate each signal’s marginal power separately, and not to believe in signals because they backtest well together (p. 29).

How to run it:

• Drop one measure at a time from each ranking, spreading its weight equally over the rest.
• Run each measure alone.
• Run the combination ladder: best single ranking, then the equal-weight average of all four, then the equal-weight best three, then your backtest-weighted best three. Each rung shows what that selection step added in-sample.

4. Test on untouched data, last and once. It comes last because it’s the most direct read on what the search inflated, but you can only use it once. Options, strongest first:

• Walk-forward replay of the whole procedure. Your search followed fixed rules, so it can be rerun mechanically.
  • Before starting, fix a numeric threshold for “meaningful.”
  • At each year-end, run the hill climb on data through that date, and redo the 3-of-4 pick and weights on the same data.
  • Hold the result for the following year, then stitch those following-year results together.
  • Compare the stitched record with your kept version’s backtest over the same years. The difference measures what the search added in-sample.
  • Keep it gross and weekly so the gap reflects the search rather than costs.
  • It’s several hundred to over a thousand runs, so script it against Portfolio123’s API.
  • These runs are out of sample by construction, so they add no selection unless you change the procedure after seeing them.
• Years after your last pass. If you stopped re-tuning in 2013 and haven’t looked at this system’s results since, 2014 onward is your cleanest untouched stretch. Any year you’ve already looked at doesn’t qualify.
• A held-back group of stocks. You tuned on every stock you had, so there’s no untouched group inside your universe. The workable version is to replay the search on half the stocks, split by a fixed identifier, and test on the other half. It’s weaker than a time split: both halves share the same years, so it catches stock-specific luck but not luck in the period.
• A market or size segment you never used. This is the weakest, because a difference between markets can explain a miss as easily as the search can.

Your live 1997–2013 results were partly forward tests, since each year’s version traded the following year. But themes, the buy list and real costs are mixed into them, so they can’t isolate the search. Nudging the screen cutoffs or the number of holdings tests robustness rather than the search, because you never tuned them, so it can wait until after these four.