Before I use a ranking system, I want to know whether its higher-ranked stocks did better, bucket by bucket, in more than one kind of market.
In a 10-year backtest of this guide’s example ranking, returns rose from a loss of 10.9% a year in the lowest-ranked tenth of stocks to a gain of 14.7% in the highest, with one break in the order. Most of the difference came from avoiding the losers, and among larger companies the ranking didn’t separate its buckets well enough to be useful.
Why I’ve relied on ranking systems
I use Portfolio123 for most of this work because it lets me backtest a ranking system and look at results by quantile, one slice of the ranked list at a time. I’ve treated my factor ranking, and choosing from rankings that have held up in backtests, as one of the main alpha-seeking parts of my process, the parts aimed at beating the market after adjusting for risk.
In October 2026 I ran simulations of my allocation. Before running them, I suspected I’d hold a concentrated portfolio of stocks from my ranking in my taxable account. The simulations showed I could best meet my tax and risk objectives in that account without the stock portfolio, because after fees and taxes it wasn’t the asset that best suited my tolerance for drawdowns. Drawdowns are drops from a portfolio’s peak.
Since 1997, nearly every decision I’ve made to hold a stock for more than a week has started from a ranked list, though until 2003 I built the ranking from screens. I ran four, for quality, value, growth, and momentum, and weighted each by how well it had performed. Many stocks showed up in more than one, so a stock’s combined screen score told me which to look at first. I then brought the top stocks’ ratios into a spreadsheet that ranked them on factors from quantitative fundamentals research and gave each a second score. I looked closely at free cash flow at that step, because I didn’t think my screens captured it well. Most of the screens and ratios compared a stock with its industry.
Last, I read the segment information, the backlog, management’s discussion and analysis, and the footnotes to the financial statements. I wanted to know whether the fundamentals could support a higher price, or only looked good for now, say because a low P/E and price-to-book put the stock high on value. The footnotes joined the routine after the collapses of Enron and WorldCom in 2001 and 2002, which showed that some companies counted revenue and earnings in odd ways that undermine the ratios built on them. I was also reading Douglas Skinner and Richard Sloan on earnings torpedoes and checking accruals. From 2001, that last step took in themes and theses too.
I’d been working toward a ranking system since at least 1999, and by 2001 I had one of sorts. It wasn’t until 2003 that I had enough data to build rankings directly, and I’ve preferred ranking to screening for choosing stocks by factors ever since.
Thirty stocks passing a filter today is no evidence that stocks like them tend to do well. A ranking system scores every stock in the universe, so you can ask the historical question directly. When it put stocks in its top bucket, what happened next, compared with the stocks in its bottom bucket?
Read the backtest bucket by bucket
A ranking backtest sorts the universe into buckets by rank at each rebalance date and follows each bucket’s return.
Most people start with the top bucket’s return, the easiest number in the test to be fooled by. Check these first:
| What to look for | Why |
|---|---|
| Returns rise steadily from the lowest bucket to the highest | A ranking that works should sort the whole universe. If only the top bucket stands out, one lucky group of stocks can explain the result. |
| The bottom buckets are clearly worse | A weak bottom is useful information too. It is where short candidates come from, and it confirms the ranking is sorting rather than picking a few winners. |
| The spread holds across sub-periods | Split the test into several multi-year windows and market environments. A system that worked in only one of them is telling you about that period. |
| The spread holds in a different universe or size range | If results disappear when you exclude micro-caps or change the universe, liquidity and trading costs may explain them. |
| Each bucket holds enough stocks | One or two big moves can make or erase the return of a bucket that holds 10 or 20 stocks. The fewer stocks in each bucket, the less a spread between buckets can tell you. |
| The data is point-in-time | Rankings must use only information available on each historical date. Restated financials and today’s index membership make backtests look better than they were. Portfolio123 keeps delisted companies in its backtests and dates financial data by when it was available, though its data provider sometimes backfills later figures. |
| Few factors and few adjustments | Every weight you tune against the same history makes the backtest fit that history better and makes it a weaker guide to the future. |
A four-factor ranking, decile by decile
This ranking system is adapted from work in Richard Tortoriello’s Quantitative Strategies for Achieving Alpha (McGraw-Hill, 2009), which reports backtests of dozens of single factors and combinations grouped under profitability, valuation, cash flow, growth, capital allocation, price momentum, and risk. It scores every stock in the universe on four factors, weighted equally:
| Factor | Measure | Ranks higher when the measure is |
|---|---|---|
| Value | Enterprise value to EBITDA, trailing 12 months | Lower |
| Cash profitability | Free cash flow (operating cash flow minus capital spending, trailing 12 months) as a percentage of invested capital (common equity, preferred equity, and long-term debt, latest quarter) | Higher |
| Profitability | Return on equity, trailing 12 months | Higher |
| Momentum | Where the latest close sits in its 12-month price range, from 0 at the low to 100 at the high. It’s there so I’m not catching falling knives: stocks still closing near their lows. | Higher |
A stock gets no value for a measure that wouldn’t mean anything, such as enterprise value to EBITDA when either one isn’t positive, or return on equity when common equity isn’t positive now or wasn’t a year earlier. A missing value ranks at the bottom for that factor.
In Tortoriello’s summary of the book for the AAII Journal, these four make a screen, with a cutoff for each, such as enterprise value to EBITDA of 8 or less (AAII Journal, 2010). Here they’re ranked instead, so every stock gets a place in the order rather than a pass or a fail.
I don’t trade this system. I’ve invested with a version of it that I modified and refined with backtests. How to refine a system that way without fooling yourself is the subject of a coming guide, Your best backtest needs to account for all the ones you discarded.
The chart shows how this ranking did over the 10 years to September 2026, with its universe split into 10 buckets, or deciles.
Annualized total return by ranking bucket
- Entire period
- First half
- Second half
- Buckets / reference
Show the numbers
| Series | Entire period | First half | Second half |
|---|---|---|---|
| Universe | 8.1 | 14.3 | 2.2 |
| Russell 2000 (IWM) | 10.1 | 14.3 | 6.1 |
| Bucket 1 | −10.9 | 6.9 | −25.8 |
| Bucket 2 | 1.0 | 10.4 | −7.7 |
| Bucket 3 | 4.3 | 14.7 | −5.2 |
| Bucket 4 | 8.8 | 14.7 | 3.2 |
| Bucket 5 | 9.5 | 16.1 | 3.2 |
| Bucket 6 | 11.7 | 14.7 | 8.7 |
| Bucket 7 | 13.0 | 15.5 | 10.6 |
| Bucket 8 | 12.5 | 13.8 | 11.2 |
| Bucket 9 | 14.5 | 16.1 | 13.0 |
| Bucket 10 | 14.7 | 16.0 | 13.4 |
The universe is US stocks priced above $3 with a median daily dollar volume above $50,000 over the last 126 trading days, excluding stocks that trade over the counter or are unlisted, master limited partnerships, real estate investment trusts (REITs), and business development companies (BDCs). Master limited partnerships are out because they send investors K-1 tax forms, and because their pass-through taxes and heavy depreciation make their ratios hard to compare with corporations’. REITs and BDCs have similar problems with their ratios. The price, volume, and listing rules are about tradability. They keep out stocks whose prices are likely stale, whose bid-ask spreads are wide for their price, or that I couldn’t trade in any size without moving the price. Those stocks often show the clearest separation between buckets in a backtest, but the comparison isn’t fair: the backtest trades them at prices I couldn’t get. In real trading I’d face heavy slippage, or be stuck in a position I couldn’t exit. Slippage is the gap between the price a backtest assumes and the price an order actually gets. Those floors are low enough to keep many small and micro-cap companies, so the benchmark is the Russell 2000.
Over the 10 years, returns rose from a loss of 10.9% a year in decile 1 to a gain of 14.7% in decile 10. The order broke once: decile 8 earned 12.5% a year, less than decile 7’s 13.0%. Deciles 6 through 10 all beat the Russell 2000’s 10.1%.
The two five-year halves hint at how the ranking behaves when conditions change. In the first half, small companies did well: the universe and the Russell 2000 both earned 14.3% a year. The ranking barely separated deciles 3 through 10, which all earned between 13.8% and 16.1%. Instead, it marked out its bottom two deciles. In the second half, small companies struggled: the universe earned 2.2% a year and the Russell 2000 6.1%. This time returns rose at every step but one, from a loss of 25.8% in decile 1 to a gain of 13.4% in decile 10. Decile 5 earned 3.20% to decile 4’s 3.22%. The top decile slipped only from 16.0% to 13.4% a year, while the universe fell from 14.3% to 2.2%.
Most of the difference comes from avoiding the losers. Decile 1 trailed the universe by 7.3 percentage points a year in the first half and by 28.0 in the second, while decile 10 beat it by 1.7 and 11.2. The winners themselves did about as well as large-company stocks: over the 10 years, decile 10 earned 14.7% a year, a little behind the S&P 500’s 15.5% but well ahead of the Russell 2000’s 10.1%.
The universe made as much difference as the ranking. On the Russell 2000, the top decile’s return beats the bottom decile’s by 18.2 percentage points, against 25.6 points in this universe. The middle deciles show the difference as clearly as the ends. Here, deciles 3 through 8 range from 4.3% to 13.0% a year. On the Russell 2000 they span 5.1 points, and deciles 4 through 7 all earned between 10.1% and 10.7%. On the Russell 1000 and the S&P 1500, the middle spans only 3.8 and 2.8 points, and the ranking doesn’t separate its buckets well enough to be useful. Some of that narrowing would happen even if the ranking worked just as well, because larger companies’ returns differ less from one another.
Annualized total return by decile in four universes
Show the numbers
| Decile | This universe | Russell 2000 | Russell 1000 | S&P 1500 |
|---|---|---|---|---|
| 1 | −10.9 | −3.9 | 4.3 | 9.6 |
| 2 | 1.0 | 5.1 | 11.2 | 10.7 |
| 3 | 4.3 | 7.1 | 9.6 | 9.4 |
| 4 | 8.8 | 10.1 | 11.0 | 10.0 |
| 5 | 9.5 | 10.7 | 11.6 | 10.3 |
| 6 | 11.7 | 10.3 | 11.1 | 11.2 |
| 7 | 13.0 | 10.1 | 12.2 | 12.1 |
| 8 | 12.5 | 12.2 | 13.5 | 12.1 |
| 9 | 14.5 | 11.4 | 14.9 | 14.5 |
| 10 | 14.7 | 14.3 | 14.7 | 14.0 |
| Span of deciles 3–8, points | 8.8 | 5.1 | 3.8 | 2.8 |
Why factor research finds many effects stronger among smaller companies is a question for a coming guide, The same ranking can look strong or useless depending on its universe.
I rebalance this test every four weeks. In an earlier run that still included REITs and BDCs, rebalancing weekly raised returns only slightly in most buckets, but neither run charged anything for slippage. Weekly rebalancing nearly doubled turnover, the share of the portfolio replaced in a year, from 304% to 583%. Realistic trading costs would almost certainly cancel that small gain or leave weekly rebalancing behind.
Small universes make small buckets
From 2005 to 2013 I worked on portfolios for ultra-high-net-worth clients, as an analyst until 2008 and then as a portfolio manager. Every stock had to come from an approved list of about 100 companies. Some screens did separate that list: they backtested well, and their top decile clearly beat their bottom decile. But I didn’t yet hold back data the test never sees, or test separate slices of time, as I would from 2013 at a proprietary trading firm. And in a list of 100 companies, a decile is 10 stocks and a quintile is 20. One stock that doubles adds 10 percentage points to a 10-stock bucket’s return for that period, so a few big moves can create or erase a spread. It’s hard to say how far a spread between buckets that small can be trusted.
The universe in my example is far broader. Each of its deciles holds about 195 stocks, so one stock that doubles adds about half a percentage point to its bucket’s return for that period.
Limits of the example
The example is a backtest. It doesn’t show that I earned these returns, and it leaves out slippage, taxes, and risk measures such as volatility and drawdowns. A performance claim would need all of those, along with the live or simulated label, the period, the benchmark, and the calculation method. A ranking with a strong quantile spread produces candidates for research, and how much to hold, or whether to own a stock at all, is decided separately.
Check a quantile backtest with AI
I gave ChatGPT the backtest above and asked it to run this guide’s checks, showing its arithmetic and using only the numbers I gave it. My first prompt is written as a template. Copy it, and the model asks for your own bucket returns, bucket sizes, and settings before it starts.
October 11, 2026
Me
I want to check a ranking system’s quantile backtest before I trust it. Don’t start yet. First, ask me for each bucket’s return from the lowest-ranked to the highest, for the whole test and each sub-period; the universe’s and the benchmark’s returns; how many stocks each bucket holds; and the settings: dates, rebalance frequency, how the buckets are weighted, trading costs, and the universe’s rules. Then: 1. Check the order. Say whether returns rise from the lowest-ranked bucket to the highest in each period, and list every place the order breaks. 2. Compare the ends. Give the spread between the top and bottom buckets in each period, and how much of it comes from the bottom bucket trailing the universe rather than the top bucket beating it. 3. Compare the periods. Say whether the pattern holds in each sub-period or rests on one of them. 4. Check the bucket sizes. Say whether each bucket holds enough stocks for its return to mean much, and how much one stock that doubled would add to a bucket’s return. 5. List what the backtest leaves out or could have gotten wrong, judging from the settings, such as trading costs, taxes, risk measures, and data that wasn’t available on each date. Show the arithmetic for every number you calculate, so I can check it. Use only the numbers I give you. If you need something I haven’t given you, say so rather than estimate it. Don’t tell me whether to use the ranking system. I’ll decide that after I’ve checked your work.
I want to check a ranking system’s quantile backtest before I trust it. Don’t start yet. First, ask me for each bucket’s return from the lowest-ranked to the highest, for the whole test and each sub-period; the universe’s and the benchmark’s returns; how many stocks each bucket holds; and the settings: dates, rebalance frequency, how the buckets are weighted, trading costs, and the universe’s rules. Then: 1. Check the order. Say whether returns rise from the lowest-ranked bucket to the highest in each period, and list every place the order breaks. 2. Compare the ends. Give the spread between the top and bottom buckets in each period, and how much of it comes from the bottom bucket trailing the universe rather than the top bucket beating it. 3. Compare the periods. Say whether the pattern holds in each sub-period or rests on one of them. 4. Check the bucket sizes. Say whether each bucket holds enough stocks for its return to mean much, and how much one stock that doubled would add to a bucket’s return. 5. List what the backtest leaves out or could have gotten wrong, judging from the settings, such as trading costs, taxes, risk measures, and data that wasn’t available on each date. Show the arithmetic for every number you calculate, so I can check it. Use only the numbers I give you. If you need something I haven’t given you, say so rather than estimate it. Don’t tell me whether to use the ranking system. I’ll decide that after I’ve checked your work.
ChatGPT
Please send the following, marking anything you don’t have as unknown. I’ll wait for your data before starting.
-
Returns for the whole test and each sub-period, with buckets ordered from lowest-ranked to highest-ranked:
Period and dates Bucket 1 (lowest) Bucket 2 … Highest bucket Universe Benchmark Whole test Sub-period 1 Sub-period 2 Add as many buckets and periods as needed. Identify the benchmark, state whether returns are cumulative or annualized, whether they include dividends, and whether they are before or after trading costs and taxes.
-
Bucket sizes and weights: How many stocks does each bucket hold? If counts vary, provide counts by rebalance date if available, or the minimum, typical, and maximum. Explain how stocks are weighted within each bucket and any position limits. This determines the contribution from one stock doubling.
-
Backtest settings:
- Exact start/end dates and sub-period boundaries.
- Rebalance frequency, when rankings are calculated, and when trades execute.
- Trading costs: commissions, spreads, slippage, and market impact included.
- Universe rules: eligibility, exclusions, liquidity/size requirements, and how membership changes.
- How the universe’s return is calculated and weighted.
- Tax assumptions and any available risk measures.
- Whether historical constituents, delisted stocks, and data available on each ranking date were used, including reporting delays or later revisions.
I’ll use only the numbers you provide, show the arithmetic for every calculation, flag missing information, and leave the decision about using the system to you.
Me
Ranking system: four factors weighted equally (enterprise value to EBITDA, lower is better; free cash flow to invested capital; return on equity; where the latest close sits in its 12-month range). 10 buckets (deciles), each equally weighted, rebalanced every four weeks, no slippage or commissions. Dates: October 1, 2016 to September 19, 2026, split into halves on September 25, 2021. Universe: US stocks priced above $3 with median daily dollar volume above $50,000 over 126 trading days, excluding over-the-counter and unlisted stocks, MLPs, REITs, and BDCs. Benchmark: Russell 2000, measured by IWM. Point-in-time data. Annualized total return, %:
| Bucket | 10 years | First half | Second half |
|---|---|---|---|
| 1 (lowest rank) | -10.9 | 6.9 | -25.8 |
| 2 | 1.0 | 10.4 | -7.7 |
| 3 | 4.3 | 14.7 | -5.2 |
| 4 | 8.8 | 14.7 | 3.2 |
| 5 | 9.5 | 16.1 | 3.2 |
| 6 | 11.7 | 14.7 | 8.7 |
| 7 | 13.0 | 15.5 | 10.6 |
| 8 | 12.5 | 13.8 | 11.2 |
| 9 | 14.5 | 16.1 | 13.0 |
| 10 (highest rank) | 14.7 | 16.0 | 13.4 |
| Universe (average of the buckets) | 8.1 | 14.3 | 2.2 |
| Russell 2000 (IWM) | 10.1 | 14.3 | 6.1 |
Stocks per bucket: 195
ChatGPT
Left out: two short progress notes, and between them the tool runs, in which ChatGPT read four of this site’s files and searched the web.
The returns broadly improve with rank, but the first half has several reversals. The top beats the bottom in both halves; the separation is much larger in the second half. Most of the spread in every period comes from the bottom trailing the reported universe.
Show the rest of ChatGPT’s answer
All calculations below use your numbers. pp means percentage points. These are differences between annualized returns, not the annualized return of a trade that buys the top bucket and shorts the bottom.
For the ordering check, these are all adjacent declines and ties. Every other adjacent step rises.
| Period | Does return rise with rank? | Declines | Ties |
|---|---|---|---|
| Whole test | Mostly, but not throughout | Bucket 7 → 8: 12.5 − 13.0 = −0.5 pp | None |
| First half | Uneven | Bucket 5 → 6: 14.7 − 16.1 = −1.4 pp; bucket 7 → 8: 13.8 − 15.5 = −1.7 pp; bucket 9 → 10: 16.0 − 16.1 = −0.1 pp | Bucket 3 → 4: 14.7 − 14.7 = 0.0 pp |
| Second half | Never declines, but does not strictly rise at every step | None | Bucket 4 → 5: 3.2 − 3.2 = 0.0 pp |
These judgments apply at the precision supplied; an apparent tie could conceal a small difference.
To compare the ends, split the spread as follows:
Top − bottom = (top − universe) + (universe − bottom).
| Period | Top minus bottom | Top beating universe | Bottom trailing universe | Bottom’s share of spread |
|---|---|---|---|---|
| Whole test | 14.7 − (−10.9) = 25.6 pp | 14.7 − 8.1 = 6.6 pp | 8.1 − (−10.9) = 19.0 pp | 19.0 ÷ 25.6 × 100 ≈ 74.2% |
| First half | 16.0 − 6.9 = 9.1 pp | 16.0 − 14.3 = 1.7 pp | 14.3 − 6.9 = 7.4 pp | 7.4 ÷ 9.1 × 100 ≈ 81.3% |
| Second half | 13.4 − (−25.8) = 39.2 pp | 13.4 − 2.2 = 11.2 pp | 2.2 − (−25.8) = 28.0 pp | 28.0 ÷ 39.2 × 100 ≈ 71.4% |
The components reconcile:
- Whole test:
6.6 + 19.0 = 25.6 pp. - First half:
1.7 + 7.4 = 9.1 pp. - Second half:
11.2 + 28.0 = 39.2 pp.
This is an arithmetic decomposition relative to your reported universe, not proof of what caused the returns.
The benchmark provides a separate comparison:
| Period | Top minus IWM | Universe minus IWM |
|---|---|---|
| Whole test | 14.7 − 10.1 = 4.6 pp | 8.1 − 10.1 = −2.0 pp |
| First half | 16.0 − 14.3 = 1.7 pp | 14.3 − 14.3 = 0.0 pp |
| Second half | 13.4 − 6.1 = 7.3 pp | 2.2 − 6.1 = −3.9 pp |
Across periods, the direction of separation holds in both halves, but its strength and orderly progression depend much more on the second half. The first half’s top bucket is below buckets 5 and 9, while the second half’s top bucket leads.
The spread widens by 39.2 − 9.1 = 30.1 pp. That widening consists of:
- A larger bottom-bucket shortfall:
28.0 − 7.4 = 20.6 pp. - A larger top-bucket advantage over the universe:
11.2 − 1.7 = 9.5 pp. - Check:
20.6 + 9.5 = 30.1 pp.
The top bucket’s own annualized return actually falls: 13.4 − 16.0 = −2.6 pp. Its larger relative advantage therefore accompanies weaker absolute performance. The aggregate halves cannot show whether the pattern holds year by year or at each rebalance, and the full test is not independent evidence from its component halves.
For bucket size, the same arithmetic applies to every bucket:
- Each stock’s weight immediately after equal weighting:
1 ÷ 195 × 100 ≈ 0.5128%. - If one stock doubles while held at its starting allocation, its contribution is
(1 ÷ 195) × 100% ≈ 0.5128 ppof bucket return over that holding interval.
That calculation assumes no intervening resizing and measures the contribution relative to that stock having a flat return, with everything else unchanged. If the doubling spans multiple rebalances, the contribution depends on its price path and changing holdings.
The holdings count dilutes a single doubling, but it does not establish statistical reliability. Stocks can move together, share sector or factor exposures, and recur across rebalances. Without holdings, contribution records, and a return series, I cannot determine whether each bucket contains enough independent evidence for its return to be reliable. Nor can I translate the doubling example into a contribution to the reported annualized backtest return.
There is also a definition to reconcile: “universe = average of the buckets.” The simple averages of the displayed annualized returns are:
- Whole test:
(-10.9 + 1.0 + 4.3 + 8.8 + 9.5 + 11.7 + 13.0 + 12.5 + 14.5 + 14.7) ÷ 10 = 79.1 ÷ 10 = 7.91%. - First half:
(6.9 + 10.4 + 14.7 + 14.7 + 16.1 + 14.7 + 15.5 + 13.8 + 16.1 + 16.0) ÷ 10 = 138.9 ÷ 10 = 13.89%. - Second half:
(-25.8 − 7.7 − 5.2 + 3.2 + 3.2 + 8.7 + 10.6 + 11.2 + 13.0 + 13.4) ÷ 10 = 24.6 ÷ 10 = 2.46%.
Those differ from the reported universe returns of 8.1%, 14.3%, and 2.2%. This need not be an error: averaging underlying periodic returns and then compounding generally differs from averaging annualized returns. The aggregation method needs confirmation. I retained your reported universe figures throughout the comparisons.
The settings leave these other limits and possible errors unresolved:
- Trading costs and capacity. Commissions and slippage are explicitly omitted. Spread, market impact, turnover, portfolio size, and trade sizes were not supplied. The dollar-volume filter alone cannot establish executable prices or capacity. I cannot calculate a cost-adjusted return. Transaction costs reduce investment returns. SEC investor bulletin
- Taxes. Tax treatment is unspecified. Rebalancing every four weeks does not tell us which positions were sold, their holding periods, or realized gains. Account type, tax lots, distributions, and applicable tax assumptions are needed for an after-tax calculation. SEC performance guidance
- Risk and uncertainty. No return series, volatility, drawdowns, market sensitivity, sector exposures, or contribution history was supplied. I cannot calculate risk-adjusted performance, confidence intervals, or whether the small differences between neighboring buckets are statistically meaningful.
- Point-in-time implementation. You state that the data are point-in-time. What remains unverified is whether fundamentals became usable only after publication, later restatements were excluded from earlier dates, and universe eligibility used information available then. Historical membership must also retain subsequently failed or delisted stocks and their losses. These are distinct look-ahead and survivorship checks. CFA Institute backtesting guidance
- Signal and execution timing. The ranking uses the latest close, but the execution price and time are unspecified. A simulation must not assume it could trade at a price before the completed signal was available. Treatment of holidays, halts, missing prices, and stocks becoming ineligible is also unknown.
- Factor definitions and exceptional values. Financial measurement periods, ranking directions beyond EV/EBITDA, missing-data rules, and tie handling were not supplied. Negative EBITDA, negative or near-zero equity or invested capital, and extreme ratios can change rankings substantially. Those rules could particularly affect the bottom bucket.
- Return and benchmark construction. Total return is specified, but dividend reinvestment, corporate actions, delisting proceeds, and IWM’s price-versus-NAV convention remain unverified. Differences in eligibility and exposures also mean that beating IWM does not, by itself, isolate the ranking system’s contribution.
- Selection and overfitting. It is unknown whether the factors, weights, filters, dates, or split were chosen after examining results, or how many alternatives were tried. A historical split is not an untouched validation period unless the model was fixed before its results were examined.
Checking the answer against the data file
A separate AI session checked every number in ChatGPT’s answer, which is quoted unchanged, against the backtest’s data file. The file carries more decimal places than the table I gave ChatGPT. The session also opened the answer’s three links. Every calculation checks out against the numbers ChatGPT was given, and the fuller data and the links add these points.
| Point | What the data file and links show |
|---|---|
| The ties | At full precision, the second half’s buckets 4 and 5 aren’t tied: bucket 5 earned 3.20% a year to bucket 4’s 3.22%, a small break in the order of the kind the answer warned a rounded tie could hide. The first half’s tie between buckets 3 and 4 is exact in the data; it’s one of the repeated figures noted under the chart above. |
| Buckets 5 and 9 | The answer’s point that the first half’s top bucket trailed buckets 5 and 9 rests on the other repeated figure: both show 16.08% for the first half. |
| The universe | The answer was right not to call the universe figures an error. They average the buckets’ returns period by period and compound the result, which gives a different number from averaging the buckets’ annualized returns. My prompt’s label, “average of the buckets,” left that out. |
| What it said was missing | The factor definitions and the rules for missing values that it asked about are earlier in this guide, as is how the data handles delisted companies and the dates figures became available. My prompt didn’t include them. |
| The links | All three open. The SEC bulletin covers how fees and expenses, including transaction fees, reduce returns. The SEC tip on mutual funds covers taxes on distributions and turnover. The CFA Institute reading on backtesting names look-ahead and survivorship bias without defining them. |
ChatGPT’s most useful points came where it declined to guess. It flagged the universe figure it couldn’t reproduce and warned that a rounded tie might hide a break in the order, and the data file bears out both. If you have your backtest at full precision, give the model that rather than a rounded table, and check its arithmetic against your own file.
Your check
Take one ranking system you use and write down its bucket returns from lowest to highest, with the number of stocks in each bucket. Mark every place where the order breaks. Then rerun it on a second universe or a different period, and note whether the breaks move and whether anything beyond the top bucket still separates.
This guide is part of the factor and screening series. Two guides take the evaluation further: Rerun a screen enough times and one version will look great by chance and a coming guide, An effect that shows up in only one kind of market may be an accident of it. Another coming guide, Turnover and taxes can erase a strategy’s lead over its index, covers what it costs to follow the ranks.