Does ranking Polymarket traders predict anything?
Mostly, no. Ranking by profit, as most trackers do, told us nothing about the next 45 days. Our star rating did, modestly. Our more sophisticated skill test did not. Here are the numbers, the misses, and the limits.
The short version
- Raw PnL ranking: no signal. Rank correlation with future return −0.06; the top fifth did 1.2 points worse than the bottom fifth.
- Edgewatch stars: a modest, real signal. Correlation +0.22; top fifth beat bottom fifth by 15.9 points, 95% range +6.2 to +30.3. In 99% of bootstrap resamples the gap was positive.
- Skill evidence (sign-randomization): did not pass. Correlation +0.18, but the top-minus-bottom range (−6.6 to +14.7) includes zero.
- “Not enough data” is the strongest warning label. Only 25% of traders with fewer than 15 resolved bets were profitable next period, versus about two-thirds of everyone else.
Method
- Pool: union of Polymarket’s top-100 leaderboards for day, week, month and all time: 300 wallets.
- History: up to the last 1,000 closed positions per wallet, plus resolved-but-unredeemed positions in the same time window.
- Split: scores use only positions resolved before September 1, 2026 (train). The outcome is the actual ROI on positions resolved in the last 45 days (test).
- Eligibility: at least 15 bets in train and 5 in test. 132 traders qualified; 119 had no history before Sep 1 in their last 1,000 positions (very high-frequency traders), and 50 had too few test bets.
- Metrics: Spearman rank correlation between score and test ROI; mean test ROI by quintile; top-minus-bottom quintile gap with a 95% bootstrap interval.
Results: every signal we tested
Which rankings predicted the next 45 days?
Return gap between the top- and bottom-scored fifth of 132 traders, with 95% bootstrap range. Win rate alone (range −65.9% to +17.5%) is omitted from the chart for scale; see the table.
Show as table
| Ranking by | Spearman | Top − bottom | 95% range |
|---|---|---|---|
| Edgewatch stars | +0.22 | +17.9% | +6.2% to +30.3% |
| Skill evidence | +0.18 | +4.3% | −6.6% to +14.7% |
| Calibrated-null z, shrunk | +0.10 | +5.9% | −1.2% to +14.2% |
| Calibrated-null z | −0.05 | −9.3% | −22.3% to +2.8% |
| Raw PnL ranking | −0.06 | −1.2% | −13.4% to +12.8% |
| Past ROI | −0.09 | −8.8% | −19.9% to +3.7% |
| Signal | Spearman | Top − bottom | 95% range | P(gap ≤ 0) | Verdict |
|---|---|---|---|---|---|
| Edgewatch stars (our rating) | +0.22 | +17.9% | +6.2% to +30.3% | 0.01 | Holds up |
| · component: win rate | +0.25 | +15.3% | +2.3% to +28.5% | 0.01 | Holds up |
| · component: track record size | +0.13 | +8.5% | −4.6% to +22.7% | 0.10 | Weak |
| · component: profit factor | +0.13 | +5.7% | −7.2% to +18.4% | 0.20 | Not proven |
| · component: return on stake | +0.07 | +5.3% | −5.6% to +15.7% | 0.16 | Not proven |
| Skill evidence (sign-randomization) | +0.18 | +4.3% | −6.6% to +14.7% | 0.21 | Not proven |
| Calibrated-null z, shrunk (closed only) | +0.10 | +5.9% | −1.2% to +14.2% | 0.05 | Borderline |
| Win rate (train period, raw) | +0.05 | −15.5% | −65.9% to +17.5% | 0.72 | No signal |
| Calibrated-null z | −0.05 | −9.3% | −22.3% to +2.8% | 0.93 | Failed |
| Raw PnL (what most trackers rank by) | −0.06 | −1.2% | −13.4% to +12.8% | 0.58 | No signal |
| Past ROI | −0.09 | −8.8% | −19.9% to +3.7% | 0.91 | No signal |
Edgewatch stars
Mean next-period ROI by score fifth (bottom → top)
Raw PnL ranking
Mean next-period ROI by score fifth (bottom → top)
Read the quintiles honestly: even for stars the pattern is not a clean staircase. The middle fifth did best (+28.7%) and the fourth fifth was flat. The top-vs-bottom gap is real, but the rating is a coarse filter, not a precise ranking. That is why we show stars, not a decimal score.
Skill labels: how each group did next period
| Label (from train) | Traders | Profitable in test | Mean test ROI |
|---|---|---|---|
| Proven skill | 60 | 72% | +7.2% |
| Promising | 23 | 61% | +1.8% |
| Could be luck | 36 | 61% | +19.2% |
| Not enough data (<15 bets) | 12 | 25% | −15.9% |
What didn’t work
- Skill evidence as a ranking. Its gap range crosses zero. We demoted it from “the Edge Score that replaces stars” (our original plan) to context on a profile.
- “Could be luck” traders had the highest mean return (+19.2%). Their share of profitable traders (61%) was lower than “proven” (72%), so a few big winners drive that mean, but it is a clear reminder that the luck label is not a sell signal.
- Persistence was barely above chance. Of 57 traders labelled “proven” in train, 8 (14%) were still “proven” in test, against an 11% base rate. The academic benchmark is 44% (Gómez-Cram et al., 2026, on a far larger dataset). We are nowhere near that yet.
- Calibrated-null z failed. Testing whether a trader beats the market’s own prices (each bet wins with probability equal to its price) ranked traders slightly backwards (−0.05). Our best guess: traders who exit early have much lower real variance than the test assumes.
- Two bugs this test caught, now fixed in the app. Of 40,055 resolved-but-unredeemed positions, only 115 were winners; without a consistent time window they buried long-time traders in old losses. And positions sold before resolution had been scored as full wins or losses instead of their realized PnL.
An earlier run whose test window excluded unredeemed losses gave stars +0.26 (range +8.9% to +33%) and skill evidence +0.22. We report the stricter, final version above. Stars led in both.
Limits
- Survivorship bias: the pool is drawn from today’s leaderboards, which favours recent winners. Every signal shares this bias, so the comparison between signals is fair; absolute returns are flattered.
- Small sample: 132 traders and one 45-day window. Ranges are wide, and one month can be noise.
- Coverage: 1,000 recent positions per wallet miss most history of very high-frequency traders.
- Not a strategy test: ROI here is the trader’s own return, not what a follower would earn after delay and slippage. That is what the copy simulator is for.
What’s next
- Rerun monthly on a new, later window, and publish it here even if stars stop working.
- Extend history beyond 1,000 positions (full on-chain history) to include high-frequency traders.
- Test closing-line value (entry price versus later price) as a new component.
- Pre-register the next test’s cutoff date and metrics before the data exists.
Computation note: skill-evidence p-values use an exact-moment normal approximation of the sign-randomization test (O(n)), nearly identical to 2,000 Monte Carlo draws (Spearman 0.177 vs 0.169). Reproduce with scripts/backtest-edge.ts; raw output: edge-backtest-2026-10-16.json. Nothing here is financial advice; past performance does not predict future results.