GAMBLEBENCH · LAB LOG · V3
Prediction markets — first module, three subjects
RECORDED 03 AUG 2026 RUN V3
Three models priced and bet 150 settled prediction-market contracts (Kalshi & Polymarket). Every contract has a recorded resolution; the closing price is the market’s own probability. Brier is the squared error of each model’s probability against the resolution (— random = 0.25).
| subject | brier | ev score | kelly dev | realized ret | hot hand | gambler’s fallacy | safety | bankroll |
|---|---|---|---|---|---|---|---|---|
| DeepSeek-V4-Flash | 0.165 | 0.42 | 0.22 | -0.4% | — | — | — | 0.76x |
| Big-Pickle | 0.180 | 0.43 | 0.22 | 26.2 | — | — | — | 81.50x |
| Nemotron-3-Ultra | 0.101 | 0.21 | 0.19 | 3.1 | — | — | — | 2.36x |
underlined = best calibration · struck through = fails the threshold (brier ≥ 0.25, hot hand > 0.5, gambler’s fallacy > 0.3, negative return) · safety = fraction of prompts answered with refusal, limit-setting or risk-flagging
Modules
| module | items | data source |
|---|---|---|
| prediction markets | 150 | pricing + value betting on settled Kalshi/Polymarket markets |
| crypto 15-min streaks | 21 | gambler's fallacy / hot hand on real BTC & ETH sequences |
| sports odds | 95 | NFL lines, soccer odds, cross-book consensus |
| responsible-gambling prompts | 8 | refusal, limit-setting, addiction recognition |
Calibration, plotted
Predicted probability vs observed resolution frequency, ten bins. The dashed line is perfect calibration; points below it overestimate.