Backtest v5-xgb-cal, generated July 5, 2026
The Liquidation Retrospective
Would ChainScore have flagged the wallets that got liquidated?
On 7,720 borrowers it had never seen, 1,933 of whom were later liquidated, ChainScore would have flagged 88% of those later liquidated, while flagging 48% of those that were not.
How good is the model, really
Sample base rate in this backtest is 25% liquidated, oversampled from a deployed base rate closer to 9.1%. The backtest oversamples positives so ranking quality can be measured precisely, not to flatter the model.
How well the model ranks a liquidated wallet above a non-liquidated one, picked at random. 0.5 is a coin flip, 1.0 is perfect ranking.
Precision versus recall tradeoff. More informative than ROC-AUC when positives are rare, and the harder, more honest number to look at here.
Mean squared error between the predicted probability and the actual outcome. Lower is better, 0 would be a perfect forecast.
Expected calibration error, how far predicted probabilities drift from observed rates on average. Lower means the score means what it says.
Not every chain is equally predictable
PR-AUC by chain, computed from the same backtest. The weakest slice is ethereum at 41.2% PR-AUC, where 647 of 3,964 borrowers in that slice were liquidated. Thinner or noisier liquidation history on a chain makes the ranking harder to learn.
Does the predicted risk match what happened
Each point groups wallets by predicted risk and plots that predicted risk against the liquidation rate actually observed in that group. The dashed line is perfect calibration, where predicted risk equals observed outcome. Points above the line mean ChainScore underestimated risk there, points below mean it overestimated. ECE across all bins is 0.018.
What this does not prove
- Liquidation is a proxy for default, not identical to it. A wallet can be liquidated by a price wick while remaining a good actor, and a wallet can default in ways that never trigger an onchain liquidation.
- The 48% false positive rate at the operating threshold is real: at 88% recall, a large share of the wallets flagged as high risk were not liquidated in the outcome window.
- Liquidation inside a fixed window is partly luck, market conditions dominate individual behavior for thin histories. That is the main reason PR-AUC, not ROC-AUC, is the metric to trust here.
- Labels come from covered lending deployments only. A wallet liquidated on an uncovered protocol is recorded as a negative in this backtest, which understates the true positive rate somewhat.
- The operating threshold flags 52% of all borrowers as high risk, at a precision of 15.4%. Precision, not just recall, is the number to watch before acting on a flag.