Evaluation and Backtesting
A single prediction chart can't tell whether a model is good—some are right by luck. After finishing this section, you'll be able to run batch predictions over the entire test period, compute RankIC, and run the official backtest; you'll also understand what "RankIC 0.057" and "annualized +65%" mean in the paper.
First Understand RankIC: Guess Rankings, Not Prices
Suppose every Monday you have to pick from 5 stocks. You don't need to know how much each will rise—you only need to know who is stronger than whom.
On Monday, the model ranks the predicted returns of 5 stocks; by Friday, you look at the ranking of actual gains. The more similar the two rankings, the more useful the model. RankIC is the correlation coefficient between these two rankings (Spearman rank correlation):
- 1: the rankings match perfectly
- 0: same as random guessing
- −1: completely reversed
Compute it once per anchor, then take the average.
Financial data is extremely noisy, and the rule of thumb cited in the paper is: the IC of stock returns rarely exceeds 0.1. So 0.05 is already a usable signal, not "only 5% correct".
Results from the Paper
On the test period from 2026-01-01 to 2026-04-10 (the official checkpoint was trained through the end of 2023 and validated through the end of 2025, so it never saw this period), the average across the seven granularities:
| Task | KiT | Kronos-base-FT | Chronos-Bolt extended | TimesFM 2.5 |
|---|---|---|---|---|
| Return prediction RankIC | 0.0571 | 0.0521 | 0.0461 | 0.0181 |
| Volatility prediction RankIC | 0.6600 | 0.4974 | 0.4234 | 0.4565 |
Volatility is far easier to predict than returns: the direction of a rise or fall is hard to guess, but "whether this stock will move violently next" follows patterns. KiT produces 16 paths at once, and volatility is simply the dispersion of the paths themselves—this is exactly the benefit of generating multiple paths.
For the complete seven granularities and more comparison models, see KiT's GitHub page.
Run It Yourself: Batch Prediction → Compute Metrics
python eval/infer_for_eval.py
python eval/cal_metrics.pyThe first command predicts 100 stocks, 7 granularities, and every anchor in the test period. You should see prediction result files appear in results/infer_for_eval/raw/ one after another; if you stop midway and rerun, already-saved ones are skipped. The second computes the metrics:
| Task | Predicted value | Ground truth |
|---|---|---|
| return | cumulative log return of each of the 16 paths, averaged | actual cumulative log return |
| vol | per-bar standard deviation of log returns for each of the 16 paths, averaged | actual per-bar standard deviation of log returns |
| price | cumulative return relative to the anchor's close for each of the 16 paths, averaged | actual value |
The full run takes a lot of time, especially on 1-minute bars. Try small-scale first:
python eval/infer_for_eval.py --scales 1d,1h --max-inst 20
python eval/cal_metrics.py --scales 1d,1h--max-inst 20 takes only the first 20 stocks, and --max-anchors N takes only the first N anchors per granularity. If you have multiple GPUs, you can start several processes running --shard 0/2 and --shard 1/2 separately.
Anchors are taken by default once every "prediction window," with no overlap between windows (--anchor-stride auto).
Backtesting: Turning Rankings into a Portfolio
After the batch prediction finishes:
python eval/backtest.py
python eval/backtest.py --top 20 --slip 5 --scales 5m,15m,1hRules (from the official README):
- 1?
- 2?
- 3?
- 4?
Click them one by one, in order.
It also simulates several A-share constraints:
- Can't buy at limit-up: stocks already at the limit-up price at the anchor (within 0.5% of it) or listed for fewer than 5 days are skipped; Can't sell at limit-down: stocks at limit-down on the last bar of the window are carried to the next window. Price limits are 10% by default, 20% for stocks with 300/301/688 prefixes; the 5% for ST stocks is not simulated.
- T+1: buy at the close, sell only in the next window—naturally satisfied.
- Costs: commission of 2.5 basis points per side, transfer fee of 0.1 basis points, stamp duty of 5 basis points on sells, plus slippage (default 10 basis points per side). Lots of 100 shares and market impact are not considered.
- Benchmark comparison: equal-weighted average and median of the same 100 stocks over the period, plus CSI 500 (downloaded once with akshare, cached in
data/csi500_daily.csv; if the download fails, it isn't plotted).
Results are in results/backtest_results/.
Section 4.3 of the paper runs a one-year (2025.05–2026.04) long-only backtest on A-shares, with the number of holdings and rebalancing frequency selected on the validation set and then fixed, and costs of 12.5 basis points to buy and 22.5 basis points to sell: net annualized return of +65.2% on 5-minute bars (Sharpe 1.70) and +64.4% on 2-hour bars (Sharpe 2.46), beating the equal-weighted same stock pool as well as CSI 300 and CSI 500.
A backtest doesn't mean you can make money
A backtest is "pretend trading" on history: it ignores lots of 100 shares and the impact of large orders, and its parameters were also chosen on history. It shows the model's rankings carry information, not that buying accordingly will earn the same money. The content of this lesson is for learning and research only and does not constitute investment advice.
Quiz timeA model's return RankIC is 0.05—what does that mean?
Check Yourself
- I can explain RankIC using the "guess the ranking" example
- I know why volatility's RankIC is much higher than returns'
- I've run an evaluation once at small scale with
--max-inst
Next section: Use Your Own Data.