Skip to content

Evaluation and Backtesting ​

A single prediction chart can't tell whether a model is good—some are right by luck. After finishing this section, you'll be able to run batch predictions over the entire test period, compute RankIC, and run the official backtest; you'll also understand what "RankIC 0.057" and "annualized +65%" mean in the paper.

First Understand RankIC: Guess Rankings, Not Prices ​

Suppose every Monday you have to pick from 5 stocks. You don't need to know how much each will rise—you only need to know who is stronger than whom.

On Monday, the model ranks the predicted returns of 5 stocks; by Friday, you look at the ranking of actual gains. The more similar the two rankings, the more useful the model. RankIC is the correlation coefficient between these two rankings (Spearman rank correlation):

  • 1: the rankings match perfectly
  • 0: same as random guessing
  • −1: completely reversed

Compute it once per anchor, then take the average.

Financial data is extremely noisy, and the rule of thumb cited in the paper is: the IC of stock returns rarely exceeds 0.1. So 0.05 is already a usable signal, not "only 5% correct".

Results from the Paper ​

On the test period from 2026-01-01 to 2026-04-10 (the official checkpoint was trained through the end of 2023 and validated through the end of 2025, so it never saw this period), the average across the seven granularities:

TaskKiTKronos-base-FTChronos-Bolt extendedTimesFM 2.5
Return prediction RankIC0.05710.05210.04610.0181
Volatility prediction RankIC0.66000.49740.42340.4565

Volatility is far easier to predict than returns: the direction of a rise or fall is hard to guess, but "whether this stock will move violently next" follows patterns. KiT produces 16 paths at once, and volatility is simply the dispersion of the paths themselves—this is exactly the benefit of generating multiple paths.

For the complete seven granularities and more comparison models, see KiT's GitHub page.

Run It Yourself: Batch Prediction → Compute Metrics ​

bash
python eval/infer_for_eval.py
python eval/cal_metrics.py

The first command predicts 100 stocks, 7 granularities, and every anchor in the test period. You should see prediction result files appear in results/infer_for_eval/raw/ one after another; if you stop midway and rerun, already-saved ones are skipped. The second computes the metrics:

TaskPredicted valueGround truth
returncumulative log return of each of the 16 paths, averagedactual cumulative log return
volper-bar standard deviation of log returns for each of the 16 paths, averagedactual per-bar standard deviation of log returns
pricecumulative return relative to the anchor's close for each of the 16 paths, averagedactual value

The full run takes a lot of time, especially on 1-minute bars. Try small-scale first:

bash
python eval/infer_for_eval.py --scales 1d,1h --max-inst 20
python eval/cal_metrics.py --scales 1d,1h

--max-inst 20 takes only the first 20 stocks, and --max-anchors N takes only the first N anchors per granularity. If you have multiple GPUs, you can start several processes running --shard 0/2 and --shard 1/2 separately.

Anchors are taken by default once every "prediction window," with no overlap between windows (--anchor-stride auto).

Backtesting: Turning Rankings into a Portfolio ​

After the batch prediction finishes:

bash
python eval/backtest.py
python eval/backtest.py --top 20 --slip 5 --scales 5m,15m,1h

Rules (from the official README):

Line them upWhat does the backtest do at each anchor? Click in order
  1. 1?
  2. 2?
  3. 3?
  4. 4?

Click them one by one, in order.

It also simulates several A-share constraints:

  • Can't buy at limit-up: stocks already at the limit-up price at the anchor (within 0.5% of it) or listed for fewer than 5 days are skipped; Can't sell at limit-down: stocks at limit-down on the last bar of the window are carried to the next window. Price limits are 10% by default, 20% for stocks with 300/301/688 prefixes; the 5% for ST stocks is not simulated.
  • T+1: buy at the close, sell only in the next window—naturally satisfied.
  • Costs: commission of 2.5 basis points per side, transfer fee of 0.1 basis points, stamp duty of 5 basis points on sells, plus slippage (default 10 basis points per side). Lots of 100 shares and market impact are not considered.
  • Benchmark comparison: equal-weighted average and median of the same 100 stocks over the period, plus CSI 500 (downloaded once with akshare, cached in data/csi500_daily.csv; if the download fails, it isn't plotted).

Results are in results/backtest_results/.

Section 4.3 of the paper runs a one-year (2025.05–2026.04) long-only backtest on A-shares, with the number of holdings and rebalancing frequency selected on the validation set and then fixed, and costs of 12.5 basis points to buy and 22.5 basis points to sell: net annualized return of +65.2% on 5-minute bars (Sharpe 1.70) and +64.4% on 2-hour bars (Sharpe 2.46), beating the equal-weighted same stock pool as well as CSI 300 and CSI 500.

A backtest doesn't mean you can make money

A backtest is "pretend trading" on history: it ignores lots of 100 shares and the impact of large orders, and its parameters were also chosen on history. It shows the model's rankings carry information, not that buying accordingly will earn the same money. The content of this lesson is for learning and research only and does not constitute investment advice.

Quiz timeA model's return RankIC is 0.05—what does that mean?

Check Yourself ​

  • I can explain RankIC using the "guess the ranking" example
  • I know why volatility's RankIC is much higher than returns'
  • I've run an evaluation once at small scale with --max-inst

Next section: Use Your Own Data.

gaodong.aiAboutPrivacyContact

Independent tutorials, not affiliated with any product mentioned. Product names and trademarks belong to their owners. Check official pages for prices and policies.