Skip to content

Train a small KiT from scratch ​

After finishing this section, you will have a KiT you trained yourself (3100 万—31 million—parameters, the same size as the paper's KiT-S) that you can use to produce forecast plots; and you will see with your own eyes how "the longer you train, the worse it gets" happens.

1. Get the training script ​

The official training code hasn't been released yet. This lesson provides a minimal version that uses only the model, feature, and window code already public in the repo, with the training objective following the paper:

Download train_mini.py

Put it in the KiT directory (at the same level as eval/). What it does is exactly the five steps from the previous section:

python
t = torch.sigmoid(torch.randn(B))                   # logit-normal 抽 t
eps = torch.randn_like(x_tgt)                       # 噪声
z = (1 - t)[:, None, None] * x_tgt + t[:, None, None] * eps
v = model(torch.cat([x_ctx, z], dim=1), t, cond, Lh)  # 历史干净 + 未来加噪
loss = ((v.float() - (eps - x_tgt)) ** 2).mean()    # 只在未来那段

The script has only four functions, about 200 lines:

FunctionWhat it does
build_windowsFor each stock, randomly picks an anchor point, calls the official prepare_inference_window to get the history x_ctx and conditions cond, then computes the Lh real bars after the anchor into five numbers, standardizes them, and uses them as x_tgt. Windows whose calendar doesn't line up due to trading halts are dropped
fm_lossThe five lines above: sample t, add noise, forward pass, compute loss
val_lossComputes the loss on validation windows with fixed-seed t and noise, so numbers across different step counts can be compared directly
mainAdamW (β=0.9/0.95, weight decay 0.1), gradient clipping 1.0, linear warmup, periodic validation, saves best.pt and last.pt

Not done (but in the paper): condition dropout, history truncation, multi-market multi-granularity mixing, multi-GPU FSDP, parameter EMA. These are exactly good practice exercises; the end of this section has two.

2. Start training ​

bash
python train_mini.py --scale 1d --steps 3000 --per-stock 60 --batch 32 --eval-every 250 --warmup 100 --lr 3e-4 --out runs/scratch-1d
  • --scale 1d: daily bars only
  • Model size comes from configs/dit_30m.yaml (default values)
  • Training windows: for each stock, randomly cut 60 segments between 2016 and 2025-06-30, 6000 segments in total
  • Validation windows: second half of 2025, 4 segments per stock, 400 segments in total, not used in training

You should see (RTX 4080 SUPER, 227 seconds end to end):

text
model 31.0M params (31.0M trainable) | scale 1d Lc=250 Lh=20 | windows train 6000 val 400 (10s)
step 0 | val 3.4190
step 250 | train 2.2832 | val 1.8879 | 19s | peak GPU 4.3 GB
  saved best.pt (val 1.8879)
step 500 | train 1.8537 | val 1.8116 | 38s | peak GPU 4.3 GB
  saved best.pt (val 1.8116)
step 750 | train 1.7955 | val 1.7866 | 57s | peak GPU 4.3 GB
  saved best.pt (val 1.7866)
step 1000 | train 1.7667 | val 1.7995 | 76s | peak GPU 4.3 GB
...
step 3000 | train 1.3409 | val 2.1159 | 227s | peak GPU 4.3 GB
saved runs/scratch-1d/last.pt

train is the average training loss over that period; val is the loss on those 400 segments of unseen data. Lower is better.

3. Read this curve ​

Drag the slider below to view each checkpoint:

Drag itTraining from scratch: loss curve (real record)RTX 4080 SUPER tested
0 步1500 步3000 步3.51.3
Training lossValidation lossValidation minimum

第 750 step, lowest validation loss (1.787). train_mini.py saves best.pt at this point.

For the first 750 steps both lines drop together—the model is learning real things. After that, the training loss keeps falling to 1.34, while the validation loss rebounds from 1.79 to 2.12: the model starts memorizing the historical details of these 100 stocks instead of learning generalizable patterns. This is called overfitting.

The reason is simple: the daily bars of 100 stocks total only 20.6 万 (206,000) bars, while the paper uses billions. Little data plus a big model leads to this. So the script saves a copy of best.pt every time the validation loss hits a new low—that's the one to use, not the final last.pt.

4. Produce a forecast plot with your model ​

The checkpoint format saved by the training script is the same as the official one, so you can hand it directly to the forecasting script:

bash
python eval/infer_for_pred.py --ckpt runs/scratch-1d/best.pt --case 300368.SZ,1d,2026-01-05 --out results/mine

You should see:

text
[warn] checkpoint has no ema key, falling back to model weights (step=750)
...
      terminal return actual -18.17%  predicted median -20.77% [q05 -43.96%, q95 +37.47%]  cumulative-return curve corr 0.900  0.4s

The warning on the first line is normal: we didn't do parameter EMA, so the script just uses the model's own weights.

Compare with the official model's forecast for the same example (the first-forecast lesson): the interval is −37% to −6%. Your small model gives an interval of −44% to +37%, much wider: it also thinks the price will fall, but is very unsure of itself. One example alone doesn't say much; for a serious comparison, use the previous chapter's infer_for_eval.py --ckpt and cal_metrics.py to compute RankIC over the whole test period.

5. Want to train better ​

  • More data: 100 stocks is too few. The paper uses 4800+ instruments, ten years, and seven granularities.
  • More granularities: replace --scale with 1h or 30m and retrain, or modify the script to mix multiple granularities in one batch.
  • Stop earlier: in this example you should stop around step 750. --steps 1000 is enough.
  • Use the official weights as a starting point: that's exactly the fine-tuning in the next section.

6. Exercise: add the paper's two techniques yourself ​

If you've learned PyTorch, each of the two changes below takes only a few lines:

  1. Parameter EMA. The open-source weights are the EMA copy. Maintain a shadow copy of the weights ema = copy.deepcopy(model), and after each step do ema_p.lerp_(p, 1 - 0.999); use ema for both validation and saving. When saving, put its state_dict() into the checkpoint's "ema" key—the official loading function reads it with priority, and the no ema key warning goes away. The paper uses 0.9999; for small data and short training, 0.999 is more appropriate.
  2. Condition dropout. In fm_loss, with probability 10% set cond["inst"] and cond["sector"] to 0 ("unknown"), and with probability 10% zero out the history x_ctx, while setting the history segment's pad_mask to False (see make_nohist_cond in kit_v1/infer/sampler.py). These correspond to inst_dropout, allcond_dropout, hist_dropout in the paper's config (0.1, 0.05, 0.1; see configs/dit_30m.yaml). Only with condition-dropout training do --cfg-w and --hist-cfg-w at inference time make sense.

Quiz timeTraining loss is still decreasing while validation loss starts rising. Which checkpoint should you use?

Check yourself ​

  • I trained my own runs/scratch-1d/best.pt
  • I can point out on the curve which step overfitting starts at
  • I produced a forecast plot with my own checkpoint

Next section: Fine-tuning on the official weights.

gaodong.aiAboutPrivacyContact

Independent tutorials, not affiliated with any product mentioned. Product names and trademarks belong to their owners. Check official pages for prices and policies.