Skip to content

Close Reading of 3.3–3.4: Structure and Conditioning Injection ​

The "Model Architecture" chapter on training covered the shape of every tensor in the code. This section switches to the paper's perspective: the rationale behind each design choice, and the evidence provided by ablation experiments.

Equation (4): One Sequence ​

text
[ r_1 … r_nr | x_1 … x_Lc | z_1 … z_Lh ]
   Registers       Clean history     Noised future

The paper's explanation of each segment:

  • Registers: learnable parameters that serve as a "global attention sink and information aggregator." They do not correspond to any point in time.
  • History: Lc clean K-lines, never noised, returned unchanged at the output (the paper says bit-identical).
  • Future: Lh bars — during training these are the noised z_t, and at inference they are pure Gaussian noise.

Why are history and future placed in the same sequence, instead of using one encoder to read the history and another decoder to generate the future? The introduction gives the answer: this way attention is bidirectional across the entire window, the conditioning information (history) is fully preserved, and every future bar can directly attend to every history bar as well as to one another.

Conditions Take Two Paths: The Paper's "Conditioning Discipline" ​

This is the core of Section 3.3. The paper divides all conditions into two categories by "granularity," each handled by a different mechanism:

ConditionExamplesHow it enters the modelRationale
Window-invariant (identity)Market, industry, instrument, granularity, plus flow time tEach embedded and summed into c_base, which together with t's embedding passes through a shallow MLP (shared AdaLN) to modulate every layerIdentical across the whole window, suitable as a global "style"
Per-bar varying (calendar)Time of day (Fourier-encoded), position within the trading session, day of week, event flags, month, day of yearEmbedded and added directly to the token of the corresponding K-lineMust be temporally aligned with that K-line; can also be determined in advance at inference

The shared AdaLN draws on PixArt-α: a single MLP computes modulation parameters shared by all layers, and each layer only adds its own bias b_l (Equation 5), saving parameters compared with giving each layer its own AdaLN.

A Clever Detail: History Is Modulated with t = 0 ​

The shared AdaLN is computed twice:

  • Once with the true t, yielding c_tgt for the future tokens
  • Once again with t = 0, yielding c_ctx for the register and history tokens

The paper's rationale: history carries no noise and inherently sits at the "clean data" end of the flow, i.e., t = 0. Since the MLP is shared and shallow, the extra computation costs almost nothing, yet lets tokens in both roles receive appropriate normalization parameters.

3.4: One KiT Block ​

Equations (6)(7) follow the standard DiT block pattern of "modulated RMSNorm → attention → gated residual," then repeat with SwiGLU in place:

  • x ← x + α₁ ⊙ Attn(RMSNorm(x) ⊙ (1 + γ₁) + β₁)
  • x ← x + α₂ ⊙ SwiGLU(RMSNorm(x) ⊙ (1 + γ₂) + β₂)

Three choices worth noting:

  1. Attention is not causal. Every token, including future ones, attends to the whole sequence. The paper says this lets the model exploit "dependencies among future bars within the noise."
  2. RoPE positional encoding + QK-Norm, for training stability.
  3. Zero initialization. The output layer producing the gate α and each layer's bias b_l are both initialized to 0, so at the start of training every block is an identity mapping and the model's output velocity is 0. This avoids huge, meaningless gradients early in training (the approach from the original DiT paper).

Evidence: Appendix E, Table 10 ​

Changing only the conditioning injection while keeping everything else fixed, average return RankIC:

ConfigurationAverage return RankIC
AdaLN identity + per-bar calendar (KiT)0.0571
Identity keeps only market and industry (removing instrument and granularity)0.0386
No identity at all (c_base = 0)0.0329
No per-bar calendar0.0372

Three conclusions:

  • Identity matters most: removing it entirely causes the biggest drop, 0.0571 → 0.0329.
  • Fine-grained identity is useful: keeping only market and industry recovers 0.0386, but still far from 0.0571, showing that "exactly which stock and which granularity" carries information that coarse-grained identity cannot provide.
  • The calendar is useful too: removing it drops performance to 0.0372. Temporal structures like market open, close, and around holidays are neither in the identity nor learnable from history alone.

The paper concludes from this: injecting each class of signal at its own granularity is what makes the whole design work.

Quiz timeWhy doesn't ”day of the week” go through AdaLN like ”instrument” does?

Check Yourself ​

  • I can state what the three segments of the sequence are and why history is not noised
  • I can state which conditions go through AdaLN, which are added to tokens, and why
  • I can explain why history is modulated with t = 0
  • I can use Table 10 to explain how much identity and the calendar are each worth

Next section: A Close Reading of 3.5 and Appendix D: Training Objective and Dual Guidance.

gaodong.aiAboutPrivacyContact

Independent tutorials, not affiliated with any product mentioned. Product names and trademarks belong to their owners. Check official pages for prices and policies.