Qwen3.8 05 · Norm and Residuals

What produced the improvement

The progress graph reflects three accepted steps. First, slightly damping the learned residual path—initializing its coefficient (resid_lambdas) at 0.9 instead of 1.0—improved the result from 1.251425 to 1.235840. This was the first clear indication that full-strength residual updates were not ideal for this short five-minute training regime.

Next, the direct path that mixes the model’s initial representation back into later computation was weakened from 0.1 to 0.05. With the residual coefficient already at 0.9, this lowered the loss substantially to 1.218093. Restoring that direct mixing coefficient to 0.1 while retaining the residual damping produced the best result, 1.215132.

That last detail matters: the baseline also used direct mixing at 0.1, but with a residual coefficient of 1.0. The evidence therefore supports the combination of 0.9 residual scaling and 0.1 direct mixing, not a claim that 0.1 alone was responsible for the gain.

The useful boundary discovered around direct mixing

Stronger direct mixing consistently hurt performance. Initial values of 0.15 and 0.2 produced losses of 1.232800 and 1.238436, respectively. Even the smaller increase from 0.1 to 0.125 degraded the result sharply to 1.260587.

In practical terms, the useful region found here is narrow and lies at or below 0.1. The run did not establish whether a value between 0.05 and 0.1 would be better: the planned 0.075 experiment was stopped before it produced a measurement. The evidence supports keeping 0.1, but it does not prove that 0.1 is the exact optimum.

Residual damping was also sensitive

Moving the residual coefficient away from 0.9 in either tested direction failed to improve the retained model. A stronger damping value of 0.8 reached 1.217550—close, but still worse than 1.215132. Increasing it to 0.95 was much worse at 1.254388.

The practical lesson is not simply “smaller residuals are better.” A modest reduction from 1.0 to 0.9 helped, while both 0.8 and 0.95 lost against the best combined configuration. For this benchmark, 0.9 is the strongest measured choice.

Recommended configuration and remaining unknowns

Keep the learned residual coefficient initialized at 0.9 and the initial-representation mixing coefficient at 0.1. This is the only tested combination that achieved the best measured loss of 1.215132, and all later completed variants were rejected under the strict-improvement rule.

The search did not produce evidence about alternative RMS-normalization placement, other normalization designs, or broader residual-stream restructuring. It also left the 0.075 direct-mixing value unmeasured. Those remain open questions; the completed experiments only justify the two initialization choices above.