Qwen3.8 10 · Initialization and Logits

What made the largest difference

Tying the token embedding and output projection weights produced the biggest single gain, lowering the result from 1.247607 to 1.220730. Sharing these weights removed the separately initialized output matrix and improved the score by 0.026877, accounting for most of the total progress.

Reducing the logit softcap from 15 to 10 then improved the result to 1.217339. The opposite direction—relaxing the cap to permit sharper output distributions—was worse at 1.226453. On this short training benchmark, tighter control of extreme logits was therefore more useful than giving the model additional confidence range.

Initialization had a useful middle range

Reducing the token-embedding initialization standard deviation from 1.0 to 0.5 yielded another substantial improvement, reaching 1.209724. Moving from 0.5 to 0.25 technically improved the result again, but only from 1.209724 to 1.209705—a difference of 0.000019.

The nearby failures reveal the practical lesson more clearly than that tiny accepted gain. A scale of 0.125 was worse in two tests, producing 1.211477 and 1.211584, while increasing the scale from 0.25 to 0.375 produced 1.209040. Thus 0.25 was the best tested token-embedding scale, but the evidence does not establish a sharply defined optimum: the difference between 0.25 and 0.375 was small, and no repeated runs were reported.

Changing the value embeddings from a uniform initialization to a normal distribution with standard deviation 0.25 delivered the final improvement, from 1.209705 to 1.208420. This was modest but strict, so it was retained.

Recommended configuration

For this pinned RTX 3090 five-minute benchmark, use:

• tied input-token embeddings and output-projection weights;
• a logit softcap of 10;
• token embeddings initialized from a normal distribution with standard deviation 0.25;
• value embeddings initialized from a normal distribution with standard deviation 0.25.

The broader lesson is that controlling initial activation and logit magnitudes helped, but only up to a point. Weight tying supplied the dominant improvement, moderate downscaling of initialization supplied most of the remainder, and further shrinking became counterproductive.

What remains unknown

These results cover ten experiment records under one fixed five-minute benchmark. The evidence contains no repeated measurements, uncertainty estimates, longer training runs, or evaluations on another GPU or data configuration. It therefore shows which settings won in this specific search, but not whether the very small late-stage differences are reproducible or whether the same initialization scales remain optimal with a longer training budget.