Autoresearch · 18 August 2026 · 2 min · Original language: English
Qwen3.8 07 · Optimizer
The best completed configuration lowered the recorded prediction loss from 1.249653 to 1.222123—an absolute improvement of 0.027530, or about 2.20%. The winning changes were a Muon momentum target of 0.98 and a matrix learning rate of 0.07. However, the evidence labels the metric as prediction loss, so it does not establish whether 1.222123 is also the requested validation bits per byte.
Measured improvement2.2% better
Started
1.2497
Best
1.2221
Best-so-far result across 10 recorded experiments. Lower text prediction loss is better. Original research record
Most of the gain came from the matrix learning rate
The progress graph should show a small initial improvement followed by a much larger drop. Raising Muon momentum from 0.95 to 0.98 moved the loss only slightly, from 1.249653 to 1.249167. That change was beneficial, but it accounted for little of the final gain.
Increasing the learning rate used for Muon’s matrix parameters mattered far more:
At 0.05, the loss fell to 1.232511.
At 0.06, it improved again to 1.231158.
At 0.07, it reached the best result, 1.222123.
In practical terms, the original matrix updates were too conservative for this fixed five-minute training budget. Larger steps let the model make more useful progress before time expired.
The useful boundary was between 0.07 and 0.08
The improvement did not continue at 0.08. Two separate runs produced 1.223939 and 1.222565, both worse than the 0.07 result. The second miss was narrow, but the strict-improvement rule correctly kept 0.07.
This makes 0.07 the best tested value, not necessarily the exact optimum. A run at 0.075 was started but stopped before producing a measurement, so the region between 0.07 and 0.08 remains unresolved. If experimentation resumes, completing 0.075—and then testing smaller steps around whichever side wins—would be the most focused next move.
What did not help
Changing the optimizer’s second-moment decay value from 0.95 to 0.98 yielded 1.249196. That was slightly worse than the current best of 1.249167 at that stage, so it provided no evidence for keeping the change.
The repeated 0.08 failures also provide a useful lesson: the earlier learning-rate trend could not simply be extrapolated upward. Once the update size crossed the useful range, performance stopped improving even though the runs remained numerically stable enough to finish.
Recommended configuration and remaining unknowns
For the tested RTX 3090 five-minute benchmark, retain the 0.98 Muon momentum target and 0.07 matrix learning rate. This is the strongest completed optimizer configuration in the evidence.
The search did not produce measured conclusions about weight decay, epsilon, parameter grouping, orthogonalization settings, or safe update scaling. It also did not finish the 0.075 learning-rate test. Those areas remain unknown rather than demonstrated to be ineffective. Finally, because the recorded metric is called prediction loss rather than validation bits per byte, the evaluator’s metric definition must be checked before reporting the result as a val_bpb improvement.