Autoresearch · 18 August 2026 · 2 min · Original language: English
Qwen3.8 04 · MLP and Activations
The best result was 1.184380 validation bits per byte, down from 1.252284. That is an absolute improvement of 0.067904, or about 5.4%. The winning choice was a compact SiLU-gated MLP; simple activation swaps and larger hidden layers did not help.
Measured improvement5.4% better
Started
1.2523
Best
1.1844
Best-so-far result across 10 recorded experiments. Lower text prediction loss is better. Original research record
What produced the improvement
The progress graph should show two meaningful steps.
First, replacing the original squared-ReLU MLP with a gated design improved the score from 1.252284 to 1.228870. The layer split its hidden representation into a value branch and a gate branch, applied SiLU to the gate, and multiplied the two. In practical terms, this let the MLP control which features passed through instead of applying one activation uniformly to every feature.
The larger gain came from making that gated MLP narrower. Reducing its hidden-width scaling from 8/3 to 2/3 preserved the gating mechanism while cutting parameters and computation. Under a fixed five-minute budget, the resulting score fell to 1.184380. This suggests that efficiency mattered more than raw MLP capacity: the compact model could use the limited training time more effectively.
What did not work
Changing only the original activation was insufficient. SiLU scored 1.260625 and GELU scored 1.256366, both worse than the 1.252284 baseline. The benefit therefore came from the multiplicative gated structure, not merely from choosing a smoother activation.
Increasing the conventional MLP expansion ratio from 4 to 6 was substantially worse at 1.289569. Increasing the compact gated MLP’s hidden-width scaling from 2/3 to 1.0 also regressed to 1.191059. Across both architectures, adding capacity hurt performance within the fixed runtime.
GELU gating was close but still inferior to SiLU gating. One completed GeGLU evaluation scored 1.191822. Another GELU substitution on the winning gated configuration scored 1.185339—only 0.000959 behind the best, but not a strict improvement. A further related run was stopped before producing a result.
Recommended configuration
Keep the compact gated MLP with SiLU gating and the 2/3 hidden-width scaling. It delivered the lowest measured validation loss, required no new dependency, and was simpler to justify than spending more compute on wider layers.
The broader lesson is to preserve the gating mechanism and favor throughput over extra MLP width on this five-minute RTX 3090 benchmark. Activation-only substitutions and capacity increases were repeatedly unproductive, so further work should explore small changes around the compact gated design rather than returning to a plain MLP or widening it.