Qwen3.8 17 · Hybrid Convolution Token Mixing

What actually improved the benchmark

The first substantial gain came from replacing the standard feed-forward block with a compute-matched gated SwiGLU block. Its separate gate and value projections improved prediction loss to 1.217373 while using a reduced hidden width to keep the computational cost comparable.

Two subsequent width increases produced strict improvements, but only very small ones: 1.217291, then 1.217136. This suggests that extra width helped, although the measured benefit had nearly flattened by the second adjustment.

The strongest final change was reducing depth from eight layers to seven. Prediction loss fell from 1.217136 to 1.206223, a much larger improvement than either width adjustment. In practical terms, the five-minute run benefited more from a shallower model that could process more training data than from spending the budget on another transformer layer.

Why additional depth was counterproductive

Increasing depth moved the result in the wrong direction. Nine layers produced 1.243839, and ten layers deteriorated further to 1.380954. Together with the seven-layer win, these results indicate that this benchmark was constrained more by training throughput than by model capacity: deeper models consumed the fixed compute budget without receiving enough training to compensate.

For this exact benchmark, the useful direction is therefore a moderately wider, shallower network—not a deeper one.

What the convolution attempts established

Three local-convolution variants were attempted: a residual convolution branch, a depthwise version, and a gated depthwise version. All three failed during execution before producing valid evaluation scores.

That means the research did not determine whether local convolution improves prediction loss. These failures should be treated as an implementation or compilation lesson, not as evidence against convolutional token mixing. A future run would need to make one minimal convolution branch execute successfully before comparing its quality.

Recommended configuration and next experiment

Keep the accepted combination: the gated SwiGLU feed-forward block, the wider model setting reached in the final width experiment, and seven transformer layers. Its measured prediction loss of 1.206223 is the best result among the ten completed experiment records.

The next useful experiment would isolate a single, compilation-safe local depthwise convolution added to this best configuration. Until that produces a valid score, the only supported conclusion about token mixing is that gated SwiGLU helped; the value of an explicit convolutional branch remains unknown.