Autoresearch · 18 August 2026 · 2 min · Original language: English
Qwen3.8 03 · Attention Windows
No attention-window change improved the benchmark. The original schedule—three short-window layers followed by one full-context layer (WINDOW_PATTERN = "SSSL")—remained best at 1.267743 validation bits per byte.
The progress graph reflects the central result: every completed alternative finished above the baseline, so none qualified to be kept under the strict-improvement rule.
Measured improvement0.0% better
Started
1.2677
Best
1.2677
Best-so-far result across 10 recorded experiments. Lower text prediction loss is better. Original research record
What the experiments showed
Increasing full-context attention from one layer in every four to two raised the score to 1.284244. Because lower is better, the extra global attention hurt performance within the fixed five-minute budget.
Several attempts introduced a medium window covering three quarters of the context. Using it in place of either a short window or the final full-context window produced scores from 1.281434 to 1.284757. The exact placement did not rescue the idea: all measured medium-window schedules were clearly worse than the baseline.
The closest result came from changing every short window from half of the context to three quarters. That scored 1.268969, only 0.001226 above the baseline, but it was still a regression and therefore correctly rejected. This suggests that the baseline’s short windows were already near the useful balance for this particular time-limited run.
Why more context did not help
Larger attention windows let layers inspect more preceding tokens, but they also make each training step more expensive. On a benchmark capped by wall-clock time, any quality gain per step must compensate for the reduced amount of training completed. These runs provide no evidence that it did: every measured move toward broader attention produced a worse final score.
That conclusion is specific to this RTX 3090 five-minute benchmark. The evidence does not include step counts, tokens processed, or direct throughput measurements, so it cannot determine how much of each regression came from lower throughput versus a less effective attention arrangement. It also does not establish whether larger windows would help with a longer training budget.
Practical recommendation
Keep the existing repeating schedule of three half-context layers and one full-context layer. Do not adopt the tested three-quarter-context window or increase the share of full-context layers for this benchmark.
If attention-window work continues, the most promising direction is a finer search close to the baseline rather than another large increase in context. The three-quarter short window came closest, but no intermediate ratios between one half and three quarters were measured, and no alternative placement achieved an improvement.
One attempted medium-window configuration failed before evaluation because the new window type was not accepted by the configuration validation, and another enlarged-short-window run was stopped. Those incomplete runs add no quality evidence. Overall, the search ended with zero strict improvements, leaving 1.267743 as both the starting and best result.