Five-Minute RTX 3090 Training Optimization

What was actually measured

The accepted baseline used the RTX 3090’s 24 GB profile with a device batch size of 32. It completed under the fixed evaluation contract and produced the sole reliable result: 1.309193 bits per byte. Lower is better, so this is the score future experiments must beat strictly before a change can be retained.

This result is a baseline, not evidence that the current training configuration is optimal. There are no comparative measurements from altered optimizer schedules, attention layouts, model widths or depths, gradient accumulation settings, or learning-rate dynamics.

Why the optimization did not progress

Two early baseline attempts failed for operational reasons. One was interrupted when the backend stopped. Another could not run because the pinned Flash Attention kernel was unavailable in the Windows environment; the recorded remedy was to use the configured WSL GPU runtime. A later attempt succeeded and established the baseline, showing that the evaluation itself could run once the environment issue was addressed.

The next stage then stalled before GPU testing. Twenty candidate-generation attempts, followed by five more after resuming, proposed no actual modification to train.py. Because the research required each candidate to change that file—and only that file—none qualified as an experiment. These repeated failures provide no information about model quality and should be treated as one automation failure rather than 25 negative training results.

Practical conclusion

There are no winning changes to adopt. Keep the measured baseline configuration as the reference, but do not infer that batch size 32 or any existing architecture and optimizer choices are best.

If the work is restarted, the immediate priority is to repair or replace the candidate-generation step so that it produces a real, reproducible one-variable edit to train.py. Run through the working WSL GPU environment, verify that the unchanged baseline remains near 1.309193, and then evaluate the requested hypothesis families one at a time. Only a completed score below 1.309193 would count as progress.

The relative merits of optimizer scheduling, attention layout, width versus depth, batch accumulation, and learning-rate changes remain entirely unknown because none of them was tested.