Parameter Golf Auto Research

I’m a software engineer, not an ML researcher. My first Parameter Golf run scored 1.148 bpb, behind the 1.1194 leaderboard result. I entered anyway with a different plan: build an agent loop that filters ideas before they consume expensive H100 time.

The constraint

Parameter Golf asks you to train the best language model you can while fitting the complete artifact, code and compressed weights, inside 16 MB. Training is capped at 10 minutes on 8×H100 SXMs. Evaluation allows no network calls. The metric is bits per byte on FineWeb, and lower is better.

At the time of writing, the best-known score was 1.1194 bpb. It used int6 quantisation, parameter banking, test-time training, and a custom bigram tokenizer. The leaderboard moves quickly, so an idea that is novel today may be old by the time it finishes running.

RunPod and OpenAI gave me $500 to continue testing. That support gave me room to replace brute force with a systems-first loop.

Extend autoresearch

Andrej Karpathy’s autoresearch provides the starting point: give an agent a training script, let it modify the code, run a five-minute experiment, keep improvements, revert failures, and repeat. On one GPU, that is roughly 12 experiments per hour.

I adapted that loop for Parameter Golf as Parameter Golf Auto Research. The system has three processes:

• An experiment agent changes the training code and runs tests.
• A research agent searches papers, repositories, and leaderboard changes for useful techniques.
• A thin supervisor manages both agents, their queues, and the compute lifecycle.

The agents communicate through JSONL files, so research can continue while an experiment is running. The supervisor restarts failed agents, up to five attempts, and controls which experiments can move to expensive verification.

Three constraints shape every run

• Artifact size. A better loss is worthless if the compressed artifact exceeds 16 MB. The agent checks parameter count, quantisation, code size, and entropy bounds before spending compute.
• Compute cost. An 8×H100 run costs roughly $20 per hour. A 10-minute run costs at least $3.33, and startup and synchronisation push the real cost closer to $3.50. MLX on Apple Silicon acts as a free scratchpad; RunPod is the expensive verification step.
• A moving leaderboard. The research agent watches public PRs, papers, repositories, and benchmark results so the experiment agent does not spend three runs rediscovering a published technique.

The loop rejects infeasible ideas before training. A local MLX run gets 500 iterations, and an experiment only moves forward if its score clears a dynamic threshold against the running baseline.

Research before expensive runs

The research pipeline does more than rank papers by abstract quality. It checks whether a technique fits the artifact budget, training window, dependency constraints, and validation rules. A promising paper that needs a new dependency or twenty minutes of training is not useful here.

Every hypothesis passes five deterministic gates:

1. Constraint checks for size, training steps, memory, and quantisation error.
2. Contamination checks for validation-data leakage.
3. A critic check for oversized diffs and repeated failed ideas.
4. A dynamic promotion threshold against the current baseline.
5. A budget check before an H100 run.

After each experiment, a reflection step updates strategy.md and technique_map.json. The agent gets a record of what worked, what failed, and which parts of the search space are still open.

This is the part that feels familiar. It is a feedback loop with a defined metric, a fixed budget, and hard failure conditions. That is a systems problem as much as an ML problem.

What would validate the approach

The central uncertainty is transfer. An idea that improves on MLX may not survive a multi-GPU H100 run. Each H100 diagnostic costs about $3.50, so the loop must reject bad ideas before they reach that stage.

The approach is worthwhile only if it:

• Uses local gates to predict which hypotheses survive H100 verification.
• Prevents the experiment agent from repeating published or failed work.
• Lowers the cost per validated improvement.

The live dashboard reports the score, artifact headroom, budget, recent experiments, and research findings. The results need enough runs before they support a conclusion.