autoresearch: An AI Agent Running LLM Training Experiments Overnight

Give an agent one editable training file, a five-minute budget, and a single metric. It mutates the code, trains, keeps the change if the number improved, and repeats while you sleep. The whole loop fits in three files.

autoresearch is the most stripped-down working instance of recursive self-improvement you can run yourself. The idea is plain: give an AI agent a small but real LLM training setup and let it experiment autonomously overnight. It modifies the code, trains for five minutes, checks whether the result improved, keeps or discards the change, and repeats. You wake up to a log of experiments and, hopefully, a better model. It is the project that turned the propose-execute-evaluate loop from a thought experiment into something that runs on one GPU for the price of a night's electricity.

The clever framing is where the human sits. You are not editing the Python a researcher would normally touch. Instead you program a program.md Markdown file that gives context to the agents and sets up your autonomous research org, while the agent edits the training code. The human iterates the instructions; the agent iterates the model.

The problem it attacks

Frontier ML research is rate-limited by humans who design experiments, run them, read the results, and decide what to try next. Most of that loop is mechanical: change one thing, train, compare to the best so far, keep or revert. autoresearch asks whether an agent can run that mechanical loop unattended on a real (if small) training task, and it answers yes, with a setup small enough that the whole thing stays reviewable. The constraint that makes it work is a fixed time budget, which turns a messy open-ended search into a stream of directly comparable experiments.

How it works

The repository is deliberately tiny: three files matter. prepare.py holds fixed constants, one-time data prep (it downloads training data and trains a BPE tokenizer), and runtime utilities like the dataloader and evaluation; the agent does not touch it. train.py is the single file the agent edits, containing the full GPT model, the optimizer, and the training loop; architecture, hyperparameters, optimizer, and batch size are all fair game. program.md is the baseline instructions for one agent, the file the human iterates on. You point a coding agent (Claude Code, Codex, or similar) at the repo with permissions disabled and prompt it to read program.md and start experimenting.

By design, training runs for a fixed five-minute wall-clock budget regardless of your hardware, excluding startup and compilation. That choice does two things. It makes every experiment directly comparable even when the agent changes model size, batch size, or architecture, because each run gets the same time rather than the same step count. And it means autoresearch finds the most capable model your platform can produce in five minutes, rather than chasing an absolute that depends on someone else's GPU. The metric, val_bpb, is vocab-size-independent so architectural changes are scored fairly.

What it produces

The headline number is throughput, not a benchmark score. The five-minute budget yields roughly 12 experiments per hour, so a single GPU runs about 100 experiments overnight while you sleep. autoresearch is self-contained: no distributed training, no complex configs, no external dependencies beyond PyTorch and a few small packages. One GPU, one editable file, one metric.

The design choices are the lesson. A single file to modify keeps the scope manageable and the diffs reviewable. The fixed budget keeps experiments comparable and squeezes the best model out of whatever hardware you have. Self-containment keeps the whole thing on one GPU with one metric. None of this is novel machine learning; it is the disciplined minimal form of an autonomous research loop, which is exactly why it is worth studying.

What it changes

autoresearch reframes where the human adds value in an automated research loop. The model code becomes the agent's mutable artifact; the human's lever moves up to the instructions, the program.md that defines how the research org behaves. That is the same shift the harness-evolution papers make at a different layer: stop hand-tuning the artifact, start shaping the process that tunes it. It is also the cleanest demonstration of why the fixed-budget judge matters. Because every experiment gets the same five minutes and the same vocab-independent metric, the keep-or-revert gate is trustworthy, and a trustworthy gate is what separates a loop that climbs from one that wanders.

Where it sits among prior work

autoresearch is the reference point several systems in this collection build on or cite as AutoResearcher. Bilevel Autoresearch takes this exact inner loop and adds an outer loop that rewrites the search mechanism itself. Industrial efforts like Recursive's automated AI research system and Poetiq's meta-system chase the same propose-evaluate-keep pattern at company scale. What autoresearch contributes is the canonical minimal form: the smallest honest thing that is still recursive self-improvement on a real training task.

Limitations

This is a working demonstration and a teaching substrate, not a benchmarked research result. The task is a small single-GPU language model, so it shows the loop runs and improves, not how far autonomous research scales. The fixed-budget design makes runs comparable within one platform but not across platforms, since results depend on your specific GPU. There is no held-out generalization claim, no variance analysis, and the judge is a single metric, which inherits the usual risk that an agent optimizes the number rather than the underlying capability if the metric has a crack. And the quality of the loop still depends on the human-authored program.md, so the autonomy is real but bounded by the instructions it starts from.

Learnings

1. The minimal honest form is three files. An editable artifact, a fixed harness, and a human-authored instruction file are enough to run recursive self-improvement on a real task. Everything fancier is an addition to this skeleton.
2. A fixed budget is what makes the gate trustworthy. Same time per run plus a vocab-independent metric means every keep-or-revert decision compares like with like, which is the whole reason the loop climbs instead of drifting.
3. Move the human up a level. The person stops editing the model and starts editing the process. That is the same lever the harness and skill papers pull, here reduced to a single Markdown file.
4. Throughput is the resource. ~100 overnight experiments on one GPU is the practical unlock: cheap, parallel-in-time iteration is what turns a loop into progress.

Glossary