Bilevel Autoresearch: Meta-Autoresearching Itself

If an autoresearch loop is itself a form of research, point it at itself. An outer loop reads the inner loop's code, finds its bottleneck, and writes new search mechanisms as Python at runtime, with no stronger model at the meta level.

An autoresearch loop runs the propose, execute, evaluate, keep-or-discard cycle that automates hyperparameter search. Every such system in the literature, from Karpathy's single-track loop to multi-batch and persistent-memory extensions, was improved the same way: a human read the code, found a bottleneck, and wrote new code. Bilevel Autoresearch asks whether an LLM can do that design step itself, and answers yes. An outer loop meta-optimizes the inner loop by generating and injecting new search mechanisms as Python at runtime.

The inner loop optimizes the task; the outer loop optimizes how the inner loop searches. Both run on the same model, so any gain comes from the structure, not from a smarter meta-model. On Karpathy's GPT pretraining benchmark the mechanism-generating outer loop delivers a 5x improvement over the inner loop alone (a val_bpb drop of -0.045 versus -0.009), while merely tuning the existing mechanism's parameters yields no reliable gain.

The problem it attacks

Autoresearch has a fixed ceiling baked in at design time: the search mechanism never changes. Karpathy's loop uses a single track with a keep/discard rule; later systems added parallel batches or cross-run memory; but in every case a human authored the improvement. The system itself cannot read its own search code, diagnose where it is stuck, and rewrite the mechanism. That is the operation Bilevel Autoresearch automates, and the distinction it draws is between adjusting the parameters of a search mechanism (which prior outer loops already do) and replacing the mechanism with newly generated code (which is new).

How it works

The framework has three nested levels, all running on the same DeepSeek model. Level 1 is the standard inner loop: propose a hyperparameter change, train a mini-run, evaluate validation bits-per-byte, keep it if it beats the current best, discard otherwise. Level 1.5 runs every 5 inner iterations and adjusts search parameters: it can freeze hyperparameters that have been tried repeatedly with no gain, or inject guidance to redirect diversity, but it cannot change the proposal logic, the acceptance rule, or the loop structure. Level 2 runs every 2 outer cycles and does the categorically different thing: it researches and writes a brand-new search mechanism as Python and injects it into the running inner loop.

The Level 2 step is a four-round structured dialogue with the same LLM: it explores relevant algorithmic domains (combinatorial optimization, multi-armed bandits, design of experiments, Bayesian optimization), critiques candidates, specifies one, and generates the code, which is then import-validated and activated in the inner loop.

Method details

The evaluation is a controlled four-group ablation, with the LLM, GPU hardware, training budget (300 seconds), search budget (30 iterations), and baseline train.py all held constant, and the baseline script restored between repeats. Each group runs three independent repeats. The metric is the change in validation bits-per-byte from baseline, where more negative is better. Group A is Level 1 only, Group B adds Level 1.5, Group C is the full three-level stack, and Group D is Level 1 plus Level 2 but without the parameter-adjusting Level 1.5.

Results

Group C's mean improvement is 5x Group A's and 7.5x Group B's by absolute size. Group D, which keeps mechanism research but drops parameter adjustment, lands at -0.034, close to and only slightly below Group C, which the authors read as confirmation that Level 2 is the primary driver and Level 1.5 is not the source of the gain. The variance is high and honestly reported: Group C's three repeats were -0.065, -0.011, and -0.058, so the mechanism-research win is real but not yet consistent run to run.

Why the generated mechanisms help

The explanation is the most interesting part. The inner loop, driven by the LLM's priors, searches in deterministic patterns and systematically avoids certain directions. The generated mechanisms (Tabu Search, a bandit allocator, orthogonal exploration) succeed precisely by breaking those patterns and forcing exploration the model would otherwise skip. In the logged runs, Level 2 mechanisms steer the search toward reducing total batch size, a direction the bare inner loop kept missing. The outer loop is not smarter; it injects a structured way to escape the inner loop's ruts.

What it changes

This is recursion applied to the search procedure itself. Where SIA's Feedback-Agent picks between editing a scaffold and editing weights, and ALMA searches over memory designs, Bilevel Autoresearch has the loop rewrite its own search operator at runtime, using the same model that runs the search. The conceptual claim is general: if autoresearch can meta-autoresearch itself, it can in principle meta-autoresearch anything with a measurable objective. The demonstrated version is narrow (one pretraining benchmark, small budgets), but the mechanism, an LLM reading and replacing its own search code, is the recursive step the field keeps gesturing at.

Where it sits among prior work

Limitations

The evidence is one benchmark (Karpathy's GPT pretraining task), three repeats per group, and small budgets (300-second training, 30-iteration search), so the 5x figure rests on a handful of runs with large variance; two of three Group C repeats drove the mean while the third barely moved. The mechanisms are drawn from well-known optimization families, so the outer loop is recombining known algorithms rather than inventing genuinely new ones. There is no held-out generalization test, and the measurable-objective requirement means the approach inherits the usual limit: it works where the objective is crisp, which a pretraining loss is.

Learnings

1. Structural edits beat parametric ones. Tuning the knobs of a fixed search mechanism barely helped; replacing the mechanism with new code produced the 5x gain. This mirrors SIA's harness-vs-weights split: the bigger lever is changing the procedure, not its parameters.
2. The same model can improve its own search. No stronger meta-model was needed; the gain came from giving the loop a way to read and rewrite its own search operator. Capability was latent and unlocked by structure.
3. Generated mechanisms work by breaking the model's own ruts. The value of Tabu Search or a bandit allocator here is forcing exploration the LLM's priors avoid, a concrete reason injected diversity helps a self-directed loop.
4. Honest variance is part of the result. The high run-to-run spread is reported plainly, which is the right posture for an early recursive-improvement result: the mechanism is real, the reliability is not there yet.