A real run, a narrow result

It does not show that the agent beats a human researcher given the same budget. It does not show that its mixture transfers to another model, training budget, random seed, or unseen evaluation. It is a blog and engineering report, not a controlled paper.

The title overstates the scope slightly. The successful work was mostly mixture tuning, filtering, repetition, and stage ordering inside an already curated pool. Its “generate” lever produced no usable dataset, and imported data was tested in only one completed recipe. This is not evidence that the agent can build a VLM dataset from scratch.

The researchers froze the model, trainer, evaluation, and compute budget. The model had about 2.1B parameters: a SigLIP2 vision encoder, Qwen3-1.7B language model, and a small connector. Each attempt trained all parts together for 16,050 updates on eight H100s, took about 38 hours, and consumed roughly 12 million samples.

The agent controlled five data choices:

1. which samples to keep, using metadata such as answer length and image-text relevance;
2. the mixture proportions;
3. stages and their order during training;
4. manifest size and deliberate repetition; and
5. outside imports or Gemma 4 31B annotation jobs.

The loop was event-driven. A timer checked jobs every 30 minutes. When something finished or failed, it started one fresh agent session for at most two hours. The session reconstructed state from three files: instructions, a prose logbook, and a machine-readable experiment log. At most four GPU jobs ran in parallel.

Humans built the tools, initial pool, baseline, grader, scheduler, and safety boundary. They also repaired the harness when it broke. The autonomous part was choosing and analysing data experiments inside that prepared environment.

That is +4.00 points absolute, or +6.81% relative. The blog calls this a 6.8% mean relative improvement; more exactly, it is the relative increase in the mean of ten normalized benchmark scores. The arithmetic mean of the ten separate relative gains is 7.44%.

“Improved all ten” is true at the benchmark-total level. It hides one reversal: MME perception rose from 1,409.3 to 1,463.5 out of 2,000, while MME cognition fell from 330.0 to 299.3 out of 800. The combined MME total still rose.

There is also broad directional evidence: every one of the 19 completed experiments scored above the baseline on the ten-benchmark mean. The worst was 59.80 and the median was 60.95. That makes pure luck an unlikely explanation for the overall conclusion that the baseline mixture was easy to improve in this setup.

Good enough for the narrow claim

The experiment is unusually expensive for a blog, the mutable surface is clear, the run data are internally consistent, and the best gain is large enough to matter. The article gives exact model settings, sample counts, benchmark splits, recipes, decontamination rules, and derived result files. This is much better than an anecdote.

Not enough for the larger claim

• No repeated seeds. There is one baseline training and generally one training per recipe. The blog itself says checkpoint-to-checkpoint movement is large, sometimes larger than the differences between recipes.
• No held-out final test. The agent read the ten benchmark data and code, chose experiments from those results, and the final winner was selected on the same benchmarks. Decontamination checked images but not overlapping text.
• Winner selection is post-hoc. The report chooses the best of 19 adaptive attempts and a favourable checkpoint. It gives no confidence intervals or correction for this search.
• Two scores, two winners. The authors’ single-checkpoint ten-benchmark mean selects experiment 5. The agent’s eleven-key, three-checkpoint score selects experiment 11. The primary objective was not fixed before the loop; the agent changed its score rule several times.
• The headline uses one checkpoint. Averaging steps 14k–16k, the same experiment still beats baseline, but by about 3.33 absolute points rather than 4.00. That is still useful, just less dramatic.
• No equal-budget human control. “The baseline took 200K+ H100 hours to develop” is asserted, not demonstrated with the comparisons. That work also built the pool and system; it is not a controlled 200K-hour human mixture search against an 11K-hour agent search.

The novelty is the scale and integration of the autonomous loop, not a new mixture algorithm. The agent is doing sequential experimental design in prose with tools. There is no learned search policy, optimizer over mixture weights, or formal treatment of uncertainty.

1. The control loop matches our architecture. Let expensive work sleep. Wake one fresh agent on a concrete event. Keep durable state in files, with facts in structured records and interpretation in a short logbook.
2. Copy the boundary checks. Declare the editable surface, hash frozen files before launch, require a clean code version, and seal each submitted input artifact with a row count and digest.
3. Freeze the primary success metric. This project let the agent revise aggregation until it had two different winners. Exploratory metrics are fine, but the main metric and held-out final test should be fixed before the first expensive run.
4. Audit the evaluator before spending the budget. A small answer-format canary suite would have found the whitespace and punctuation bugs before several 38-hour runs were interpreted through them.
5. Make exploration an explicit portfolio choice. The agent optimised near-term benchmark return and abandoned slow annotation. If the operator values a high-upside long-horizon branch, reserve budget for it rather than hoping a myopic loop volunteers.
6. Separate development and final evaluation. Evaluator access helped the agent debug real bugs, but it also invited benchmark-specific training choices. Give the research loop a development suite and keep a final suite unavailable until selection is over.
7. Parallel jobs still need one experiment ledger. The authors worry that 50 simultaneous hypotheses would confuse the agent. They are right about the failure mode. Parallel workers are useful, but branch ownership, immutable experiment IDs, and one shared decision record become mandatory.

• The 200K+ H100-hour history behind FineVision is not shown, so the claim that this is the strongest human mixture they found cannot be checked from the page.
• The page says it releases the full agent loop, trainer, data pipeline, and tests, but it links no such repository. The public blog Space contains the article and graphics. A separate public tracker Space contains only the dashboard, and its README says the result dataset is private. I could not verify the claimed full release from the two public Spaces.
• The compact JSON behind the blog supports the displayed tables, but the raw per-item predictions, complete run records, random settings, failed-run details, and training data are not available there.
• No result is replicated on another model size or compute budget. The authors themselves acknowledge that the best mixture can depend on model size, sample budget, and mixture ratio.

So this is not thin—it contains real work and unusually useful operational detail—but it is an engineering report with one adaptive experiment campaign. Treat the scientific findings as leads to test, not settled conclusions.

I used the direct hosted page to confirm the rendered article, then read the exact MDX chapters and underlying JSON from the Hugging Face Space repository. This avoids relying on the Hub wrapper or on the article’s own summary box.

1. Results chapter and benchmark values.
2. All 19 experiment summaries and late checkpoint curves.
3. Findings and mistakes chapter and technical appendix.
4. Separate public tracker Space and its data and release notes.