In a MirrorCode benchmark run in the summer of 2026, the then-flagship Claude model (Opus 4.7) rewrote a sixteen-thousand-line program from scratch in 14 hours, spending $251 on inference. Epoch AI and METR estimated the same job at two to seventeen weeks for human engineers working without AI.
The number has already made the rounds as proof that agents are ready for real work. The benchmark holds a detail that matters more than the record: the condition that made the record possible in the first place. That condition, not model size, decides whether automation works in your company.
What MirrorCode is, and why it isn't another coding benchmark
Epoch AI and METR built MirrorCode to test AI on long tasks rather than on patching individual bugs. The setup is strict: the model gets a working program, but only through its command line. No source code, no internet, nowhere to peek. It has to figure out what the program does and write its own implementation that behaves identically.
The result is checked with end-to-end tests the agent has never seen, and the output has to match exactly. You can't fake the work with a lookup table of answers, which is why the tests stay hidden.
The twenty-five target programs cover different corners of computer science: unix utilities, serialization and data queries, bioinformatics, interpreters, static analysis, cryptography, compression. Three examples for scale:
Program | What it does | Size |
|---|---|---|
pkl | a programmable configuration language, built by Apple | 61,000 lines |
qsv_select | selecting and reordering CSV columns | 87,000 lines |
gotree | working with phylogenetic trees, 40+ commands | 16,000 lines |
The main difference from familiar benchmarks is money. Most tests cap spending on a single run at $1–10, even when the task would cost a human weeks. Here the budget was set so the attempt could be serious. On the leaderboard each attempt gets up to 10 billion tokens and up to 7 days; the runs in the paper followed a different rule — the limit was on tokens, with no time ceiling at all. That's where the 19 days of continuous work below comes from.
The numbers worth keeping in view
Below is what Epoch AI published as the result, together with what usually gets lost in the retellings: how many targets the models didn't take.
What the MirrorCode benchmark measured | Result |
|---|---|
gotree, Claude Opus 4.7 | 14 hours, $251 |
Estimate of the same work for an engineer without AI | 2–17 weeks |
gotree in different languages, Opus 4.7 and GPT-5.5 | $100–400 per run |
Largest task in the set (paper run, no time limit) | $2,600, 19 days of work with no human intervention |
Targets with at least one flawless run | 17 of 25 |
Targets never taken to 100% | 8 of 25 |
Of those eight: reached 99% but not 100% | 4 |
Of those eight: didn't even reach 99% | 4 |
Leading models a year earlier | around 30% on the benchmark's metric, and only simple programs like a calendar |
In a year the frontier moved from a calendar utility to Apple's configuration language, and that pace means any verdict of "AI can't do that" has a shelf life of about a year.
Where the AI failed
Eight programs out of twenty-five were never taken to a hundred percent, and four never even to ninety-nine. The models did worst on ruff, a Python linter and code formatter. The same list includes giac_subset, a math package, and mailauth, an email authentication library.
The benchmark's authors state the failures but don't explain them, so the next paragraph is our reading, not a conclusion from Epoch AI.
The project that failed wasn't the biggest one: the models took the enormous qsv_select and not the linter. A linter is agreements made concrete — hundreds of small decisions about what counts as correct style, settled over years and recorded in behavior rather than in a spec. Ask two Python developers how many blank lines go before a nested function and you'll get an argument that appears in no specification. Reconstructing that kind of system from behavior is harder than reconstructing a large but consistent one. The same goes for cryptographically sensitive logic in mailauth, where "almost right" equals "wrong".
The same line runs through a business: AI can carry the volume, but not the unwritten agreements a company has about what counts as correct.
The condition that makes long autonomy possible at all
MirrorCode has something almost no business process has: an uncontested machine definition of done. Here "done" means the output matched the reference on tests the agent never saw. Not "close", not "broadly fine", and not "the manager looked at it and nodded".
That's why the agent holds out for 14 hours, and on the paper's largest task for 19 days straight with no human. The model can make as many mistakes inside a run as it likes, and each one costs tokens: a machine catches it, instantly and for free.
How long an agent can work on its own is set by how cheaply the result can be checked, not by how strong the model is. No cheap check, no long autonomy, whichever model you put in.
We covered the same relationship from the other side in the piece on why the same agent gives different companies different results.
An honest caveat from the authors belongs here too. The target programs are real open source code, so the models almost certainly saw them during training. Epoch AI ran a memorization check and notes that the AI successfully rewrote several programs that passed it, while failing on some of the programs that showed signs of memorization. So the result doesn't reduce to recall, though the authors won't rule out that recall contributed. Treat the numbers as an upper estimate, not a guarantee.
The cost-of-checking test: carrying this over to an SMB
Before asking "can an agent handle this process", ask three questions about the process itself.
- Is there a reference? Can you describe correctly finished work in one sentence, so that two of your employees read the description the same way.
- What does one check cost? A minute of a junior's time or an hour of the manager's. That sets how many attempts you can afford at all.
- What does a missed error cost? A corrected record in the database is one thing; an amount already sent to a client is another.
Sort your processes by that logic and the picture usually looks like this:
Checking is cheap and mechanical | Checking is expensive or subjective |
|---|---|
normalizing a product catalog against a given schema | choosing the positioning for a new service |
finding and merging duplicate customers | negotiation tactics in a complex deal |
turning a call recording into structured CRM fields | deciding which of two vendors gets the contract |
reconciling an export against source documents | writing a proposal for one specific client |
checking an inbound request for missing data | judging whether a leaving client is worth keeping |
Processes in the left column can go to an agent right now, with a ROI calculation that holds up. The right column isn't "AI will never touch this" — it's "here the agent works under a human, in short steps, and the saving is counted differently". We laid out the line between AI embedded in a process and full autonomy separately, in when autonomy is unnecessary.
The most common mistake at this step is starting with the process that hurts most. Start with the process that's cheapest to check: it gives you a working measurement of the effect, and with that in hand you can go after the expensive ones.
Is $251 a lot or a little
Comparing $251 with two to seventeen weeks of engineering work looks devastating, and that's the form the comparison usually gets quoted in. Three corrections belong with it.
That's the cost of one successful run. On the leaderboard each task is run three times, and failed attempts burn budget too, so the real cost of the result includes them. Next, this is a lab: the task is specified, the environment is assembled, the done criterion is written before the start. In a company that preparation is the expensive line, not inference. And finally, $2,600 for the largest task is a reminder that long autonomy has a price that grows with the ambition.
The full breakdown of what an agent costs outside a lab is in our piece on what an AI agent actually costs, and the method for counting the return is in how to calculate AI agent ROI.
What to do this week
- Pick one process that repeats every week and is done by hand today.
- Write the done criterion in one sentence, as a check: "the result is correct if it matches X". If you can't phrase it, that's your first task, and it isn't about AI.
- Count the cost of one check in minutes, and in whose time those minutes are.
- Estimate the cost of a missed error. If it's high, keep a human at the end and cut the preparation first, not the control.
- Measure one metric before and after: hours per process, or errors per hundred operations.
If the metric hasn't moved in two weeks, the problem is almost always in step two. A wider view of what actually works in small business right now is in our read on the state of AI agents for SMBs in 2026, and the basic introduction is AI agents for small business.
Sources
- Epoch AI × METR, MirrorCode: What's the largest software project AI can complete on its own? — the benchmark page with results and the authors' caveats.
- Tom Adamczewski, David Owen, David Rein et al., MirrorCode: AI can rebuild entire programs from behavior alone, arXiv 2606.30182, 2026 — the full paper.
- Jack Clark, Import AI 466, 27 July 2026 — a read on the results and the argument that these systems can find their own way around an unfamiliar environment.
- Source code and 22 of the 25 target programs (Epoch Research, GitHub).
Next step. Take the free AI audit in two minutes: it shows which of your processes have a cheap check on the result, and where the data needs sorting out first. Our lineup of front-office agents is on a separate page.
If someone around you is choosing their first process for automation right now, forward them the two-column table above: it saves a month of trying from the wrong end.
Published by Andrew Maryasov, founder of Grow2.ai — AI agents and AI consulting for small and mid-sized business. Grow2.ai is the AI arm of Auspex.