The production issue. Agents stopped demoing and started shipping work — coding the repo, running the tests — and ran straight into the reliability tax nobody had priced in.
| 01 | From the editor's desk Issue two. Capability was never the question. Reliability is. | 03 |
| 02 | Into production Coding agents leave the chat window and start editing the repo, running the tests, opening the PR. | 04 |
| 03 | The reliability tax The metric that separates a clever agent from a dependable one — and why most fail it. | 06 |
| 04 | Required Reading Four papers on shipping and measuring agents, each with a citation, a link, and the line that matters. | 07 |
| 05 | Case study — Copilot's 55.8% A randomized trial, not a testimonial. What measured productivity actually looks like. | 08 |
| 06 | The Ticker Four real signals from agents in production, sourced and dated. | 09 |
| 07 | Notes from the floor RDLB Agentic in practice — evals before prod, reliability as a gate, judgment kept human. | 10 |
| 08 | Sources & colophon Every claim, linked. Plus the type, the tools, and next month. | 11 |
Last month's question was "can it act?" This month's is harder and far more expensive: "can it act dependably, on the hundredth try, unattended?" The agentic economy does not turn on a model's best day. It turns on its worst one.
Production is where the romance ends and the engineering begins. A coding agent that resolves a GitHub issue in a demo is a marvel. A coding agent you can point at a backlog overnight is a business. The distance between those two sentences is this entire issue.
We walk through the systems that crossed it — agents with a real interface to a computer, editing files and running tests — and then we do the unglamorous thing and measure them. The headline result of the year is not a capability score. It is a reliability score, and it is sobering: even strong agents that succeed once often cannot succeed consistently.
The enemy this month is vibes-based autonomy — the decision to let an agent run unattended because a demo felt impressive, without a number for how often it actually finishes the job correctly. Feelings are not a deployment criterion. Pass rates are.
The encouraging half: when you do measure honestly, the value is real and large. A randomized trial — not a testimonial — is our case study. Read it, then go ask your vendors for their pass rates.
— RDLB Agentic, editorial. New York. rdlbagentic.com
The clearest proving ground for working agents is software itself, because the work is verifiable: did the test pass or not? That is why the most important production benchmark of the era is about resolving real bugs in real codebases.
"SWE-bench" (Jimenez et al., 2023) set the bar: take real GitHub issues from popular Python projects and ask a model to produce a patch that actually resolves them, across multiple files.1 When it launched, the verdict was humbling — existing models could handle only the simplest tasks. The benchmark made "can it code?" precise: not "write a function," but "fix this repository."
The leap came from giving the model a real way to work. "SWE-agent" (Yang et al., 2024) introduced an agent-computer interface — a purpose-built set of commands to navigate a repo, edit files, and run tests — and showed that interface design, not just model size, drives results.2 With it, an agent reached a pass@1 of 12.5% on SWE-bench, far above the non-interactive baselines of the time. A small number, but a structural one: the agent was now operating the codebase, not describing it.
This is the production pattern in miniature. Localize the problem, edit, run the tests, read the failure, try again — the perceive-reason-act-reflect loop from last issue, pointed at a build system. When the environment can tell the agent whether it succeeded, the loop closes and real work gets done.
The catch is that "12.5% of the time" is also a sentence about the other 87.5%. Which is where reliability — the subject of the next page — stops being academic.
Read the issue, search the repository, narrow thousands of lines to the few that matter. Most failures start here — fix the wrong place and nothing downstream helps.1
Apply a patch through a real interface — open the file, edit the lines, save. SWE-agent's insight: the quality of that interface is itself a major lever on success.2
Run the suite. The test result is the ground truth that closes the loop — the rare case where an agent gets unambiguous, automatic feedback on whether it actually worked.
Parse the stack trace, revise, run again — until green or until the budget is spent. The discipline is knowing when to stop and escalate to a human.
The reason coding agents work better than most is that the environment grades them automatically. Your marketing work rarely does. So the production question for any non-code agent is: what is your test suite? If you cannot define an automatic check for "did it do the job right," you cannot safely let it run unattended — you can only let it draft.2
Capability and reliability are different axes, and the gap between them is the single most under-priced cost in agentic AI. A 2024 benchmark from Sierra named it, and the number is the one to remember.
"τ-bench" (Yao et al., 2024) does something most benchmarks don't: it talks back. It simulates a real user conversing with a tool-using agent under domain rules — a retail or airline help desk — and checks the actual end state of the database, not just the agent's confident summary.3
Then it does the thing that matters for production: it runs each task many times. Its pass^k metric measures whether an agent can succeed not once but consistently. The finding is the headline of this issue — even state-of-the-art agents solved fewer than half the tasks on a single try, and when asked to repeat the same task eight times, consistency fell to roughly 25%.3 The capability was there. The dependability was not.
This is the reliability tax. An agent that works 80% of the time is not "mostly done" — at scale it is a steady stream of wrong actions you must catch, reverse, and apologize for. The metric is now serious enough that frontier model cards report pass^k directly.3,5
From benchmark to interface to reliability to measured value. Each freely available and verified live for this issue.
Why it mattersRedefined "can it code" as "can it fix this repository." The benchmark that turned coding agents from a demo into a measurable discipline.
Why it mattersThe interface is the product. A well-designed agent-computer interface, not just a bigger model, is what let agents actually operate a codebase.
Why it mattersIntroduced pass^k and proved the reliability tax: strong agents that succeed once routinely fail to succeed consistently. The number every buyer should demand.
Why it mattersA randomized controlled trial, not a testimonial: the treatment group finished 55.8% faster. This is what measured value looks like.
Almost every productivity claim in AI is a vibe with a logo. This one is a randomized controlled trial — which is exactly why it is worth more than a hundred case-study quotes.
In 2023, researchers at Microsoft and GitHub ran a controlled experiment: developers were asked to build an HTTP server in JavaScript, randomly assigned to use GitHub Copilot or not.4 The group with the AI pair programmer finished 55.8% faster. The result was replicated across field experiments at Microsoft, Accenture, and a large anonymized firm.
Two details make it credible rather than hype. First, randomization: the only systematic difference between groups was the tool, so the speed-up is attributable to it, not to motivated volunteers. Second, the heterogeneity: less-experienced developers adopted it more and gained the most — a clue that these systems compress the distance between junior and senior on well-scoped work.
But read it next to τ-bench and the lesson sharpens. Copilot's win is a throughput result on a bounded task with a human in the loop at every keystroke. It is not a claim that an unattended agent finishes the job correctly 56% more often. Throughput scales with assistance; reliability is what you need before you remove the human.
Measured properly, the value is real and large. Measured loosely, it becomes the excuse for vibes-based autonomy. The difference is whether there was a control group.
Demand two numbers, not one. Throughput (how much faster, with a human assisting) and reliability (pass^k: how often it finishes correctly, unattended). Copilot is a throughput story. Autonomy is a reliability story. Do not pay for one and deploy the other.
Dated, sourced, and free of the word "10x."
SWE-bench raises the bar. Resolving real GitHub issues becomes the standard test for coding agents — and the early scores make clear how far there is to go.1
The interface becomes the product. SWE-agent shows a purpose-built agent-computer interface lifts performance sharply — design, not just scale, moves the number.2
Reliability gets a number. Sierra's τ-bench introduces pass^k and the uncomfortable finding: consistency, not capability, is the bottleneck. Frontier model cards begin reporting it.3,5
Productivity, measured. The Copilot RCT puts a defensible number — 55.8% faster — on assisted developer work, and finds the largest gains among less-experienced developers.4
From shipping to speaking: how agents coordinate and communicate across systems — the protocol wars, and why the brand that picks well stops paying the integration tax. That is Issue Nº 03.
A production magazine should run on a production system. Here is how this issue's research disciplines our own build.
Every agent has a test suite. We borrow the coding-agent trick: define an automatic check for "did it do the job right" before anything runs unattended. No check, no autonomy — only drafting.2
We gate on pass^k, not pass^1. A capability that works once is a demo. We measure consistency across repeats and set a reliability bar an agent must clear before it touches a client's brand — because the worst day is the one that ships.3
Throughput to the agent, judgment to the human. Like the Copilot result, our agents compress the time-to-draft dramatically. They do not get the final say. A person signs the work, every time.4
What we refuse. Vibes-based autonomy. We will not let a system run unattended because a demo felt good. If we cannot show you the pass rate, we have not earned the right to remove the human — and we tell you so.
Book a 30-minute strategy blueprint call. You leave with a one-page map of your highest-cost work, sorted three ways — what to replace, what to augment, and what to leave alone — against real numbers, not vibes.
dashboardrdlbagency.com/book→