Agents in practice — Juan Reyero

Spend your compute on correctness

Sat, 16 May 2026 00:00:00 +0000

For developers who have fully embraced AI agents, productivity is no longer the bottleneck. Agents produce code faster than humans, and often better. The bottleneck is correctness: whether the code solves the problem, whether the architecture can still hold the feature set you’re growing into, whether the implementation encodes the right premises. The practical shift is that a significant fraction of your agent compute should now go to validation (tests, reviewers, scope checks), rather than to producing more code.

Productivity is no longer the bottleneck

A year ago, the argument for agentic programming was throughput: agents let you ship more code in less time.

That is not controversial anymore. With the current generation of models inside agentic harnesses (Claude Code, Codex, and the like), a single developer supervising one agent can run circles around a small team writing by hand.

For people fully inside this workflow, the question is no longer “how do I produce more?”. It is “how do I know what I produced is right?”.

Three layers where “right” can fail

Each layer has its own remedies.

The product layer: does this match what a user wants?

Part of “right” is still human judgment. An actual human has to look at the running product and say whether it does what they actually need. No amount of agent compute substitutes for that.

(Incidentally, this will probably change fast, as more and more products are targeted at agents.)

The most you can do is make this human’s job easy: ship into a real environment quickly, with real data, so the human can poke at it and react.

The architecture layer: is this the right shape of code?

This is the hardest layer, and the one where current agents are weakest.

A lot of what experienced software engineers do is architectural judgment:

structuring code so it can accommodate the features that will land on it, without knowing in advance what’s coming;
knowing what features not to implement;
recognizing when the feature set has outgrown the original architecture, and the codebase needs to be reshaped before more code can land cleanly.

Agents are not yet good at any of these. They will happily add the feature you asked for, in the place where it fits least, with abstractions that make the next three features harder.

They will implement the configurability you asked for instead of pushing back on whether you should want it. They will keep extending a structure that has outlived its assumptions, because each individual diff still looks reasonable in isolation.

The remedy at this layer is human-driven architectural review, backed by an agent whose explicit job is to check each diff against a source-of-truth document for scope and shape.

The implementation layer: is the code doing what it claims?

The third failure mode is the easiest to miss: code that looks right, has passing tests, and is actually wrong. The premises it encodes are off by a degree. There is a workaround in the middle that the tests happen not to exercise. A helper that “handles edge cases” is actually swallowing errors that should be propagated.

This is the failure mode the writer agent cannot see, because it inherited the premises from its own earlier reasoning. It needs a separate pair of eyes (a reviewer agent, a fresh-context subagent, or a different agent kind entirely) to read the diff without the writer’s assumptions.

Validation is what you spend compute on

The remedy is simple to state and harder to commit to: spend a significant fraction of your agent budget on validation.

The point of running teams of coding agents is no longer to produce more code faster. It is to make it more likely that the code is correct.

Validation, in practice, means several things at once:

Tests, including the expensive ones. Real end-to-end tests against a real database. Browser tests with Playwright. Runs against representative data.
Constant code review, not just at PR time. A reviewer agent that reads each diff as it lands, with the architectural document and the task list in its context.
Scope rechecks against a source-of-truth document. Has the diff drifted from the original ask? Has the agent quietly expanded the scope? Has it implemented something the SOT explicitly said not to do?

The mistake is to treat validation as a tax on productivity. It isn’t. The output of one coding agent whose work has been validated is worth meaningfully more than the output of two coding agents whose work has not.

Three patterns to apply

There are several ways to deploy validation compute. They are not exclusive.

A code-reviewer subagent inside the same agent. Both Claude Code and Codex ship with strong code-reviewer subagents. The writer spawns one mid-task, gets local feedback, applies it. Catches bugs, missing edge cases, style violations. Cheap and easy first step.
A different agent kind reviewing the writer. Have Claude Code spawn Codex (or vice versa) for review. Different training, different priors, catches a slightly different class of issue. Works, but is qualitatively close to (1).
A persistent, specialized reviewer agent. Two agents per significant task. One programmer, one reviewer; they persist, share a task list, and talk after each TDD cycle. This is the pattern I have found most useful. I wrote separately about it; in this article’s framing, it is the highest-context, most expensive, and most useful form of validation compute.

The shift

The earlier argument for agentic programming was: you can produce more code per hour of attention. That is still true, and still useful.

The argument that matters more for people already inside the workflow is: you can produce more correct code per hour of attention, if you spend the compute on correctness.

The bottleneck has moved. The budget should move with it.

Our team behind aweb uses this pattern daily. aweb is an open-source coordination layer for AI coding agents — identity, task claims, messaging across worktrees and machines. MIT-licensed. Hosted at aweb.ai

Pair your coding agent with a reviewer

Fri, 08 May 2026 00:00:00 +0000

The simplest improvement you can make to your agentic programming workflow is to run two agents instead of one. One writes code in its own worktree; the other, in a parallel worktree, reviews it. They share a task list and talk after every TDD cycle. The reviewer is the holder of the lamp: it keeps the end goal fresh in context across the whole task, catching drift before it accumulates.

I wrote previously that the bottleneck in AI-assisted programming is shifting from individual productivity to coordination. The two-agent pair is the simplest possible coordination, and the most useful next step for anyone who has gotten comfortable running one agent and is ready for more.

One agent works, but it drifts

A single coding agent gets a lot done. It picks up a task, writes the test, writes the implementation, runs code-reviewer subagents, runs the test suite, fixes what’s broken, commits.

However, when a task is longer than its working context can hold cleanly, the agent tends to lose sight of the original brief. It makes locally-correct decisions that drift from the overall goal.

Two agents: writer and reviewer

The pattern is simple: two shells, each with its own worktree of the same repo, each with an aweb identity. Run a coding agent in each.

One agent writes code, the other agent reviews. They share tasks, because they are in the same aweb team. When the coding agent finishes a task, it sends mail to the reviewing agent requesting feedback. The reviewing agent does a thorough code review, and then chats with the coding agent. They discuss the findings, and either move on to the next task or agree to have the coding agent go back and fix the implementation.

The cycle, per task:

The writer picks up the next task.
Writes the test first (TDD: red, then green).
Implements until the test passes.
Asks the reviewer for a review on the diff.
Waits for the review to come back.
Discusses anything unclear with the reviewer.
Applies the changes the review asked for.
Marks the task done; moves to the next.

The reviewer holds the lamp

A coding agent that has been writing for an hour has its working context filled with implementation details: the function it just refactored, the test it just made green, the import it just moved. The original goal — the why of the task — has been crowded out by the how.

The reviewer agent has a different job. It reads the original brief, the task list, the architectural constraints. When the diff arrives, the reviewer asks the question the writer can no longer ask: does this still solve the original problem? Does it move us toward the end state we agreed on?

I think of the reviewer as the holder of the lamp — the one who keeps the destination visible while the writer is heads-down in the trench. Without the lamp, the writer eventually walks in a circle and convinces itself the new spot is the destination.

The reviewer also catches a class of error that is invisible to the writer: implementations that are correct but make the next task harder. The writer is optimizing the current diff. The reviewer is optimizing the trajectory.

On top of subagents

Both Claude Code and Codex have strong code-reviewer subagents built in. The writer agent can spawn a code-reviewer subagent mid-task to catch local issues — bugs, missing edge cases, style violations. These work well and you should keep using them.

A dedicated reviewer agent is a different layer. The code-reviewer subagent runs once inside the writer’s session and forgets when it returns. It sees the diff but not the trajectory. The dedicated reviewer is a separate, persistent session whose context is kept fresh on the end goal across the whole task list.

The two complement cleanly. Subagents catch local issues at the diff level. The dedicated reviewer catches drift at the trajectory level. Use both; they do not compete for context, and they catch different classes of mistake.

Scaling up: a product / engineering owner agent

Once the writer/reviewer pair works for one task list, the next step is adding a third agent: a product / engineering owner. This one is not in the loop on every diff. It scopes work, breaks larger goals into the task list each pair consumes, and reviews the result before it ships.

In our team, this agent is named Athena. She owns engineering direction for the aweb codebase: architecture, invariants, review of every diff that lands on main. She does not write code on most tasks; she scopes, dispatches, reviews. When a substantial feature needs authoring, Athena asks for a writer/reviewer pair as a worktree-scoped task, then reviews the gate-input commit before it goes to ship.

Athena pairs with another agent, Hestia, who handles operations. Hestia runs the gate chain (full end-to-end tests, release-ready checks) then tags releases, deploys, and verifies that the new version is live. Athena hands off the reviewed code to Hestia, and Hestia ships releases. Neither is the other’s manager; they are peers in different roles.

This is the working pattern across our codebase today. Two-agent pairs implement; Athena scopes and reviews; Hestia runs gates and deploys. Even multi-day shipping efforts, like a customer-blocking bug we recently worked through across several back-to-back releases, fit this shape without any extra orchestration layer.

I believe that the lesson generalizes. Most agentic-programming bottlenecks are no longer capability problems. They are coordination problems. The simplest form of coordination is a peer that holds context the working agent has lost. Two agents can do this for each other with no infrastructure beyond the conventions of how they talk.

BeadHub: coordination for AI programming teams

Mon, 16 Feb 2026 00:00:00 +0000

Note added on 2026-05-08: I am moving away from beadhub. Having a tool that messed up with my .git was too painful. I have been focussing all my efforts in https://aweb.ai (OSS at https://github.com/awebai/aweb) which fully replaces beads and beahdub.

I wrote previously that the bottleneck in AI-assisted programming is shifting from individual productivity to coordination. I’ve spent the past several months building a tool to address that.

BeadHub is an open-source coordination server that lets AI programming agents claim work, talk to each other, reserve files, and escalate to humans—across machines and across programmers. I use it daily to manage around fifteen agents working on two or three products.

Beads

Around the time I wrote that article, I started using Steve Yegge’s beads, a git-native issue tracker designed for AI agents. Your agent runs bd create "Fix the login redirect bug" and it appends a JSON line to .beads/issues.jsonl, right in the repository. Issues travel with the code. When you push a branch, the issues come along.

Yegge calls it the “50 First Dates” problem: agents wake up every session with no memory of yesterday’s work. Beads fixes that. An agent reads the issue list and knows where things stand. My agents got much more done.

Which meant more agents, more worktrees, more parallel work—and the coordination problem became even more acute. Two agents modify the same file. One refactors a function while another adds to it. An agent picks up a task already in progress in a different worktree. Nobody knows who’s working on what.

But beads is also the right scaffolding for coordination. If everyone in a team uses beads, all agents share a picture of what needs doing. Beads gives agents something useful to talk about; BeadHub gives them a way to talk.

The coordination gap

The major platforms are moving in this direction. Anthropic just shipped Agent Teams in Claude Code: a lead session that spawns independent teammates who communicate directly and self-coordinate. OpenAI’s Codex app runs parallel agent threads in isolated worktrees.

Yegge built Gas Town on top of beads to tackle the single-machine case: a “Mayor” agent orchestrates dozens of coding agents, tracks work in convoys, and persists state so agents can pick up where they left off.

These are real steps forward, but they’re solving a specific version of the problem: multiple agents for one programmer, on one machine, within one tool.

The version I am interested in is Maria in Buenos Aires running a frontend agent while Juan in San Francisco runs a backend agent, and they need their agents to not destroy each other’s work, and to figure out how to work together.

BeadHub

BeadHub is a server that agents connect to through bdh, a wrapper around the beads bd command. When an agent runs any bdh command it registers with the server. The server tracks which agents are online across the project—what machine they’re on, what branch, what files they’re touching.

Communication. Agents can send each other mail (async, fire-and-forget) or chat (sync, block-until-reply). With mail an agent finishes a task and drops a note: “Done with bd-42, tests passing.” Chat is for when agents need to think together: “I’m adding a role field to the user model—will that break your permission checks?” / “It will, but the fix is small. Go ahead and I’ll update my side.”

Claims. When an agent marks a bead as in-progress, that claim is immediately visible to every other agent in the project, regardless of whose machine they’re on. If another agent tries to claim the same bead, it gets rejected with a message telling it who has it.

File reservations. When an agent modifies a file, the server records an advisory lock. Other agents see a warning if they touch the same file. Advisory, not blocking—hard locks caused deadlocks immediately in early versions. Agent A locks file X, agent B locks file Y, both need the other’s file. Warnings work better. Agents are cooperative; they just need information.

Escalation. An agent runs bdh :escalate with a description of what it’s stuck on and a human gets notified with full context. Without this, agents either fail silently or spin retrying things that need human judgment.

The multi-machine part is where it comes together. BeadHub recognizes Maria’s and Juan’s clones as the same repo. Maria’s agents and Juan’s agents see each other’s claims, locks, and messages. If Maria’s frontend agent reserves src/components/Auth.tsx, Juan’s backend agent sees the warning even though they’re in different cities on different machines.

A project can span multiple repositories. The frontend repo agents can message the backend repo agents. A bead in the frontend can be marked as blocked by a bead in the backend.

You can see what this looks like in practice on the BeadHub project’s own dashboard, where we coordinate BeadHub’s development using BeadHub. Make sure to check the chat page, it is almost magical to see them figuring things out.

What I’ve learned

A few things I got wrong before getting them right.

The client is the source of truth. My instinct was to make the server authoritative. But agents work locally, in git repos, and their local state is the ground truth. The server aggregates and distributes. If the server and the client disagree, the client wins. If the server goes down, bdh falls back to local bd with a warning. Work continues. Coordination catches up later.

Async by default. My first instinct was real-time negotiation between agents. Doesn’t scale. Agents work at different speeds, on different schedules, and blocking one while waiting for another is expensive. Mail is the default. Chat is the exception.

Advisory over mandatory. Advisory file locks that warn instead of block. Bead claims that can be overridden with --:jump-in "reason" (which notifies the other agent). The system provides information and trusts agents to act on it.

The coordinator role. I assign one agent per project the “coordinator” role. The coordinator doesn’t write code. It watches the dashboard, assigns work, checks on progress, nudges stuck agents, and keeps the end goal in sight. The implementer agents are heads-down in their worktrees; the coordinator is the one who knows what the project needs next. BeadHub serves each agent a role-specific policy—markdown documents describing how agents in that role should behave—and the coordinator’s policy is fundamentally different from an implementer’s. This turned out to matter more than any of the technical decisions.

Where this goes

The single-machine problem is getting solved. Agent Teams, Codex—within a few weeks, running multiple agents in parallel on your laptop will be table stakes.

The multi-programmer problem is next. Five engineers, fifty agents, three repositories, two time zones. That’s where the coordination problem changes in kind, not just degree. It’s not enough that your agents can talk to each other. They need to talk to your teammate’s agents, on a different machine, in a different time zone, working on a different repo in the same project.

BeadHub is open source and free for open-source projects.

Data cleaning and anonymizing with GPT-3.5

Sun, 06 Aug 2023 00:00:00 +0000

Note added on 2025-07-28: It’s been two years of programming with and for LLMs. This article sounds so naïve now.

Say you made a website in which customers buy personalized gifts. Each gift comes with a message, written by the customer in whatever language the customer wants. The use of grammar, punctuation and capitalizations in the messages is often creative.

You would like to be able to offer reasonably normative messages to your customers. You would also like to store a fully anonymized version of the messages; replace all proper names with a [proper_name] placeholder, place names with [place_name], dates with [date], times with [time], and geographical coordinates with [coordinates].

Is it possible to use OpenAI’s gpt-3.5-turbo to do this?

Luckily you happen to have some tens of thousands of messages laying around, so you can actually test whatever solution you come up with.

The naïve approach: just ask the model to do it

The first thing I tried was the straight-forward option: just ask the genie to do it for you, with a system prompt and a user prompt like this:

system_prompt = ("You are a proof-reader. You will be given "
 "a text that may be poorly written, perhaps "
 "with bad punctuation and wrong capitalization. "
 "Your task is to first correct the grammar, "
 "punctuation, and capitalization of the text. "
 "Once this is done you should anonymize it by "
 "replacing all the person's proper names in the text "
 "with [proper_name], place names with [place_name], "
 "dates with [date], times with [time], and "
 "geographical coordinates with [coordinates]")

user_prompt = f'Correct and anonymize the following text: {message}'

This doesn’t work with GPT-3.5. It sometimes produces the desired result, but it often misses the anonymization elements, and generally makes a mess of things.

Function calling

Function calling is a wonderful tool that allows you to get structured data from the model. You describe functions and their inputs in your call to the model; if the model decides that it wants to call one of the functions, it will come back with a request to call the function and, crucially, with a dictionary containing the function’s arguments.

And, most interesting for the problem at hand, you can also force the model to call a function. If you do, you will get the structured data you would need to call the function.

So what we do is ask the model to call a function that needs, as arguments, the corrected text and the anonymization elements. Once we get back the dictionary with the arguments for the function we can anonymize by simply searching and replacing the anonymization elements in the corrected text.

This actually works. Interestingly, the GPT-3.5 model is not able to replace the anonymization elements in the text by their placeholder, but it can reliably find them.

Sample code

This is the function I have been using to play with the OpenAI api:

import json
from typing import List, Dict

import openai


# Ensure that the openai key is loaded, with something like
Api_loaded = False

def ensure_api(api_key):
 global Api_loaded
 if not Api_loaded:
 openai.api_key = api_key
 Api_loaded = True


def chat_completion_request(messages: List[Dict],
 api_key: str,
 functions: List[Dict] = None,
 call_function: str = None,
 temperature: float = 1, # 0 to 2
 model_name: str = 'gpt-3.5-turbo-0613',
 attempts: int = 4) -> Dict:
 args = {'model': model_name,
 'messages': messages,
 'temperature': temperature}

 if functions:
 args['functions'] = functions

 if call_function:
 if not functions:
 raise RuntimeError(f'Cannot call function {call_function} '
 'if no functions are defined')

 found_function = len([fdesk for fdesk in functions
 if fdesk['name'] == call_function])
 if not found_function:
 raise RuntimeError(f'Cannot call undefined function {call_function}')
 if found_function > 1:
 raise RuntimeError(f'Found more than one definitions of {call_function}')

 args['function_call'] = {'name': call_function}

 try:
 ensure_api(api_key)
 response = openai.ChatCompletion.create(**args)["choices"][0]["message"]
 function_call = response.get('function_call', None)
 if function_call:
 try:
 return {
 'call_function': function_call['name'],
 'with_arguments': json.loads(
 function_call['arguments']),
 'success': True}
 except json.decoder.JSONDecodeError:
 # The model may have failed to produce correct json
 if attempts:
 return chat_completion_request(
 messages=messages,
 api_key=api_key,
 functions=functions,
 call_function=call_function,
 temperature=temperature/2,
 model_name=model_name,
 attempts=attempts-1)
 return {'success': False}
 return {'reply': response['content'],
 'success': True}
 except openai.InvalidRequestError as exc:
 raise RuntimeError('Invalid request error from OpenAI') from exc

We define a store_correct_text function like this:

functions = [{
 "name": "store_correct_text",
 "description": "Store a corrected version of the input text",
 "parameters": {
 "type": "object",
 "properties": {
 "correct_text": {
 "type": "string",
 "description": "A corrected version the text"
 },
 "person_proper_names": {
 "type": "string",
 "description": ("A list of the proper names of "
 "persons in the text separated "
 "by semicolons")
 },
 "place_names": {
 "type": "string",
 "description": ("A list of the names of places in "
 "the text separated by semicolons")
 },
 "dates": {
 "type": "string",
 "description": ("A list of the dates in the text "
 "separated by semicolons")
 },
 "geographical_coordinates": {
 "type": "string",
 "description": ("A list of the dates in the text "
 "separated by semicolons")
 },
 "times": {
 "type": "string",
 "description": ("A list of the times (hours and minutes) "
 "in the text separated by semicolons")
 }
 },
 "required": ["correct_text",
 "person_proper_names",
 "place_names",
 "dates",
 "geographical_coordinates",
 "times"]
 }
}]

The system and user prompts are:

system_prompt = ("You are a proof-reader. You will be "
 "given a text that can be poorly written, "
 "maybe with bad punctuation and wrong "
 "capitalization. Your task is to first correct "
 "the grammar, punctuation, and capitalization "
 "of the text, and once this is done to find all "
 "the person's proper names in the text. "
 "Person's proper names are those that identify "
 "people, like Mark or Anne Frank. You should also "
 "find all the proper names of places in the text. "
 "You should also find the dates. You should also "
 "find geographical coordinates (latitude and "
 "longitude). You should also find the times "
 "mentioned in the text (hour and minute).")

user_prompt = ('Correct the following text, and make it into '
 f'a template:\n\n{message}')

We can now call chat_completion_request:

result = chat_completion_request(
 messages=[{'role': 'system',
 'content': system_prompt},
 {'role': 'user',
 'content': user_prompt}],
 api_key='your api key possibly loaded from .env',
 functions=functions,
 call_function'store_correct_text')

Finally, if things worked, simply replace the anonymization elements:

if result['success']:
 final_text = result['with_arguments'].get('correct_text', '')

 for argname in ('person_proper_names',
 'place_names',
 'dates',
 'geographical_coordinates',
 'times'):
 names = result['with_arguments'].get(argname, '')
 for name in names.split(';'):
 if name.strip():
 # The placeholder is the argument name
 # minus the final s
 final_text = final_text.replace(name.strip(),
 f'[{argname[:-1]}]')