- An orchestration layer owns six things the model cannot own: state, retries, tool routing, the stop condition, the cost ceiling and observability.
- The stop condition is where most homegrown loops break, because a model asked whether it is done is a poor judge of whether it is done.
- Research on agentic experience learning names this the self confirmation trap: wrong but internally consistent trajectories get recorded as successes and the error compounds.
- Anthropic's own guidance separates workflows, which follow predefined code paths, from agents, which direct their own process. Most teams need the first and build the second.
- Retries in a chained system multiply context rather than requests, because each retry re sends the whole accumulated conversation at full price.
- A survey of thirteen open source agent scaffolds found tool counts from zero to thirty seven and seven different context compaction strategies, so there is no default shape to copy.
It starts as one call. Then a second call to check the first one's output. Then a tool. Then a retry when the tool fails. Then a loop, because sometimes two tools are needed and you cannot tell in advance which. By the fifth call there is a file in the codebase nobody wants to open, full of nested conditionals and a while loop with a magic number in it, and the honest description of that file is that it is an orchestration layer that nobody designed.
AI orchestration is the name for that layer once you decide to design it on purpose. This piece is about what it has to own, what it must not own, and the three failures that only show up after the second call.
What is an orchestration layer, exactly?
It is the code that decides what happens between model calls. That is the whole definition, and the boundary is sharper than it sounds: the model decides what to say and which tool to ask for, the orchestration layer decides everything else, a split that is easiest to see when you write the loop out yourself with no framework in the way.
Anthropic's engineering guidance draws the useful line between workflows and agents. A workflow is a system where models and tools are orchestrated through predefined code paths. An agent is a system where the model dynamically directs its own process and tool use. The same document names five patterns worth knowing by name: prompt chaining, routing, parallelisation, orchestrator workers and evaluator optimiser. Its recommendation is to start simple and add complexity only when the simpler version demonstrably underperforms.
That advice is worth taking literally, because the two things are not on a quality ladder. A workflow is not a primitive agent. It is a different trade: you give up flexibility and get determinism, testability and a bill you can predict. Most production features want that trade. Teams reach for the agent because it is more interesting to build.
What must the orchestration layer own?
Six responsibilities, and a system that leaves any of them undefined has not removed the responsibility, it has assigned it to chance, which is the same reason each rung of conversational AI needs a named owner for its failures.
| Responsibility | What it means concretely | What covers it if you do nothing | Failure if unowned |
|---|---|---|---|
| State | What the model sees on turn N, and what got dropped to fit | The provider's conversation array, until it overflows | Silent truncation of the instruction that mattered |
| Retries | Which errors retry, how many times, with what backoff | The SDK retries transport errors only | Bad output retried forever, or not at all |
| Tool routing | Which tools exist on this turn and who executes them | Every tool exposed on every turn | Token cost and wrong tool selection both rise |
| Stop condition | What ends the loop, decided outside the model | A turn counter, if you remembered one | Loops that circle, or stop one step early |
| Cost ceiling | A hard token or dollar budget per task | Nothing | One pathological request bills like a thousand |
| Observability | Per turn traces with token counts and tool results | Application logs, if any | You cannot tell why yesterday's run was worse |
The academic framing lands in the same place from a different direction. A June 2026 systems paper by Ankur Sharma and Deep Shah on agent operating systems argues that classical operating system abstractions, built around deterministic programs and explicit control flow, stress at exactly these boundaries once the workload is a long lived goal directed agent. Their decomposition of what a control plane must provide is close to the table above: schedulers, context and memory management, tool and capability registries, policy enforcement, and observability with audit.
You do not need an operating system to run five chained calls. But if two independent lines of thinking, one from a model vendor and one from a systems paper, produce nearly the same list of responsibilities, that list is probably not arbitrary.
What are the three failures that only appear once you chain calls?
A single call fails in one way: it returns something wrong and you can see it. Chained calls fail in ways that are invisible at the level of any individual call.
The loop circles instead of progressing. The agent tries an approach, evaluates its own output, decides more work is needed, and tries a rephrased version of the same approach. Each turn looks productive. Nothing moves. This is the failure a body of 2026 research keeps returning to under different names. Work on agentic experience learning by Shiding Zhu and colleagues calls it the self confirmation trap, where wrong but self consistent trajectories are misidentified as successful experience and the errors accumulate. Their proposed fix is structural rather than prompt based: separate the agent that executes from the agent that distils what happened, then verify by consensus rather than by self report.
Retries multiply context, not requests. In a single call system, a retry costs one more call. In a chained system, a retry on turn six re sends turns one through five along with it, at full price, with no discount for repetition. A 10% retry rate on a six turn loop is not a 10% cost increase, it is closer to 30% depending on where in the loop the failures land. This is also why retry policy belongs to the orchestration layer rather than to the HTTP client: the client knows the request failed, only the orchestrator knows what that failure costs.
State drifts from what the model believes. Your application updates a record. The model, working from a context assembled three turns ago, still believes the old value. It then calls a tool using the stale value and the tool succeeds, because the value is well formed, just wrong. Nothing errors. The only defence is deciding explicitly, in code, what gets re read from source on each turn and what is allowed to persist from earlier context.
How do you write a stop condition that works?
By making it external. The rule to hold onto is that the model may propose stopping and the orchestrator decides, never the reverse.
A working stop condition is a disjunction of four clauses, evaluated in code after every turn, and it terminates when any one of them is true.
The first clause is goal satisfaction, and it must be checked against something outside the model. Tests pass. The API returned 200. The JSON validates against the schema, which is a check worth enforcing during decoding rather than after the fact. The row exists in the database. If your goal cannot be expressed as a check that runs without asking a model, you do not yet have a task an agent should be running unattended.
The second clause is a hard turn limit. Pick a number, make it a constant with a name, and log every time it fires. If it fires often you have learned something real about the task rather than about the model.
The third clause is a budget ceiling in tokens or dollars, tracked cumulatively across the whole task rather than per call. Every provider returns token counts in the usage block of each response; add them up and stop when the total crosses the line. This is the clause that turns a runaway loop from an incident into a log entry, and it is the one most often missing.
The fourth clause is a no progress detector. Hash whatever the task is meant to change, the file contents, the record, the plan, and compare it across turns. Two consecutive turns that change nothing observable means the loop is circling, and circling for a third turn will not fix it. This is the clause that catches the self confirmation failure without needing the model to admit anything.
The clause people write first, asking the model "are you finished?" and parsing the answer, belongs nowhere in this list. It is the only one of the five that cannot fail safely, because the same reasoning that produced a wrong answer produces the confidence that the answer was right.
Does a framework give you this, or do you write it?
Partly, and the honest answer is that the six responsibilities are not covered evenly by anything off the shelf.
Benjamin Rombaut's source code survey of thirteen open source coding agent scaffolds is the most concrete evidence of how little consensus exists. Tool counts across those thirteen ran from zero to thirty seven. Seven distinct context compaction strategies appeared. Eleven of the thirteen combined more than one control primitive, drawing from a set of five: ReAct, generate test repair, plan execute, multi attempt retry and tree search. The survey's own conclusion is that agents converge where constraints force convergence, on tool categories and edit formats, and diverge everywhere the design is open, which is precisely context management and state handling.
Read that as a buying signal. Frameworks are strong on the parts that converged, tool calling plumbing and message formatting, and weak on the parts that did not, which are the parts your specific task actually depends on. Take the plumbing, write the loop.
The tool interface is the piece most worth standardising rather than inventing, because it is a protocol problem rather than a logic problem. Pointing an assistant at a documented tool server removes an entire category of routing bugs, and we walked through the mechanics in a guide to setting up an MCP server. The tools MaShop exposes over that protocol are listed on our MCP page, which is also a fair example of what a tool registry looks like when it is written down rather than assembled at runtime.
Where does the cost ceiling belong?
In the same loop as the stop condition, incremented from real usage numbers rather than estimates.
Each response carries its own token counts, split by input, cached input and output. Anthropic's pricing documentation spells out where the numbers live and what feeds them: the tools parameter itself is billed as input, as are every tool_use block and every tool_result block you send back, on top of a per model system prompt that appears whenever any tool is present. Accumulate all of it per task, price it against the rates on that page, and compare against a per task budget you decided in advance. The arithmetic for that, including why the cached and uncached split matters more than the headline rate, is worked through in our piece on what a real feature costs once retries are counted.
Two design details make the difference between a ceiling that works and one that only exists in a config file. The budget must be per task, not per call, because the failure mode is many cheap calls rather than one expensive one. And the ceiling must terminate the task rather than throttle it, because a loop that is out of budget and still running is a loop that will find more budget somewhere.
If you are building something merchants pay for, that ceiling is also the thing standing between a pathological request and your margin. It is the same reasoning behind how we meter credits per build on MaShop: a ceiling that is visible to the person spending it is a ceiling that gets respected.
What does observability mean here, and why is it different?
In ordinary software, observability means you can answer questions about a running system. In an orchestrated model system it means something more specific and more awkward: you can reconstruct exactly what the model saw.
The reason this differs from normal tracing is that the input is assembled rather than passed. By turn six, the prompt is a composite of a system message, a compacted history, a set of tool schemas chosen by your routing logic, and several tool results. If a run goes wrong and all you logged was the user's original request and the final answer, you have logged the two things that were never in question.
What to capture per turn is short: the assembled input, the token counts split three ways, which tools were exposed, which tool was called with what arguments, what came back, and how long each step took. Capture all of it or the trace cannot answer the question you will actually have, which is always some version of why did this run differ from that one.
This is also the part that decays fastest. Compaction strategies get tuned, tool sets get trimmed, a prompt gets edited, and a trace recorded under the old configuration stops being comparable. Version the orchestration configuration and stamp the version onto every trace. It costs an integer and it is the difference between a regression you can bisect and one you argue about.
Where does parallelisation fit?
It is the cheapest structural win available and the one most often skipped, because a loop is easier to reason about than a fan out.
Two of the patterns in Anthropic's list are parallel rather than sequential. Sectioning splits independent subtasks and runs them at once, which converts a latency problem into a concurrency problem. Voting runs the same task several times and compares outputs, which converts a reliability problem into a cost problem you can choose to pay.
Voting deserves particular attention next to the self confirmation research, because it is the same insight arriving from the practical direction. Three independent attempts compared against each other give you a signal that one attempt evaluating itself cannot produce, for the simple reason that the errors are not correlated in the same way. That is the structural argument behind the execute, distil and verify separation, and you can approximate a useful fraction of it with three parallel calls and a majority rule.
The orchestration cost is real: parallel branches need their own state, their own retry accounting, and a merge step that can fail on its own. But when the alternative is a sequential loop that circles for eight turns, three parallel attempts are often cheaper in both money and wall clock.
When is an orchestration layer the wrong answer?
When the task has a fixed shape. If you know the sequence, write the sequence. Three model calls in a row with typed inputs and outputs is not an inferior design to an agent that discovers the same three calls, it is a better one, because you can test it, cost it and debug it.
The orchestration layer earns its complexity when the number of steps genuinely depends on the input, when tools can fail in ways that change the plan, or when the same pipeline must serve tasks of very different sizes. Short of that, the loop is overhead with a research paper attached.
A reasonable progression: start with a chain, add routing when the inputs split into categories, add a loop only when you cannot enumerate the steps in advance. At each stage the previous one keeps working, which is not true if you start at the end. The related question of what changes when a prompt becomes a loop is one we took apart separately in a look at what an agent loop actually changes about prompting.