Multi-Agent Systems: Orchestrating Multiple LLM Agents

One agent can do a lot. Multiple agents can do more, but only if you solve the problems single agents never face: who decides what, how agents share state, how they communicate, and what happens when one of them fails. This guide covers the topologies, the coordination patterns, and the failure modes of multi-agent systems.

Why Multiple Agents at All

A single agent with many tools hits three walls:

1. Context bloat: every tool result stays in one context window. Ten tools × long outputs = a confused model.
2. Role confusion: a model that is great at code review is mediocre at SQL, and one prompt cannot be great at both.
3. Single point of failure: one bad tool call poisons the whole run.

Specialized agents with narrow roles and narrow tools keep each context small and each behavior predictable.

Topology 1: Hub-and-Spoke (Orchestrator)

One orchestrator agent decides which specialist agent handles each task. The orchestrator never does the work itself; it routes.

                 ┌──► researcher
orchestrator ────┼──► coder
                 ├──► reviewer
                 └──► writer

from langgraph.graph import StateGraph, START, END
from typing import TypedDict, List

class State(TypedDict):
    task: str
    specialist: str
    result: str

def route_task(state: State) -> State:
    # Orchestrator: classify and delegate
    specialist = llm_choose_specialist(state["task"])
    return {"specialist": specialist}

def run_specialist(state: State) -> State:
    # Dispatch to the right specialist agent
    agents = {"researcher": researcher_agent, "coder": coder_agent}
    result = agents[state["specialist"]].invoke(state["task"])
    return {"result": result}

graph = StateGraph(State)
graph.add_node("route_task", route_task)
graph.add_node("run_specialist", run_specialist)
graph.add_edge(START, "route_task")
graph.add_edge("route_task", "run_specialist")
graph.add_edge("run_specialist", END)

Pros: clear ownership, easy to reason about, each specialist stays focused. Cons: the orchestrator is a bottleneck and a single point of failure; if it misroutes, everything fails.

Topology 2: Pipeline (Sequential Handoff)

Each agent does one stage and hands off to the next. The output of one is the input of the next.

planner ──► researcher ──► writer ──► reviewer ──► formatter

def planner(state: State) -> State:
    return {"plan": plan_agent.invoke(state["task"])}

def researcher(state: State) -> State:
    return {"research": research_agent.invoke(state["plan"])}

def writer(state: State) -> State:
    return {"draft": writer_agent.invoke(state["research"])}

def reviewer(state: State) -> State:
    feedback = reviewer_agent.invoke(state["draft"])
    # Reviewer can send work back up the pipeline
    if feedback["approved"]:
        return {"approved": True}
    return {"approved": False, "revision": feedback["notes"]}

Pros: simple, testable stage by stage, natural for content and data flows. Cons: total latency is the sum of all stages; one slow agent stalls the pipeline.

Topology 3: Swarm (Peer-to-Peer)

Agents talk to each other directly, with no central coordinator. Each agent decides who to hand off to.

from langgraph.prebuilt import create_react_agent

def transfer_to_billing():
    """Hand off to the billing specialist."""
    return billing_agent

def transfer_to_support():
    """Hand off to the support specialist."""
    return support_agent

support_agent = create_react_agent(
    model=model,
    tools=[transfer_to_billing, transfer_to_support, search_kb],
    prompt="You are support. Hand off to billing for payment issues.",
)

Pros: flexible, scales to many agents, no bottleneck. Cons: hard to trace, easy to loop (A hands to B hands to A), hard to test.

Shared State and Communication

Agents need a shared memory. Three levels:

1. Shared state object: a typed dict passed through the graph (LangGraph's State). Best for structured data.
2. Shared message list: all agents append to one conversation. Simple but bloats context.
3. External store: a database or vector store for long-lived memory. Necessary when agents run across sessions.

class AgentState(TypedDict):
    messages: List[dict]       # shared conversation
    artifacts: dict            # structured outputs
    budget_used: int           # cross-agent budget tracking

Rule: pass artifacts (structured results) between agents, not raw conversations. Each agent should receive only what it needs, not everything everyone said.

Failure Handling

Multi-agent systems fail in ways single agents do not:

• Agent loops: A and B hand off to each other forever. Fix: max handoff counter in state.
• Cascading errors: one agent's bad output poisons every downstream agent. Fix: validate outputs at each handoff boundary.
• Orchestrator misrouting: the router sends a coding task to the writer. Fix: add a validation step after routing.
• Budget exhaustion: agents burn tokens without finishing. Fix: track budget_used in shared state and force-stop at a cap.

def check_budget(state: AgentState) -> str:
    if state["budget_used"] > MAX_BUDGET:
        return "end"          # hard stop
    if state["handoffs"] > MAX_HANDOFFS:
        return "end"          # loop protection
    return "continue"

When NOT to Build a Multi-Agent System

Multi-agent is a complexity tax. Do not pay it when:

• One agent with a few tools solves the task (most tasks).
• The steps are fixed (use a workflow).
• Your single-agent baseline is not yet solid (multi-agent amplifies bad foundations).
• You cannot afford the latency of sequential agent calls.

Start single-agent. Add a second agent only when you can name the specific failure the second agent fixes.

Implementation Checklist

• Start with one agent; add specialists only for named failures
• Choose hub-and-spoke for clarity, pipeline for stage flows, swarm only for flexibility
• Pass structured artifacts between agents, not raw conversations
• Add handoff counters and token budgets to shared state
• Validate every agent's output at handoff boundaries
• Log every handoff for tracing and debugging
• Test each agent in isolation before wiring them together

---

MatterAI builds frontier AI infrastructure for engineering teams — from inference-optimized models to autonomous coding agents and agentic code reviews.

Explore what we're building:

• Orbital IDE — Autonomous AI coding agent with background agents and deep codebase memory
• AI Code Reviews — Agentic pre-commit reviews across GitHub, GitLab, and Bitbucket
• Axon Models — Frontier-grade reasoning models at 70% lower inference cost

Get started free - https://app.matterai.so

• Read the docs
• View pricing

---

Follow us on X · LinkedIn · GitHub