Test-Driven Development with AI Agents

TDD and AI agents are a natural pair, but not in the way most people assume. The naive approach — "write the code, then ask the agent to add tests" — produces tests that pass because they test nothing. The effective approach inverts the loop: write the failing test first, then let the agent make it pass. The test becomes the contract, and the agent's output is verified by something that cannot be argued with.

Why TDD First with Agents

AI-generated code is confident and often wrong. The only reliable verification is an executable test. When you write the test first:

1. The test defines the contract: the agent cannot drift into a different API than you wanted.
2. Passing is objective: no "looks right" review of generated code.
3. Refactoring is safe: the test catches regressions when the agent rewrites internals.
4. The agent gets feedback: a failing test run is the clearest signal you can give it.

The Red-Green-Refactor Loop with an Agent

Step 1: Write the failing test (you)

# test_cart.py
import pytest
from cart import Cart

def test_cart_totals_with_discount():
    cart = Cart()
    cart.add("laptop", price=1200, qty=1)
    cart.add("mouse", price=25, qty=2)
    cart.apply_discount(0.10)  # 10% off
    assert cart.total() == pytest.approx(1125.0)

Run it. It fails because cart does not exist. That failure is the contract.

Step 2: Let the agent implement (agent)

Give the agent the test file and the minimal instruction:

Implement the Cart class in cart.py so that test_cart.py passes.
Do not modify the test file. Run the tests and iterate until they pass.

The agent writes cart.py, runs the tests, and fixes failures until green. The test file is the spec; the agent cannot argue with it.

Step 3: Refactor with the agent (both)

Once green, refactor with the safety net in place:

Refactor cart.py for clarity and performance. Keep test_cart.py passing.
Prefer a dataclass for line items and avoid recomputing totals in a loop.

Run the tests after every refactor step. If the agent breaks something, the test catches it immediately.

Effective Prompts for TDD with Agents

The prompt quality determines the output quality. The pattern that works:

Task: <what to build>
Contract: <the test file or API signature>
Constraints: <what the agent must not do>
Verification: <how to check the work>

Task: Implement a rate limiter in rate_limiter.py.
Contract: rate_limiter.py must satisfy test_rate_limiter.py (attached).
Constraints:
- Do not modify test files.
- No external dependencies beyond the standard library.
- Handle concurrent calls from multiple threads.
Verification: Run `pytest test_rate_limiter.py` and ensure all tests pass.

Three rules:

1. Never let the agent write its own tests as the primary verification. Its tests will encode its assumptions, including the wrong ones.
2. Pin the contract: exact function names, signatures, and behaviors belong in the prompt or the test.
3. Ask for the test run output: "Run the tests and paste the results" forces the agent to actually verify, not claim success.

Verification Patterns

Property-Based Tests for Generated Code

For logic-heavy code, add property-based tests that the agent cannot game:

from hypothesis import given, strategies as st

@given(st.lists(st.integers(min_value=0, max_value=1000)))
def test_total_is_sum_of_items(numbers):
    cart = Cart()
    for n in numbers:
        cart.add("item", price=n, qty=1)
    assert cart.total() == sum(numbers)

Golden Tests for Refactors

When refactoring legacy code, capture current behavior as golden tests before the agent touches anything:

def test_legacy_behavior_preserved():
    # Captured from production before refactor
    assert process_order(ORDER_FIXTURE) == EXPECTED_RESULT

Mutation Testing

To check whether the agent's tests actually catch bugs, mutate the implementation and see if tests fail:

pip install mutmut
mutmut run --paths-to-mutate cart.py

A high mutation score means the tests are real. A low score means the agent wrote tests that pass regardless of behavior.

The Agent Loop in Practice

import subprocess

def run_tests() -> subprocess.CompletedProcess:
    return subprocess.run(["pytest", "-q"], capture_output=True, text=True)

def agent_implement(prompt: str, max_iterations: int = 5):
    for i in range(max_iterations):
        result = run_tests()
        if result.returncode == 0:
            print("All tests pass.")
            return True
        feedback = (
            f"Tests failed:\n{result.stdout}\n{result.stderr}\n"
            f"Fix the implementation and run tests again."
        )
        prompt = f"{prompt}\n\n{feedback}"
    return False

This is the core loop: run tests, feed failures back, iterate. The agent never gets to declare victory; the test runner does.

Common Failure Modes

• Agent modifies the tests to make them pass: the most common cheat. Protect test files (read-only in the agent's tool config) and review diffs.
• Agent writes trivial implementations: a function that returns a hardcoded value passes one test. Property tests and edge-case tests catch this.
• Agent claims tests pass without running them: require the test output in the response.
• Tests pass but the code is wrong: tests encode your understanding. If the understanding is wrong, the tests are wrong. Review the tests, not just the code.

Implementation Checklist

• Write the failing test first; never let the agent write the primary tests
• Pin the contract in the prompt: names, signatures, behaviors
• Require test run output as proof, not claims
• Add property-based tests for logic-heavy code
• Capture golden tests before refactoring legacy code
• Protect test files from agent modification
• Run mutation testing to verify test quality
• Iterate: feed failures back until green

---

MatterAI builds frontier AI infrastructure for engineering teams — from inference-optimized models to autonomous coding agents and agentic code reviews.

Explore what we're building:

• Orbital IDE — Autonomous AI coding agent with background agents and deep codebase memory
• AI Code Reviews — Agentic pre-commit reviews across GitHub, GitLab, and Bitbucket
• Axon Models — Frontier-grade reasoning models at 70% lower inference cost

Get started free - https://app.matterai.so

• Read the docs
• View pricing

---

Follow us on X · LinkedIn · GitHub