Building reliable AI agents in production

Last year I shipped three LLM-powered agents into production. Two of them failed spectacularly in the first week. The third one is still running, handling ~50k requests per day with 99.7% uptime. Here's what made the difference.

The naive approach (and why it breaks)

Most tutorials show you something like this:

from openai import OpenAI
 
client = OpenAI()
 
def agent(user_query: str) -> str:
    response = client.chat.completions.create(
        model="gpt-4",
        messages=[
            {"role": "system", "content": "You are a helpful assistant."},
            {"role": "user", "content": user_query}
        ],
        tools=my_tools,
    )
    return response.choices[0].message.content

This works in a notebook. It does not work in production. Here's what's missing:

1. No retry logic — LLM APIs fail. A lot.
2. No output validation — the model can return anything.
3. No timeout — some queries hang for 60+ seconds.
4. No cost tracking — you'll get a surprise $2,000 bill.

The retry pattern that actually works

After burning through several approaches, I settled on exponential backoff with jitter. The key insight: don't just retry on HTTP errors — retry on semantic failures too.

interface AgentResult<T> {
  data: T;
  tokens_used: number;
  latency_ms: number;
  retries: number;
}
 
async function resilientAgent<T>(
  prompt: string,
  validator: (output: unknown) => output is T,
  maxRetries = 3
): Promise<AgentResult<T>> {
  for (let attempt = 0; attempt <= maxRetries; attempt++) {
    try {
      const start = performance.now();
      const raw = await callLLM(prompt);
      const parsed = JSON.parse(raw);
 
      // Semantic validation — not just "did the API respond"
      // but "is the response actually correct"
      if (!validator(parsed)) {
        throw new SemanticError(`Validation failed: ${JSON.stringify(parsed)}`);
      }
 
      return {
        data: parsed as T,
        tokens_used: raw.usage.total_tokens,
        latency_ms: performance.now() - start,
        retries: attempt,
      };
    } catch (err) {
      if (attempt === maxRetries) throw err;
 
      const delay = Math.min(1000 * 2 ** attempt, 30000);
      const jitter = delay * (0.5 + Math.random() * 0.5);
      await sleep(jitter);
    }
  }
  throw new Error("Unreachable");
}

The validator function is the secret weapon. Instead of hoping the LLM returns the right format, you verify it:

function isValidInvoice(obj: unknown): obj is Invoice {
  return (
    typeof obj === "object" &&
    obj !== null &&
    "amount" in obj &&
    typeof obj.amount === "number" &&
    obj.amount > 0 &&
    "currency" in obj &&
    ["USD", "EUR", "GBP"].includes(obj.currency)
  );
}

The math behind backoff timing

The expected wait time after $n$ retries with exponential backoff and jitter is:

Where $b$ is the base delay (1 second) and $W_{\max}$ is the maximum delay cap (30 seconds). The $\frac{3}{4}$ factor comes from the uniform jitter over $[0.5d, d]$.

For our configuration, the expected total wait times are:

This gives us the three nines we need. The probability of all $n$ attempts failing is $(1-p)^n$, so with $p = 0.85$ and $n = 4$ total attempts:

Monitoring: the forgotten piece

Here's my Prometheus-style metrics setup. Every agent call gets instrumented:

from dataclasses import dataclass, field
from time import time
from collections import defaultdict
 
@dataclass
class AgentMetrics:
    total_calls: int = 0
    total_tokens: int = 0
    total_cost_usd: float = 0.0
    latency_p50: float = 0.0
    latency_p99: float = 0.0
    error_rate: float = 0.0
    _latencies: list[float] = field(default_factory=list)
 
    def record(self, tokens: int, latency: float, cost: float):
        self.total_calls += 1
        self.total_tokens += tokens
        self.total_cost_usd += cost
        self._latencies.append(latency)
 
        # Update percentiles
        sorted_l = sorted(self._latencies)
        n = len(sorted_l)
        self.latency_p50 = sorted_l[int(n * 0.50)]
        self.latency_p99 = sorted_l[int(n * 0.99)]
 
metrics = AgentMetrics()

The metrics that actually matter:

• Token burn rate — how fast you're spending money
• Semantic error rate — how often the model returns valid JSON but wrong answers
• p99 latency — the worst 1% of user experiences
• Retry amplification — are retries causing cascading load

Architecture overview

After a lot of iteration, here's the architecture that stuck:

┌──────────────┐     ┌──────────────┐     ┌──────────────┐
 │   API Layer  │────▶│  Agent Core  │────▶│   LLM API    │
│  (FastAPI)   │     │  (retry +    │     │  (OpenAI /   │
│              │     │   validate)  │     │   Anthropic) │
└──────────────┘     └──────┬───────┘     └──────────────┘
                            │
                     ┌──────▼───────┐
                     │   Tool       │
                     │   Registry   │
                     ├──────────────┤
                     │ • Search API │
                     │ • Database   │
                     │ • Calculator │
                     └──────────────┘

The key insight: the Agent Core is a pure function of (prompt, tools, history) → action. It doesn't hold state between calls. All state lives in the caller. This makes it trivially testable and horizontally scalable.

What I'd do differently

If I started over, three things:

1. Start with evals, not prompts. Write 50 test cases before you write a single prompt. Seriously. The eval suite is the specification — without it you're just vibing.

2. Log everything. Every prompt, every response, every tool call. Storage is cheap; debugging blind is expensive. I use a structured format:

{
  "trace_id": "abc-123",
  "timestamp": "2026-08-15T10:30:00Z",
  "prompt_tokens": 1247,
  "completion_tokens": 389,
  "model": "claude-sonnet-4-6",
  "tools_called": ["search", "calculator"],
  "semantic_valid": true,
  "latency_ms": 1843
}

3. Set a cost budget with a circuit breaker. If daily spend exceeds $X, shut it down. You'll thank yourself when a retry loop goes infinite at 3am.

Conclusion

Building reliable AI agents isn't about better prompts — it's about better engineering. Treat the LLM as an unreliable external service (because it is), validate everything, monitor everything, and build in graceful degradation.

The agents that survive production are the boring ones. The ones with retry logic, validation, monitoring, and circuit breakers. The magic is in the infrastructure, not the model.