The architecture that separates prototypes from production systems

This article is about that layer. Not the agent itself, not the model, the boundary that connects the three: where the data lives, where the agent decides, and where FastAPI comes in to expose this as a stable API contract.

The real role of FastAPI in this architecture

The most common mistake I see is treating FastAPI as if it were the brain of the system. It is not. FastAPI is the asynchronous HTTP layer, responsible for input validation with Pydantic, authentication, rate limiting, CORS, and response serialization. The agent's decision logic, the conversation state, the execution checkpoints, none of this should reside within a route handler.

This directly connects with what I already established in the first article of this series: LangGraph, CrewAI, and Agno solve orchestration, LangChain solves composition, and none of them compete for the same space as FastAPI. FastAPI exposes what these layers do; it does not do their work. A route handler that directly calls an LLM, builds a prompt, and handles the return is the same trap of agent-as-prompt that I mentioned before, only at the API layer.

Where each piece fits into the architecture

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
 Observability: OpenTelemetry, LangSmith or Phoenix, Prometheus and Grafana
 Structured logs and tracing per execution, traversing all the layers below
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Client
  │
  ▼
FastAPI (asynchronous HTTP layer)
  Pydantic (input and output validation)
  JWT (authentication)
  Rate limiting per endpoint
  CORS
  │
  ▼
Service layer
  Data repository (access to Postgres, no SQL loose in the handler)
  Embedding service (generates and caches vectors)
  Agent invocation (starts or resumes the graph)
  │
  ▼
Orchestrator (LangGraph, for example)
  Agent state
  Persistent checkpoint
  Tools via direct SDK or via MCP server
  │
  ├──────────────┬──────────────┐
  ▼              ▼              ▼
PostgreSQL     pgvector /     Redis
(relational     Qdrant        (embedding cache,
data)           (vector data)  session, intermediate result)
  │
  ▼
LLM (Anthropic, OpenAI, or another provider)
  │
  ▼
FastAPI → StreamingResponse (SSE) → Client

The detail that most architecture diagrams hide is in the bottom right corner: the response does not come back as a single payload. It comes back as events, token by token, transmitted via Server-Sent Events. This is not a frontend detail; it is an API architecture decision that changes how the endpoint is designed from the start.

The lifecycle of a request

The layered diagram shows where each piece resides. It is worth complementing with the view of a specific request traversing this system from start to finish, because it is this sequence that is usually missing in the minds of those designing the endpoint for the first time.

Request
  │
  ▼
Validation (Pydantic)
  │
  ▼
Authentication (JWT)
  │
  ▼
Context retrieval (history, agent state)
  │
  ▼
Vector search (pgvector or Qdrant)
  │
  ▼
Agent execution (LangGraph)
  │
  ▼
Tools (direct SDK or MCP)
  │
  ▼
Persistence (checkpoint in PostgreSQL)
  │
  ▼
Streaming (SSE back to the client)
  │
  ▼
Response

Observability spans this entire execution, from the first to the last step; it is not a stage that happens afterward.

Each arrow in this chain is a point where something can fail, and each failure point needs a different response: a failed validation returns 422 before spending a token, an empty vector search should not stall the agent, and a tool failure needs to appear in the log before leaking as a generic error to the client.

The decisions that really matter

The agent's state resides in the database, not in the process. A traditional HTTP handler is stateless; it processes the request and forgets. An agent does not; it accumulates conversation history, tool usage, and context between turns. If this state lives in the memory of the Python process, a server restart wipes out all ongoing conversations. The correct practice is for the orchestrator to persist checkpoints in PostgreSQL at each node transition, and the FastAPI handler becomes a thin wrapper that starts the execution and returns immediately, without holding the request captive to the entire processing.

For conversational experiences, streaming becomes essential. An agent that takes twenty-five seconds to respond completely loses the user in the eighth second. The same response, transmitted token by token via SSE, delivers the first token in fractions of a second, and the abandonment rate plummets, even if the total processing time is identical. This changes the design of the endpoint: instead of returning a JSON at the end, it returns a StreamingResponse with media_type="text/event-stream", and it is necessary to disable buffering on the reverse proxy in front of it, because an nginx with buffering enabled nullifies the entire streaming without warning. It is worth noting: embedding, classification, OCR, scoring generation, and batch processing do not have this requirement; the user is not waiting on a screen watching blinking dots, and forcing streaming in these cases is complexity without purpose.

Decouple the HTTP lifecycle from the agent's processing. A request tied to the process that serves it does not survive connection drops, load balancer timeouts, or server restarts. This does not need to come all at once. The first step is often BackgroundTasks from FastAPI itself or asyncio.create_task(), sufficient while the volume is low. The next step, when the volume justifies it, is to move long-running processing to a dedicated asynchronous worker, Celery, Dramatiq, Arq, or Temporal, and leave FastAPI responsible only for starting the work and serving the progress stream, allowing the client to reconnect to an execution that is already in progress.

Observability is not what you add after something breaks. Every agent in production needs to answer five questions at any moment: how much did this execution cost, which tool was called, which prompt ran, where exactly did it fail, and how many tokens were spent. This means tracing per execution, not just error logging, with OpenTelemetry, LangSmith, or Phoenix for what is specific to LLM, and Prometheus with Grafana for the infrastructure metrics on top. Without this layer, each incident in production becomes a manual investigation of loose logs.

Access to data consumed by the agent is not the same as the usual CRUD. An agent does not make a query; it makes several, iteratively, and sometimes it fails the first attempt and tries again with another filter. A well-designed data repository for this consumer exposes methods thought out for exploratory search, not just for ad-hoc reading and writing, and isolates the agent from schema changes, just as a repository isolates any other consumer.

Security comes in layers, not in a single middleware. Schema validation with Pydantic blocks malformed input before it gets close to the agent. JWT authentication and rate limiting per endpoint contain abuse and cost. Defense against prompt injection and masking of PII before any sensitive data touches the model protect the input. On output, the logic is the same applied in reverse: preventing a response outside the expected schema from producing any side effects. The output parser validates the format, the schema guarantees the structure, and, when the domain requires, a policy engine decides whether that action can actually be executed before releasing it. This also applies to how the agent calls tools: the service layer can invoke directly via SDK or through an MCP server, and this choice changes where this output validation needs to happen.

Where the deterministic boundary comes in

This point I do not compromise on in any architecture I design: AI analyzes and recommends, deterministic engine decides, system executes. In practice for this layer, this means that the endpoint exposing an agent's recommendation and the endpoint executing an action with a real effect on the business are not the same endpoint, and often they should not even have the same response contract. If the agent suggests approving a transaction, canceling an order, or generating a legal document, the API layer needs to make it explicit, in the response format, that this is a recommendation, not a decision already made.

A real example of decision

"

Real case

In a payment gateway I architected, the data and intelligence layer in Python with FastAPI functioned as a Data and AI BFF, consuming transactional microservices written in Go. The transactional core, in Go, handled the payment flow itself, fast, deterministic, with no AI in the critical path.

FastAPI exposed the AI Risk Agent as an assistive capability, querying PostgreSQL for transactional data and Qdrant and pgvector for risk pattern similarity, always returning a recommendation, never a final decision about the transaction.

Separating these layers by language, responsibility, and API contract made it explicit, at every point in the system, where AI analyzes and where the deterministic engine decides, without relying on anyone to remember this rule when programming. This also allowed evolving the AI layer independently of the financial core, without touching the code that processes the payment itself.

When this complete architecture is overkill

Not every project needs the full five layers from the first commit. If the usage volume is still low and the conversation fits into a single request-response, a dedicated worker to decouple processing is premature complexity; BackgroundTasks suffices for now. In many initial architectures, pgvector alone, within the same PostgreSQL that already holds the rest of the relational data, resolves for much longer than one might think, while index, volume, and latency still comfortably fit in the relational database, and a dedicated vector database only pays off when pgvector shows real signs of limitation, not before. The criterion I use is simple: I add a layer when the problem it solves has already appeared, not because it appeared in a reference architecture article.

The real cost of the choice

Perceived latency versus real latency. Streaming does not reduce the total processing time; it changes the perception of waiting. Ignoring this and delivering a response only at the end is the cheapest way to lose users in a product that technically works.

Operational complexity of multiple storages. PostgreSQL, vector database, and Redis together mean three failure surfaces, three backup strategies, and three monitoring points, not one. Each additional storage needs to justify this recurring operational cost.

Coupling to the reverse proxy. A streaming endpoint that works perfectly in development and fails in production almost always has the same cause: buffering in nginx or the load balancer swallowing the events before they reach the client. This is the type of bug that only appears after deployment, and that no local test catches.

Audit of distributed state. When the agent's state lives in a checkpoint in the database and the processing runs in a worker separate from the handler that received the request, tracking what happened in a specific execution requires correlating logs between at least three components, not one.

The question that almost no one asks

Before designing this layer, I answer five questions about the system I am building.

Does the agent need to maintain state between conversation turns, or is each call independent enough not to require a checkpoint?

Does the response take long enough for the user to perceive the wait, to the point that streaming becomes non-optional?

Is there an action with a real effect on the business at the end of the flow, or does the agent only recommend, and something deterministic decides afterward?

Does the usage volume already justify decoupling the processing from the HTTP request lifecycle, or is this still anticipated complexity?

Will someone need to reconstruct what happened in a specific execution after it has finished?

Each answer decides how many of the five layers of this