The harness is the product

The first prototype looked convincing. A user typed a request, a capable model called two tools, and the screen filled with a correct result. Then the process restarted between the second tool call and the final reply. The user saw a spinner forever. The external side effect had happened, the database claimed no reply existed, and the model had no record of what it had done.

That failure settled the architecture. The model was only one component. The real product was the system around it: durable input, bounded execution, visible work, recovery, and a record that could explain the outcome.

The machine around the model

user action
    |
    v
durable request ---> worker ---> model loop ---> tool boundary
    ^                  |              |               |
    |                  v              v               v
synced result <--- transaction <--- operation log <--- result
                       |
                       v
                 retry or stop

The line that matters most is not the arrow into the model. It is the path from a committed request to a committed outcome. If that path survives a restart and reports a clear failure, the system can recover. If it exists only in process memory, every deployment and crash becomes a small data-loss event.

Terms used here

• Harness: The code that accepts work, builds model context, exposes tools, records progress, applies limits, and commits the result.
• Turn: One bounded attempt to answer a user request, including its tool calls.
• Operation: One recorded tool invocation inside a turn.
• Authoritative store: The database whose committed state decides what happened.
• Replica: A local copy used for fast reads and optimistic interaction.
• Recovery: The rules that convert interrupted work into pending, failed, or completed work after restart.

How this book uses evidence

The chapters mix three kinds of material. Dated field notes report production observations and state the date, deployment scope, and evidence limits. Mechanisms described as requirements are design guidance, even when the reference system has not completed them. Alternatives such as richer merge modes and authoritative game loops are options to implement and verify, not claims about a hidden finished product.

That distinction matters. A measured failure can justify a design direction without proving that every proposed fix has shipped. When a chapter gives a number without a source note, treat it as an example to calibrate against your own workload.

Start with invariants

Choose a few statements that must remain true even when the model behaves badly.

For an interactive agent, a useful starting set is:

1. A request accepted by the UI exists in durable storage before work begins.
2. At most one turn for a conversation runs at a time.
3. Every tool call has a durable status.
4. The final reply and the completion of its request commit together.
5. Restarting the worker leaves no request permanently stuck in an in-flight state.

These are stronger than "the demo works." They describe behavior under concurrency, failure, and deployment.

One practical schema is small:

CREATE TABLE work_items (
  id          BIGSERIAL PRIMARY KEY,
  stream_id   TEXT NOT NULL,
  request_id  TEXT NOT NULL UNIQUE,
  status      TEXT NOT NULL
              CHECK (status IN ('pending', 'running', 'done', 'failed')),
  attempts    INTEGER NOT NULL DEFAULT 0,
  created_at  TIMESTAMPTZ NOT NULL DEFAULT now()
);

CREATE INDEX work_pending
  ON work_items (stream_id, created_at)
  WHERE status = 'pending';

The model does not need to know this table exists. The worker does. It claims rows, runs the turn, and changes status in transactions with the visible output.

Local first changes what fast means

A chat input should not wait for an LLM or even for a round trip to the server. The browser can write optimistically to a local replica, render the message, and synchronize it to the authoritative database. The server-side write then creates the work item in the same transaction as the message.

This split gives the user immediate feedback without pretending that model work has begun. The UI can distinguish three facts:

• the message exists locally;
• the message has synchronized;
• the worker is processing it.

Collapsing those facts into one spinner creates misleading states. A disconnected user may have a safe local message that has not reached the worker. A connected user may have a durable request waiting behind another turn. The interface should report the state it knows, not infer progress from elapsed time.

Why one worker is often enough

The obvious alternative is a distributed job system on day one. That adds leases, heartbeats, duplicate delivery, deployment order, and another store to operate. For a personal or small-team agent, one worker plus Postgres can support a surprising amount of work.

Serialize by conversation, not across the whole service. Different conversations may run together. Requests in one conversation must preserve order because each reply changes the context for the next.

const inFlight = new Set<string>();

async function processStream(streamId: string) {
  if (inFlight.has(streamId)) return;
  inFlight.add(streamId);
  try {
    const ids = await claimPending(streamId);
    if (ids.length === 0) return;
    await runTurn(streamId, ids);
  } finally {
    inFlight.delete(streamId);
  }
}

This in-memory guard is valid only when there is one worker. That is a deliberate limit. If throughput later requires several workers, move the claim rule into the database with row locks and skip-locked selection. Do not pay that complexity before the second worker exists.

Decisions and rejected options

Commit a complete reply once. Token streaming feels responsive, but it makes recovery harder. A partial assistant row can mean a live stream, a disconnected client, or a dead worker. Committing one finished reply gives the database a clean boundary. Stream ephemeral progress separately if users need it.

Keep the queue in the authoritative database. An in-memory queue is simple until the first restart. A separate broker can be correct, but it creates a two-store consistency problem when a message and its job must appear together. A database outbox gives one transaction.

Record tool calls before execution. Logging only after success loses the most useful evidence. Insert a running operation, execute the tool, then finalize it. Interrupted operations are visible and can inform retry policy.

Use plain functions before framework machinery. Agent frameworks often bundle routing, memory, tools, retries, and tracing. That is convenient, but their hidden control flow becomes expensive when production behavior needs a precise answer. A short explicit loop is easier to inspect.

Failure modes worth testing

A worker can die after the provider returns but before the database commit. The next attempt may produce another reply. You cannot remove every duplicate window without a provider-supported request key or a durable resume protocol. State the accepted window and make side-effecting tools idempotent.

A user can send another message during a turn. If the worker loads all messages again before committing, it may accidentally answer the new message before its request is claimed. Build the turn from the exact claimed request IDs, and leave later messages pending.

A tool can succeed and the model loop can fail afterward. Blindly replaying the whole turn may repeat the effect. Once any side-effecting operation has run, prefer a visible failed turn that a later user action can resume.

A process can restart with rows marked running. Startup recovery must reset safe work to pending or mark uncertain work as failed. Leaving it untouched is not recovery.

Field checklist

• User input becomes durable before model work starts.
• Message and work-item creation share one transaction.
• Work is serialized at the context boundary, usually per conversation.
• Tool calls have running, done, and error records.
• Final reply and work completion commit together.
• Startup recovery handles every in-flight status.
• Retry behavior changes after a side effect.
• The UI separates local, synchronized, running, and failed states.
• A restart test kills the worker at several points in a turn.
• Operational limits are written down before scaling the worker count.

The surrounding machine determines whether model output becomes dependable software. The next chapter narrows that machine into explicit trust zones, because durable execution is still unsafe if untrusted code and secrets share the wrong boundary. Continue with Draw the trust boundaries.