Outboxes before agents

The UI inserted a message, then called an HTTP endpoint to wake the agent. The message commit succeeded. The wake-up request died with the user's Wi-Fi. The conversation showed the request forever, but no worker knew it needed an answer.

Reversing the order did not help. Waking the worker before inserting the message created the opposite race: the worker ran, found no input, and exited. The two writes needed one commit.

The durable path

browser replica
      |
      v
server mutation
      |
      +-- insert message --+
      |                    | one transaction
      +-- insert outbox ---+
                           |
                           v
                     worker claim
                           |
                           v
                  reply + done commit

The outbox is a trigger record. The message table remains the source of conversation content. This separation keeps queue mechanics out of user-visible data while preserving atomic creation.

Short glossary

• Outbox: A table of durable work created in the same transaction as application state.
• Producer: Code that inserts the state change and outbox row.
• Consumer: The worker that claims and processes rows.
• Claim: A status change that assigns rows to an attempt.
• At-least-once delivery: Work may run more than once after uncertain failures.
• Debounce window: A short delay used to batch nearby requests.
• Orphan operation: A tool record created by an attempt that never produced a reply.

Keep the schema boring

A useful first schema needs status, attempts, ownership by a context stream, and a unique request key:

CREATE TABLE agent_outbox (
  id             BIGSERIAL PRIMARY KEY,
  conversation_id TEXT NOT NULL REFERENCES conversations(id) ON DELETE CASCADE,
  message_id      TEXT NOT NULL REFERENCES messages(id) ON DELETE CASCADE,
  status          TEXT NOT NULL DEFAULT 'pending'
                  CHECK (status IN ('pending', 'processing', 'done', 'error')),
  attempts        INTEGER NOT NULL DEFAULT 0,
  created_at      TIMESTAMPTZ NOT NULL DEFAULT now(),
  processed_at    TIMESTAMPTZ,
  UNIQUE (message_id)
);

CREATE INDEX agent_outbox_pending
  ON agent_outbox (conversation_id, created_at)
  WHERE status = 'pending';

The unique message ID matters when an offline client retries a mutation. The second server call should observe that the message and its outbox record already exist. It must not create a second turn.

Insert both rows in one transaction:

await withTransaction(pool, async (db) => {
  const inserted = await db.query(
    `INSERT INTO messages (id, conversation_id, role, content)
     VALUES ($1, $2, 'user', $3)
     ON CONFLICT (id) DO NOTHING
     RETURNING id`,
    [messageId, conversationId, content]
  );

  if (inserted.rowCount === 1) {
    await db.query(
      `INSERT INTO agent_outbox (conversation_id, message_id)
       VALUES ($1, $2)`,
      [conversationId, messageId]
    );
  }
});

The conditional insert prevents a duplicate outbox row when the original message already exists.

Claim by conversation

Agent context creates an ordering boundary. Claim all pending rows for one conversation after a brief debounce. Process different conversations concurrently if capacity allows.

With one worker, an in-memory set can prevent overlap while an atomic update claims the rows. With several workers, use a database transaction with FOR UPDATE SKIP LOCKED on candidate streams or leases with expiry. Do not use a plain select followed by an update across processes. Two workers can select the same rows.

The claim increments attempts and returns IDs. Build the logical-turn ID from those IDs, then load content. The returned set, not "whatever is pending now," defines the turn.

Finish all related state together

After a successful model loop, one transaction should:

1. insert the assistant message;
2. attach this attempt's operation rows to it;
3. mark the claimed outbox rows done;
4. update conversation ordering and unread metadata.

If any step fails, none should commit. A reply with pending work will run twice. Done work without a reply disappears from the user's view.

Operations need attempt ownership. A simple early design may identify unattached operations with message_id IS NULL, but this can mix concurrent work if one conversation ever allows more than one attempt. A run ID is safer:

UPDATE operations
SET message_id = $1
WHERE run_id = $2 AND message_id IS NULL;

Make the ownership link explicit before the system grows.

Recovery is part of the write path

On startup, inspect every nonterminal state. A small single-worker service may reset all processing outbox rows to pending. That is safe only if operations are read-only or idempotent, or if interrupted attempts with operations are handled differently.

A stronger recovery rule is:

• no tool operation started: reset to pending;
• only read operations completed: reset to pending;
• any side-effecting operation completed: mark the attempt failed and show it;
• final transaction committed: rows are already done and require no recovery.

This rule needs operation effect metadata. Add it before side-effecting tools, not after the first duplicated email.

Why Postgres often wins

A queue product may offer better throughput, delayed delivery, and consumer groups. It also creates a dual-write problem. If the database commit succeeds and queue publish fails, work is lost unless you add a database outbox anyway. If queue publish succeeds first, the consumer may run before the state exists.

Using Postgres as the initial queue keeps atomicity and operational load simple. Polling every few hundred milliseconds is adequate for human-facing agent latency. Partial indexes keep pending scans cheap.

Move to a broker when measured load, retention, or fan-out requires it. The database outbox can then publish to the broker reliably. The pattern survives the migration.

Rejected shortcuts

Do not call the model inside the request transaction. A model call can last minutes. Holding a database transaction open creates lock pressure and makes client retries ambiguous.

Do not let the browser create queue rows. The browser can optimistically insert its message, but queue creation belongs to the authoritative server transaction. A malicious or outdated client should not choose attempts or statuses.

Do not derive "thinking" only from the latest role. Offline replicas and delayed synchronization can show a local user message that the server has never received. Use connection and synchronization state so the UI does not claim server work is running.

Do not delete failed work silently. Keep an error outcome tied to the conversation. A user can act on a visible failure; a missing reply looks like neglect.

Failure modes

The worker can die after the provider returns and before the final transaction. The row returns to pending, and another attempt may pay for the model call again. This is an accepted at-least-once window unless the provider offers durable request replay.

A conversation can be deleted during a turn. The final insert should fail on its foreign key. Treat that as cancellation and discard the result. Recreating deleted context behind the user's back is worse.

Poison work can fail forever. Cap automatic attempts. After the cap, commit a visible error and mark the outbox rows terminal.

A long-running row can remain processing after a machine loss. Startup reset handles restarts, but multi-worker systems also need lease expiry because another process may survive while one disappears.

Rapid messages can create one call per row. A short debounce groups them. Keep the window small and measured because every added millisecond affects perceived response time.

Outbox field checklist

• State change and outbox insertion share a transaction.
• A stable client request ID prevents duplicate queue rows.
• The outbox stores triggers, not a second copy of message content.
• Claims return the exact IDs included in a turn.
• Work serializes at the conversation boundary.
• Reply, operations, and done statuses commit together.
• Attempts have a cap and a visible terminal error.
• Startup recovery covers every in-flight status.
• Recovery distinguishes attempts with side effects.
• Multi-worker claims use locks, skip-locked selection, or expiring leases.
• A kill test covers before claim, during the model call, after a tool, and before final commit.

The turn defined in Design the turn, not the demo now has durable transport. Once work reaches the model loop, the system needs a strict language for actions. The next chapter treats each tool as a versioned protocol. Continue with Tools are protocols.