Error analytics without transcripts

A useful failure record can be this small:

{
  "phase": "schema",
  "tool": "schedule_intent",
  "issueCodes": ["invalid_type"],
  "schemaPaths": ["expiresAt"],
  "logicalTurnId": "turn_7"
}

It says where the contract failed and lets later events join the same group. It does not preserve the attempted value.

The dangerous version starts as a debugging convenience. Provider bodies, prompts, arguments, stack traces, and outputs go into one searchable table. The report works, but the table becomes a second transcript store with weaker retention, broader operator access, and no working connection to product forget controls.

Record structural events only. A failure fingerprint describes what went wrong through bounded fields. It never copies conversation content, attempted values, raw errors, stacks, or provider responses.

Error analytics should answer "Which failure class is recurring?" It does not need to answer "What did the user say?"

Concept map

physical run attempt
  -> run_id
  -> protocol, schema, runtime, result, repeat, terminal events

automatic retry
  -> new run_id
  -> same logical_turn_id

later user request
  -> new logical_turn_id

event
  -> allowlisted structural fields
  -> stable fingerprint
  -> counts across runs and logical turns
  -> 30-day expiry

never stored
  prompts, messages, arguments, outputs, raw errors, stacks, provider bodies

The logical turn boundary prevents retries from looking like repeated user failures.

Model attempts and user turns are different

Give every execution attempt a run_id. Give the user-visible unit of work a logical_turn_id. Automatic retries share the logical turn id. A later message, resume action, or new scheduled firing gets a new one.

This lets reports distinguish three cases:

• one malformed call retried three times in one turn
• the same failure recurred across two separate user turns
• many conversations hit the same structural contract problem

A multi-turn candidate should require the same fingerprint in at least two logical turns. That threshold is evidence for review, not proof that the user needs a new feature.

Classify at execution boundaries

The tool wrapper is the common boundary for schema validation, runtime exceptions, and structured unsuccessful results. Provider loops must add events for failures that happen before a valid tool reaches the wrapper.

Useful event classes include:

• malformed provider tool JSON
• unknown tool name
• schema validation failure
• tool runtime exception
• unsuccessful structured result
• unchanged repeated failure
• provider failure
• budget pause
• tool-iteration exhaustion
• completed turn totals

An app check returning ok: false is unsuccessful. A successful check with advisory design issues is still successful. Classification must follow tool semantics, not whether the output contains words such as "error."

Both provider adapters need the same event contract. If one loop handles unknown tools internally without emitting an event, the report becomes provider-dependent.

Fingerprint structure, not values

A fingerprint can hash a canonical object made from bounded fields:

type FailureShape = {
  source: "tool" | "provider" | "turn";
  phase: "protocol" | "schema" | "runtime" | "result" | "terminal";
  event: string;
  tool?: string;
  issueCodes?: string[];
  expectedTypes?: string[];
  schemaPaths?: string[];
  errorCode?: string;
};

function fingerprint(shape: FailureShape): string {
  const canonical = stableJson({
    ...shape,
    issueCodes: [...(shape.issueCodes ?? [])].sort(),
    expectedTypes: [...(shape.expectedTypes ?? [])].sort(),
    schemaPaths: [...(shape.schemaPaths ?? [])].sort(),
  });
  return sha256(canonical);
}

For schema failures, retain issue codes, expected primitive types, and paths known by the schema. Discard attempted values and raw unknown keys. Even an unknown key may contain user text.

For runtime failures, keep only an allowlisted code or class. Error messages often embed filenames, URLs, SQL, response bodies, or user input. Normalize first and discard the raw string.

For provider events, retain provider, returned model, bounded status category, iteration, and whether fallback occurred. Do not store response bodies.

Correlation without a hidden transcript

An event can carry conversation id, operation id, release revision, cell, model, attempt, and timestamps. These identifiers support scoped investigation while the detailed operational row still exists.

The operation id should be a non-foreign-key pointer if operation retention differs. Analytics expiry must not prevent normal cleanup. Conversation deletion should cascade events when the contract requires it.

Private conversations should emit no agent events. Omitting message text is not enough because event timing, tool names, and conversation correlation still reveal activity.

Analytics writes must be best effort. A failed event insert cannot replace the original tool result or turn error. Observability that changes user-facing behavior is a new failure source.

Reports should stay content-free

A command-line report can group by fingerprint and show:

• count and distinct logical turns
• first and last seen
• affected cells and releases
• tools, phases, models, and bounded outcomes
• sample event, run, conversation, and operation ids

The sample ids support a controlled follow-up through existing retained records. They do not justify copying those records into analytics.

Prometheus counters should expose event kind and phase only. Fingerprints, tool inputs, conversation ids, and error text are poor metric labels because they create high cardinality and another retention problem.

Expire live events after a short fixed period such as 30 days. Exclude table data from backups if the report does not need historical recovery.

Terminology

• Agent event is an immutable structural record of an execution outcome.
• Run id identifies one physical attempt.
• Logical turn id groups retries for one user-visible unit of work.
• Failure shape is the bounded canonical metadata used for grouping.
• Fingerprint is the stable hash of that shape.
• Multi-turn candidate is one fingerprint observed in at least two logical turns.
• Allowlist is the finite set of safe error codes or classes retained.
• Content-free report groups behavior without storing message content.

The word "candidate" matters. Structural repetition does not reveal user intent or satisfaction.

Decisions and rejected alternatives

Append-only events preserve attempt order, release boundaries, and retry distinctions. A mutable failure summary loses that evidence.

Deterministic fingerprints keep grouping stable. A model-generated incident summary costs money, varies between runs, and requires reading the content this design avoids storing.

Short retention and backup exclusion fit the job. These events support current engineering work. They are not product history.

Use one contract across providers and tools so reports compare execution paths instead of inheriting adapter gaps.

Rejected alternatives include raw transcript archives, semantic classifiers over assistant prose, and metric labels containing fingerprints. Transcript archives expand privacy scope. Prose classifiers can mistake apologies for failures and miss silent contract errors. Fingerprint labels create unbounded metric series.

Failure modes

• Automatic retries count as separate user turns.
• Attempted schema values enter event details.
• Runtime messages bypass the allowlist.
• Unknown tool arguments are stored for debugging.
• One provider emits protocol events while another does not.
• Advisory warnings count as failed tools.
• Private turns emit supposedly anonymous events.
• Event insertion failure masks the original result.
• Fingerprints change because arrays are not canonicalized.
• Conversation deletion leaves correlated analytics behind.
• Backups retain events after live expiry.
• A report labels repeated structure as proven user dissatisfaction.

Field checklist

• Assign run and logical turn ids separately.
• Reuse logical turn ids across automatic retries.
• Emit events at tool and provider boundaries.
• Define bounded event and outcome enums.
• Build fingerprints only from structural allowlisted fields.
• Discard prompts, values, outputs, raw errors, stacks, and bodies.
• Treat private conversations as no-event paths.
• Make event writes best effort.
• Group by distinct logical turns, not only counts.
• Keep reports content-free and claims modest.
• Use bounded metric labels.
• Apply deletion, retention, and backup policy.

Structural events show which failure classes deserve work. The final chapter turns those signals, production trials, and explicit cost units into a repeatable improvement method: The improvement loop. Model and quota events originate in the routing system described in Model routing, cost, and quota.