Measure the machine

Users reported that "builds sometimes get stuck." The logs contained provider status codes and stack traces, but no shared turn ID, no tool iteration count, and no record of whether work had paused on cost or exhausted its loop. Engineers could see errors. They could not reconstruct the machine's decision.

Adding more log text would have made the privacy problem worse. The system needed structured, bounded evidence.

Concept map: one logical turn

logical turn
|
+-- attempt 1
|    +-- provider calls
|    +-- tool operations
|    +-- retryable failure
|
+-- attempt 2
     +-- model fallback
     +-- tool operations
     +-- terminal outcome

aggregates: latency, tokens, cost, iterations, failures

The logical turn is the unit a user recognizes. Attempts explain automatic recovery. Operations explain actions. Provider calls explain cost and routing. Keep all four identities.

Terms

• Metric: A numeric time series used for rates, totals, and alerts.
• Event: A bounded structured record of one notable occurrence.
• Trace correlation: Shared identifiers that connect turns, attempts, and operations.
• Fingerprint: A stable hash of safe structural failure fields.
• Cardinality: The number of distinct metric label combinations.
• Retention: How long records remain available.
• Content-free analytics: Measurements that omit prompts, arguments, outputs, and raw errors.

Measure outcomes before internals

Start with questions that affect users and cost:

• How many logical turns complete, fail, pause, or exhaust?
• What is p50 and p90 time to a visible reply?
• How many provider calls and tool calls does a turn use?
• What do prompt, cached, and completion tokens cost?
• How often does routing fall back?
• Which tools produce contract, runtime, or structured failures?
• How often does the same fingerprint recur across distinct turns?

CPU and memory still matter, but they do not explain a turn that took eight minutes because it reread 2,000 lines five times.

Emit one terminal event per attempt:

type TurnFinished = {
  event: "turn_finished";
  logicalTurnId: string;
  runId: string;
  attempt: number;
  source: "chat" | "intent" | "nudge";
  outcome: "completed" | "failed" | "paused" | "exhausted";
  role: "chat" | "build";
  model: string;
  iterations: number;
  toolCalls: number;
  promptTokens: number;
  cachedTokens: number;
  completionTokens: number;
  fallbacks: number;
  costUsd: number;
  durationMs: number;
};

This shape contains no conversation text. A conversation ID can be included when local investigation needs correlation, but it deserves tighter access and retention.

Classify failures where they happen

The tool wrapper sees schema validation, runtime errors, and structured results. The provider loop sees malformed tool JSON, unknown tool names, repeated unchanged calls, quota, and exhaustion. The scheduler sees claim and occurrence failures.

Record each at the narrowest boundary. A global catch block knows too little and tends to store raw exception text.

Useful bounded classes include:

• tool_contract with issue code and schema-known path;
• unknown_tool with normalized requested name;
• tool_runtime with allowlisted error class;
• tool_result_failure with tool-defined outcome code;
• provider_protocol with phase and model;
• provider_fallback with reason class;
• turn_exhausted with iteration count;
• budget_pause with budget kind.

Do not store attempted values for schema errors. A value may contain a password, document body, or email address. The path ["recipient"] and expected type string are enough to find a contract mismatch.

Fingerprint structure, not prose

Exception messages contain IDs and provider wording that change across attempts. Build a stable fingerprint from bounded fields:

function fingerprint(failure: {
  kind: string;
  phase: string;
  tool?: string;
  code?: string;
  path?: string[];
}) {
  const safe = {
    kind: failure.kind,
    phase: failure.phase,
    tool: failure.tool ?? null,
    code: failure.code ?? null,
    path: failure.path ?? [],
  };
  return sha256(stableJson(safe));
}

Count the same fingerprint within one run and across distinct logical turns. A repeated call inside one run may be model looping. The same fingerprint across later turns is stronger evidence of a broken contract or missing feature.

It is still evidence, not a verdict. A recurring quota fingerprint does not mean a new capability is needed. Reports should avoid model-written interpretations such as "user dissatisfaction."

Use metrics without cardinality accidents

Prometheus labels should come from small allowlists:

agent_turns_total{outcome,source,role}
agent_tool_calls_total{tool,outcome}
agent_provider_calls_total{provider,model,role,result}
agent_tokens_total{provider,model,role,type}
agent_cost_usd_total{provider,model,role}
agent_turn_duration_seconds{role,outcome}

Do not put conversation IDs, operation IDs, fingerprints, error text, paths, or arbitrary model names in labels. Those belong in bounded event rows or logs. High-cardinality labels increase memory and make queries expensive.

Normalize models through a catalogue. Unknown provider-returned names become unknown, while the event row may keep a bounded normalized identifier for investigation.

Privacy and retention are design constraints

Agent observability can accidentally become a second memory system. Set strict rules:

• private conversations emit no agent events;
• prompts, tool arguments, outputs, provider bodies, stack traces, and raw errors stay out of analytics rows;
• event writes are best effort and never alter the user result;
• rows expire after a fixed period, such as 30 days;
• conversation deletion cascades related events;
• backups exclude short-lived analytics table data when long retention is unnecessary.

Metrics can outlive event rows because they contain only aggregates with bounded labels. Document the difference.

Decisions and rejected approaches

Keep operation records and analytics separate. Operations support the user-visible audit trail and may contain compact result details. Analytics supports fleet-wide patterns and should contain less.

Use an append-only event table. Editing historical failures complicates counts and investigation. Retention deletes are the exception.

Do not make analytics writes transactional with the turn. Losing one event is preferable to failing a reply because the analytics table is unavailable.

Prefer reports before dashboards. A command that groups the last seven days by fingerprint can validate whether the data is useful. Build dashboards and alerts after recurring questions appear.

Measure context economy directly. Counts of file reads, lines returned, cache tokens, and calls per first versus follow-up build expose waste that latency alone cannot.

Failure modes in observability

• Nested catches emit a terminal event twice. Give one layer ownership and test for exactly one terminal event per attempt.
• Best-effort writes create an unbounded in-memory queue during a database outage. Cap the queue and emit a dropped-event metric after the cap.
• Raw provider model strings increase metric cardinality. Normalize them.
• Sampling hides rare, consequential failures. Sample high-volume success events if needed, but retain all bounded failure events within the retention window.
• A gateway omits price, making cost totals look exact when they are not. Record estimated and reported cost separately, or mark the source.
• Average latency stays flat while p90 climbs. Report percentiles and separate roles because agent turns have long tails.

Field checklist

• Logical turns, attempts, provider calls, and operations have separate IDs.
• Every attempt emits one terminal outcome.
• Tokens separate prompt, cached, and completion usage.
• Cost records its source as reported or estimated.
• Tool failures distinguish contract, runtime, and structured outcomes.
• Fingerprints use bounded structural fields only.
• Cross-turn recurrence counts distinct logical turns.
• Metric labels come from small allowlists.
• Private work emits no analytics events.
• Analytics excludes prompts, values, outputs, raw errors, and stacks.
• Event writes cannot fail user work and have a bounded queue.
• Retention, cascade deletion, and backup policy are explicit.
• Reports compare first and follow-up work, model roles, fallbacks, and tail latency.

Measurement closes the first engineering loop. The system can now accept durable work, enforce boundaries, resume safely, control context, route by risk, and show where those choices fail. When later changes are proposed, return to the routing assumptions in Route work by risk, then test them against turn-level evidence rather than a polished demo.

The next part moves that discipline onto the device. Local-first or local-later asks whether accepted work survives disconnection and reload, where server metrics alone cannot help.