Browser tests for generated apps

A page can pass HTML parsing and JavaScript syntax checks while its Save button does nothing. A shared game can render perfectly for the host while the guest never receives updates. Generated apps need a browser test, but giving an agent unrestricted Playwright code is the wrong interface.

Run the draft inside a controlled browser service. Let the agent describe user actions through a small flow language. Capture runtime errors, visible outcomes, URL state, layout evidence, and screenshots. Keep publication separate from the check environment.

This chapter supplies the runtime evidence used by the policy in Checks that report, not block.

Concept map

draft files -> loopback server -> browser context -> flow steps
                                           -> errors
                                           -> layout
                                           -> screenshot

The browser driver is an implementation detail. The agent-facing contract is a bounded test language.

Terms

Check origin is the fixed local HTTP origin that serves instrumented draft files.

Instrumentation is code inserted for tests to record uncaught errors, rejected promises, failed requests, and console failures.

Flow DSL is the allowed set of user-level test steps.

Client is one isolated browser page with its own storage and network connections.

Flow fingerprint is a stable hash of normalized steps used for caching.

Checker version invalidates cached reports when test behavior changes.

Fail-fast index is the first step that did not complete or meet its assertion.

Serve drafts locally

Deploying every check to a hosting provider wastes quota and adds alias propagation delays. It also makes the checker depend on external control-plane availability.

A loopback file server can expose an in-memory draft to headless Chromium. The server should:

• Bind only to loopback.
• Route requests by an unguessable check token or internal app key.
• Serve normalized draft paths and the instrumented entry document.
• Set predictable content types and no-store caching.
• Shut out directory traversal and arbitrary filesystem reads.
• Use one fixed origin allowed by any shared data service.

The browser sees a normal origin, so modules, service workers, storage, fetch, and WebSockets behave more like production than they do under file: URLs.

Do not persist an instrumented deployment. Instrumentation is test-only. Publication should upload the original prepared files.

Keep the flow language small

Most mini-app checks need a short vocabulary:

[
  {"type": {"label": "Name", "text": "Mina"}},
  {"click": "Add player"},
  {"expectText": "Mina"},
  {"expectUrl": "#round"}
]

Useful operations include:

• Click an accessible name or stable selector.
• Type into a labelled field.
• Expect visible text.
• Expect a URL path, query, or fragment.
• Wait for a bounded interval.
• Open another client.
• Repeat a short setup block.

Prefer accessible labels because they test the user interface and encourage good markup. Permit stable selectors for canvas controls or repeated rows where labels are insufficient.

Stop at the first failed step. Later failures are usually consequences and add noise. Return the failing index, client name, action, expected value, actual evidence, and a screenshot.

repeat is worth adding because generated forms often need several similar records. The checker can expand it internally while the model sends one compact structure.

Test real client isolation

Multiplayer and shared-data apps need separate pages, not iframes inside one page. Iframes on the same origin share local storage and may reuse state that should belong to separate devices.

Give each named client its own browser context or isolated page storage. Let one client join another's capability link:

[
  {"openClient": {"name": "guest", "join": {"client": "host", "access": "rw"}}},
  {"click": "Add item", "client": "host"},
  {"expectText": "Milk", "client": "guest"}
]

Cap the number of clients. Host plus three guests is enough for normal smoke tests and prevents one generated flow from exhausting memory.

Capability fragments require care. Test helpers must preserve the fragment used by the app's data SDK. Navigation code must not overwrite it for screen state. Shared-app checks should prove both directions when collaboration matters: host writes and guest observes, then guest writes and host observes.

Gather evidence after the flow

The browser run should capture:

• Uncaught exceptions and unhandled promise rejections.
• Console errors, kept separate from crashes.
• Failed fetches and module loads.
• Dialogs that block fresh-page use.
• Visible text or accessibility landmarks.
• A screenshot of the host.
• Layout measurements at phone and desktop widths.

Layout checks can detect document overflow, overlapping visible elements, clipped controls, and single words broken in narrow containers. These checks produce candidates, not final visual truth. Report them under the advisory policy unless launch promises require more.

Run the phone viewport near 390 by 844 and a desktop viewport near 1280 by 800. Test light and dark schemes when the design system supports both. Use reduced-motion passes for interaction components that animate.

Cache what is safe to cache

Browser checks can take seconds. Cache reports by:

draft revision + flow fingerprint + checker version

A changed flow reruns the browser even when files are unchanged. A code edit invalidates all results for the old revision. A checker change invalidates earlier reports because the same inputs may now produce different findings.

Do not include transient screenshots in the identity. Store them as evidence attached to the cached report.

Concurrency needs a hard bound. A small browser pool, perhaps two checks at once, protects the service. When full, return an exact retry time as an advisory infrastructure issue. Do not make the model spin in a tight retry loop.

Decisions and rejected paths

Use a local browser instead of remote snapshot APIs. Remote browser services add permissions, quotas, provider outages, and draft deployments. A packaged Chromium costs image size but gives direct control.

Expose a DSL, not Playwright scripts. Raw scripts allow arbitrary network access, infinite loops, brittle selectors, and large tool payloads.

Use separate clients. Same-origin iframes do not model device storage correctly.

Fail at the first broken step. Cascading assertion failures obscure the cause.

Keep browser checks advisory except for first-load failure. A failed optional flow should not hide a usable live link.

Do not deploy drafts for checks. Local serving avoids quota use and public noindex previews.

Failure modes

The check creates real user data or consumes quotas. Mark check-created spaces as ephemeral, exclude them from normal counts, and expire them.

Top-level network setup leaves a blank page. Generated apps should render local content first and report sync failures in the page.

Storage leaks between clients. Isolate browser contexts and verify device identifiers differ.

The flow waits forever. Put deadlines on each step and the full run.

A busy pool throws a generic error. Return retryAt and keep it advisory.

Service workers make offline tests misleading. Browser network emulation may not cover worker-controlled requests. Use the context's true offline mode and verify requests.

The checker tests instrumented code, then publishes it. Keep test injection in memory only.

A screenshot replaces assertions. Screenshots aid diagnosis. They do not prove behavior.

Field checklist

• Are drafts served from loopback without provider deployment?
• Is the check origin explicitly allowed by shared services?
• Does the flow language cover click, type, text, URL, wait, repeat, and clients?
• Do selectors prefer accessible names?
• Does execution stop at the first failed step?
• Are browser clients storage-isolated?
• Can shared-app flows join by capability without leaking keys to logs?
• Are runtime errors distinct from console noise?
• Are phone, desktop, and reduced-motion states checked?
• Is the cache keyed by revision, flow, and checker version?
• Is browser concurrency bounded with a retry time?
• Are check-created data spaces ephemeral?
• Are screenshots evidence rather than pass criteria?

Browser tests reveal what happened. They should not force agents to reinvent every visual rule. A design system agents can read describes a versioned design system exposed as queries rather than raw CSS.