Hosted ongabo.esvia theHypermedia Protocol

The Harness

Testing and GatesWhat must pass before anything ships, the harnesses that drive a real server, and honest coverage markers.

The Harness

14 August 2026, 19:32
1

The agents suite is the backbone: 307 tests passing across 27 files, run with bun test in agents/. Around it sit build smoke tests, model-facing gates, and end-to-end harnesses that drive the real service.

The gate list

Before a push, all of these (agents/docs/development.md:26-40):

cd agents && bun check          # typecheck (tsgo) + format
cd agents && bun test           # the full suite, no skips added
cd agents && bun run test:build # compile the binary + smoke it
cd agents && bun run test:docker
cd agents && bun run test:trigger
pnpm typecheck                  # the whole frontend workspace
pnpm test

test:trigger deserves its own mention: it boots a real daemon against a stand-in activity feed, creates an agent and a user-mention trigger over the signed API, and asserts that one comment-mention fires exactly one session. It exists because of a real bug: a comment and its citation arrive as sibling events, and a naive matcher fires twice or drops the second.

Desktop runs vitest for units and Playwright for e2e.

What the suite covers

Area

Files

Verbs and service surface

api-service.test.ts, verbs.test.ts

Tool documents

tool-documents.test.ts, agent-tools-api.test.ts

Runs and time

runs.test.ts, run-time.test.ts

Script engine

workflow-host.test.ts

Triggers

activity-triggers.test.ts, trigger-events.test.ts, activity-trigger-race.test.ts, schedule-triggers.test.ts

Memory, attachments, exec, web tools

agent-memory.test.ts, session-attachments.test.ts, code-exec.test.ts, web-tools.test.ts

Platform

auth.test.ts, sqlite.test.ts, main.test.ts, config.test.ts, json-schema.test.ts, poll-loop.test.ts, provider-oauth.test.ts

Invariant tests

A recurring technique worth copying: when two functions must agree, a test asserts the agreement directly rather than trusting review. workflow-host.test.ts:519 checks that the signal a park advertises (answerSignalFor) is one that would actually satisfy it (signalMatchesWait) — so the Answer button can never send a signal the wait would reject. Others pin content-keyed replay against reordered ctx.parallel continuations, and exactly-once event delivery against wait cleanup.

The end-to-end harnesses

Four programs in agents/e2e/, each solving a different half of "does this work against reality":

    run.ts — the Tier-3 live-model harness. Scenarios (chat-smoke, sub-basic, sub-typed, sub-restraint, wf-hello, todo-adoption) run against a real provider with --record, or replay from cassettes by default.

    live-gate.ts — signs envelopes with the desktop daemon's real key and drives an already-running server over CBOR. It creates a throwaway agent mirroring a real one, runs scenarios (trivial, memory, parallel-delegate, script-parallel, script-narration), writes transcripts, and deletes the agent. Enforced checks are deterministic properties; model behavior is reported, not asserted.

    scripted-provider.ts — an OpenAI-compatible server with the model's half pinned, so the real runtime can be driven over real HTTP and real SQLite deterministically.

    obligations-live-check.ts — drives a real server with that scripted provider to verify the obligations loop end to end.

    narration-check.ts — model-free; drives the workflow VM directly to assert that a ctx.call description reaches both the journal and the tool-call adapter.

Honest coverage: the STALE cassettes

The replay cassettes were recorded against the pre-verb-collapse tool surface (sub_session, run_workflow, update_plan, memory_*, execute_code). Cassette fingerprints include tool names, so every one of them is now invalid.

Rather than delete the tests or let them pretend, agents/e2e/recordings/STALE.md exists as a marker: while it is present, bun e2e/run.ts prints that recordings are stale, skips replay, reports 0 pass / 0 fail, and exits 0 — so e2e-replay.test.ts stays green without claiming coverage it does not have. Restoring them means re-recording live with an API key, verifying every scenario, then deleting the marker.

This is the pattern the whole test strategy uses: when coverage is missing, say so in a machine-readable way rather than leaving a green check that lies.

Simulated-model gates

Where a live model is unavailable, a blind subagent is given only the system prompt and the real tool schemas and asked to complete a scripted task — delegation fan-out, tool discovery, plan maintenance. Its output is then validated mechanically (did it choose delegation, does the JSON validate, does the script lint and run). This is what gated the M1 verb migration, since cassettes could not survive a tool-surface change.

The review record

agents/docs/harness/reviews/ holds a document per milestone, each with what changed, how it was verified, known gaps, and a manual test script. Their status lines are deliberately unequal — 01-verbs.md reads "verified — reviewed by Eric", while 05-exec-time.md reads "implemented, gates green, not live-verified".

Related

Do you like what you are reading? Subscribe to receive updates.

Unsubscribe anytime

Powered by Seed Hypermedia