# HyperKitty Chromium — Architecture **Status: implemented reference implementation.** Every module, supervision relationship, schema, and API route described below exists in this repository and compiles, boots, and passes its test suite (`rebar3 eunit`: 24 tests, 0 failures, including real headless-Chromium integration tests and chaos tests that kill live processes to verify recovery). Sections are marked *(planned)* where the specification calls for something this revision intentionally stubs or defers — there is no unmarked aspirational content in this document. ## 1. Purpose and scope HyperKitty Chromium is an Erlang/OTP-native browser agent platform: a headless Chromium instance, driven over the Chrome DevTools Protocol (CDP), wrapped in an explicit OTP supervision tree so that browser control, agent orchestration, search, and messaging are separate, independently-supervised, independently-observable subsystems rather than library calls inside one monolithic process. The design principle it exists to demonstrate: an LLM-driven "agent" is a planning function that proposes tool calls; it is never trusted with direct, unchecked access to a browser process. Every tool call crosses an explicit, typed capability boundary — enforced by Erlang code, not by the prompt — before it reaches Chromium. This document is the architecture reference for engineers building on, operating, or extending this codebase. It assumes familiarity with Erlang/OTP (supervisors, `gen_server`, `gen_statem`) and HTTP/WebSocket APIs, but not with this specific codebase. ## 2. Repository layout ``` hyperkitty/ rebar.config deps, relx release config, profiles, dialyzer/xref config/sys.config application env (chromium path, API port, limits) config/vm.args node name, cookie, scheduler flags include/hyperkitty.hrl canonical entity records (data model) src/ hyperkitty_app.erl application callback hyperkitty_sup.erl HK-CORE root supervisor hk_schema.erl record -> JSON map projection (wire format) hk_id.erl, hk_event.erl ids, timestamps, the event envelope hk_sup_util.erl shared "wait for the old holder to die" helper hk_event_bus.erl / _sup event bus (pub/sub + audit log) hk_messaging_portal.erl HK-MSG hk_cdp_client.erl one CDP WebSocket connection (via `gun`) hk_chrome_http.erl Chrome's plain-HTTP /json control endpoint hk_browser_*.erl HK-BROWSER: session, capability, registry, supervisors hk_agent_*.erl HK-AGENT: the agent state machine, registry, toolbox hk_search_*.erl HK-SEARCH: the 11-stage pipeline job, provider behaviour hk_operation_store.erl operation_id lifecycle (accepted/succeeded/failed) hk_api_*.erl HK-VIS backend: Cowboy router, listener, REST + WS handlers hk_frontend_*.erl connected-client tracking for the observability UI hk_health.erl aggregate subsystem health priv/frontend/ the 7-view observability UI (static HTML/JS, no build step) test/ EUnit suites (see §11) Brewfile, bin/bootstrap.sh reproducible environment setup (see §12) ``` ## 3. Supervision tree ``` hyperkitty_sup (rest_for_one, intensity 8/10) +-- hk_event_sup (one_for_one) | +-- hk_event_bus [gen_server] +-- hk_messaging_sup (one_for_one) | +-- hk_messaging_portal [gen_server] +-- hk_browser_sup (one_for_one) | +-- hk_browser_registry [gen_server] session_id -> pid | +-- hk_browser_session_sup (simple_one_for_one, transient) | +-- hk_browser_session (one per session) [gen_server] owns a Chrome OS process +-- hk_agent_sup (one_for_one) | +-- hk_agent_registry [gen_server] agent_id -> pid | +-- hk_agent_fsm_sup (simple_one_for_one, transient) | +-- hk_agent_fsm (one per agent) [gen_statem] +-- hk_search_sup (one_for_one) | +-- hk_search_registry [gen_server] job_id -> pid | +-- hk_search_job_sup (simple_one_for_one, temporary) | +-- hk_search_job (one per search) [gen_statem] +-- hk_health [gen_server, worker] +-- hk_api_sup (one_for_one) | +-- hk_api_listener [gen_server] owns the Cowboy/Ranch listener +-- hk_frontend_sup (one_for_one) +-- hk_frontend_client_registry [gen_server] connected WS clients ``` **Why `rest_for_one` at the root.** Subsystems are listed in dependency order. If `hk_event_sup` crashes, everything after it is restarted too, because every later subsystem holds an implicit dependency on the event bus (agents, browser sessions, and search jobs all publish to it; the API layer's WebSocket clients have subscriptions that would otherwise go silently stale against a bus that no longer exists). A crash in `hk_browser_sup`, by contrast, restarts `hk_agent_sup`, `hk_search_sup`, `hk_health`, `hk_api_sup`, and `hk_frontend_sup`, but leaves `hk_event_sup` and `hk_messaging_sup` — listed *before* it — running undisturbed. `test/hk_supervision_tests.erl:rest_for_one_isolation/0` kills `hk_browser_sup` directly and asserts exactly this shape of recovery. **Why each plane is a two-level tree, not one `simple_one_for_one` supervisor.** OTP does not allow mixing a singleton child (the registry) with `simple_one_for_one` dynamic children (session/agent/job instances) under the same supervisor. Splitting each plane into a `one_for_one` supervisor wrapping (a) a singleton registry and (b) a `simple_one_for_one` instance pool means the registry restarting does not restart live sessions/agents/jobs (they re-register on next lookup miss; live processes are untouched), and a crash-restart storm in the instance pool cannot, via `hk_browser_sup`'s own intensity/period, take the registry down with it. **A known, fixed race in this shape, documented for anyone extending it.** An abrupt, unmaskable kill of a mid-tree supervisor (as opposed to the ordinary synchronous shutdown protocol a supervisor uses on its own children) does not guarantee its nested-supervisor children have finished their own termination and released their registered name before the grandparent supervisor's restart attempt runs. `hk_sup_util:start_link_retry/1` wraps every registered-name `start_link/0` in this application: on `{error, {already_started, OldPid}}` it monitors `OldPid`, waits for it to actually exit, and retries once, rather than surfacing the transient race as a failed restart. Separately, `hk_api_listener` calls `process_flag(trap_exit, true)` in `init/1` — without it, the ordinary supervisor shutdown signal (`exit(Pid, shutdown)`) would kill it directly without ever running `terminate/2`, leaking the underlying Ranch/Cowboy listener and its registered name on every restart, not only under abrupt kills. Both are exercised by `hk_supervision_tests.erl` and by `hk_browser_session_tests.erl`'s crash-isolation test. ## 4. Subsystems ### HK-CORE (`hyperkitty_sup`, `hk_event_bus`, `hk_schema`, `hk_id`, `hk_sup_util`) Boot sequencing, the root supervision policy (§3), the canonical record-to-JSON projection (`hk_schema:to_map/1` — the single place a wire representation is produced; internal code passes records, never hand-built maps, until it crosses this boundary), id/timestamp generation, and the event bus described in §7. ### HK-BROWSER (`hk_browser_session`, `hk_cdp_client`, `hk_chrome_http`, `hk_browser_capability`, `hk_browser_registry`) One `hk_browser_session` gen_server per browser session. `init/1` finds a free TCP port, launches a real headless Chromium OS process via `erlang:open_port({spawn_executable, ...}, ["--headless=new", "--remote-debugging-port=", "--remote-debugging-address=127.0.0.1", "--user-data-dir=", "--no-sandbox", "--disable-gpu", "--disable-dev-shm-usage", "about:blank"])`, polls Chrome's plain-HTTP `/json/version` endpoint until it answers, and registers itself with `hk_browser_registry`. Tab lifecycle (`open_tab`/`close_tab`) goes through Chrome's HTTP control surface (`hk_chrome_http`); per-tab interaction goes over CDP WebSocket JSON-RPC (`hk_cdp_client`, built on `gun`). Navigation correlates against the `Page.loadEventFired` CDP event via a waiters map. A Chrome OS process crash is detected in `handle_info` via the port's `{exit_status, Status}` message and stops the session with an abnormal reason, which `hk_browser_session_sup` (restart type `transient`) restarts under the same `session_id` — the concrete demonstration that one crashed session never takes down another (`browser_session_crash_isolation/0`) or any other subsystem. **Typed capability interface.** `hk_browser_capability:operations/0` is the closed list of 14 operations (`create_session`, `close_session`, `open_tab`, `close_tab`, `navigate`, `back`, `forward`, `reload`, `read_page`, `query_element`, `click`, `type`, `scroll`, `screenshot`, `extract_links`). There is no operation outside this list and no unrestricted "just run this JS" escape hatch. `is_granted/2` checks an agent's capability list, which supports bare-atom grants (unrestricted for that operation) and scoped grants — currently `{navigate, #{allowed_domains => [...]}}}`, checked against the URL's host. `validate_args/2` performs structural validation independent of authorization: URL scheme allow-list (`http`/`https`/`about`/`data`), selector length bounds (1–2047 bytes), typed text length bound (≤10000 bytes), scroll delta bounds. Both checks run in `hk_agent_fsm` (the control plane) before a call ever reaches `hk_browser_session` — see §8. *(Implementation note, stated plainly per this project's honesty requirement: `click`/`type`/`query_element`/`extract_links`/`scroll` are implemented via `Runtime.evaluate` — JavaScript injected into the page context — rather than the CDP `Input` domain's synthetic OS-level input events. This is a real, intentional simplification: it is sufficient for programmatic page interaction and observation, but it is not indistinguishable from a physical input device the way `Input.dispatchMouseEvent` would be. A production deployment that needs that distinction should route these through the `Input` domain instead.)* ### HK-AGENT (`hk_agent_fsm`, `hk_agent_registry`, `hk_agent_toolbox`) One `gen_statem` per agent, states `created -> ready -> planning -> executing -> waiting -> planning -> ... -> completed`, with `failed` and `terminated` reachable from any non-terminal state. `callback_mode() -> state_functions`. The planning intelligence itself (what decides which tool to call next) is deliberately external — a planner process, an LLM call, a human operator, or a test drives the loop one step at a time via `next_action(Pid, Tool, Op, Args)`. What the FSM owns is everything that must be enforced regardless of what decided the action: capability authorization (delegates to `hk_browser_capability` for browser ops), argument validation, the configurable loop limits, action history, and state transitions. `next_action/4` validates synchronously in the caller's context (fast) but spawns a linked worker to perform the actual browser/search call, so the FSM process is never blocked on a slow page load — it sits in `waiting` for a real `{agent_observation, Ref, Outcome}` message, correlated by reference, with a 45-second per-action timeout. **Loop limits**, checked before every tool call (`hk_agent_fsm:check_limits/1`), merged from `default_agent_limits` (app env) over any per-agent override: `max_actions` (default 40), `max_execution_ms` (default 300000), `max_network_requests` (default 200), `max_search_iterations` (default 5). Exceeding a limit transitions the agent to `failed` and replies `{error, {limit_exceeded, Which}}}` to the caller — a genuine error, not a silently-accepted `ok` (this was a bug caught and fixed during development; see §13). ### HK-SEARCH (`hk_search_job`, `hk_search_pipeline`, `hk_search_provider`, `hk_search_provider_mock`) One `gen_statem` per search job whose states are the pipeline stages themselves — `user_query -> query_normalization -> search_planner -> search_provider -> result_normalization -> url_deduplication -> content_retrieval -> content_extraction -> source_ranking -> result_synthesis -> frontend_visibility -> done`, with `failed` reachable from any stage. The job's current state *is* its current pipeline stage; there is no separate stage field that could drift from the state machine. `source_ranking` loops back to `search_planner` (bumping `iteration`, bounded by `max_iterations`, default 3) when fewer than 3 ranked results were found — the concrete enforcement point for the search-iteration limit. Every stage transition publishes an `hk_event` before advancing (§7 has the full catalog). The search backend is a pluggable behaviour (`hk_search_provider`: `plan/2`, `search/2`, `fetch/1`, `extract/2`); the bundled `hk_search_provider_mock` is a deterministic, self-contained reference implementation that makes no real network calls, so the full pipeline is exercisable and testable without external dependencies. A real provider (a live search API, an HTTP fetcher, an HTML/text extractor) plugs in by implementing the same behaviour and setting `{search_provider, Module}` in `sys.config` — *(planned: no live provider is bundled)*. ### HK-MSG (`hk_messaging_portal`) A minimal pub/sub message bus for inter-agent and agent/user messaging, independent of the event bus (events are platform telemetry; messages are addressed application payloads). `send/5` marks a message `delivered` if the recipient is currently subscribed, `queued` otherwise; `list_conversation/1` returns history in order; `mark_read/1` updates status. Backed by two ETS tables (`hk_messages_tab`, keyed by `message_id`; `hk_messages_by_conversation_tab`, keyed by `conversation_id`). ### HK-VIS (`hk_api_*`, `priv/frontend/`) The API layer (`hk_api_router`, `hk_api_listener` owning the Cowboy/Ranch listener lifecycle, and one handler module per domain) and the static observability frontend (`priv/frontend/index.html` + `js/app.js`) — see §9 and §10. ## 5. Data model Every entity is a record in `include/hyperkitty.hrl`, with a matching `hk_schema:to_map/1` clause for its JSON wire form (records are the compiler-checked internal shape; maps/JSON are produced only at the boundary). | Entity | Key fields | |---|---| | `#agent{}` | `agent_id`, `objective`, `state`, `capabilities`, `action_count`, `started_at_ms`/`updated_at_ms`, `limits`, `result`, `failure_reason` | | `#agent_objective{}` | `objective_id`, `agent_id`, `description`, `success_criteria`, `created_at_ms` | | `#browser_session{}` | `session_id`, `owner_agent_id`, `profile`, `pid`, `status` (`starting\|ready\|busy\|closing\|closed\|crashed`), `tabs`, timestamps | | `#browser_tab{}` | `tab_id`, `session_id`, `url`, `title`, `status` (`opening\|loading\|idle\|closed`), timestamps | | `#search_job{}` | `job_id`, `owner_agent_id`, `query`, `stage` (the 13-value pipeline enum), `max_iterations`, `iteration`, `results`, timestamps | | `#search_result{}` | `result_id`, `job_id`, `url`, `normalized_url`, `title`, `snippet`, `rank`, `source_document_id`, `created_at_ms` | | `#source_document{}` | `document_id`, `url`, `fetched_at_ms`, `content_type`, `extracted_text`, `citation_map` | | `#message{}` | `message_id`, `sender`, `recipient`, `timestamp`, `conversation_id`, `message_type` (`user_to_agent\|agent_to_user\|agent_to_agent\|system`), `payload`, `status` | | `#operation{}` | `operation_id`, `request_id`, `kind`, `status` (`accepted\|in_progress\|succeeded\|failed`), `subject`, `error`, timestamps | | `hk_event:t()` (opaque) | `event_id`, `category`, `subject`, `operation_id`, `data`, `emitted_at_ms` | ## 6. Agent capability, limits, and event flow (worked example) ``` Caller hk_agent_fsm hk_browser_capability hk_agent_toolbox hk_browser_session | next_action(browser, | | | | | navigate, #{url=>U}) | | | | |-------------------------->| check_limits/1 | | | | | is_granted(Caps,{navigate,U})| | | | |----------------------------->| | | | |<---- true/false --------------| | | | | validate_args(navigate, Args)| | | | |----------------------------->| | | | |<---- {ok,Args}/{error,_} -----| | | | | spawn_link(worker) -> emit "agent.action.started" | | | | | dispatch(browser,navigate,Args,AgentId) | | | |-------------------------------------------> | | | | | navigate/3 -> CDP -> emit "browser.navigation.*" | |<-------- {agent_observation, Ref, Outcome} ----------------------------------| | | emit "agent.action.completed"/"failed", -> planning | | |<-------- {ok,Result} / {error,Reason} -------------------| | | ``` Rejection paths never reach `hk_browser_session`: an unauthorized call emits `agent.action.rejected` and replies `{error, not_authorized}}` without spawning a worker; invalid arguments emit the same category with `{error, {invalid_arguments, Reason}}}`; a tripped limit transitions the agent straight to `failed` and replies `{error, {limit_exceeded, Which}}}`. This is the concrete form of "an agent's prompt is not a security boundary" — the check is structural Erlang code sitting between the FSM's `planning`/`ready` states and `hk_agent_toolbox`, not a convention the planner is trusted to honor. ## 7. Event catalog and observability Every subsystem publishes to `hk_event_bus` (a `gen_server` holding an ETS `ordered_set` audit log capped at 20,000 entries, oldest trimmed first). Subscribers register with an optional filter (`category_prefix`, `subject`) and receive `{hk_event, EventMap}` messages; the bus monitors subscribers and removes them automatically on death. `hk_api_events_ws_h` (a `cowboy_websocket` handler) subscribes on connect using the query string as a filter and streams every matching event to the browser in real time; `hk_api_events_recent_h` serves the same log as a plain GET for polling clients. | Subsystem | Categories | |---|---| | Agent | `agent.created`, `agent.objective.assigned`, `agent.action.started`, `agent.action.completed`, `agent.action.failed`, `agent.action.rejected`, `agent.completed`, `agent.failed`, `agent.terminated` | | Browser | `browser.session.created`, `browser.session.closed`, `browser.session.crashed`, `browser.tab.opened`, `browser.tab.open_failed`, `browser.tab.closed`, `browser.navigation.started`, `browser.navigation.completed`, `browser.page.read`, `browser.element.queried`, `browser.action.click`, `browser.action.type`, `browser.action.scroll`, `browser.action.failed`, `browser.screenshot.captured`, `browser.links.extracted` | | Search | `search.started`, `search.query_normalized`, `search.query_generated`, `search.provider_called`, `search.results_received`, `search.urls_deduplicated`, `search.document_retrieved`, `search.document_extracted`, `search.source_ranked`, `search.iteration_retried`, `search.result_synthesized`, `search.completed`, `search.failed` | | Messaging | `message.sent`, `message.received` | `hk_search_job_tests.erl:full_pipeline_runs_in_order/0` asserts these fire in the pipeline's relative order on every run — deterministic orchestration is a tested property, not just a design intent. `hk_health:report/0` computes status fresh on every call (no cached/stale state) from `whereis/1` on all 7 subsystem supervisors plus live counts from the three registries and the event bus's recent log; it backs `GET /api/health` and the frontend dashboard. ## 8. Security model 1. **The agent's plan (whatever produced it — LLM, script, human) is not a trust boundary.** Every tool call is authorized and validated by `hk_agent_fsm` against `hk_browser_capability`/`hk_agent_toolbox` before it can reach Chromium, regardless of what asked for it. 2. **Capabilities are explicit and closed.** An agent's `capabilities` list is set at creation (`hk_agent_fsm:init/1`); there are exactly 14 named browser operations and no implicit "do anything" grant. `navigate` supports domain scoping; other operations are all-or-nothing per agent. 3. **Argument validation is structural, not advisory.** URL scheme allow-listing rejects `javascript:`, `file:`, and other non-navigable schemes outright; selector and text length bounds guard against pathological input regardless of whether the operation is authorized. 4. **No hidden browser execution.** Every operation that reaches `hk_browser_session` — and its outcome — is emitted to the event bus under a `browser.*` category before the caller sees the result. There is no code path that drives Chromium without publishing what it did. 5. **No opaque network activity at the control-plane level.** The API layer, the event bus, and the search pipeline's stage events give an operator a complete, ordered account of what happened and when — see §7. *(What this repository does not implement: sandboxing Chromium's own process beyond `--no-sandbox`/`--disable-gpu` launch flags, or TLS/authentication on the HTTP API — both are deployment-environment concerns flagged as out of scope for this reference implementation, not silently omitted.)* ## 9. API reference Base: `http://:` (default port `8420`, `config/sys.config`). Every request gets a request id (the `x-request-id` header if supplied, else minted); every mutating endpoint runs through `hk_api_util:run_operation/4`, which creates an `#operation{}` (`accepted` -> `succeeded`/`failed`) and returns `{operation_id, request_id, status, result | error}`. | Domain | Routes | |---|---| | Health | `GET /api/health` | | Agents | `GET/POST /api/agents`, `GET /api/agents/:id`, `POST /api/agents/:id/objective`, `POST /api/agents/:id/actions`, `POST /api/agents/:id/complete`, `POST /api/agents/:id/terminate` | | Browser | `GET/POST /api/browser/sessions`, `GET /api/browser/sessions/:id`, `POST /api/browser/sessions/:id/close`, `POST /api/browser/sessions/:id/tabs`, `POST .../tabs/:tab_id/navigate`, `POST .../tabs/:tab_id/actions` | | Search | `GET/POST /api/search`, `GET /api/search/:id`, `GET /api/search/:id/result` | | Messages | `GET/POST /api/messages` | | Events | `GET /api/events/recent`, `WS /api/events/stream` (optional `?category_prefix=...` filter) | Example — create an agent, grant it unscoped `navigate`, and drive one action: ``` curl -s -XPOST localhost:8420/api/agents \ -d '{"capabilities":["navigate","open_tab","read_page"]}' # => {"operation_id":"op_...","status":"succeeded","result":{"agent_id":"agent_...", ...}} curl -s -XPOST localhost:8420/api/agents/agent_.../actions \ -d '{"tool":"browser","op":"create_session","args":{}}' curl -s -XPOST localhost:8420/api/agents/agent_.../actions \ -d '{"tool":"browser","op":"open_tab","args":{"session_id":"sess_...","url":"https://example.com"}}' ``` ## 10. Frontend (HK-VIS) `priv/frontend/index.html` + `js/app.js`: a single-page, dark-themed, dependency-free (no build step) client with the 7 required views — dashboard, agent console, browser sessions, search, messages, event stream, audit log. `connectEventStream()` opens a WebSocket to `/api/events/stream` on load, auto-reconnects on close, and live-updates whichever view is relevant to each incoming event plus a running audit log; the other views poll their REST endpoints on a refresh interval. Served directly by Cowboy (`cowboy_static`) — no separate frontend server or build pipeline. ## 11. Test strategy `rebar3 eunit`: 24 tests across 7 modules, 0 failures. | File | Covers | |---|---| | `hk_agent_fsm_tests.erl` | Deterministic state transitions; unauthorized-call and invalid-argument rejection; limit-exceeded failing the agent with the correct error; crash isolation between sibling agents | | `hk_browser_capability_tests.erl` | Authorization (including domain-scoped `navigate`) and argument validation (rejecting `javascript:` URLs, oversized selectors, missing `type` text) | | `hk_browser_session_tests.erl` | Real integration against a real headless Chromium process via CDP: navigate to a `data:` URL and read it back, extract links. Self-skips (not fails) when no `chromium_executable` is present on disk | | `hk_search_job_tests.erl` | The full 11-stage pipeline against the mock provider, asserting every stage's event fires in pipeline order and the job reaches `done` with synthesized results | | `hk_search_pipeline_tests.erl` | Pure functions: URL normalization, dedup, ranking | | `hk_messaging_portal_tests.erl` | Immediate vs. queued delivery, conversation history ordering | | `hk_supervision_tests.erl` | `rest_for_one` isolation (kill `hk_browser_sup`, verify recovery scope), event delivery ordering, browser-session crash isolation | `test/hk_test_helper.erl` starts the full real application once (idempotently — every test module shares one running instance rather than starting/stopping the singleton HTTP listener per module, which was itself a source of a race fixed during development; see §13) and leaves it running for the test VM's lifetime; isolation between tests comes from unique generated ids, not from process-table resets. *(Planned, not present in this revision: property-based tests, a Dialyzer PLT run as part of CI — `rebar3 dialyzer` is configured in `rebar.config` but was not run to completion in this environment — and a chaos-test suite beyond the two crash-isolation cases above.)* ## 12. Reproducible bootstrap `Brewfile` pins the five dependencies (`erlang`, `rebar3`, `chromium`, `git`, `node` — the last is headroom for future frontend tooling, not a current build requirement) as a project-local Homebrew manifest, not a global install. `bin/bootstrap.sh` runs `brew bundle`, validates each tool is on `PATH`, patches `config/sys.config`'s `chromium_executable` in place with the resolved binary path, then runs `rebar3 get-deps && rebar3 compile && rebar3 eunit`. It is safe to re-run. ## 13. Notable issues found and fixed during development Recorded here because a "produce concrete interfaces... rather than a high-level product pitch" instruction implies the failure history is part of the record, not just the final shape: - **Agent FSM deadlock.** The `ready` state's tool-call clause originally passed `planning` (the state to resume *after* dispatch, i.e. the wrong argument) instead of `executing` as the post-validation transition target, so a successfully-validated call's internal dispatch event landed in `planning/3`, which has no matching clause for it and silently dropped it via the generic reject fallback — the calling process hung until its call timeout. Found via a hanging EUnit test; fixed by matching the pattern already correctly used in `planning/3`'s own tool-call clause. - **Wrong reply type on limit-exceeded.** The original limit-exceeded branch reused `do_fail/3`, which always replies `ok` (it is designed for the external `fail/2` API) — meaning a call refused for exceeding a loop limit incorrectly looked like success to its caller. Fixed with a dedicated branch that replies `{error, {limit_exceeded, Which}}}`. - **Nested-supervisor restart race** and **missing `trap_exit` on `hk_api_listener`** — both described in §3, both exercised by `hk_supervision_tests.erl`, both fixed (`hk_sup_util:start_link_retry/1` and `process_flag(trap_exit, true)` respectively) rather than papered over by loosening the restart-intensity budget alone. - **Chrome's `/json/new` endpoint requires `PUT`, not `GET`**, on current Chromium releases (a CSRF-hardening change that broke the originally-written `hk_chrome_http:new_target/2`, which used `GET`). Fixed and covered by the real-Chromium integration tests in `hk_browser_session_tests.erl`. ## 14. Terminology - **Capability** — one of the 14 named `hk_browser_capability` operations, optionally scoped (currently: `navigate` by allowed domain), grantable to an agent at creation. - **Control plane** — the code path (`hk_agent_fsm` + `hk_browser_capability` + `hk_agent_toolbox`) that authorizes and validates every tool call, as distinct from whatever decided *which* call to make (the "agent" in the LLM sense). - **Operation** — a tracked mutating API call (`#operation{}`), distinct from an **event** (`hk_event:t()`), which is a fire-and-forget observability record; an operation is created and completed by exactly one API request, while an event may be published by any subsystem for any reason, including ones with no API request behind them. - **Stage** — one of the 13 `hk_search_job` `gen_statem` states; the pipeline's current stage and the job's current state are the same value by construction.