Add ARCHITECTURE.md

0088be9 verified 2 days ago

30 kB

HyperKitty Chromium — Architecture

Status: implemented reference implementation. Every module, supervision relationship, schema, and API route described below exists in this repository and compiles, boots, and passes its test suite (rebar3 eunit: 24 tests, 0 failures, including real headless-Chromium integration tests and chaos tests that kill live processes to verify recovery). Sections are marked (planned) where the specification calls for something this revision intentionally stubs or defers — there is no unmarked aspirational content in this document.

1. Purpose and scope

HyperKitty Chromium is an Erlang/OTP-native browser agent platform: a headless Chromium instance, driven over the Chrome DevTools Protocol (CDP), wrapped in an explicit OTP supervision tree so that browser control, agent orchestration, search, and messaging are separate, independently-supervised, independently-observable subsystems rather than library calls inside one monolithic process. The design principle it exists to demonstrate: an LLM-driven "agent" is a planning function that proposes tool calls; it is never trusted with direct, unchecked access to a browser process. Every tool call crosses an explicit, typed capability boundary — enforced by Erlang code, not by the prompt — before it reaches Chromium.

This document is the architecture reference for engineers building on, operating, or extending this codebase. It assumes familiarity with Erlang/OTP (supervisors, gen_server, gen_statem) and HTTP/WebSocket APIs, but not with this specific codebase.

2. Repository layout

hyperkitty/
  rebar.config              deps, relx release config, profiles, dialyzer/xref
  config/sys.config         application env (chromium path, API port, limits)
  config/vm.args             node name, cookie, scheduler flags
  include/hyperkitty.hrl    canonical entity records (data model)
  src/
    hyperkitty_app.erl       application callback
    hyperkitty_sup.erl       HK-CORE root supervisor
    hk_schema.erl            record -> JSON map projection (wire format)
    hk_id.erl, hk_event.erl  ids, timestamps, the event envelope
    hk_sup_util.erl          shared "wait for the old holder to die" helper
    hk_event_bus.erl / _sup  event bus (pub/sub + audit log)
    hk_messaging_portal.erl  HK-MSG
    hk_cdp_client.erl        one CDP WebSocket connection (via `gun`)
    hk_chrome_http.erl       Chrome's plain-HTTP /json control endpoint
    hk_browser_*.erl         HK-BROWSER: session, capability, registry, supervisors
    hk_agent_*.erl           HK-AGENT: the agent state machine, registry, toolbox
    hk_search_*.erl          HK-SEARCH: the 11-stage pipeline job, provider behaviour
    hk_operation_store.erl   operation_id lifecycle (accepted/succeeded/failed)
    hk_api_*.erl             HK-VIS backend: Cowboy router, listener, REST + WS handlers
    hk_frontend_*.erl        connected-client tracking for the observability UI
    hk_health.erl            aggregate subsystem health
  priv/frontend/            the 7-view observability UI (static HTML/JS, no build step)
  test/                     EUnit suites (see §11)
  Brewfile, bin/bootstrap.sh  reproducible environment setup (see §12)

3. Supervision tree

hyperkitty_sup (rest_for_one, intensity 8/10)
  +-- hk_event_sup (one_for_one)
  |     +-- hk_event_bus                          [gen_server]
  +-- hk_messaging_sup (one_for_one)
  |     +-- hk_messaging_portal                    [gen_server]
  +-- hk_browser_sup (one_for_one)
  |     +-- hk_browser_registry                    [gen_server]  session_id -> pid
  |     +-- hk_browser_session_sup (simple_one_for_one, transient)
  |           +-- hk_browser_session (one per session)  [gen_server]  owns a Chrome OS process
  +-- hk_agent_sup (one_for_one)
  |     +-- hk_agent_registry                       [gen_server]  agent_id -> pid
  |     +-- hk_agent_fsm_sup (simple_one_for_one, transient)
  |           +-- hk_agent_fsm (one per agent)       [gen_statem]
  +-- hk_search_sup (one_for_one)
  |     +-- hk_search_registry                       [gen_server]  job_id -> pid
  |     +-- hk_search_job_sup (simple_one_for_one, temporary)
  |           +-- hk_search_job (one per search)      [gen_statem]
  +-- hk_health                                       [gen_server, worker]
  +-- hk_api_sup (one_for_one)
  |     +-- hk_api_listener                           [gen_server]  owns the Cowboy/Ranch listener
  +-- hk_frontend_sup (one_for_one)
        +-- hk_frontend_client_registry               [gen_server]  connected WS clients

Why rest_for_one at the root. Subsystems are listed in dependency order. If hk_event_sup crashes, everything after it is restarted too, because every later subsystem holds an implicit dependency on the event bus (agents, browser sessions, and search jobs all publish to it; the API layer's WebSocket clients have subscriptions that would otherwise go silently stale against a bus that no longer exists). A crash in hk_browser_sup, by contrast, restarts hk_agent_sup, hk_search_sup, hk_health, hk_api_sup, and hk_frontend_sup, but leaves hk_event_sup and hk_messaging_sup — listed before it — running undisturbed. test/hk_supervision_tests.erl:rest_for_one_isolation/0 kills hk_browser_sup directly and asserts exactly this shape of recovery.

Why each plane is a two-level tree, not one simple_one_for_one supervisor. OTP does not allow mixing a singleton child (the registry) with simple_one_for_one dynamic children (session/agent/job instances) under the same supervisor. Splitting each plane into a one_for_one supervisor wrapping (a) a singleton registry and (b) a simple_one_for_one instance pool means the registry restarting does not restart live sessions/agents/jobs (they re-register on next lookup miss; live processes are untouched), and a crash-restart storm in the instance pool cannot, via hk_browser_sup's own intensity/period, take the registry down with it.

A known, fixed race in this shape, documented for anyone extending it. An abrupt, unmaskable kill of a mid-tree supervisor (as opposed to the ordinary synchronous shutdown protocol a supervisor uses on its own children) does not guarantee its nested-supervisor children have finished their own termination and released their registered name before the grandparent supervisor's restart attempt runs. hk_sup_util:start_link_retry/1 wraps every registered-name start_link/0 in this application: on {error, {already_started, OldPid}} it monitors OldPid, waits for it to actually exit, and retries once, rather than surfacing the transient race as a failed restart. Separately, hk_api_listener calls process_flag(trap_exit, true) in init/1 — without it, the ordinary supervisor shutdown signal (exit(Pid, shutdown)) would kill it directly without ever running terminate/2, leaking the underlying Ranch/Cowboy listener and its registered name on every restart, not only under abrupt kills. Both are exercised by hk_supervision_tests.erl and by hk_browser_session_tests.erl's crash-isolation test.

4. Subsystems

HK-CORE (`hyperkitty_sup`, `hk_event_bus`, `hk_schema`, `hk_id`, `hk_sup_util`)

Boot sequencing, the root supervision policy (§3), the canonical record-to-JSON projection (hk_schema:to_map/1 — the single place a wire representation is produced; internal code passes records, never hand-built maps, until it crosses this boundary), id/timestamp generation, and the event bus described in §7.

HK-BROWSER (`hk_browser_session`, `hk_cdp_client`, `hk_chrome_http`, `hk_browser_capability`, `hk_browser_registry`)

One hk_browser_session gen_server per browser session. init/1 finds a free TCP port, launches a real headless Chromium OS process via erlang:open_port({spawn_executable, ...}, ["--headless=new", "--remote-debugging-port=<port>", "--remote-debugging-address=127.0.0.1", "--user-data-dir=<tmp>", "--no-sandbox", "--disable-gpu", "--disable-dev-shm-usage", "about:blank"]), polls Chrome's plain-HTTP /json/version endpoint until it answers, and registers itself with hk_browser_registry. Tab lifecycle (open_tab/close_tab) goes through Chrome's HTTP control surface (hk_chrome_http); per-tab interaction goes over CDP WebSocket JSON-RPC (hk_cdp_client, built on gun). Navigation correlates against the Page.loadEventFired CDP event via a waiters map. A Chrome OS process crash is detected in handle_info via the port's {exit_status, Status} message and stops the session with an abnormal reason, which hk_browser_session_sup (restart type transient) restarts under the same session_id — the concrete demonstration that one crashed session never takes down another (browser_session_crash_isolation/0) or any other subsystem.

Typed capability interface. hk_browser_capability:operations/0 is the closed list of 14 operations (create_session, close_session, open_tab, close_tab, navigate, back, forward, reload, read_page, query_element, click, type, scroll, screenshot, extract_links). There is no operation outside this list and no unrestricted "just run this JS" escape hatch. is_granted/2 checks an agent's capability list, which supports bare-atom grants (unrestricted for that operation) and scoped grants — currently {navigate, #{allowed_domains => [...]}}}, checked against the URL's host. validate_args/2 performs structural validation independent of authorization: URL scheme allow-list (http/https/about/data), selector length bounds (1–2047 bytes), typed text length bound (≤10000 bytes), scroll delta bounds. Both checks run in hk_agent_fsm (the control plane) before a call ever reaches hk_browser_session — see §8.

(Implementation note, stated plainly per this project's honesty requirement: click/type/query_element/extract_links/scroll are implemented via Runtime.evaluate — JavaScript injected into the page context — rather than the CDP Input domain's synthetic OS-level input events. This is a real, intentional simplification: it is sufficient for programmatic page interaction and observation, but it is not indistinguishable from a physical input device the way Input.dispatchMouseEvent would be. A production deployment that needs that distinction should route these through the Input domain instead.)

HK-AGENT (`hk_agent_fsm`, `hk_agent_registry`, `hk_agent_toolbox`)

One gen_statem per agent, states created -> ready -> planning -> executing -> waiting -> planning -> ... -> completed, with failed and terminated reachable from any non-terminal state. callback_mode() -> state_functions. The planning intelligence itself (what decides which tool to call next) is deliberately external — a planner process, an LLM call, a human operator, or a test drives the loop one step at a time via next_action(Pid, Tool, Op, Args). What the FSM owns is everything that must be enforced regardless of what decided the action: capability authorization (delegates to hk_browser_capability for browser ops), argument validation, the configurable loop limits, action history, and state transitions. next_action/4 validates synchronously in the caller's context (fast) but spawns a linked worker to perform the actual browser/search call, so the FSM process is never blocked on a slow page load — it sits in waiting for a real {agent_observation, Ref, Outcome} message, correlated by reference, with a 45-second per-action timeout.

Loop limits, checked before every tool call (hk_agent_fsm:check_limits/1), merged from default_agent_limits (app env) over any per-agent override: max_actions (default 40), max_execution_ms (default 300000), max_network_requests (default 200), max_search_iterations (default 5). Exceeding a limit transitions the agent to failed and replies {error, {limit_exceeded, Which}}} to the caller — a genuine error, not a silently-accepted ok (this was a bug caught and fixed during development; see §13).

HK-SEARCH (`hk_search_job`, `hk_search_pipeline`, `hk_search_provider`, `hk_search_provider_mock`)

One gen_statem per search job whose states are the pipeline stages themselves — user_query -> query_normalization -> search_planner -> search_provider -> result_normalization -> url_deduplication -> content_retrieval -> content_extraction -> source_ranking -> result_synthesis -> frontend_visibility -> done, with failed reachable from any stage. The job's current state is its current pipeline stage; there is no separate stage field that could drift from the state machine. source_ranking loops back to search_planner (bumping iteration, bounded by max_iterations, default 3) when fewer than 3 ranked results were found — the concrete enforcement point for the search-iteration limit. Every stage transition publishes an hk_event before advancing (§7 has the full catalog). The search backend is a pluggable behaviour (hk_search_provider: plan/2, search/2, fetch/1, extract/2); the bundled hk_search_provider_mock is a deterministic, self-contained reference implementation that makes no real network calls, so the full pipeline is exercisable and testable without external dependencies. A real provider (a live search API, an HTTP fetcher, an HTML/text extractor) plugs in by implementing the same behaviour and setting {search_provider, Module} in sys.config — (planned: no live provider is bundled).

HK-MSG (`hk_messaging_portal`)

A minimal pub/sub message bus for inter-agent and agent/user messaging, independent of the event bus (events are platform telemetry; messages are addressed application payloads). send/5 marks a message delivered if the recipient is currently subscribed, queued otherwise; list_conversation/1 returns history in order; mark_read/1 updates status. Backed by two ETS tables (hk_messages_tab, keyed by message_id; hk_messages_by_conversation_tab, keyed by conversation_id).

HK-VIS (`hk_api_*`, `priv/frontend/`)

The API layer (hk_api_router, hk_api_listener owning the Cowboy/Ranch listener lifecycle, and one handler module per domain) and the static observability frontend (priv/frontend/index.html + js/app.js) — see §9 and §10.

5. Data model

Every entity is a record in include/hyperkitty.hrl, with a matching hk_schema:to_map/1 clause for its JSON wire form (records are the compiler-checked internal shape; maps/JSON are produced only at the boundary).

Entity	Key fields
`#agent{}`	`agent_id`, `objective`, `state`, `capabilities`, `action_count`, `started_at_ms`/`updated_at_ms`, `limits`, `result`, `failure_reason`
`#agent_objective{}`	`objective_id`, `agent_id`, `description`, `success_criteria`, `created_at_ms`
`#browser_session{}`	`session_id`, `owner_agent_id`, `profile`, `pid`, `status` (`starting\|ready\|busy\|closing\|closed\|crashed`), `tabs`, timestamps
`#browser_tab{}`	`tab_id`, `session_id`, `url`, `title`, `status` (`opening\|loading\|idle\|closed`), timestamps
`#search_job{}`	`job_id`, `owner_agent_id`, `query`, `stage` (the 13-value pipeline enum), `max_iterations`, `iteration`, `results`, timestamps
`#search_result{}`	`result_id`, `job_id`, `url`, `normalized_url`, `title`, `snippet`, `rank`, `source_document_id`, `created_at_ms`
`#source_document{}`	`document_id`, `url`, `fetched_at_ms`, `content_type`, `extracted_text`, `citation_map`
`#message{}`	`message_id`, `sender`, `recipient`, `timestamp`, `conversation_id`, `message_type` (`user_to_agent\|agent_to_user\|agent_to_agent\|system`), `payload`, `status`
`#operation{}`	`operation_id`, `request_id`, `kind`, `status` (`accepted\|in_progress\|succeeded\|failed`), `subject`, `error`, timestamps
`hk_event:t()` (opaque)	`event_id`, `category`, `subject`, `operation_id`, `data`, `emitted_at_ms`

6. Agent capability, limits, and event flow (worked example)

Caller                    hk_agent_fsm                 hk_browser_capability   hk_agent_toolbox        hk_browser_session
  |  next_action(browser,      |                              |                       |                       |
  |   navigate, #{url=>U})     |                              |                       |                       |
  |-------------------------->| check_limits/1               |                       |                       |
  |                            | is_granted(Caps,{navigate,U})|                       |                       |
  |                            |----------------------------->|                       |                       |
  |                            |<---- true/false --------------|                      |                       |
  |                            | validate_args(navigate, Args)|                       |                       |
  |                            |----------------------------->|                       |                       |
  |                            |<---- {ok,Args}/{error,_} -----|                      |                       |
  |                            | spawn_link(worker) -> emit "agent.action.started"    |                       |
  |                            |                              |     dispatch(browser,navigate,Args,AgentId)   |
  |                            |                              |------------------------------------------->  |
  |                            |                              |                       |  navigate/3 -> CDP -> emit "browser.navigation.*"
  |                            |<-------- {agent_observation, Ref, Outcome} ----------------------------------|
  |                            | emit "agent.action.completed"/"failed", -> planning  |                       |
  |<-------- {ok,Result} / {error,Reason} -------------------|                       |                       |

Rejection paths never reach hk_browser_session: an unauthorized call emits agent.action.rejected and replies {error, not_authorized}} without spawning a worker; invalid arguments emit the same category with {error, {invalid_arguments, Reason}}}; a tripped limit transitions the agent straight to failed and replies {error, {limit_exceeded, Which}}}. This is the concrete form of "an agent's prompt is not a security boundary" — the check is structural Erlang code sitting between the FSM's planning/ready states and hk_agent_toolbox, not a convention the planner is trusted to honor.

7. Event catalog and observability

Every subsystem publishes to hk_event_bus (a gen_server holding an ETS ordered_set audit log capped at 20,000 entries, oldest trimmed first). Subscribers register with an optional filter (category_prefix, subject) and receive {hk_event, EventMap} messages; the bus monitors subscribers and removes them automatically on death. hk_api_events_ws_h (a cowboy_websocket handler) subscribes on connect using the query string as a filter and streams every matching event to the browser in real time; hk_api_events_recent_h serves the same log as a plain GET for polling clients.

Subsystem	Categories
Agent	`agent.created`, `agent.objective.assigned`, `agent.action.started`, `agent.action.completed`, `agent.action.failed`, `agent.action.rejected`, `agent.completed`, `agent.failed`, `agent.terminated`
Browser	`browser.session.created`, `browser.session.closed`, `browser.session.crashed`, `browser.tab.opened`, `browser.tab.open_failed`, `browser.tab.closed`, `browser.navigation.started`, `browser.navigation.completed`, `browser.page.read`, `browser.element.queried`, `browser.action.click`, `browser.action.type`, `browser.action.scroll`, `browser.action.failed`, `browser.screenshot.captured`, `browser.links.extracted`
Search	`search.started`, `search.query_normalized`, `search.query_generated`, `search.provider_called`, `search.results_received`, `search.urls_deduplicated`, `search.document_retrieved`, `search.document_extracted`, `search.source_ranked`, `search.iteration_retried`, `search.result_synthesized`, `search.completed`, `search.failed`
Messaging	`message.sent`, `message.received`

hk_search_job_tests.erl:full_pipeline_runs_in_order/0 asserts these fire in the pipeline's relative order on every run — deterministic orchestration is a tested property, not just a design intent.

hk_health:report/0 computes status fresh on every call (no cached/stale state) from whereis/1 on all 7 subsystem supervisors plus live counts from the three registries and the event bus's recent log; it backs GET /api/health and the frontend dashboard.

8. Security model

The agent's plan (whatever produced it — LLM, script, human) is not a trust boundary. Every tool call is authorized and validated by hk_agent_fsm against hk_browser_capability/hk_agent_toolbox before it can reach Chromium, regardless of what asked for it.
Capabilities are explicit and closed. An agent's capabilities list is set at creation (hk_agent_fsm:init/1); there are exactly 14 named browser operations and no implicit "do anything" grant. navigate supports domain scoping; other operations are all-or-nothing per agent.
Argument validation is structural, not advisory. URL scheme allow-listing rejects javascript:, file:, and other non-navigable schemes outright; selector and text length bounds guard against pathological input regardless of whether the operation is authorized.
No hidden browser execution. Every operation that reaches hk_browser_session — and its outcome — is emitted to the event bus under a browser.* category before the caller sees the result. There is no code path that drives Chromium without publishing what it did.
No opaque network activity at the control-plane level. The API layer, the event bus, and the search pipeline's stage events give an operator a complete, ordered account of what happened and when — see §7. (What this repository does not implement: sandboxing Chromium's own process beyond --no-sandbox/--disable-gpu launch flags, or TLS/authentication on the HTTP API — both are deployment-environment concerns flagged as out of scope for this reference implementation, not silently omitted.)

9. API reference

Base: http://<host>:<api_port> (default port 8420, config/sys.config). Every request gets a request id (the x-request-id header if supplied, else minted); every mutating endpoint runs through hk_api_util:run_operation/4, which creates an #operation{} (accepted -> succeeded/failed) and returns {operation_id, request_id, status, result | error}.

Domain	Routes
Health	`GET /api/health`
Agents	`GET/POST /api/agents`, `GET /api/agents/:id`, `POST /api/agents/:id/objective`, `POST /api/agents/:id/actions`, `POST /api/agents/:id/complete`, `POST /api/agents/:id/terminate`
Browser	`GET/POST /api/browser/sessions`, `GET /api/browser/sessions/:id`, `POST /api/browser/sessions/:id/close`, `POST /api/browser/sessions/:id/tabs`, `POST .../tabs/:tab_id/navigate`, `POST .../tabs/:tab_id/actions`
Search	`GET/POST /api/search`, `GET /api/search/:id`, `GET /api/search/:id/result`
Messages	`GET/POST /api/messages`
Events	`GET /api/events/recent`, `WS /api/events/stream` (optional `?category_prefix=...` filter)

Example — create an agent, grant it unscoped navigate, and drive one action:

curl -s -XPOST localhost:8420/api/agents \
  -d '{"capabilities":["navigate","open_tab","read_page"]}'
# => {"operation_id":"op_...","status":"succeeded","result":{"agent_id":"agent_...", ...}}

curl -s -XPOST localhost:8420/api/agents/agent_.../actions \
  -d '{"tool":"browser","op":"create_session","args":{}}'
curl -s -XPOST localhost:8420/api/agents/agent_.../actions \
  -d '{"tool":"browser","op":"open_tab","args":{"session_id":"sess_...","url":"https://example.com"}}'

10. Frontend (HK-VIS)

priv/frontend/index.html + js/app.js: a single-page, dark-themed, dependency-free (no build step) client with the 7 required views — dashboard, agent console, browser sessions, search, messages, event stream, audit log. connectEventStream() opens a WebSocket to /api/events/stream on load, auto-reconnects on close, and live-updates whichever view is relevant to each incoming event plus a running audit log; the other views poll their REST endpoints on a refresh interval. Served directly by Cowboy (cowboy_static) — no separate frontend server or build pipeline.

11. Test strategy

rebar3 eunit: 24 tests across 7 modules, 0 failures.

File	Covers
`hk_agent_fsm_tests.erl`	Deterministic state transitions; unauthorized-call and invalid-argument rejection; limit-exceeded failing the agent with the correct error; crash isolation between sibling agents
`hk_browser_capability_tests.erl`	Authorization (including domain-scoped `navigate`) and argument validation (rejecting `javascript:` URLs, oversized selectors, missing `type` text)
`hk_browser_session_tests.erl`	Real integration against a real headless Chromium process via CDP: navigate to a `data:` URL and read it back, extract links. Self-skips (not fails) when no `chromium_executable` is present on disk
`hk_search_job_tests.erl`	The full 11-stage pipeline against the mock provider, asserting every stage's event fires in pipeline order and the job reaches `done` with synthesized results
`hk_search_pipeline_tests.erl`	Pure functions: URL normalization, dedup, ranking
`hk_messaging_portal_tests.erl`	Immediate vs. queued delivery, conversation history ordering
`hk_supervision_tests.erl`	`rest_for_one` isolation (kill `hk_browser_sup`, verify recovery scope), event delivery ordering, browser-session crash isolation

test/hk_test_helper.erl starts the full real application once (idempotently — every test module shares one running instance rather than starting/stopping the singleton HTTP listener per module, which was itself a source of a race fixed during development; see §13) and leaves it running for the test VM's lifetime; isolation between tests comes from unique generated ids, not from process-table resets.

(Planned, not present in this revision: property-based tests, a Dialyzer PLT run as part of CI — rebar3 dialyzer is configured in rebar.config but was not run to completion in this environment — and a chaos-test suite beyond the two crash-isolation cases above.)

12. Reproducible bootstrap

Brewfile pins the five dependencies (erlang, rebar3, chromium, git, node — the last is headroom for future frontend tooling, not a current build requirement) as a project-local Homebrew manifest, not a global install. bin/bootstrap.sh runs brew bundle, validates each tool is on PATH, patches config/sys.config's chromium_executable in place with the resolved binary path, then runs rebar3 get-deps && rebar3 compile && rebar3 eunit. It is safe to re-run.

13. Notable issues found and fixed during development

Recorded here because a "produce concrete interfaces... rather than a high-level product pitch" instruction implies the failure history is part of the record, not just the final shape:

Agent FSM deadlock. The ready state's tool-call clause originally passed planning (the state to resume after dispatch, i.e. the wrong argument) instead of executing as the post-validation transition target, so a successfully-validated call's internal dispatch event landed in planning/3, which has no matching clause for it and silently dropped it via the generic reject fallback — the calling process hung until its call timeout. Found via a hanging EUnit test; fixed by matching the pattern already correctly used in planning/3's own tool-call clause.
Wrong reply type on limit-exceeded. The original limit-exceeded branch reused do_fail/3, which always replies ok (it is designed for the external fail/2 API) — meaning a call refused for exceeding a loop limit incorrectly looked like success to its caller. Fixed with a dedicated branch that replies {error, {limit_exceeded, Which}}}.
Nested-supervisor restart race and missing trap_exit on hk_api_listener — both described in §3, both exercised by hk_supervision_tests.erl, both fixed (hk_sup_util:start_link_retry/1 and process_flag(trap_exit, true) respectively) rather than papered over by loosening the restart-intensity budget alone.
Chrome's /json/new endpoint requires PUT, not GET, on current Chromium releases (a CSRF-hardening change that broke the originally-written hk_chrome_http:new_target/2, which used GET). Fixed and covered by the real-Chromium integration tests in hk_browser_session_tests.erl.

14. Terminology

Capability — one of the 14 named hk_browser_capability operations, optionally scoped (currently: navigate by allowed domain), grantable to an agent at creation.
Control plane — the code path (hk_agent_fsm + hk_browser_capability + hk_agent_toolbox) that authorizes and validates every tool call, as distinct from whatever decided which call to make (the "agent" in the LLM sense).
Operation — a tracked mutating API call (#operation{}), distinct from an event (hk_event:t()), which is a fire-and-forget observability record; an operation is created and completed by exactly one API request, while an event may be published by any subsystem for any reason, including ones with no API request behind them.
Stage — one of the 13 hk_search_job gen_statem states; the pipeline's current stage and the job's current state are the same value by construction.