docs/PRODUCTION_HARDENING.md

# Production Hardening

This document defines the work and evidence required before Sovereign Event Bus
(SEB) can be operated as a production event system. It is a release-gate
specification, not a statement that the current repository meets these
requirements.

## Status contract

Use the following terms consistently:

| Term | Meaning |
| --- | --- |
| Source present | An interface or implementation exists in the repository |
| Builds | The declared target compiles from a clean checkout in CI |
| Tested | Repeatable automated tests execute and assert behavior |
| Integrated | Two or more components pass a versioned contract test |
| Hardened | Threat controls, recovery, observability, and operational limits are demonstrated |
| Production ready | Every required gate in this document is closed with reviewable evidence |

Documentation, generated reports, proof certificates, hashes, and signatures
must use these terms according to evidence. A command returning exit code zero
is not sufficient when the command skipped a tool, discovered no tests, or ran
a simulated backend.

## Definition of production ready

SEB is production ready only when all of the following are true:

1. One canonical envelope encoding and signature domain is implemented and
   validated across every supported language.
2. Authentication, authorization, replay defense, key lifecycle, and policy
   failure behavior are fail closed.
3. Accepted events, offsets, decisions, and receipts survive the documented
   crash and disaster scenarios.
4. The runtime and native boundary compile, load, negotiate compatible
   versions, and fail without corrupting the host VM.
5. Human approval is authenticated, authorized, durable, idempotent, and linked
   to exactly one eligible event transition.
6. Formal claims build in CI without placeholders and are connected to the
   implementation through explicit refinement or conformance artifacts.
7. A clean source checkout passes the full test matrix, security analysis,
   cross-language vectors, and release build without silently skipped work.
8. Operators can observe saturation, lag, failures, and evidence health; they
   have tested backup, restore, rollback, and incident procedures.
9. Capacity and latency limits are measured on declared hardware and workloads.
10. Release artifacts are reproducible enough to audit, have an SBOM and
    provenance, and are signed by a controlled release identity.

Any unmet item keeps the release in reference, experimental, or preview status.

## Assurance principles

### One event, one canonical representation

Every implementation must consume and produce the same byte-level envelope.
Language-native structs are views over that representation, not independent
definitions. The specification must define:

- field order, field widths, endianness, and alignment;
- string encoding and Unicode normalization;
- timestamp precision, epoch, and allowed clock skew;
- integer overflow behavior;
- absent, empty, and null semantics;
- map ordering and duplicate-key handling;
- maximum envelope, field, and evidence sizes;
- canonical serialization and signature domain separation;
- version negotiation and unknown-field behavior; and
- hash-chain input, receipt input, and stable test vectors.

The present Ada, test-vector, Rust, TypeScript, Python, RPG, and PL/I shapes are
not yet one interoperable wire format.

### Claims follow evidence

The following words require named evidence:

| Claim | Minimum evidence |
| --- | --- |
| Deterministic | Repeat tests across clean processes, hosts, time zones, and supported architectures produce identical decision artifacts |
| Fail closed | Fault injection shows every unavailable dependency denies or queues work according to policy |
| Cryptographically sealed | A reviewed algorithm implementation, controlled keys, canonical input, test vectors, negative tests, and verification at every trust boundary |
| Durable | Crash, power-loss, partial-write, corruption, restore, and retention tests against the selected storage system |
| WORM | Storage-level retention and deletion controls, administrative separation, and provider or device evidence |
| Formally verified | CI-checked theorem set without admitted terms, plus a documented connection between model and deployed implementation |
| Production ready | Every required gate in this document closed |

### No silent success

Build, test, audit, proof, policy, and release commands must fail when a required
tool is absent, a test suite discovers zero tests, a component is skipped, an
artifact is stale, or a required report cannot be produced. Optional work must
be explicitly labeled optional in both console and machine-readable output.

## System and trust boundaries

| Boundary | Trusted input | Untrusted or fallible input | Required control |
| --- | --- | --- | --- |
| Producer to ingress | Registered identity and negotiated schema version | Envelope bytes, timestamps, evidence references | Authentication, size limits, canonical parse, replay check, rate limit |
| Ingress to policy | Parsed immutable event and identity context | Policy bundle, external facts, policy service availability | Signed policy bundle, version pin, timeout, deny-on-error |
| Policy to router | Authorized decision tied to event digest | Route name, adapter metadata | Decision signature, exact event binding, route allowlist |
| Runtime to native kernel | Versioned NIF ABI and bounded buffers | Native code, allocation, file and crypto errors | ABI handshake, input bounds, dirty schedulers or isolation, crash containment |
| Router to adapter | Typed request and capability grant | Remote platform behavior and partial completion | Deadlines, idempotency key, least privilege, reconciliation |
| Review service | Eligible event and policy-required review state | Reviewer browser/session, duplicate actions, stale approval | Strong authentication, RBAC, expiry, single-use decision token |
| Evidence writer | Final transition and prior committed tip | Storage failures, partial writes, administrative action | Atomic append, integrity tree, retention controls, restore verification |
| Build to release | Reviewed source and locked dependencies | Compromised runner, registry, toolchain, secret | Hermetic build, provenance, SBOM, signing, separated approval |

Native code, external policy engines, IBM platform calls, human interfaces,
artifact registries, and evidence storage are separate trust boundaries even
when deployed on the same host.

## Threat model

### Assets

- event intent, authority, and payload confidentiality;
- ordering, offsets, partition ownership, and continuation state;
- policy bundles and authorization decisions;
- signing, verification, and release keys;
- human-review identity and decision records;
- evidence-chain integrity and retention;
- deployment packages, manifests, and provenance; and
- operator credentials, audit access, and recovery material.

### Adversary capabilities

Plan for an adversary who can:

- submit malformed, oversized, duplicated, delayed, or replayed envelopes;
- control payload bytes, identifiers, timestamps, and selected metadata;
- observe or interrupt network traffic;
- crash a process between any two storage operations;
- compromise one service account or adapter credential;
- race approval, cancellation, retry, and timeout operations;
- introduce a malicious dependency or build artifact;
- modify mutable logs or repository metadata; and
- exploit parser differences across language and legacy-platform boundaries.

### Explicit assumptions

Production design must name and test its assumptions about:

- host and hypervisor integrity;
- trusted time and maximum skew;
- entropy availability;
- hardware security modules or managed key services;
- storage consistency and retention guarantees;
- network identity and service discovery;
- administrator separation of duties; and
- supported operating systems, CPU architectures, and legacy platforms.

An assumption is not a control. Each assumption needs an owner, validation
method, and documented behavior when it is violated.

## Required readiness gates

### Gate 1: Canonical protocol

Required controls:

- Publish a versioned binary or canonical text specification.
- Generate language bindings from one machine-readable schema where practical.
- Define strict parsing and rejection behavior.
- Use a dedicated signature context string and versioned hash preimage.
- Define forward/backward compatibility and deprecation windows.
- Reject ambiguous, duplicate, non-canonical, and out-of-range input.

Exit evidence:

- A checked-in conformance corpus with positive and negative vectors.
- Round-trip and byte-equality tests for every supported implementation.
- Property tests for parser/serializer invariants.
- Differential fuzzing across at least two independent implementations.
- A compatibility table for every released schema and runtime version.

### Gate 2: Identity, keys, and cryptography

Required controls:

- Replace placeholder hashes and signatures with reviewed library
  implementations.
- Establish producer, service, reviewer, evidence, and release identities.
- Keep private keys outside source and ordinary configuration files.
- Define issuance, activation, rotation, revocation, expiry, and compromise
  response.
- Bind every signature to protocol version, environment, tenant, event digest,
  transition, and intended verifier.
- Use constant-time verification paths where secret-dependent behavior exists.
- Reject zero, malformed, expired, revoked, unknown, or wrong-purpose keys.

Exit evidence:

- Published key hierarchy and data-flow review.
- Known-answer, negative, mutation, and cross-language cryptographic tests.
- HSM/KMS access policy with administrative separation.
- Rotation and emergency-revocation exercise.
- Independent security review of algorithms, input construction, and key use.

### Gate 3: Replay, idempotency, and ordering

Required controls:

- Assign an immutable event identifier and producer-scoped idempotency key.
- Persist replay decisions for at least the maximum accepted replay window.
- Define ordering per partition and behavior across partitions.
- Make adapter dispatch and commit retry-safe.
- Distinguish duplicate submission, duplicate delivery, and duplicate effect.
- Detect conflicting reuse of the same idempotency key.

Exit evidence:

- Concurrent duplicate and retry tests.
- Restart tests around every transition.
- Delayed-message and clock-skew tests.
- Reconciliation evidence proving one externally visible effect per committed
  idempotency key where the adapter supports it.

### Gate 4: Durable append and recovery

Required controls:

- Select a storage engine with documented atomicity and durability semantics.
- Connect the kernel append path to the WAL or replace it with a defined durable
  store.
- Record event, decision, dispatch intent, effect receipt, and committed offset
  with an explicit transaction model.
- Protect segment metadata and chain tips against torn or reordered writes.
- Define corruption detection, quarantine, repair, and read-only recovery.
- Separate immutable evidence retention from ordinary application logs.

Exit evidence:

- Kill-at-every-write-point tests.
- Power-loss and partial-sector simulation where supported.
- Backup and point-in-time restore exercise.
- Chain verification against corrupted, truncated, reordered, and duplicated
  records.
- Measured recovery point and recovery time under the declared deployment.
- WORM retention evidence from the actual storage control plane if WORM is
  claimed.

### Gate 5: Runtime and native isolation

Required controls:

- Reconcile the Erlang facade and C NIF names, arities, handles, and return
  contracts.
- Load the NIF through a versioned initialization path.
- Bound all native allocations and copied buffers.
- Move blocking file, crypto, and platform operations off normal schedulers.
- Convert native failures to stable Erlang error values without leaking
  resources.
- Decide whether the kernel belongs in a NIF, port, or separate supervised
  service based on crash impact.
- Make shutdown, drain, partition transfer, and restart state machines explicit.

Exit evidence:

- Clean C/Ada build with warnings treated as errors.
- ABI conformance tests and version-mismatch tests.
- Erlang property tests and EUnit tests that are confirmed to execute.
- Native fuzzing under sanitizers and leak detection.
- Host-VM crash-containment and scheduler-latency tests.
- Rolling upgrade and mixed-version compatibility tests.

### Gate 6: Policy enforcement

Required controls:

- Package policy as an immutable, signed, versioned artifact.
- Record the exact policy and fact-set digest used for each decision.
- Apply strict deadlines and deny or durably quarantine on timeout.
- Separate policy administration from event submission.
- Validate external facts for freshness and provenance.
- Make authorization decisions deterministic or record every nondeterministic
  input needed to replay them.

Exit evidence:

- Policy decision vectors, mutation tests, and deny-on-error tests.
- Policy service loss, slow response, malformed output, and stale-fact tests.
- Rollback exercise for a faulty policy bundle.
- Audit showing who approved and deployed each policy version.

### Gate 7: Human review

Required controls:

- Persist review requests and state transitions transactionally.
- Authenticate reviewers with phishing-resistant controls appropriate to risk.
- Authorize by tenant, action, amount, environment, and separation-of-duty rule.
- Bind each decision to one event digest, one policy, one action, and an expiry.
- Make approval, rejection, cancellation, timeout, and reassignment atomic.
- Prevent self-approval where policy forbids it.
- Connect an approved decision to an idempotent commit gateway.
- Store evidence in append-only storage with an independently verifiable digest.

Exit evidence:

- Race tests for two reviewers, timeout versus approval, and cancel versus
  commit.
- Session theft, stale page, replayed decision, and revoked-role tests.
- Accessibility and operator-error review of the decision interface.
- Recovery test proving pending reviews survive restart without duplicate
  commits.

### Gate 8: Formal assurance

Required controls:

- Name the exact theorem set included in a release.
- Remove `sorry`, admitted axioms, trivial crypto definitions, and unused test
  files from the release proof target.
- Model failure, overflow, ordering, replay, and bounded-resource behavior.
- Document which implementation artifacts refine or are generated from the
  model.
- Pin the Lean toolchain and dependency graph.

Exit evidence:

- `lake build` from a clean, network-controlled environment.
- Automated scan rejecting admitted terms in the release theorem closure.
- Proof manifest containing source digests, toolchain digests, theorem names,
  and command output.
- Independent review of model assumptions and implementation correspondence.

Formal verification applies only to the model and theorem closure named in the
evidence. It must not be generalized to adapters, storage, crypto libraries, or
runtime code that the model does not cover.

### Gate 9: Supply chain and release integrity

Required controls:

- Define one root build entry point and fail on missing required toolchains.
- Lock dependencies and verify registry or vendor integrity.
- Pin CI actions, containers, compilers, and proof toolchains by immutable
  identifiers.
- Generate an SBOM for each distributed artifact.
- Produce build provenance and sign releases with a protected identity.
- Scan source, dependencies, containers, and release archives.
- Keep build output and credentials out of source history.

Exit evidence:

- Clean-checkout CI run for the exact release commit.
- Machine-readable test manifest with executed, skipped, and failed counts.
- SBOM, provenance, checksums, vulnerability disposition, and signatures.
- Rebuild comparison from an independent runner.
- Release approval by a principal other than the artifact builder.

### Gate 10: Observability and operations

Required controls:

- Use structured logs with stable event, trace, partition, policy, and release
  identifiers.
- Export queue depth, oldest-event age, decision latency, dispatch latency,
  commit latency, replay rejects, policy failures, native failures, evidence
  verification failures, and recovery progress.
- Trace one event across ingress, policy, routing, review, adapter, and evidence.
- Redact payloads, credentials, signatures, and sensitive evidence metadata.
- Define health, readiness, and dependency status separately.
- Provide runbooks for saturation, corrupted evidence, key compromise, policy
  failure, adapter outage, stuck review, and failed rollout.

Exit evidence:

- Alert tests and dashboard review.
- On-call exercise using only published runbooks.
- Log-redaction tests with representative sensitive payloads.
- Demonstrated event-level diagnosis without direct database mutation.

### Gate 11: Capacity, resilience, and performance

Do not set release targets from placeholder configuration values. Establish
targets from a declared workload and business requirement.

Required test dimensions:

- payload and evidence size distributions;
- number of producers, tenants, partitions, reviewers, and adapters;
- steady-state, burst, backlog catch-up, and hot-partition traffic;
- policy cache hit and miss behavior;
- review-required and automatic paths;
- storage latency, network loss, and dependency degradation;
- key rotation and evidence verification load; and
- supported CPU, memory, filesystem, and platform combinations.

Exit evidence:

- Reproducible benchmark harness and raw results.
- p50/p95/p99 latency with confidence intervals where appropriate.
- Throughput at the declared durability level.
- Saturation point, queue growth behavior, and recovery time.
- Soak test with memory, descriptor, scheduler, and storage trends.
- Fault-injection results and accepted residual risks.

### Gate 12: Release, rollback, and incident response

Required controls:

- Version schema, runtime, policy, proof manifest, and deployment package
  independently.
- Define compatible upgrade and downgrade paths.
- Use staged rollout with automated health and evidence-integrity gates.
- Keep database and evidence migrations backward compatible through rollback.
- Reconcile external adapter effects before retry or rollback.
- Maintain a signed release inventory and supported-version policy.
- Define severity, containment, evidence preservation, notification, and
  post-incident review procedures.

Exit evidence:

- Staging promotion using the exact production artifact.
- Failed-canary rollback exercise.
- Mixed-version and downgrade tests.
- Key-compromise and corrupted-evidence incident exercises.
- Signed release record linking source, tests, proof manifest, SBOM,
  provenance, configuration schema, and operator approval.

## Edge-case test matrix

Every row requires an automated test or a documented platform exercise.

| Condition | Expected system property |
| --- | --- |
| Empty, maximum, and over-limit payload | Accept only documented sizes; reject before expensive work |
| Invalid UTF-8, normalization variants, and embedded nulls | Canonical parser behavior is identical across languages |
| Duplicate map key or reordered fields | Reject ambiguity or serialize to exactly one byte sequence |
| Unknown schema version | Reject or negotiate without guessing |
| Integer boundary and offset wrap | No truncation, sign conversion, panic, or reused offset |
| Timestamp before epoch, far future, or excessive skew | Stable policy result with recorded clock evidence |
| Missing, zero, wrong-purpose, expired, or revoked key | Authentication fails closed |
| Valid signature over the wrong tenant or environment | Domain binding rejects it |
| Replayed event before and after restart | Durable replay policy produces the same rejection |
| Two writers append concurrently | Ordering and chain tip remain valid |
| Crash before WAL flush | Recovery follows the documented acceptance contract |
| Crash after effect but before receipt | Reconciliation avoids a duplicate external effect |
| Truncated or reordered evidence segment | Verification detects and quarantines corruption |
| Policy process absent, slow, or malformed | No unauthorized execution; queue/dead-letter behavior is bounded |
| Review approved twice | One durable transition and one external effect |
| Approval races timeout or cancellation | Exactly one terminal state wins |
| Reviewer loses role after loading request | Commit-time authorization prevents stale approval |
| Adapter returns success after caller timeout | Reconciliation resolves unknown outcome before retry |
| One partition becomes hot | Backpressure is bounded and unrelated partitions remain available |
| Native call blocks or crashes | Scheduler and host-VM failure stay within the declared boundary |
| Disk full, read-only, or high latency | Acceptance and health signals match durability guarantees |
| Log or metrics backend unavailable | Processing follows a documented policy without leaking data |
| Key rotation during in-flight event | Verification uses the correct historical and active key versions |
| Rolling upgrade changes schema | Mixed versions interoperate or rollout stops automatically |
| Identifier is short, long, or multibyte | Diagnostics never panic or split invalid bytes |

## CI release pipeline

A production branch should require these stages in order:

1. Repository policy: formatting, generated-file drift, license consistency,
   secret scan, and forbidden-artifact scan.
2. Component builds: Rust all targets, Erlang release, native kernel/NIF, Lean
   theorem set, code generators, and supported adapters on their real
   platforms.
3. Component tests: unit, property, negative, and mutation-sensitive tests with
   explicit discovered-test counts.
4. Contract tests: canonical vectors and cross-language round trips.
5. Security tests: fuzzing, sanitizers, dependency review, static analysis, and
   authorization/replay suites.
6. Integration tests: policy, routing, review, adapter, evidence, restart, and
   reconciliation.
7. Resilience tests: dependency loss, crash points, corrupted state, resource
   exhaustion, and mixed-version rollout.
8. Assurance: Lean build, admitted-term gate, proof manifest, and traceability
   to implementation or conformance tests.
9. Artifact build: locked inputs, SBOM, provenance, checksums, and signature.
10. Promotion: independent approval, canary, automatic stop conditions, and
    evidence-integrity verification.

Each stage must emit a machine-readable result. A skipped required stage blocks
promotion.

## Operational acceptance

Before the first production deployment, operators must be able to answer:

- What exactly has been accepted if the process dies at this instruction?
- Where is the durable idempotency decision?
- Which policy and key version authorized this event?
- Has an external effect occurred when the adapter response is unknown?
- Can this evidence be deleted or altered by the service administrator?
- How is a corrupt chain isolated without hiding subsequent valid records?
- What happens when every reviewer is unavailable?
- Which metric proves the system is keeping up?
- Which artifact, schema, configuration, and proof manifest are running?
- Can the previous version be restored without reapplying an irreversible
  effect?

If any answer depends on reading source during an incident, the runbook and
telemetry are incomplete.

## Current readiness ledger

| Gate | State | Primary reason |
| --- | --- | --- |
| Canonical protocol | Open | Cross-language layouts and hash inputs differ |
| Identity and cryptography | Open | Placeholder verification, signing, and key material remain |
| Replay and idempotency | Open | No durable cross-component replay store |
| Durable append and recovery | Open | WAL is not connected to the kernel append path |
| Runtime and native isolation | Open | Runtime/NIF interfaces do not compile and integrate as one boundary |
| Policy enforcement | Open | Policy execution and failure behavior are stubbed |
| Human review | Open | Crate compile and durable authorization workflow are incomplete |
| Formal assurance | Open | Lean build fails and proof placeholders remain |
| Supply chain | Open | No required CI/release provenance pipeline |
| Observability and operations | Open | No integrated telemetry, SLOs, or exercised runbooks |
| Capacity and resilience | Open | No reproducible benchmark or soak evidence |
| Release and incident response | Open | No staged release, rollback, or incident evidence |

Closing a gate requires a pull request that links the implementation, tests,
generated evidence, owner review, and any accepted residual risk. Documentation
language may be upgraded only in the same change or a later change that cites
that evidence.