Threat model

This document states what isolation sandbox provides today, what it intends to provide, and the current status of every boundary. It is written against the code in this repository, not the README. Statuses: mitigated, partial, open.

For the operator-facing summary of how tenant secrets are delivered, isolated across tenants, and encrypted at rest (consolidated with docs/encryption.md and docs/fork-correctness.md), see docs/secrets.md.

Honest summary for the current codebase: do not run untrusted code with this project in production yet. The KVM/Firecracker boundary is real, but several defense-in-depth layers around it are still open, and an extended security review (recorded in this document) found controls that the code did NOT implement. The most serious of those, the husk default path having NO egress enforcement and NO cloud-metadata block, is now IMPLEMENTED and KVM-VERIFIED end to end on a real VM: the husk default path enforces in-pod default-deny egress with an unconditional cloud-metadata block and the threaded per-template allowlist, and the husk-network KVM cluster e2e (test/cluster-e2e/husk-network-e2e.sh, the cluster-husk-network-e2e suite) now PASSES on the Hetzner Talos KVM cluster: all three in-VM enforcement assertions are green inside a real restored VM (metadata-blocked: a connect to 169.254.169.254 fails with exit 1 and no IAM theft; default-deny: a non-allowlisted host fails to connect, exit 1; and allowlist-works: the allowlisted name example.com IS reachable, connect exit 0). The claim reaches Ready and the pool warms a dormant husk pod. This was verified TWICE: once with a node kubelet sysctl allowance in place (proving the datapath), then again with that node change fully reverted (the probe confirmed SysctlForbidden), proving the feature needs NO node prerequisite. The remaining open items below (item 1 is now resolved) are why this is not yet a production posture. No external security review has happened.

Running untrusted code: what is and is NOT warranted today

The microVM (KVM + Firecracker) is the isolation boundary, and the design DOES target running untrusted code. But advertising this project as “safe for untrusted code” is NOT yet warranted. The extended review found controls claimed in this document that the code does not implement; those are corrected per row below. Until the following are all true, do not run untrusted multi-tenant workloads in production on this project, and do not claim it is safe to:

1. Husk egress default-deny plus a cloud-metadata (169.254.169.254) block. This was the top blocker; it is now IMPLEMENTED and KVM-VERIFIED end to end, so it is RESOLVED (mitigated). The husk default path enforces default-deny egress via the in-pod nftables filter the husk-stub programs in the pod’s own netns at activation (internal/husk/netfilter.go, applied in internal/husk/stub.go, reusing internal/netconf and internal/dnsproxy). The cloud metadata endpoints (169.254.169.254, the 169.254.0.0/16 link-local range, and IPv6 fd00:ec2::254) are an UNCONDITIONAL hard drop rendered before any allow (netconf.RenderMetadataBlock), so the allowlist can never reach them and a guest cannot steal the node’s cloud IAM credentials. The per-template egress allowlist is threaded husk-side (huskNotifyNetwork + huskEgressConfig in internal/controller/sandboxclaim_controller.go, carried in husk.ActivateRequest.Egress/Allow), with a per-pod DNS proxy for name entries (internal/dnsproxy), and the husk pod adds exactly the scoped NET_ADMIN capability needed to program the filter. A best-effort Kubernetes NetworkPolicy (internal/controller/husknetworkpolicy.go) is created as defense in depth; HONEST CNI caveat: it enforces only on a CNI that implements NetworkPolicy and it cannot express name-based allows, so the in-pod nft filter is the guarantee that holds with no CNI policy at all (now KVM-verified). VERIFIED: the husk-network KVM cluster e2e (test/cluster-e2e/husk-network-e2e.sh, the cluster-husk-network-e2e suite) PASSES on the Hetzner Talos KVM cluster. The claim reaches Ready, the pool warms a dormant husk pod, and all three in-VM enforcement assertions are green inside a real restored VM: metadata-blocked (connect to 169.254.169.254 fails, exit 1, no IAM theft), default-deny (a non-allowlisted host fails to connect, exit 1), and allowlist-works (the allowlisted name example.com IS reachable, connect exit 0). It was verified TWICE: once with a node kubelet sysctl allowance in place (proving the datapath), then again with that node change fully reverted (the probe confirmed SysctlForbidden), proving the feature needs NO node prerequisite. See section 4 and section 0 surface 5.
2. Fork-correctness fail-closed on ALL engines. The raw-forkd and sandbox-server paths do NOT fail closed on an un-reseeded fork; only the husk path does (section 6, docs/fork-correctness.md row 1).
3. Node-CAS bounds and integrity. The node CAS is tenant-writable, unbounded, and activated with --allow-unverified-snapshots on the fork child path (section 3, W4 row).
4. The secondary hardening below: the raw-forkd privileged DaemonSet remains (the privileged-no-jailer DaemonSet is why raw-forkd is still NOT for untrusted multi-tenant). Four earlier secondary items are now SHIPPED: the husk pod no longer automounts the default SA token, the forkd gRPC surface fails closed without mTLS (opt-in --allow-insecure-grpc for dev), the host-side vsock client applies a per-request read deadline, and the raw-forkd shared-rootfs cross-fork write channel is FIXED via per-fork rootfs CoW (mirroring the husk fix). See the per-row table.
5. The fork-correctness and failure/GC CI suites green (a precondition already stated in CLAUDE.md and ROADMAP).
6. A CVE / patch pipeline for the guest kernel and Firecracker (today CVE watch is a manual process, section “Supply chain”).
7. An EXTERNAL security audit (none has happened; see the review gate).

Must-fix-first set (the items that make the current posture actively unsafe, not merely incomplete): item 1 (husk egress default-open + metadata reachable) is now RESOLVED (mitigated): the husk default path enforces in-pod default-deny + unconditional metadata block + threaded allowlist + per-pod DNS proxy via the scoped NET_ADMIN capability (best-effort NetworkPolicy is defense in depth), and the husk-network KVM cluster e2e (cluster-husk-network-e2e) PASSES on the Hetzner Talos KVM cluster with all three in-VM enforcement assertions green (metadata-blocked, default-deny, allowlist-works), verified twice (once with and once without a node sysctl allowance), proving NO node prerequisite. The remaining must-fix items are item 2 (fork-correctness not fail-closed off the husk path), the raw-forkd privileged-no-jailer DaemonSet (item 4, which is why raw-forkd is NOT for untrusted multi-tenant, even now that its shared-rootfs cross-fork write channel is fixed by per-fork rootfs CoW), and item 3 (node-CAS write/integrity/DoS). The per-row honest status discipline (mitigated / partial / open, with a severity on the review rows) is preserved throughout.

Components and trust

0. Default execution surface: the unprivileged husk pod (issue #18)

Pod-native execution is now the DEFAULT (the controller runs --enable-husk-pods by default; --enable-raw-forkd and --mock select the fork-per-claim fallback). This is a deliberate change to the per-sandbox execution surface. This section is the FULL re-derivation for the unprivileged-stub escape surface (issue #18); it re-derives the surface boundary by boundary, states which CI-proven mechanism backs each claim, and names every residual. It does not contradict the per-row sections below; it reconciles with them (the networking egress row in section 4, the encryption/key-custody row in section 5, the sandbox-API row in section 3) and points at each.

The build-vs-run split is the core idea: a SNAPSHOT is BUILT once per node by a privileged process and RUN many times by unprivileged pods.

• Old default (raw-forkd, now the fallback behind --enable-raw-forkd). A sandbox VM was forked by forkd: a root DaemonSet with /dev/kvm, an explicit capability set (CAP_SYS_ADMIN, SYS_CHROOT, and others; section 3), and a hostPath to the node data dir. The per-sandbox EXECUTION surface WAS that privileged process.
• New default (husk pods). A sandbox VM runs inside an UNPRIVILEGED husk pod: privileged: false, allowPrivilegeEscalation: false, drop ALL capabilities, seccompProfile: RuntimeDefault, /dev/kvm injected by the device plugin (not a hostPath or privilege), and the template snapshot mounted READ-ONLY. It adds exactly one capability, NET_ADMIN, scoped to the pod’s own netns, so the stub can program the in-pod egress firewall (surface 5). The three documented PSA exceptions are runAsNonRoot: false (the device exception), the read-only snapshot hostPath (surfaces 1 and 3 below), and NET_ADMIN (the in-pod firewall, surface 5). On name-based-egress pools ONLY (the controller flag --husk-dns-upstream set) the pod additionally carries a short-lived PRIVILEGED init container (enable-ip-forward) that sets net.ipv4.ip_forward=1 in the shared pod netns and exits before the workload runs (surface 5); it is privileged but one-shot, and the long-lived husk-stub container itself stays unprivileged (NET_ADMIN only).
• forkd-the-builder is the residual host privilege, but it is no longer a privileged container (#352). Building a template snapshot still needs /dev/kvm and the jailer, so forkd remains the privileged BUILDER role on the KVM nodes (and the --enable-raw-forkd fork-per-claim engine). That role now runs as a NON-privileged pod confined to the explicit builder capability set, not privileged: true, and the privileged surface is confined to the BUILD path, run once per node per template, rather than every sandbox execution.
• forkd is NON-privileged (deploy/daemon/daemonset.yaml, #352). forkd is the per-node BUILDER and the raw-forkd fallback: building a snapshot runs the jailer, which needs CAP_SYS_ADMIN (chroot-base mount setup, cgroup, and namespace setup), CAP_CHOWN/CAP_SETUID/CAP_SETGID (hand and drop to the per-VM uid), CAP_MKNOD (in-chroot /dev/kvm and /dev/net/tun nodes), CAP_NET_ADMIN (the build-time placeholder tap), and CAP_DAC_OVERRIDE (reopen the template rootfs.ext4 the builder wrote after the jailer chowned the shared inode to the per-VM uid, #426). forkd runs with EXACTLY that set (drop: [ALL], privileged: false, allowPrivilegeEscalation: false, seccompProfile: RuntimeDefault), and /dev/kvm//dev/net/tun come from the device plugin (mitos.run/kvm), not a privileged hostPath. Status: mitigated. Mitigation: the forkd pod runs only on labelled KVM nodes (mitos.run/kvm), is one-per-node (not one-per-sandbox), is not exposed to tenant traffic, and its build/raw-forkd VMMs run jailed under throwaway uids; husk pods, not forkd, are the tenant execution surface.
• The per-VM Firecracker jailer is deliberately NOT run inside the husk pod. jailer-in-pod was implemented and VERIFIED achievable on real KVM (branch feat/jailer-in-pod, closed PR #96), but it requires the full 9-cap jailer set, Unconfined seccomp, and a writable exec+dev hostPath chroot, which makes EVERY husk pod privileged-class and breaks the PSA-restricted model. It is declined: the jailer isolates many VMs sharing one process (raw-forkd), but in the husk model each pod runs exactly ONE VM, so the pod itself is the per-VM boundary (its own uid, netns, cgroup, PSA-restricted securityContext). Per-VM isolation comes from one-VM-per-unprivileged-pod plus the microVM, not an in-pod jailer.

Unprivileged-stub escape surface (issue #18 re-derivation)

The honest framing up front: the per-sandbox EXECUTION surface is strictly improved (an unprivileged, capability-dropped, restricted-minus-two, pod-netns-governed container instead of a root process), while the INHERENT microVM-host-escape risk (a KVM or host-kernel bug reachable from any /dev/kvm-holder) is UNCHANGED, and forkd-the-builder remains a smaller privileged control-plane surface (run once per node per template, not per sandbox). “Provably better” is argued PER SURFACE below and tallied honestly; it is NOT claimed globally, because the /dev/kvm-and-kernel axis is EQUAL, not better.

Surface 1: GUEST -> HUSK-STUB CONTAINER (the post-VMM-escape blast radius). A guest that breaks out of the microVM (a Firecracker or KVM escape) lands in the process that hosts the VMM. Under the old default that process was forkd: ROOT, with /dev/kvm, an explicit capability set including CAP_SYS_ADMIN, and a hostPath to the node data dir (section 3). Under the husk default that process is the husk stub inside an UNPRIVILEGED pod whose securityContext (internal/controller/huskpod.go, proven in internal/controller/huskpod_test.go at the object level and against the v1.31 PodSecurity admission plugin on the kind-e2e-husk job, slice 4, conformance criterion 4) sets, each control load-bearing:

• privileged: false,
• allowPrivilegeEscalation: false (no setuid path regains privilege),
• capabilities.drop: [ALL], with exactly NET_ADMIN added back so the stub can program the in-pod egress firewall (surface 5) in the pod’s OWN netns; this is scoped to the pod netns (not hostNetwork, not privileged) and cannot reach the host netns or another pod,
• seccompProfile: RuntimeDefault at BOTH the pod and the container level,
• the ONLY host mount is the READ-ONLY snapshot dir plus the read-only kernel file (surface 3),
• the ONLY device is /dev/kvm (and /dev/net/tun) via the device plugin, not a hostPath or privilege (surface 4).

This securityContext satisfies EVERY PSA restricted control except three documented exceptions; the husk pod is kept out of a restricted namespace by EXACTLY those three, each intrinsic to the model: the read-only snapshot hostPath and runAsNonRoot: false (uid 0 so Firecracker can open the injected /dev/kvm without privileged), both recorded as docs/adr/0003-kvm-device-plugin-psa-exception.md, plus NET_ADMIN (forbidden under PSA baseline/restricted “Capabilities”) for the in-pod egress firewall, recorded as docs/adr/0006-husk-netadmin-egress-firewall.md. Justification: the capability is scoped to the pod’s own netns (not hostNetwork, not privileged, allowPrivilegeEscalation: false, seccomp: RuntimeDefault), so it cannot reach the host netns or other pods; the net security change is strongly positive: it trades one scoped capability in an already-unprivileged pod for closing default-open egress and node IAM-credential theft. The husk pod is thus “restricted EXCEPT the read-only snapshot hostPath, runAsNonRoot: false, and NET_ADMIN (for in-pod firewalling)”. A genuinely privileged pod IS rejected in the same namespace (PSA is enforcing), asserted on kind-e2e-husk (slice 4, section 6e of docs/husk-pods.md).

This is the core “provably better” claim, and it is bounded to THIS surface: a guest that escapes the microVM lands with NO root authority, NO Linux capabilities, NO privilege-escalation path, NO broad host filesystem, only a read-only base-image mount and the pod’s own netns and cgroup, instead of forkd’s root with CAP_SYS_ADMIN and host data-dir access. The post-escape blast radius is strictly smaller. What this does NOT change is whether the guest can reach the host kernel through /dev/kvm in the first place (surface 4).

Surface 2: the CONTROL CHANNEL (activation + secret delivery). Activating a husk pod delivers the tenant’s claim-time env, secrets, and the per-sandbox bearer token into the pod. The channel is mTLS, RequireAndVerifyClientCert, authorized to the controller identity ONLY: internal/husk.ServeTLS plus AuthorizeControllerIdentity accept a connection only when the VERIFIED mTLS peer (read from VerifiedChains, never from a merely-presented cert) carries the pki.ControllerName SAN, and a nil TLS config or nil authorize hook is refused (fail-closed: an unauthenticated activate channel that delivers secrets is rejected before any request is read). CI-proven: the KVM husk network-activation phase asserts a WRONG-CA controller cert is REJECTED by the mTLS gate before any secret is read (slice 2, section 6b of docs/husk-pods.md). So an in-cluster actor cannot activate or hijack a husk pod, or inject secrets into one. Residual: a compromised CONTROLLER can activate any husk pod and deliver secrets to it. The controller is the trust anchor here, the same anchor as in the raw-forkd model and the encryption key custody (section 5); this is not a regression, it is the same boundary.

Surface 3: the READ-ONLY SNAPSHOT HOSTPATH. The node template snapshot is mounted READ-ONLY into the husk pod (huskpod.go: the snapshot hostPath and the kernel file are both ReadOnly: true). The husk stub RE-VERIFIES the snapshot ON ACTIVATE, before it loads it, applying the SAME fail-closed gate as raw-forkd: the stub decodes the mounted CAS manifest, binds it to the controller-passed recorded digest (husk.ActivateRequest.ExpectedDigest, fed from the NodeRegistry’s forkd-reported TemplateDigests), re-hashes the loaded mem+vmstate against it (a sha256 digest verify, #9), and runs internal/snapcompat.Check against THIS node’s detected environment (#32), all in internal/husk verifySnapshot (the production verifier, shared with the fork path via the internal/cas chunk/hash primitives so the two cannot drift). Both checks fail closed: a snapshot tampered on the node disk after forkd’s build-time verification, or one incompatible with this node, is REFUSED on the husk path too and never loaded into the VM (proven by internal/husk TestActivateVerifyRefusesTamperedSnapshot and TestActivateVerifyRefusesIncompatibleSnapshot). So the husk path is no longer a verify gap relative to raw-forkd: it is the same digest + snapcompat gate. Residual, stated honestly: all husk pods on a node share the SAME read-only snapshot dir. This IS a shared read-only host mount and is one of the two documented PSA-restricted exceptions (the hostPath, surface 1). It is acceptable because (a) it is READ-ONLY: a husk pod cannot WRITE it, so it cannot tamper with the base image another pod loads; (b) it is integrity-verified and content-addressed, and the husk stub re-checks the digest + compatibility on EACH activate before loading, so a tampered-on-disk or incompatible snapshot is refused at activate time on the husk path; and (c) it is a BASE IMAGE, not tenant data: tenant secrets are delivered post-restore over the control channel (surface 2), never baked into the shared snapshot (section 6). Cross-pod isolation of the snapshot mem/vmstate is the read-only property, not a per-pod copy. The ROOTFS isolation is from a per-activation copy-on-write clone rebound while the guest is FROZEN, not from a read-only template mount. The template dir (which holds rootfs.ext4) is mounted read-write, because Firecracker opens the snapshot’s baked rootfs path with O_RDWR during /snapshot/load (a read-only mount fails the load EROFS, verified on real KVM); isolation does NOT rely on the mount mode. Each activation gets its OWN clone: internal/husk Stub.Prepare reflink-clones <dataDir>/templates/<id>/rootfs .ext4 to a PER-POD file <dataDir>/husk-rootfs/<pod-name>/rootfs.ext4 (the clone path is scoped to the per-pod VM id the controller passes via the downward API metadata.name, so two husk pods sharing the node CoW hostPath never collide on, overwrite, or delete each other’s clone), and Stub.Activate loads the snapshot PAUSED (resume=false), rebinds the baked rootfs drive to that clone with PatchDrive while the guest is frozen, THEN resumes. The template is only OPENED (never written) during the paused load and the drive fd is replaced by PatchDrive before resume, so the guest writes only its own clone, never a single block of the shared template, and concurrent activations of one template never leak one tenant’s filesystem state into another. The clone is removed on pod teardown (Stub.Close). Fully pod-native snapshot delivery (a CAS pull into the pod, removing the shared read-only mem/vmstate hostPath) remains a documented follow-up.

Warm-pool autoscaling (no integrity-gate move)

Demand-driven warm-pool autoscaling changes only WHEN and HOW MANY dormant husk pods the controller creates or deletes. It does NOT change the snapshot integrity gate: every dormant pod still runs the same fail-closed Prepare-time verify (digest + snapcompat) against the read-only mounted CAS manifest before it can be offered for a claim (Surface 3). The autoscaler reads only pod labels (dormant vs claimed) and a process-local claim-arrival timestamp; it trusts no tenant-controlled input, holds no secret, and a compromised husk pod cannot influence the desired count beyond appearing claimed (which only makes the pool create MORE warm capacity, never fewer or unverified pods). Scale-down deletes only surplus DORMANT pods, never a claimed/in-use one. Security surface: unchanged.

A SEPARATE follow-up (a per-node verify cache so the second dormant pod on a node skips the ~680 MiB re-hash) WILL touch the integrity gate and must land with its own threat-model delta; it is intentionally out of scope here.

Husk fork snapshot (live fork on the husk path)

A fork (a Sandbox with source.fromSandbox) of a husk-backed source drives a ForkSnapshot control op against the SOURCE husk pod’s stub over the SAME mTLS channel as activate (authorized to the controller identity only: internal/husk.ServeTLS plus AuthorizeControllerIdentity; the op rides the same op-dispatched channel that delivers secrets on activate). The op carries NO secrets (a fork id and a node-local snapshot path). The stub pauses the running VM, writes a Full Firecracker snapshot to a node hostPath <dataDir>/forks/<fork-id> (read-write only to the source pod that owns the VM; read-only to the child pods on the SAME node), then resumes the source unless pauseSource. The fork snapshot is a LIVE, EPHEMERAL artifact created by a trusted node-local stub and consumed by child stubs on the same node within the same trust boundary; it is NOT content-addressed, so the children activate it with verify disabled (--allow-unverified-snapshots), the same posture a pre-digest pool uses. This is acceptable because the artifact is root-owned, never tenant-writable, and re-hashing would gate on a digest that does not exist for a live fork. The child still runs the full fail-closed RNG/clock reseed handshake (see docs/fork-correctness.md, husk fork children). Per-child independence: each child is its own husk pod + dormant VMM + per-activation rootfs CoW clone, so guest writes never cross between children or back to the source; the children share only the read-only fork snapshot mem+vmstate as a restore image, exactly as warm pods share the template snapshot. Each child mints its OWN bearer token (the source’s token never opens a child). Lifecycle: the fork snapshot is owned by the forking Sandbox and removed by its finalizer (RemoveForkSnapshot op) on delete; the child pods are owner-ref’d to the fork and reaped by Kubernetes GC. Residual: a compromised controller can drive a fork-snapshot of any husk pod it can reach, the same residual already stated for activate (Surface 2).

Husk workspace hydrate/dehydrate (W4 transport on the husk path)

A bound claim’s /workspace is persisted and restored over TWO new control ops on the SAME op-dispatched mTLS channel as activate and fork-snapshot, authorized to the controller identity ONLY (internal/husk.ServeTLS plus AuthorizeControllerIdentity). The controller is not on the node and cannot reach the guest vsock or the node CAS, so it DELEGATES the transfer to the husk-stub that owns both:

• dehydrate-workspace(excludePaths, capturePaths, parentManifestDigest): the stub runs the guest vsock TarDir over /workspace, stores the content-addressed chunks plus manifest into the node CAS (a <dataDir>/cas hostPath mounted read-write into the pod), and returns the manifest digest. It reuses internal/workspace.Dehydrate (the KVM-proven tar round trip), not a reimplementation. When parentManifestDigest is set (a {diff: true} terminate) it ALSO computes the content-hash diff of the new revision against that parent and returns it: the diff is computed on the node from the two MANIFESTS (path -> chunk-digest lists) in the node CAS via internal/workspace.DiffManifests, never the chunk bytes, because the off-node controller cannot read either node-CAS manifest. The diff carries content path NAMES only; no chunk bytes ride the result.
• hydrate-workspace(manifestDigest): the stub reads the manifest plus chunks from the node CAS and UntarDirs them into the guest /workspace.

The ops carry NO secrets: the request is path lists / content-address manifest digests, and the result is a manifest digest plus an optional content-path diff and latency. Secret/credential paths are stripped from the captured tree by the dehydrate exclude list (WorkspaceSecretExcludePaths), so a committed revision is content only. Workspace CONTENT bytes never appear in a log line or an error on either side. The ops FAIL CLOSED: the stub requires an active VM and a configured node CAS (an unset --cas-dir disables them), and the controller delegate refuses on an unreachable pod or a not-OK result rather than committing a revision the node never produced; the controller still owns the WorkspaceRevision commit + head advance. Surface delta vs the prior model: the node CAS is now mounted READ-WRITE into the husk pod (it was read-only manifests before) so the stub can persist a revision. The content-addressed store stays plaintext-content-addressed (or per-workspace encrypted at rest under spec.store.encryptionKeyRef, section 6). Residual: a compromised controller can drive a dehydrate/hydrate of any husk pod it can reach (the same activate/fork residual, Surface 2); a compromised husk pod already had write access to its node <dataDir> subtree (forks dir, rootfs CoW), and the CAS mount widens that to the content store on the same node, bounded to the node’s own data dir.

Surface 4: the DEVICE /dev/kvm. KVM access is injected by the device plugin (cmd/kvm-device-plugin, internal/deviceplugin): the pod requests mitos.run/kvm like any extended resource and the kubelet bind-mounts /dev/kvm (and /dev/net/tun) on Allocate, so the pod sets NO privileged: true and carries NO /dev/kvm hostPath. CI-proven: the kind-e2e job drives the full advertise -> schedule -> inject path with a NON-privileged probe pod (privileged: false, escalation false, drop ALL, read-only rootfs) and kubectl exec confirms /dev/kvm is present inside the container, coming entirely from Allocate, not from any privilege (section 5 of docs/husk-pods.md). Residual, stated honestly: /dev/kvm IS exposed to the pod, so a KVM-or-host-kernel escape from the VMM is STILL a host-escape vector. The device plugin removes the PRIVILEGED requirement, NOT the /dev/kvm attack surface itself; that surface is inherent to ANY Firecracker host and is UNCHANGED between raw-forkd and husk. This is the axis on which the two models are EQUAL, not better. The device-plugin DaemonSet has its own small surface (deploy/device-plugin/daemonset.yaml): it runs as root because the kubelet device-plugins dir is root-owned, but it is privileged: false, allowPrivilegeEscalation: false, ALL capabilities dropped, and readOnlyRootFilesystem: true; its only host access is the kubelet device-plugins dir (to serve and register its socket) and a read-only /dev (to stat /dev/kvm); it creates NO device nodes and starts NO VMs.

Surface 5: the POD NETNS (egress). IMPLEMENTED and KVM-VERIFIED end to end (in-pod default-deny + metadata block proven inside a real restored VM on the husk path; raw-forkd unchanged). In the husk default the VM’s tap lives inside the HUSK POD’s network namespace, so the sandbox’s traffic IS the pod’s traffic. This was the top must-fix-first blocker (default-open egress). The remediation is an in-pod nftables filter the husk-stub programs in the pod’s own netns at activation, and it is now VERIFIED working inside a real restored VM by the husk-network KVM cluster e2e (see the status note below). The datapath is:

• The in-pod nftables filter is the guarantee (CNI-independent). The husk-stub brings up the VM’s tap and installs a default-deny egress chain in the husk pod netns at activation (internal/husk/netfilter.go, applied in internal/husk/stub.go), reusing the raw-forkd dataplane rendering (internal/netconf: per-tap chain, ip saddr anti-spoof) and the controlled DNS proxy (internal/dnsproxy: name-allowlist resolution + IP pinning). Because the tap is in the POD’s netns, this holds regardless of the cluster CNI. It needs exactly NET_ADMIN scoped to the pod netns (surface 1, ADR 0006).
• The cloud-metadata block is unconditional. netconf.RenderMetadataBlock emits a hard drop for 169.254.169.254, the 169.254.0.0/16 link-local range (covers ECS task metadata), and IPv6 fd00:ec2::254 BEFORE any allow rule and regardless of the template policy (even EgressAllow), so the allowlist can never reach the metadata endpoint and a guest cannot steal the node’s cloud IAM credentials.
• The guest agent configures its NIC via rtnetlink, not an ip binary. The guest agent brings up its own eth0 (address, route) through rtnetlink syscalls (internal/guestnet), not by shelling out to an ip binary, so the datapath works on an arbitrary OCI-image rootfs that may carry no networking tools.
• Name-based egress runs an in-pod DNS proxy with a failover upstream list. The in-pod DNS proxy resolves ONLY allowlisted names and pins each resolved IP into the per-tap nftables allow set, forwarding to a comma-separated upstream resolver list tried in failover order. The recommended value is the public pair 1.1.1.1:53,8.8.8.8:53, deliberately NOT cluster DNS, so an untrusted sandbox cannot resolve internal service names. The proxy also REFUSES to pin a resolved address in any non-publicly-routable range (RFC1918, IPv6 ULA, loopback, link-local, multicast, unspecified, RFC6598 CGNAT, deprecated site-local fec0::/10) and strips it from the answer, so an allowlisted name whose authoritative DNS an attacker influences cannot be rebound at internal cluster services or node-local targets (DNS-rebinding-to-internal defense in depth, complementing the unconditional IMDS block). The check also decodes IPv6 addresses that embed an IPv4 target inside the NAT64 well-known prefix (64:ff9b::/96) or the 6to4 prefix (2002::/16) and applies the same policy to the embedded IPv4, so on a DNS64/NAT64 cluster a wrapper such as 64:ff9b::a9fe:a9fe (NAT64 of 169.254.169.254) is refused while a wrapper of a public IPv4 stays allowed. RESIDUAL: a non-default, operator-configured NAT64 prefix (other than the well-known 64:ff9b::/96) is not yet decoded; clusters using a custom NAT64 prefix should set it via the resolver config once that knob lands.
• Guest-to-pod-local traffic is filtered on the INPUT hook (husk path). The per-tap forward chain governs transit egress, but a packet the guest sends to an address LOCAL to the pod netns (the tap gateway, the resolver, the husk-stub sandbox API on :9091 and mTLS control on :9443) is delivered on the kernel input hook, which a forward-only filter never evaluates. So without an input rule a guest could reach those pod-local listeners regardless of egress policy (their own auth gates limited impact, but the egress allowlist offered no protection and any future in-pod listener was exposed). applyEgressFilter now also installs an input-hooked base chain plus a per-tap input chain (netconf.RenderSharedInputTable / RenderSandboxInputChain) that accepts the guest only to the resolver on udp/tcp 53 and drops every other guest-sourced packet to a pod-local address. The base chain policy is accept so non-sandbox input (kubelet probes, the controller’s mTLS dial arriving on the pod uplink) is untouched; isolation is via the per-tap dispatch jump. This is on the HUSK path only: the filter lives in the isolated pod netns, whereas the raw-forkd path runs in the node netns where a node-wide input hook is not added (raw-forkd’s host-local exposure is tracked separately). Renderer and wiring are unit-tested (TestRenderSandboxInputChainBlocksGuestToPodLocal, TestApplyEgressFilterInstallsInputGuard); end-to-end enforcement inside a real restored VM is gated by the husk-network KVM e2e (test/cluster-e2e/husk-network-e2e.sh) which MUST be green before merge.
• The per-template allowlist is threaded husk-side. huskNotifyNetwork delivers the fixed in-pod /30 plus the in-pod resolver, and huskEgressConfig carries the template egress policy + allowlist in the activate request (internal/controller/sandboxclaim_controller.go, husk.ActivateRequest.Egress/Allow). IP:port entries become static chain accepts; name entries are resolved + pinned by the in-pod DNS proxy.
• First-class network-posture knobs (issue #219): block_network, CIDR allowlists, deny-by-default inbound. The per-sandbox NetworkPolicy (api/v1, threaded through husk.ActivateRequest and the proto NetworkConfig) now also expresses, in addition to the egress policy + name allowlist above:
  • blockNetwork (total deny). When set, the per-tap chain drops ALL egress, v4 and v6, with NO accept of any kind (not even established/related): the Modal block_network=True primitive for a sandbox that must never reach the network. It overrides the egress policy and every allowlist. Rendered by netconf.RenderSandboxChainSpec (the block branch emits only the metadata block and the v4/v6 drops).
  • allowCidrs (CIDR egress allowlist). A destination IP inside an allowed block is accepted, v4 saddr-pinned and v6 family-scoped, exactly like the static IP:port accepts. The CIDR list is parsed fail-closed (netconf.ParseCIDRList): a malformed CIDR fails the whole activation rather than silently dropping the rule, so a sandbox never comes up with a partially applied allowlist.
  • Deny-by-default INBOUND (inbound, inboundCidrs). The input-hook chain (above) already dropped all guest-sourced pod-local traffic except DNS; the secure default is now made explicit as inbound: deny. inbound: allow (optionally narrowed to inboundCidrs source blocks) accepts unsolicited inbound to the guest IP for a sandbox that intentionally hosts a listener. Return traffic for the guest’s own egress is always accepted via the forward chain’s established,related rule, so deny-by-default inbound never breaks the guest’s outbound flows. The SECURE DEFAULT for an untrusted sandbox, applied by the SDK and the sandbox-server when no policy is supplied, is deny-by-default in BOTH directions: egress deny (no allows) and inbound deny. The rule rendering of each new dimension is unit-tested in internal/netconf (TestRenderSandboxChainBlockNetwork, TestRenderSandboxChainCIDRAllowlist, TestRenderSandboxInputChainAllowCIDR, and the deny-by-default cases); the real in-VM packet enforcement is KVM-gated by the existing husk-network e2e (the new dimensions reuse the SAME proven datapath: per-tap dispatch, ip saddr anti-spoof, terminal drop).
• Per-sandbox egress byte counter (the #211 metering seam). Each sandbox’s chain carries a named nftables counter (sb_<tap>_egress) incremented on every guest-sourced egress packet at the top of the chain, so the metering pipeline (#211) reads per-sandbox egress bytes by name (netconf.NftReadEgressCounterArgs
  • ParseEgressCounterBytes, surfaced via metering.Sample.EgressBytes). It is a passive counting rule with no verdict, so it never changes enforcement; it is a usage-accounting and abuse-signal source, not a security control.
• SNAT masquerade plus IPv4 forwarding let allowed traffic egress and return. An nftables SNAT masquerade scoped to the guest source address, plus IPv4 forwarding enabled in the pod netns, let allowed traffic reach the internet and return. net.ipv4.ip_forward=1 is set in the pod netns by a SHORT-LIVED PRIVILEGED INIT CONTAINER on the husk pod (name enable-ip-forward), which writes the sysctl in the shared pod netns and exits before the workload runs. This needs NO node/kubelet change. It is added ONLY when name-based egress is configured (controller flag --husk-dns-upstream set). It runs in the privileged-PSA namespace husk already requires (for NET_ADMIN and hostPath), so it adds no new operator prerequisite. Blast radius, stated honestly: it is privileged but SHORT-LIVED (runs once, sets one sysctl, exits), versus a privileged long-lived workload; the long-lived husk-stub container itself stays unprivileged (NET_ADMIN only, via the device plugin for KVM, not privileged: true). It is gated to name-egress pools only.
• A best-effort Kubernetes NetworkPolicy adds defense in depth. The controller now creates one networking.k8s.io/v1 NetworkPolicy per pool selecting mitos.run/husk=true with default-deny egress, a DNS allow, and one egress rule per enforceable IP:port allow, owner-referenced to the pool for GC (internal/controller/husknetworkpolicy.go). HONEST CNI caveat: a NetworkPolicy only enforces on a CNI that implements it, so it is defense in depth ONLY; the in-pod nft filter above is the guarantee that holds with no CNI policy at all.

Status: mitigated (IMPLEMENTED and KVM-VERIFIED end to end). The in-pod default-deny filter, the unconditional metadata block, the threaded per-template allowlist, the per-pod DNS proxy, and the scoped NET_ADMIN are all implemented, the controller creates the best-effort NetworkPolicy, and the husk-network KVM cluster e2e (test/cluster-e2e/husk-network-e2e.sh, the cluster-husk-network-e2e suite) now PASSES on the Hetzner Talos KVM cluster. The claim reaches Ready, the pool warms a dormant husk pod, and all three in-VM enforcement assertions are green inside a real restored VM: metadata-blocked (connect to 169.254.169.254 fails, exit 1, no IAM theft), default-deny (a non-allowlisted host fails to connect, exit 1), and allowlist-works (the allowlisted name example.com IS reachable, connect exit 0). It was verified TWICE: once with a node kubelet sysctl allowance in place (proving the datapath), then again with that node change fully reverted (the probe confirmed SysctlForbidden), proving the feature needs NO node prerequisite (the enable-ip-forward init container sets the only sysctl required, in the pod netns). The raw-forkd nftables egress remains CI-proven in-VM on KVM (section 4) and is unchanged; it gains the same unconditional metadata block because the rendering is shared. The earlier hand-applied allow-all husk-egress NetworkPolicy in .github/workflows/ci.yaml “Conformance 3” proved nothing about restriction and is superseded by the controller-emitted object plus the in-pod filter. Residuals: NET_ADMIN is a documented PSA exception (surface 1, ADR 0006), and on name-egress pools the enable-ip-forward init container is a short-lived privileged surface (one-shot, sets one sysctl, exits; gated to name-egress pools; the long-lived husk-stub container stays unprivileged). The trade is strongly positive: it closes default-open egress and node IAM-credential theft for one scoped capability in an already-unprivileged pod plus a one-shot privileged init step, and it is now KVM-verified end to end.

Surface 6: the IN-POD SANDBOX API (exec and files). After activation the husk stub serves the SAME internal/daemon.SandboxAPI forkd serves, IN the pod, on the sandbox port; the claim’s Status.Endpoint is podIP:sandboxPort. Every exec/files request is gated by the per-sandbox bearer token (32-byte crypto/rand, constant-time compare, fail-closed: a sandbox with no registered token rejects everything). The token is delivered to the stub over the mTLS control channel (surface 2), never logged, never in argv. CI-proven: a tokened HTTP exec reaches the guest over vsock and an UNTOKENED or WRONG-TOKEN request is rejected with the token value absent from host-side logs (slice 2, internal/husk TestActivateServesTokenGatedSandboxAPI, and the KVM network-activation phase over the real endpoint; section 3 row below). Residual: tokens are static per sandbox (no rotation or expiry); anyone with namespace-wide Secret read can take them (section 3).

Serving-workload start is host-trusted only (issue #460). StartWorkload (the serving-workload RPC the build uses to start a long-running app in its own session, captured running in the snapshot) is on the host-trusted Control service (vsock, reached only by forkd during the template build), NOT the public Sandbox service. A tenant exec cannot call it, so it cannot start arbitrary detached processes that escape the exec process-group kill or the per-fork SIGUSR2 reset outside its own budget. The workload command and env are non-secret build-time config (env values are non-secret; secrets are still injected per fork via Configure); the agent logs only the workload session id and env-key count, never argv or values. The per-fork SIGUSR2 reset no longer default-terminates a non-handler process: the broadcast delivers only to confirmed SIGUSR2 handlers (the SigCgt bit in /proc/<pid>/status), so a captured serving workload that installs no handler survives the fork rather than being silently killed (issue #467). This closes a self-DoS/availability hazard on the fork path; it adds no new escape surface (the signal set only shrinks).

Surface 7: the ENCRYPTION KEY (#31 PR2). When --enable-encryption is on, the per-template 256-bit key reaches the node ONLY over the mTLS control channel (the CreateTemplate/Fork gRPC requests; the controller refuses to deliver the key to a node whose connection is not mTLS, and forkd refuses to start encrypted without its TLS flags), is held in node process memory while a container is open, and is NEVER written to the node data disk (section 5: RequestKeyProvider, key-not-on-disk proven by unit and envtest, key-never-logged enforced by grep in CI). On the HUSK path the key reaches FORKD (the builder) ONLY, over the same mTLS gRPC; forkd uses it to open the per-template LUKS container and the snapshot is decrypted BELOW the page cache by forkd’s dm-crypt mount. The husk pod mounts that mount’s PRE-DECRYPTED snapshot bytes read-only and NEVER receives the encryption key: the key does not cross the controller-to-husk mTLS control channel and is never present in the husk pod’s address space. So a compromised husk pod cannot exfiltrate the template key. Residual, stated honestly: the IN-MEMORY KEY WINDOW on the FORKD process. While a container is open the key is necessarily in forkd’s process memory; a root attacker with a node-memory dump of FORKD while a container is open recovers it. Zeroize-on-close is the current mitigation; HSM/envelope custody is the follow-up.

Surface 8: EVICTION and DRAIN (slice 4b). A husk pod is an ordinary pod, so it is subject to drain, eviction, preemption, and delete. A policy/v1 PodDisruptionBudget (<pool>-husk, minAvailable = max(1, Replicas-1)) BOUNDS voluntary disruption to at most one warm slot at a time; a lost husk pod re-pends the claim (Phase Pending, endpoint cleared) and the warm pool self-heals a replacement; a drainPolicy governs an active sandbox (Kill re-pends, Checkpoint snapshots the live VM first where the VMM still runs). CI-proven object-level on kind-e2e-husk (slice 4b, section 6f of docs/husk-pods.md). This is an AVAILABILITY surface, not a new ESCAPE surface: a drained or evicted husk pod is gone, not escalated. The honest availability note vs the old model: raw-forkd’s VMs were not pods and did not feel drains, but they also had no bounded, self-healing disruption story; the husk model trades that for ordinary, self-healing pod disruption with a documented budget. The live-VM Checkpoint-on-drain actually SURVIVING end to end is bare-metal work (it needs the VMM running in the husk pod on a KVM-capable kubelet).

Per-axis tally: old forkd vs husk pod

This compares the per-sandbox EXECUTION surface. forkd-the-builder (the privileged snapshot builder, run once per node per template) is NOT a per-sandbox surface and is discussed separately below the table.

Honest conclusion: on the privilege and capabilities axes the husk model is clearly BETTER. On the host-FS axis it is better for the BASE IMAGE (read-only) but it still holds writable node hostPaths (the CoW dirs and the read-write node CAS), so it is not the read-only-only surface earlier claimed. On the NETWORK axis the husk default now MATCHES opt-in raw-forkd and is KVM-VERIFIED: it enforces an in-pod default-deny egress filter in the pod’s own netns (CNI-independent), plus an unconditional cloud-metadata block and the threaded per-template allowlist (surface 5, section 4), the same posture raw-forkd has with --enable-networking, and it adds the metadata block raw-forkd now also gains because the rendering is shared. This is IMPLEMENTED and KVM-VERIFIED end to end: the husk-network cluster e2e PASSES on the Hetzner Talos KVM cluster with all three in-VM enforcement assertions green (metadata-blocked, default-deny, allowlist-works), verified twice (once with and once without a node sysctl allowance, proving NO node prerequisite). It costs one scoped capability (NET_ADMIN in the pod netns, ADR 0006) plus, on name-egress pools only, a short-lived privileged enable-ip-forward init container (one-shot, sets one sysctl, exits; the long-lived husk-stub container stays unprivileged). The residuals named (shared read-only snapshot mount, in-memory key window) stand. On the inherent /dev/kvm-and-kernel axis the two are EQUAL: a KVM or host-kernel bug reachable from a /dev/kvm-holder is the same risk in both models, and the device plugin removes the privileged requirement, NOT that attack surface. The per-sandbox EXECUTION surface is therefore IMPROVED on privilege and capabilities, the inherent microVM-host-escape risk is UNCHANGED, and the husk EGRESS surface now MATCHES raw-forkd (in-pod default-deny + metadata block) and is KVM-verified end to end. Separately, forkd-the-builder remains a PRIVILEGED control-plane surface (root, CAP_SYS_ADMIN, /dev/kvm, the jailer), but it is SMALLER than the old per-sandbox surface: it runs the BUILD path once per node per template, not on every sandbox execution, so the privileged surface is confined to the build path and amortized across all sandboxes a template serves. Removing forkd’s privilege entirely (a builder redesign) is out of scope.

RESOLVED gaps the review surfaced (were must-fix-first; now mitigated):

• HUSK EGRESS is now IMPLEMENTED and KVM-VERIFIED end to end (was the top blocker): the husk-stub enforces an in-pod default-deny egress filter in the pod’s own netns with an unconditional cloud-metadata block and the threaded per-template allowlist, plus a best-effort controller-emitted NetworkPolicy (surface 5, section 4). FIX/PROOF: the husk-network KVM cluster e2e (test/cluster-e2e/husk-network-e2e.sh, cluster-husk-network-e2e) PASSES on the Hetzner Talos KVM cluster with all three in-VM enforcement assertions green (metadata-blocked, default-deny, allowlist-works), the claim reaches Ready, and the feature was verified twice (once with and once without a node sysctl allowance), proving NO node prerequisite. The warm-pool over-create that blocked activation (below) is fixed by the husk pod readiness probe.
• DEHYDRATE-ON-DELETE FINALIZER HOT-LOOP is FIXED. A Sandbox bound to an already-deleted Workspace now treats the missing workspace as a TERMINAL no-op: it drops the finalizer and the sandbox is garbage-collected instead of retrying the dehydrate step forever. Proven by envtest.
• HUSK WARM POOL OVER-CREATE is FIXED by the husk pod readiness probe gating dormant-Ready on the control listener: a dormant pod is only counted Ready once its control listener is up, so the pool no longer creates an unbounded number of never-Ready pods toward a deficit it cannot close.

The husk default-SA-token automount, the fail-open gRPC default, and the missing host-side vsock read deadline that this list previously carried are now SHIPPED (mitigated); see the per-row table entries above for the mechanism and the test.

Accepted residuals (tracked, see ROADMAP W1 #18):

• the SHARED READ-ONLY SNAPSHOT MOUNT across husk pods on a node (read-only, integrity-verified, non-tenant base image; fully pod-native CAS delivery is the follow-up);
• the WRITABLE node hostPaths a husk pod holds (per-pod CoW dirs and the read-write node CAS, bounded to the node data dir; see the W4 CAS row in section 3 for the unbounded-disk and cross-tenant-destruction concerns);
• the /dev/kvm INHERENT host-escape surface (unchanged from raw-forkd; inherent to any Firecracker host);
• the IN-MEMORY ENC-KEY WINDOW while a container is open (HSM custody is the follow-up, #31);
• the FORKD-BUILDER PRIVILEGE (it stays the privileged builder; a builder redesign is out of scope);
• the live-Checkpoint-on-drain survival, proven only on a KVM-capable kubelet (a bare-metal reference node, #16).

The device-plugin surface itself is in section 3; the per-mode networking reconciliation is in section 4; the encryption-key custody is in section 5.

1. Guest → host escape

The primary boundary is KVM hardware virtualization via Firecracker.

2. Guest → guest

3. Sandbox / forkd → cluster

forkd is the highest-value HOST component: uid 0 in a NON-privileged container with an explicit, minimal capability set, /dev/kvm + /dev/net/tun via the device plugin (not a privileged hostPath), and a hostPath to /var/lib/mitos, on every KVM node. Since #352 it is the privileged BUILDER concentrated to that audited cap set, not a privileged-and-long-lived daemon; its build/raw-forkd VMMs run jailed under throwaway uids.

Code interpreter (run_code) surface

run_code (the Connect sandbox.v1.Sandbox.RunCode RPC; the legacy POST /v1/run_code/stream JSON route was removed in #358) runs tenant code in a long-lived Python kernel (/opt/mitos/kernel_driver.py, ipykernel) that the guest agent spawns lazily and keeps for the VM lifetime. Status: mitigated for host isolation, partial for in-guest blast radius.

• The kernel runs INSIDE the untrusted guest VM, at the same privilege as any exec child. It crosses no new host boundary: the KVM/Firecracker boundary and the unprivileged husk pod (section 0) bound it exactly as they bound exec. The host treats kernel output (frames) as data only.
• It is a PERSISTENT interpreter holding tenant state across calls. Within one sandbox this is by design (statefulness is the feature); across tenants there is no sharing because each sandbox is its own VM.
• Fork inheritance: a forked VM inherits the live kernel and its namespace (it is part of the snapshot). This is the same fork-shared-state surface as any in-VM process; the RNG/clock caveat is in docs/fork-correctness.md.
• Optionality: a base image without the kernel returns a KernelUnavailable error frame (exit 127); no new attack surface exists on minimal images, and plain exec is unaffected.
• The kernel driver reads only newline-delimited JSON {id, code} on its stdin from the agent, never from the network, so there is no kernel-protocol exposure beyond the existing vsock/HTTP authz (per-sandbox bearer token).

Residual (open): the kernel inherits the configured env+secrets exactly as exec does; secret values are never logged in frames (only stdout the tenant itself prints, which is the tenant’s own choice). No CPU/memory cgroup bounds the kernel beyond the VM’s own limits.

Interactive terminal surface (GET /sandbox.v1.Sandbox/Exec over WebSocket)

The interactive terminal is the sandbox.v1.Sandbox.Exec bidi RPC carried over a WebSocket upgrade (subprotocol connect.sandbox.v1, handleExecWS, internal/daemon/exec_ws.go); it bridges to a dedicated vsock gRPC Exec stream on which the guest agent allocates a pseudo-terminal and starts /bin/sh as a session leader. This is a LIVE interactive shell into the VM: stdin frames flow client to guest and stdout/stderr frames flow back, for as long as the connection is held. The bespoke /v1/pty JSON WebSocket wire was removed in #358; the row above (GET /sandbox.v1.Sandbox/Exec) is the authoritative surface entry. The threat properties carry over unchanged:

• Authentication. The upgrade is a bodyless GET, so it does NOT pass through the JSON-body-peeking requireBearer middleware. handleExecWS authenticates the upgrade itself (ptyAuth): the sandbox id comes from the ?sandbox= query parameter and the token from the Authorization: Bearer header, compared in constant time. Semantics match requireBearer: a sandbox with no registered token fails closed with 401 (unless AllowTokenless, the standalone sandbox-server only); a missing, malformed, or mismatched token is 401. Token values are never logged. Status: mitigated (same per-sandbox token custody as exec/files).
• Process containment. The shell runs in its own session and process group; a host hangup (WebSocket close, ctx cancel, or vsock drop) cancels the guest Exec stream, which the guest turns into a process-group SIGKILL, so a terminal session and its children do not outlive the connection. Status: mitigated.
• No command injection at the edge. handleExecWS does NOT take the shell command from the client for the interactive case; the guest defaults to /bin/sh. Only the PTY window size crosses from the open frame. The shell, of course, can run any command the guest user can, exactly like exec; the terminal does not widen the in-guest capability set, only the interactivity of access. Status: accepted by design, identical to the exec surface.
• Concurrent-session cap. The terminal holds a dedicated vsock connection for the session lifetime, so it counts against the SAME per-sandbox concurrent-stream ceiling as exec and run_code (production-blocker #2): handleExecWS reserves a slot via ExecPTY; because the open frame arrives after the upgrade, a session over the --max-streams-per-sandbox ceiling is refused with a terminal error exit frame plus a policy-violation close (not a pre-upgrade 429); existing sessions are never killed. So a tenant cannot open unbounded terminals to exhaust host connections and goroutines. Status: mitigated (tested in TestExecWSStreamCapRejected).
• Residual. The terminal inherits the exec surface’s residuals (the in-guest user is unconfined within the VM; isolation is the microVM boundary, not in-guest privilege separation). It adds no new host-side privilege. The auditor records an exec_ws op with the pty flag only, never terminal contents.

4. Sandbox → network

See docs/networking.md for the full design (tap-per-sandbox, nftables dispatch model, per-fork identity). Networking is opt-in: with forkd’s --enable-networking off, restored VMs have no NIC and egress is denied by absence. With it on, each fork gets its own tap and a host-side default-deny egress ruleset.

PER-MODE ENFORCEMENT (reconciled with the husk default, section 0 surface 5). The nftables egress dataplane described in the rows below is the SAME rendering (internal/netconf) used in both modes. In RAW-FORKD mode (--enable-raw-forkd) with --enable-networking on, it is applied host-side in forkd’s netns; those rows are CI-proven in-VM on KVM. On the HUSK DEFAULT path (the shipped default) the SAME rendering is applied IN-POD by the husk-stub in the pod’s OWN netns at activation (internal/husk/netfilter.go, internal/husk/stub.go), so a husk sandbox gets a CNI-independent default-deny egress chain, the unconditional cloud-metadata block, and the threaded per-template allowlist; 169.254.169.254 (and the v4 link-local range and IPv6 IMDS) is dropped before any allow. Name-based egress runs the in-pod DNS proxy that resolves only allowlisted names and pins each resolved IP into the per-tap allow set, forwarding to a comma-separated failover upstream list (recommended 1.1.1.1:53,8.8.8.8:53, NOT cluster DNS); allowed traffic egresses via an nftables SNAT masquerade scoped to the guest source plus IPv4 forwarding, the latter enabled by a short-lived privileged enable-ip-forward init container (name-egress pools only, no node prerequisite). The controller additionally emits a best-effort networking.k8s.io/v1 NetworkPolicy selecting mitos.run/husk=true (internal/controller/husknetworkpolicy.go); HONEST CNI caveat: a NetworkPolicy enforces only on a CNI that implements it and cannot express name-based allows, so the in-pod nft filter is the guarantee and the NetworkPolicy is defense in depth. The earlier CI step that hand-applied an allow-all (0.0.0.0/0) policy proved no restriction and is superseded. This in-pod enforcement is IMPLEMENTED and KVM-VERIFIED end to end: the husk-network KVM cluster e2e (test/cluster-e2e/husk-network-e2e.sh, cluster-husk-network-e2e) PASSES on the Hetzner Talos KVM cluster, the claim reaches Ready, and all three in-VM enforcement assertions are green inside a real restored VM (metadata-blocked, default-deny, allowlist-works), verified twice (with and without a node sysctl allowance), proving NO node prerequisite. The raw-forkd in-VM KVM proof is unchanged. See docs/husk-pods.md section 6d.

5. Snapshot integrity and supply chain

Snapshots are executable memory images; loading one is equivalent to running arbitrary code at sandbox privilege.

Device passthrough (GPU/VFIO) and the GPU node tier (issue #221)

GPU support attaches a physical device into the microVM via VFIO passthrough. This MOVES the security surface: a passthrough device is a new, privileged path between the guest and host hardware, and a GPU node is a DISTINCT, higher-trust tier from a CPU-only node. NONE of this is implemented or hardware-validated here; the device-attach and fork-with-device code is a documented, HARDWARE-GATED follow-up (see docs/platforms/gpu.md). What ships today is the scheduling/metering/spec plumbing only (GPU node selection in the registry, resources.gpu on the template, GPU-seconds metering). The rows below state the surface so it is not silently expanded when the device path lands.

Isolation tiers: not every node is the same assurance (issue #40)

Mitos’s default and strongest isolation is the hardware-virtualization microVM (KVM + Firecracker). Two WEAKER mechanisms exist as run-anywhere or fallback tiers, and the threat model’s job is to make sure they are NEVER silently treated as equivalent to hardware virt:

• PVM (Firecracker on the PVM kernel module, pagetable-based virtual machine, Ant/Alibaba): runs guests in ring 3 via pagetable switching with NO VMX/SVM, so Firecracker runs on a plain cloud VPS that exposes no /dev/kvm nested virt. Ring-3 pagetable isolation is WEAKER than hardware virt: there is no VT-x/AMD-V root-mode boundary, the host kernel runs out-of-tree patches, and the guest shares more of the host’s privilege machinery. PVM is EVALUATED, NOT ADOPTED (docs/platforms/pvm-evaluation.md); this row exists so that if a PVM tier is ever enabled it is a documented lower-assurance tier from day one.
• gVisor (userspace-kernel syscall interposition): not a VM at all. The software-isolation tier, relevant to the gVisor fallback and to ADR 0005 (raw forkd is not multi-tenant). Weaker still than a VM boundary.

The MITIGATION that makes a mixed fleet safe is a NODE isolation tier plus a per-pool/template assurance FLOOR, both shipped here:

6. Secrets

• Cross-namespace secret replication. The controller projects ONLY mitos-ca (ca.crt) from its namespace into every pool namespace where it creates husk pods (ReplicateHuskSecrets). The CA private key (ca.key) is never copied, and the forkd server leaf is no longer copied either: the husk control channel serves a per-namespace leaf (mitos-husk-tls, issued in-namespace by EnsureHuskTLS), so a pool namespace holds only the public CA cert and its own husk server key, never the CA signing key and never the shared forkd key. Scope: the cluster-wide secrets grant is the enabling privilege for writing ca.crt into pool namespaces; a namespaced grant scoped to pool namespaces is a follow-up once pool namespaces are enumerable at install time. This pool-namespace-holds-no-CA-signing-key property holds for MULTI-namespace deployments. In the single-namespace self-host topology (a pool created in the controller’s own namespace, now supported so husk pods start, #414) the pool IS the controller namespace and therefore naturally holds ca.key: the cross-namespace isolation boundary does not exist there because there is no second namespace. This is an accepted simplicity/isolation tradeoff for single-tenant self-host; multi-tenant deployments MUST give each pool (tenant) its own namespace distinct from the controller’s, where the property above applies.
• Husk activation target authenticity (warm-slot impersonation). Activation delivers the claim’s resolved secrets plus the per-sandbox bearer token to the husk pod’s self-reported PodIP over the mTLS control channel. So if pod selection trusted only the husk LABELS (which any principal with pod-create in the pool namespace can set), a tenant could stand up a decoy pod pointing at their own listener and receive another claim’s secrets and token. Mitigated by two independent, composing controls:
  1. PROVENANCE: selectDormantHuskPod requires the controller owner reference reconcileHuskPods stamps (a controller reference of Kind SandboxPool naming the pool with BlockOwnerDeletion=true), so only pods the controller actually created are activation targets. The forgery barrier is BlockOwnerDeletion: the OwnerReferencesPermissionEnforcement admission plugin refuses to let a tenant set it referencing a pool whose finalizers subresource they cannot update.
  2. PER-NAMESPACE IDENTITY (now implemented): the husk control channel no longer uses the shared forkd server key. Each pool namespace gets its OWN server leaf, husk.<namespace>.mitos with a distinct key, issued by EnsureHuskTLS into a namespace-local mitos-husk-tls Secret; the controller pins husk.<namespace>.mitos when dialing a pod in that namespace (HuskDialTLSConfig), and ReplicateHuskSecrets projects only ca.crt into a pool namespace, never the forkd server key. So even a per-namespace key leaked in namespace A cannot impersonate a husk in namespace B (the controller dials B pinning husk.B.mitos, which A’s leaf does not satisfy); within A, impersonation yields only A’s own tenant secrets. This does not depend on the OwnerReferencesPermissionEnforcement plugin, so it is the durable control on clusters where control (1) is weaker. End-to-end verified by the husk KVM e2e (the controller-pins-vs-husk-serves handshake). ROTATION: EnsureHuskTLS now reissues the per-namespace leaf when it is within HuskCertRenewBefore (30 days) of NotAfter, so a long-lived pool rotates its husk serving cert ahead of expiry across reconciles rather than serving an aging leaf until the control channel breaks.

7. Multi-tenancy statement

What a namespace boundary buys you today: RBAC on the CRDs, and nothing else. Pools, claims, and forks are namespace-scoped objects, but:

• Snapshots on a node are a flat directory shared by all tenants; no per-namespace separation, no enforcement that a claim only forks snapshots its namespace published. open
• VMs of different namespaces share nodes, host kernel, and forkd BY DEFAULT. open
• A pool MAY opt into node separation via .spec.placement (PoolPlacement: a nodeSelector ANDed onto the husk pods plus tolerations for tainted dedicated nodes). The controller pins the pool’s husk pods to the matching nodes AND constrains the template snapshot build/distribution to that same node set (placementFilter / createSnapshotsOnNodes, internal/controller/sandboxpool_controller.go, issue #172), so a placed pool’s VMs and their snapshot never land on a node outside its dedicated set. This is SCHEDULING-enforced separation (Kubernetes nodeSelector / taints), NOT a hardware or hypervisor isolation guarantee: it depends on the operator labeling and tainting the dedicated nodes and not co-scheduling other tenants there. Node-side taint enforcement and per-tenant node-pool provisioning remain the operator’s responsibility. partial

Until the above are closed, treat the whole cluster as one trust domain unless an operator has provisioned dedicated, tainted node pools and placed each tenant’s pools onto them. This posture is recorded as a residual decision in docs/adr/0004-node-flat-snapshot-trust-domain.md.

7a. Per-org namespace tenancy boundary (issue #288)

The hosted offering hardens the namespace boundary above into a declarative per-org isolation namespace. An Org CR (cluster-scoped; its name is the org id) is reconciled into mitos-org-<id> by the OrgReconciler (internal/controller/org_controller.go), gated behind --enable-org-tenancy (default OFF, so a self-host single-tenant install is unaffected). Each org’s pools, warm husk pods, claims, sandboxes, and secrets live in its own namespace. The reconciler is self-healing: it watches the objects it owns, so a manually deleted or drifted stack object enqueues the Org and is recreated; org deletion cascades through owner references (the cluster-scoped Org owns the cluster-scoped Namespace, a valid owner edge).

What isolates two orgs: the separate namespace + the per-org default-deny NetworkPolicy + the per-org ResourceQuota ceiling + the microVM. No single one of these is the whole boundary; the microVM remains the in-pod isolation guarantee, the namespace + NetworkPolicy is the cross-tenant control-plane boundary, and the ResourceQuota is the abuse-control ceiling.

PRODUCTION GATE: per the #208 sequencing, public self-serve untrusted multi-tenancy does NOT switch on until fork-correctness (#1), failure/GC (#163), and the external security review (#194) are green and this per-boundary checklist passes. Until then, run in waitlist / design-partner mode.

7b. Customer front door: accounts, orgs, and the public API gateway (issue #210)

The hosted offering adds a NEW public attack surface layered ABOVE the internal mTLS and per-sandbox token plane: an internet-facing gateway (cmd/gateway, internal/saas), customer-presented API keys, external accounts and organizations, and cross-tenant (org) isolation. This surface does NOT replace the internal plane; it sits in front of it and forwards org-scoped, authenticated requests to the control plane. Design: docs/saas/accounts-gateway.md.

PRODUCTION GATE: this front door is NOT cleared for production tenants until the external security review (issue #194) covers it. Until then it is a development and review artifact only.

7c. Expose URL ingress: per-sandbox port exposure (issues #126, #230)

Expose URLs add a NEW public ingress that maps a single-label hostname, <label>.<expose-domain>, to a port INSIDE a running sandbox through the Mitos expose edge proxy (cmd/preview-proxy, internal/preview). <label> is an opaque routing key; the operator configures <expose-domain>. This is the E2B get_host(port) / Daytona signed-preview-URL surface. It is a deliberate, caller-initiated hole in the otherwise deny-by-default inbound posture (section 4): a tenant who mints an expose URL is asking for that port to be reachable from the internet for the URL’s lifetime. Design: docs/preview-urls.md.

PRODUCTION GATE: this ingress is NOT cleared for production tenants until the external security review (issue #194) covers the public attack surface it adds and the #213 abuse-control envelope lands. Until then it is a development and review artifact only. Slice 3 ships TLS termination directly at the Go proxy with a wildcard cert and hybrid post-quantum key exchange; the proxy is now a deployed public ingress surface gated by expose.enabled (default false) in the Helm chart.

Architecture as of slice 2b: the proxy resolves the <label> in the incoming hostname to a route (NodeEndpoint, SandboxID, Port, Token) via the route table, verifies the signed expose token, and reverse-proxies to the forkd expose handler at http://<NodeEndpoint>/v1/sandboxes/<id>/expose/<port>/ (the slice-1 forkd surface, section 3 row ANY /v1/sandboxes/{id}/expose/{port}/). The upstream is NEVER derived from request input; it comes only from the route table. The per-sandbox bearer crosses the cluster network to forkd :9091 in CLEARTEXT: this is the SAME trust model as the existing SDK path (the SDK reaches forkd :9091 in cleartext with a per-sandbox bearer); the cluster network is the trust boundary. In-cluster TLS for forkd :9091 is a recorded follow-up.

The route table is populated by the controller ExposeRouteReconciler (slice 2b). On each reconcile it lists all Sandboxes, selects those that are Ready (Status.Phase==Ready) with spec.expose set and a non-empty Status.Endpoint, reads each one’s per-sandbox bearer from its <name>-sandbox-token Secret (key token, namespace-scoped so two same-named Sandboxes in different org namespaces never collide), builds the full route set, and POSTs it to POST /internal/routes authenticated by the shared admin bearer. The POST body carries per-sandbox bearer tokens over the in-cluster hop in CLEARTEXT, the same cluster-network trust model as the forkd :9091 hop. The admin token is sourced from EXPOSE_PROXY_ADMIN_TOKEN (environment variable, never argv, never logged). The reconciler is disabled by default; it activates only when --expose-proxy-admin-url is set. This completes the end-to-end path: a Sandbox with spec.expose set and Status.Phase==Ready is reachable at <label>.<expose-domain> with routes kept current by the reconciler.

Headless-browser CDP exposure (issue #314)

The computer-use template (images/computer-use) exposes a Chrome DevTools Protocol (CDP) endpoint over the section-7c expose ingress. CDP is FULL remote control of the browser: a CDP client can navigate file://, read local files, and execute arbitrary JavaScript in pages. The exposed CDP endpoint is therefore treated as a privileged control surface and inherits the section-7c PRODUCTION GATE: it is not cleared for untrusted tenants until the #194 external review and the #213 abuse-control envelope land, and it MUST stay behind the expose access ladder, private by default; the recipe never demonstrates the public tier for CDP.

8. What we explicitly do NOT claim

• No pod-scoped Kubernetes mechanism (NetworkPolicy, PodSecurity, pod quotas) applies to sandbox VMs. Where we provide an equivalent, it is documented as ours.
• No external security review has been performed. The README must continue to say so until one has.
• Side-channel resistance between forks of the same snapshot is not claimed.

Supply chain and artifact provenance (issue #35)

Review gate

An external security review is required before any 1.0 (or “production-ready”) claim. Tracking: not scheduled. The hosted customer front door (section 7b, issue #210: the public gateway, customer keys, and cross-org isolation) is explicitly gated on this review (issue #194) before it serves production tenants.

Change discipline

Any PR that moves the security surface (new listener, new privilege, new artifact type, new cross-component call) must update this file in the same PR.