Skip to content

Controller and node-worker deployment model

CertiStack has two execution locations with deliberately different trust boundaries:

Dedicated controller VM / unprivileged OCI container
  - plans, PVE and PBS credentials, signing key, reports
  - strict SSH host-key validation and PVE TLS validation
  - creates no PVE-local permanent CertiStack service
                    |
                    | PVE API + verified SSH
                    v
Selected PVE node
  /var/lib/certistack/workers/<run UUID>/certistack-node
  - PBS map, loop device, COW delta, QMP/QGA, isolated VNet probe endpoint
  - guestfish for optional COW-only guest network recovery
  - node-local journal and cleanup while a run is active
  - removed when the job completes

The controller is the product installation point. The Community source distribution includes an unprivileged OCI image definition under deploy/controller and a Packer-built controller VM template under deploy/appliance. Neither is privileged and neither needs /dev/kvm, a host filesystem bind mount, a Docker socket, or a PVE shell. Their step-by-step instructions are in the controller image README (secrets, the published image, Compose) and the appliance README (the Packer template and first boot).

Why a node worker exists

Zero-copy validation cannot be PVE-API-only. proxmox-backup-client map makes a loop device in the PVE node kernel; qemu-img creates an overlay QEMU opens on that same node; QMP and the isolated VNet bridge are also node-local. A controller can orchestrate them remotely, but it cannot move those kernel resources into its own container while retaining node-local backup mapping.

For isolated wire probes, the worker automatically uses guestfish from libguestfs-tools to write a MAC-bound, no-gateway profile to the disposable overlay before boot. network_recovery.mode: preserve is the explicit opt-out. This is not a guest-agent installation or a source-VM change. Bundle the dependency into the signed Enterprise connector or install it as a documented worker prerequisite for Community; do not silently substitute DHCP, Cloud-init, or a production VNet.

Stage 2 TCP, HTTP, DNS, and LDAP probes execute on that worker through its temporary isolated VNet address. The worker requires the node's nft and sysctl tools, refuses a VNet with IPv4 forwarding enabled, and installs run-scoped input/forwarding drops around the probe address. They are evidence of worker-to-guest service reachability, not proof of a guest-originated client route or DNS policy. The signed probe record identifies this worker_host origin; use a QGA probe for an assertion that executes inside the guest.

For a firewalld guest, the worker may also add a source-restricted allow rule for the sandbox probe address and configured probe ports to that guest's effective default zone on the same overlay, keeping the zone's own rules. It is a recovery-test adapter, not a production firewall change, and disappears with the overlay. It opens nothing between restored guests: they reach each other on the isolated VNet only as far as their own firewalls allow.

The worker receives a single validated plan plus only the PVE/PBS connection material required for that run. It never receives the controller report signing key. It returns unsigned evidence; the controller signs and stores the JSON report.

Access modes

Bootstrap worker — Community and lab

The Community controller uploads its own static binary to a unique, root-owned <state-dir>/workers/<uuid> directory through SSH (by default, /var/lib/certistack/workers/<uuid>). This avoids relying on /run being executable, which hardened PVE nodes commonly prohibit. It requires a verified known_hosts entry and either a root SSH account or a passwordless sudo policy that can install and execute that temporary worker.

This is intentionally explicit: permission to execute an uploaded binary as root is root-equivalent. It is convenient for a lab but it is not a least-privilege remote-execution boundary. Restrict its key to the controller host, protect the controller as a privileged management system, and use a separate PVE service account/API token for normal lifecycle calls.

Signed connector — Enterprise target

Not shipped. The current Enterprise package runs a node-local, root-owned certistackd on each PVE node it validates, which executes the engine in process; it offers no distributed execution. This section is the contract such a feature must meet.

Any Enterprise release that offers distributed managed execution must replace bootstrap SSH with a small, independently auditable, signed node connector. Its RPC surface must be fixed to map/unmap, overlay lifecycle, PVE-local VM lifecycle, QMP/QGA, sandbox probe operations, journal/recovery, and exact cleanup. Authenticate controller to connector with mTLS and per-run authorization; do not expose a general shell or arbitrary command endpoint.

It is a deliberately small PVE-side component, not the full CertiStack product: no UI, database, schedule engine, tenant data, license service, reports, or signing key live on the hypervisor.

Controller secret contract

Store these only in the controller secret manager or protected mount:

  • PVE API URL, API-token ID, API-token secret, and private CA where needed.
  • Read-only PBS repository credential and fingerprint.
  • SSH private key and approved known_hosts file for each PVE node.
  • Ed25519 report signing key.

Do not put secrets in YAML plans, Terraform variables committed to source, container layers, command-line arguments, Docker Compose files, or PVE environment files. The controller rejects storage.pbs_password in a remote run. Its SSH/SCP command line never contains a PVE or PBS secret.

Operational requirements

  • Make PVE and PBS TLS verification mandatory; mount a CA file rather than setting insecure-skip-verify.
  • Pin controller OCI images by digest in production; the Packer appliance template enforces a sha256 image reference.
  • Keep controller reports and signing keys on encrypted, backed-up controller storage with retention and access controls appropriate to the customer.
  • Allow only controller-to-node management traffic. Restored workloads remain inside their temporary VNet with no gateway, SNAT, or physical uplink; the worker also verifies per-interface forwarding is disabled and applies its run-scoped host-input/forwarding containment.
  • Use the default network.lifecycle: ephemeral for per-run PVE SDN objects. For a deliberately platform-managed sandbox, set network.lifecycle: preprovisioned; the worker validates its exact zone, VNet, subnet, no-gateway, and no-SNAT properties read-only and never adopts, reloads, or deletes that SDN configuration.
  • Run controller and PVE recovery checks before scheduling a new job. A hard crash must preserve enough node-side journal information to reconcile only resources recorded for that run. If a completed worker reports incomplete teardown, the controller retries that exact journal before it removes the worker workspace. When that retry cannot be proven, the exact mode-0700 workspace is retained rather than deleting the recovery environment. The next run on that node that holds the host lock with a clean journal removes the credentials of any workspace whose controller has been silent for 15 minutes: its runtime.env, PBS encryption key and worker binary. It keeps the logs and results, and writes a credentials-removed.txt note saying when and why.
  • Keep sandbox VMs out of scheduled backup jobs. A job that selects all VMs (vzdump --all) also picks up the sandbox VMs of a validation that is running when the job starts. CertiStack attaches every overlay with backup=0, so such a job stores only the sandbox VM's configuration and never restored guest data. It still creates a backup group under the sandbox VMID, and it briefly locks the VM. Add the sandbox VMIDs to the job's exclusions, select the job's VMs by pool, or schedule validations outside the backup window.