Skip to content

CertiStack: Technical Architecture & System Mechanics

This document details the underlying engineering mechanics, low-level OS interactions, hypervisor protocols, and execution workflows powering CertiStack.

1. System Invariants & Core Guarantees

CertiStack Community runs as a controller on a dedicated management VM or unprivileged OCI container. A compiled, short-lived node worker runs only on the selected PVE host for the kernel- and QEMU-local operations that cannot cross the PVE API. The full controller, signing key, reports, plans, database, and management UI never live on a hypervisor. It enforces five non-negotiable engineering invariants:

┌─────────────────────────────────────────────────────────────────────────────┐
│                       CERTISTACK SYSTEM INVARIANTS                          │
├──────────────────────────┬──────────────────────────┬───────────────────────┤
│ Read-only Backup Sources │ Air-Gapped L2/L3 Isolation│ Deterministic Probing │
│ • Optional local copies  │ • Ephemeral SDN Bridges  │ • QMP Status (Stage 1)│
│ • PBS on-demand chunks   │ • Strictly NO Gateways   │ • Wire Probes(Stage 2)│
│ • COW .qcow2 local delta │ • Strictly NO SNAT/NAT   │ • In-Guest QGA(Stage 3│
├──────────────────────────┴──────────────────────────┴───────────────────────┤
│ Fail-Safe LIFO Teardown Stack      │ Admission Control Guardrails           │
│ • Unwinds loop devices, deltas, VMs │ • Memory usage ceiling: <= 85%         │
│ • Traps SIGINT, SIGTERM, Panics    │ • CPU I/O wait ceiling: <= 12%         │
└─────────────────────────────────────┴───────────────────────────────────────┘

2. Mapped Backups and Temporary Storage

Traditional disaster recovery testing requires restoring full virtual machine disk images to hypervisor storage. For a 2TB database VM, a full restore consumes 2TB of SAN/NVMe disk space, hours of network transfer, and severe disk I/O load.

CertiStack can boot from read-only mapped backups through the pipeline below. Guest writes go to temporary COW overlays. When disks exceed the available cache, the default copy_before_boot: auto can copy them to local storage during the integrity scan and boot from those copies. Firmware state is also restored as temporary raw copies. Capacity planning must include copies and overlay growth; storage.max_copy_gib: 0 means no copy cap. The source backup stays read-only in either path.

┌──────────────────────────────────────────────────────────────────────┐
│                    PROXMOX BACKUP SERVER (PBS)                       │
│        Deduplicated Content-Addressable Chunk Store (.fidx)          │
└──────────────────────────────────┬───────────────────────────────────┘
                                   │ HTTPS On-Demand Blocks
                                   ▼
┌──────────────────────────────────────────────────────────────────────┐
│                  TEMPORARY PVE NODE WORKER + KERNEL                   │
│               Loopback Block Device (/dev/loopX)                     │
│                (Read-Only Backing Block Device)                      │
└──────────────────────────────────┬───────────────────────────────────┘
                                   │ Read Operations Only
                                   ▼
┌──────────────────────────────────────────────────────────────────────┐
│                 EPHEMERAL LOCAL SCRATCH STORAGE                      │
│ /var/lib/certistack/scratch/certistack-delta-vm<id>-<timestamp>.qcow2 │
│         Copy-on-Write (COW) Overlay — Captures all Boot Writes       │
└──────────────────────────────────┬───────────────────────────────────┘
                                   │ Attached as scsi0
                                   ▼
┌──────────────────────────────────────────────────────────────────────┐
│                     SANDBOXED KVM VIRTUAL MACHINE                    │
│            OS Boot, Journal Replays, Pagefile, Temp Files            │
└──────────────────────────────────────────────────────────────────────┘

Execution Protocol:

  1. Remote Chunk Mapping: The temporary PVE node worker invokes proxmox-backup-client map <snapshot> <drive_archive> --repository <pbs_url>.
  2. The client negotiates an authenticated, read-only session with the PBS API.
  3. It reassembles the remote deduplicated chunk index (.fidx) into a local Linux block device (e.g., /dev/loop0).
  4. Blocks are fetched over HTTPS on-demand only when requested by the hypervisor.
  5. Ephemeral Delta Overlay Creation: CertiStack executes: 
     qemu-img create -f qcow2 -b /dev/loop0 -F raw /var/lib/certistack/scratch/certistack-delta-vm9001.qcow2
  6. /dev/loop0 acts as the immutable backing file.
  7. The .qcow2 overlay file captures all guest disk writes (OS boot logs, swap files, database crash recovery journal replays).
  8. Hypervisor Attachment: The worker registers an ephemeral PVE VM pointing its drive directly to the node-local delta overlay (scsi0: /var/lib/certistack/scratch/certistack-delta-vm9001.qcow2,discard=on,iothread=1,backup=0; backup=0 keeps the restored data out of any scheduled backup job). The scratch directory is owner-only (0700); shared temporary directories such as /tmp are rejected.
  9. Teardown: The VM is destroyed, the .qcow2 delta file is deleted from scratch storage, and proxmox-backup-client unmap /dev/loopX releases the kernel loop device.

3. Isolated Software-Defined Networking (SDN)

Restoring virtual machines into production broadcast domains causes catastrophic collisions: IP conflicts, split-brain database syncs, and rogue Active Directory domain controllers.

CertiStack leverages Proxmox VE SDN for layer-2 isolation and adds a node-local layer-3 containment boundary for the temporary probe endpoint:

Single-Node Topologies (Simple Zones)

  • The engine calls POST /cluster/sdn/zones with type=simple (e.g. csSb01).
  • A corresponding VNet is created under POST /cluster/sdn/vnets (e.g. csVnet01).
  • PVE SDN provisions an isolated Linux bridge without physical interface bindings (ethX or bondX).
  • The configuration is activated cluster-wide via PUT /cluster/sdn.
  • Packet containment: All Ethernet frames remain within host kernel memory because the bridge has no physical uplink. Before assigning the temporary probe address, the worker verifies that per-interface IPv4 forwarding is disabled and installs a run-scoped nft policy that permits established probe replies, drops new traffic from the VNet to the host, and drops traffic forwarded through the VNet. Restored NICs also enable the PVE firewall. This is defense in depth; the no-gateway, no-SNAT, and no-uplink invariants remain required deployment controls.

Multi-Node Cluster Topologies (VXLAN Zones)

  • For multi-tier stacks spanning physical nodes, an ephemeral vxlan zone is created with an explicit peer IP list. The VNI is assigned to the VNet, must be in the 24-bit VXLAN range, and is checked against every existing PVE VNet before creation.
  • Traffic is encapsulated within UDP packets over the private cluster interconnect, preserving sandbox routing while preventing packet leakage onto physical access switches.

The No-Gateway Invariant

  • Ephemeral subnets strictly omit gateway parameters.
  • No SNAT (Source Network Address Translation) or masquerade directives are ever applied.
  • Restored virtual machines cannot resolve public DNS queries or initiate connections to external cloud APIs.

4. Deterministic Multi-Stage Verification Pipeline

CertiStack rejects OCR and screen-scraping heuristics as pass/fail gating criteria. Visual heuristics are brittle and easily fooled by background updates or splash screens. Service verification relies on a three-stage deterministic pipeline:

  ┌────────────────────────────────────────────────────────┐
  │ Stage 1: Hypervisor State Verification                │
  │ • Connects to QMP socket (/run/qemu-server/<id>.qmp)   │
  │ • Negotiates capabilities (qmp_capabilities)           │
  │ • Asserts status == "running" within 60 seconds        │
  └──────────────────────────┬─────────────────────────────┘
                             │ PASS
                             ▼
  ┌────────────────────────────────────────────────────────┐
  │ Stage 2: Synthetic Gateway Wire-Level Probing          │
  │ • TCP 3-Way Handshake against target ports (389, 5432) │
  │ • HTTP/HTTPS GET request with status assertion (200 OK)│
  │ • TLS handshake negotiation & certificate check        │
  │ • DNS UDP/53 lookup & LDAP TCP/389 rootDSE query       │
  └──────────────────────────┬─────────────────────────────┘
                             │ PASS
                             ▼
  ┌────────────────────────────────────────────────────────┐
  │ Stage 3: In-Guest Inspection via QEMU Guest Agent      │
  │ • Connects to QGA socket (/run/qemu-server/<id>.qga)   │
  │ • Dispatches guest-exec ("systemctl is-active <svc>")  │
  │ • Asserts exit code == 0 & stdout substring match      │
  │ • Fallback: Gracefully skipped if QGA not installed    │
  └──────────────────────────┬─────────────────────────────┘
                             │
                             ├──────────────────────────┐
                       ALL PASS                       ANY FAIL
                             │                          │
                             ▼                          ▼
                  ┌──────────────────────┐   ┌──────────────────────┐
                  │ PASS: Attestation    │   │ FAIL: Forensic       │
                  │ Certificate Signed   │   │ Screendump Captured  │
                  └──────────────────────┘   └──────────────────────┘

Forensic Screendump Fallback

If any stage fails, CertiStack immediately dispatches {"execute": "screendump"} over QMP. The raw framebuffer is read as binary P6 PPM, converted to PNG in pure Go, base64-encoded, and embedded in the failure certificate to provide human operators with visual proof of kernel panics, Windows BSOD stop codes, or GRUB bootloader errors.

5. Fail-Safe LIFO Rollback Stack

Aborted simulations, system panics, or termination signals (SIGINT, SIGTERM) must never leave orphaned loop devices, dangling delta files, or uncontained VMs.

CertiStack implements an atomic Last-In, First-Out (LIFO) teardown stack:

// Every allocated resource registers an idempotent teardown closure immediately:
loopDev, err := pbsClient.Map(...)
cleanupStack.Push("unmap loop device", func() error {
    return pbsClient.Unmap(loopDev)
})

deltaPath, err := sysutil.CreateDeltaOverlay(...)
cleanupStack.Push("remove delta overlay", func() error {
    return os.Remove(deltaPath)
})

upid, err := pveClient.CreateVM(...)
cleanupStack.Push("destroy ephemeral VM", func() error {
    pveClient.StopVM(node, vmid)
    return pveClient.DestroyVM(node, vmid)
})

Upon normal completion, test failure, process panic, or signal interception, the stack unwinds in exact reverse order of creation:

  1. Ephemeral VMs are stopped and destroyed.
  2. Ephemeral COW .qcow2 delta overlays are unlinked.
  3. PBS loopback block devices are detached.
  4. Ephemeral SDN VNets and zones are deleted and reloaded.

Crash recovery is journal-scoped rather than heuristic. certistack recover reconciles only resources recorded for the interrupted run, verifies their ownership and process identity, and refuses to guess about unrecorded host resources. There is no scratch-directory or host-wide orphan sweep.

The signed report's teardown_evidence.orphan_sweep_clean field keeps its historical name for report-schema compatibility. It records that every resource this run recorded as owned was cleaned; it never indicates that a host-wide scan was performed. verify-report labels it "Owned Resources Clean".

6. Cryptographic Attestation & Non-Repudiation

To satisfy compliance audits (DORA, SOC 2, HIPAA), generated certificates must be non-repudiable and tamper-evident:

  1. RFC 8785 Canonicalization (JCS): JSON payloads are serialized deterministically (sorted keys, uniform float formatting, stripped whitespace).
  2. Ed25519 Asymmetric Signature: The canonical payload is signed on the controller with the operator's Ed25519 private key. The key never leaves the controller; the node worker returns unsigned evidence.
  3. Pinned Signer Verification: The hex-encoded signature and public key are embedded in the report, but the verifier also requires a trusted signer keyring or explicitly supplied trusted public key. A report cannot establish trust in its own embedded key. Any modification to a single byte invalidates the signature when checked via certistack verify-report --keyring <trusted-signers>.