Disaster recovery validation
for Proxmox VE & PBS.
CertiStack boots the virtual machines in your PBS backups inside isolated, zero-copy sandboxes. Prove they start and their services answer before disaster strikes, and keep tamper-evident Ed25519-signed evidence of every run.
How CertiStack Validates Recovery
Engineered with fail-closed safety, air-gapped sandboxing, and zero lingering state. Recovery validation drills never impact production workloads.
01. Read-Only PBS Chunk Ingest
CertiStack accesses Proxmox Backup Server datastores using non-privileged, read-only snapshot credentials. Chunk indexes and SHA-256 manifests are validated in memory before any hypervisor resources are allocated.
- ✓ Strict Read-Only Access: Impossible to overwrite, mutate, or corrupt production snapshots.
- ✓ Cryptographic Chunk Verification: Verifies PBS deduplicated chunk trees and crypt-hashes against the manifest.
- ✓ Bandwidth Optimization: Only chunks required for boot and probed services are pulled from the PBS datastore.
{
"backup_id": "vm/104/2026-09-30T16:00:00Z",
"datastore": "pbs01-nvme",
"chunks_count": 4218,
"verified_hash": "sha256:7e9b0429f43c8b82...",
"access_mode": "RO_SNAPSHOT_LOCK",
"source_node": "proxmox-pve-01"
}02. Zero-Copy Copy-On-Write Provisioning
Rather than performing lengthy restores that consume terabytes of disk space and hours of I/O, CertiStack maps a thin, ephemeral copy-on-write (COW) scratch disk atop the read-only PBS backing store.
- ✓ Instant Provisioning: Storage is prepared in under 500 milliseconds regardless of VM disk size.
- ✓ Zero Disk Waste: Writes made during boot are contained within a temporary memory or NVMe scratch overlay.
- ✓ Safe Concurrency: Multiple validation drills can run simultaneously on the same PBS cluster.
qemu-img create -f qcow2 \
-b "nbd://pbs.internal:10809/vm-104-disk-0" \
-F raw /run/certistack/scratch_vm104.qcow2
# Allocation: 0 bytes copied
# Time elapsed: 0.28s03. Air-Gapped SDN Sandbox Boot
Test workloads boot within isolated virtual bridges controlled by Proxmox SDN. Strict iptables and nftables policies drop all egress traffic to corporate and production subnets, preventing IP conflicts and accidental production writes.
- ✓ Complete LAN Isolation: Test VM retains original production IP address safely inside an isolated VLAN/VNet.
- ✓ Synthetic Default Gateway: Local test runner emulates essential gateway responses to satisfy OS network daemons.
- ✓ Fail-Closed Firewall: Any attempted outbound connection outside the test sandbox is dropped immediately.
ZONE: sdn_zone_dr_isolated
BRIDGE: vmbr99 (VLAN 999)
GATEWAY STUB: 10.99.0.1 (Synthetic Runner)
TARGET VM IP: 10.99.0.10 (Preserved)
POLICY: DROP ALL TO 10.0.0.0/8 & WAN04. Synthetic Health & State Probing
A green hypervisor indicator doesn't mean your application is functional. CertiStack executes synthetic multi-layer checks to prove that database engines start, web endpoints respond, and critical systemd units reach target state.
- ✓ Database Query Readiness: Executes non-mutating test queries against PostgreSQL, MySQL, and Redis.
- ✓ HTTP/TLS Endpoint Probes: Validates SSL certificates, HTTP response status codes, and response headers.
- ✓ Guest-Agent Telemetry: Verifies kernel startup, filesystem mounts, and service targets without SSH keys.
[PROBE: OS] systemd: default.target -> ACTIVE (140ms)
[PROBE: DB] tcp://10.99.0.10:5432 -> SYN/ACK (12ms)
[PROBE: SQL] SELECT count(*) FROM db -> 2,419,082 (24ms)
[PROBE: HTTP] GET /healthz -> 200 OK (8ms)
[RESULT] ALL 4 PROBES PASSED (0 FAILS)05. Cryptographic Attestation & Clean Teardown
Upon probe completion, CertiStack seals run telemetry with an Ed25519 digital signature. The temporary VM is immediately halted and the COW scratch disk unlinked, leaving zero lingering state or storage drift.
- ✓ Ed25519 Proof Bundle: Tamper-evident attestation artifact verifiable with standard open-source tools.
- ✓ Automated Garbage Collection: Atomic teardown ensures zero orphaned disks, stale taps, or dangling locks.
- ✓ Audit Compliance: Machine-readable reports ready for ingestion into GRC platforms and insurance binders.
{
"status": "VALIDATED",
"timestamp": "2026-09-30T16:00:05Z",
"runtime_duration": 4.168,
"signing_key_id": "node01-clift-infra",
"signature": "0a174f88e1bc98d47b0e45f917540263f350c388..."
}PBS Backup Verify vs. CertiStack Execution
Proxmox Backup Server provides exceptional backup verification at rest, but a healthy chunk hash does not prove your guest operating system or production application will boot and recover.
| Validation Capability | PBS Backup Verify (At Rest) | CertiStack Engine (In Sandbox) |
|---|---|---|
| Verification Scope | Read-only SHA-256 chunk checks at rest | Live boot execution in air-gapped sandbox |
| OS & Kernel Driver Health | Untested (treated as opaque disk blocks) | Boots the guest and confirms it is running over QMP; optional guest-agent service checks |
| Filesystem & Database Recovery | Undetected broken WALs or dirty states | Boots the guest so its own filesystem and database recovery runs, then probes the service |
| Application Layer Health | No service port or API validation | Synthetic TCP, HTTP, DNS, and LDAP probes |
| Restore Speed & Storage Overhead | Reads all chunks sequentially across network | Disposable zero-copy COW overlays |
| Network & Routing Isolation | N/A (offline backup store check) | Ephemeral SDN bridge with zero default gateway |
| Audit & Compliance Proof | PBS daemon syslog text entries | Portable Ed25519-signed JSON evidence; HTML/PDF auditor binders in Enterprise Edition |
Inspect the evidence shape
This redacted example shows the fields produced by the CLI. It is not retained lab evidence or a signed production report; generate and verify a report from your own approved run.
{
"report_schema_version": "1.5",
"report_id": "example-report-id",
"timestamp": "2026-01-01T00:00:00Z",
"plan_id": "example-recovery-plan",
"plan_name": "Illustrative recovery validation",
"environment": "example-lab",
"node_fqdn": "pve-node-01.example.com",
"validation_mode": "full_integrity",
"engine_version": "dev",
"git_commit": "not-published",
"frameworks": [],
"retention_days": 0,
"rto_seconds": 0,
"vm_records": [
{
"vmid": 9001,
"source_vmid": 101,
"name": "example-http-01",
"boot_duration_seconds": 0,
"source_snapshot": "pbs.example.com:backup/vm/9001/2026-01-01T00:00:00Z",
"source_disks": [{"slot": "scsi0", "archive": "drive-scsi0.img.fidx"}],
"network_recovery_address": "192.0.2.11",
"restored_hardware": {"from_backup_config": true, "cores": 2, "sockets": 1, "memory_mb": 2048, "cpu": "host", "scsihw": "virtio-scsi-single", "ostype": "l26", "nic_model": "virtio"},
"probe_results": [
{"type": "http", "target": "http://192.0.2.11:80/healthz", "description": "Example HTTP health probe", "passed": true, "duration": 0}
]
},
{
"vmid": 9002,
"source_vmid": 102,
"name": "example-service-01",
"boot_duration_seconds": 0,
"source_snapshot": "pbs.example.com:backup/vm/9002/2026-01-01T00:00:00Z",
"source_disks": [
{"slot": "scsi0", "archive": "drive-scsi0.img.fidx"},
{"slot": "scsi1", "archive": "drive-scsi1.img.fidx"}
],
"network_recovery_address": "192.0.2.31",
"probe_results": [
{"type": "tcp", "target": "192.0.2.31:5432", "description": "Example TCP service probe", "passed": true, "duration": 0}
]
}
],
"teardown_evidence": {
"vm_stopped_and_destroyed": true,
"overlay_files_removed": [],
"loop_devices_unmapped": [],
"orphan_sweep_clean": true,
"all_cleaned": true,
"verified_at": "example-only"
},
"all_passed": true
}Report field summary
Comprehensive Documentation
A complete, self-hostable documentation suite compiled with Material for MkDocs, covering deployment, architecture, command and configuration references, YAML schemas, report verification, scheduling, and lab runbooks.
Ten commands to a signed report
Install the CLI, create a signing key, copy the minimal plan, run it against one backup, and verify the evidence.
Controller Deployment
Deployment patterns for management VMs, temporary node workers, and isolation boundaries.
Automation & Scheduling
Run validations nightly with systemd or cron, verify each report, and get told when a recovery fails.
CLI Reference
Every command and flag, the exit statuses scripts rely on, and the checks the node diagnostic runs.
Configuration Reference
Every environment variable, the protected environment file, precedence, and the files kept on the controller and the node.
Test-Plan Schema
Complete YAML reference for VMs, disks, ephemeral networks, timeouts, and synthetic probes.
Example Plans
Copy-ready plans for a single VM, a three-tier Linux application, and a Windows domain, with a guide to choosing probes.
Reports & Verification
What a signed report contains, how keys and keyrings work, and how an auditor verifies one with or without CertiStack.
System Architecture
Detailed recovery flows, zero-copy COW loopbacks, air-gapped sandboxes, and fail-closed teardown.
Troubleshooting
Operational diagnostics, SSH and setup errors, QEMU guest agent timeouts, SDN bridge states, and recovery logging.
Frequently Asked Questions
Short answers on safety, supported versions, scheduling, reports, licensing, and where to get help.
Release Readiness
Documented test-plan coverage, required lab evidence, campaign safety controls, and approval criteria.
Commercial licensing
Community Edition is free and open source. Homelab+, Team, and MSP Fleet are Enterprise Edition subscriptions bought online; the signed license file is emailed on payment. The Enterprise daemon is installed on each Proxmox VE node it validates. Enterprise Scale is a negotiated agreement.
Prices are in US dollars. To subscribe or request a license, contact sales directly at sales@certistack.dev. Already subscribed? Manage your subscription.
Community
Free and open source under AGPLv3, for any use that complies with the license.
- GNU Affero General Public License v3.0 (AGPLv3)
- No node or VM cap — commercial use permitted
- Full local CLI runner (
certistack run) - Zero-copy COW loopback on PVE & PBS
- Synthetic probes (TCP, HTTP/HTTPS, DNS, LDAP, guest agent)
- Ed25519-signed JSON evidence and pinned-key verification
Homelab+
The Enterprise management daemon and dashboard for one personal lab, under a commercial license.
- Management daemon, dashboard, and HTTP API for one lab
- Commercial license — no copyleft obligation
- Up to 10 protected workloads
- Signed license emailed on payment
Team
Commercial licensing and the management plane for internal engineering teams.
- Up to 25 protected workloads
- Management daemon, dashboard, and HTTP API
- Compliance binders and outbound webhooks
- Need more capacity? Contact sales
MSP Fleet
Multi-tenant licensing for managed service providers validating client fleets.
- Up to 100 pooled workloads across 5 environments
- Tenant separation for per-client reporting
- Management daemon, dashboard, API, and webhooks
- One daemon per Proxmox VE node, plus a central evidence hub across nodes and clients
- Larger fleets priced by agreement
Enterprise Scale
Multi-datacenter, sovereign air-gapped, and compliance-mandated enterprise infrastructure.
- Deployment, security, and capacity scope defined in writing
- Support commitments agreed per contract
- Security review and procurement handled directly