NexGate Deployment Guide

NexGate Deployment Guide (New)
The paved road for shipping every NexGate service — architecture, flow, secrets, setup, and operations. 
 Owner: Kibuti · Organization: NexGate / nexgate-hq · Status: living document · v2 
 
 How to read this 
 Single source of truth for how services are built, deployed, secured, and operated. Five things: 
 
 Architecture — what the pieces are and how they fit 
 Flow — what happens when you push code 
 Secrets — how credentials reach a running container without ever touching git 
 Setup — standing it up from zero 
 Operations — running it day to day, adding services, rolling back, scaling 
 
 If you remember one sentence: 
 
 You write code, drop one folder, and register one secret role. An image is built once when you push, and pulled once per environment by the reconciler. No credential ever enters git, and no human ever SSHes to deploy. 
 
 v2 changes: the secrets model is rebuilt from §5 onward. v1 described AppRole correctly at a high level but left the mechanics undefined — where Role IDs and Secret IDs physically live, how Komodo authenticates to Vault, how the trust chain terminates, and how any of it is verified. It also shipped a policy that granted one service both staging and prod. All fixed below. 
 
 Table of contents 
 
 Core principles 
 Architecture 
 Repository layout 
 Deploy flow 
 Secrets model 
 The trust chain 
 Setup from scratch 
 Adding a new service 
 Deploying a change 
 Operations 
 Scaling 
 Conventions and invariants 
 Glossary 
 
 
 1. Core principles 
 The design turns on one shift away from the old Jenkins pipeline: 
 Stop editing shared state imperatively. Declare desired state, and let a reconciler make reality match it. 
 Everything follows: 
 
 Every service has the same shape — onboarding the 40th is no harder than the 4th 
 Each service is self-contained — its own folder describes containers, config, and secret permissions. One service's deploy can never touch another's. 
 Build once, promote by reference — the image tested in staging is the exact artifact in production. Prod never rebuilds. 
 Git is the control plane — approvals are pull requests, history is the audit log, rollback is a revert 
 No credential is ever in git — only pointers to credentials 
 Every access is attributable — audit logs name the service and environment, never "root" 
 Substrate-agnostic — the same repo and flow run on one VPS today and many hosts later 
 
 
 2. Architecture 
 2.1 The four moving parts 
 
 
 
 Part 
 What it is 
 Who owns it 
 
 
 
 
 Service repos 
 App code + a thin CI workflow ( nexgate-hq/<service> ) 
 One repo per service 
 
 
 nexgate-infra 
 Declarative desired state — what runs where, at which version 
 One shared repo 
 
 
 Reconciler (Komodo) 
 Watches nexgate-infra , applies changes on the host(s) 
 Installed once per host 
 
 
 Vault 
 Issues every credential any container ever uses 
 Central, vault.qbitspark.com 
 
 
 
 There is no Jenkins . The reconciler is off-the-shelf — not code you write. It does what Jenkins did on the deploy side ( docker compose pull + up -d ), driven by git instead of a webhook. 
 2.2 Runtime layers 
 
 Base layer — always-on platform: Traefik, Vault Agent, PostgreSQL, Redis, RabbitMQ, MinIO. Rarely redeployed. Deploying an app service never touches it. 
 Service layer — application services (backend, notifications, File Thunder, ai , …), each deployed independently against the base layer. 
 
 2.3 Component roles 
 
 GHCR ( ghcr.io/nexgate-hq/* ) — image registry. CI pushes; the reconciler pulls. 
 Komodo — the reconciler. Watches nexgate-infra , applies stack changes, dashboard for logs/redeploy/rollback, emits deploy notifications. Also mints per-service Vault Secret IDs at deploy time (§6). 
 Vault ( vault.qbitspark.com ) — secrets. Services authenticate via AppRole , scoped per service and per environment. 
 Vault Agent — sidecar per host. Handles AppRole login, token renewal, and secret rendering so application containers never handle Vault credentials directly. 
 Traefik — edge router. Auto-discovers by container labels, serves *.nexgate.co over HTTPS (Let's Encrypt). 
 
 
 3. Repository layout 
 nexgate-infra is the heart. Everything a service needs is in its own folder — the information that used to be crammed into two giant shared compose files, cut apart so each service owns its slice. 
 nexgate-infra/
├── base/ # always-on platform layer
│ ├── traefik.yml
│ ├── vault-agent.yml
│ ├── postgres.yml # optional shared Postgres for light services
│ ├── redis.yml
│ ├── rabbitmq.yml
│ └── minio.yml # always-on (File Thunder needs it live)
│
├── services/
│ ├── nexgate-backend/
│ │ ├── service.yml # compose block: image, labels, depends_on
│ │ ├── config.env # non-secret config ONLY
│ │ ├── vault-policy.hcl # which Vault paths this service may read
│ │ ├── vault-agent.hcl # secret rendering template
│ │ └── tag.env # IMAGE_TAG=... ← the one line CI edits
│ ├── notification-server/
│ ├── file-thunder/
│ └── ai/
│
├── policies/ # generated from services/*/vault-policy.hcl
│ └── .gitkeep
│
├── scripts/
│ ├── vault-bootstrap.sh # create policies + approles from service folders
│ └── verify-isolation.sh # prove cross-service access is denied
│
├── staging/
│ └── stack.yml # include: base + services (staging tags)
└── prod/
 └── stack.yml # include: base + services (prod tags)
 
 Nothing in a service folder is a program. service.yml is a compose snippet. tag.env is one line. vault-policy.hcl is a few lines of permissions. The layout guarantees one service's deploy is isolated to its own files. 
 Example service folder 
 # services/ai/service.yml
services:
 ai:
 image: ghcr.io/nexgate-hq/ai-service:${AI_TAG:-latest}
 env_file: [config.env, tag.env]
 networks: [nexgate-prod, proxy]
 depends_on: [postgres, redis, rabbitmq]
 volumes:
 # Vault Agent renders secrets here. The app reads a file.
 # It never sees a role_id or secret_id.
 - vault-secrets-ai:/run/secrets:ro
 labels:
 - traefik.enable=true
 - traefik.http.routers.ai.rule=Host(`ai.nexgate.co`)
 - traefik.http.routers.ai.tls.certresolver=le
 restart: unless-stopped
 
 # services/ai/config.env ← non-secret ONLY. Committed to git.
SPRING_PROFILES_ACTIVE=prod
LOG_LEVEL=INFO
VAULT_ADDR=https://vault.qbitspark.com
 
 # services/ai/tag.env ← CI rewrites ONLY this file
AI_TAG=0.1.0
 
 Because tag.env is per-service, deploying ai cannot reset the backend's tag. The shared- .env clobber problem is structurally impossible. 
 
 4. Deploy flow 
 4.1 The one rule: built once, pulled per environment 
 
 Created (built + pushed to GHCR): once, by CI, when you push. Tied to the code, not the environment. 
 Pulled : by the reconciler, once per environment, when that environment's tag changes. 
 
 Staging pulls right after the change. Prod pulls the same image after approval. Prod never rebuilds. 
 4.2 End-to-end 
 git push (staging/master)
 │
 ▼
CI (GitHub Actions, reusable workflow)
 • build image via Buildx
 • push to GHCR (version tag + latest)
 • bump services/<svc>/tag.env in nexgate-infra
 │
 ▼
nexgate-infra ← declarative desired state
 │
 ▼ (prod only: approval gate — PR review)
 │
 ▼
Komodo reconciler on the VPS
 • detects the tag change
 • mints a fresh Secret ID for <svc>-<env> ← §6
 • writes it to /etc/vault/<svc>-<env>-secret-id (0400 root)
 • docker compose pull <svc> && up -d <svc> (only that service)
 • on health-check pass: destroys prior Secret IDs
 │
 ▼
Vault Agent authenticates via AppRole → renders secrets to /run/secrets
 │
 ▼
Service container starts, reads secret files
 │
 ▼
Traefik routes it over HTTPS (*.nexgate.co)
 
 No SSH. No Jenkins job. No credential in git. 
 4.3 Approvals 
 The gate lives in git — stronger than a button because it is reviewable and permanent. 
 
 Staging — no gate. Merge to staging , auto-syncs. Fast feedback. 
 Prod — promotion is a pull request bumping the prod tag. The PR is the approval; you see exactly which service and tag are moving. Merge = deploy. 
 
 Optional hard gates: 
 
 A GitHub production environment with a required reviewer (may need a paid plan for private repos; branch protection requiring review before merge to master gives the same gate free) 
 Komodo manual-sync for prod — auto-sync staging, hold prod for a manual click. Closest one-to-one replacement for the old Jenkins button. 
 
 Recommended: PR promotion + Komodo manual-sync for prod. 
 4.4 Notifications 
 
 CI → webhook posts build + tag-bump status to Telegram 
 Komodo → deploy success/failure to the same channel, reported after a health check rather than "compose exited 0" 
 GitHub → free email/mobile pings on workflow failures and pending approvals 
 Vault audit → alert on root token use, permission-denial spikes, seal events (§10) 
 
 Post-deploy runtime health comes from the Grafana stack — §10. 
 
 5. Secrets model 
 
 This is the section that was thin in v1. It is now the longest, because it's where the real risk lives. 
 
 5.1 The rule 
 No credential is ever in git. Only pointers to credentials. 
 
 
 
 Artifact 
 In git? 
 Why 
 
 
 
 
 config.env (non-secret config) 
 ✅ Yes 
 Log levels, profiles, Vault address 
 
 
 vault-policy.hcl 
 ✅ Yes 
 Permission rules , not credentials 
 
 
 Role ID 
 ✅ Yes 
 Identifies a role. Useless alone. 
 
 
 Secret ID 
 ❌ Never 
 Delivered at deploy, file on host only 
 
 
 Actual secrets 
 ❌ Never 
 Live in Vault, fetched at runtime 
 
 
 
 5.2 Path hierarchy 
 v1 used a flat nexgate/data/ai_staging — one segment for service_env . That blocks globbing, so every new component needs a new policy line, and eventually someone "fixes" it with a wildcard that grants too much. 
 v2 uses a proper hierarchy: 
 nexgate/<service>/<environment>/<component>

nexgate/ai/staging/postgres
nexgate/ai/prod/postgres
nexgate/ai/prod/openai
nexgate/backend/prod/postgres
nexgate/backend/prod/jwt-signing
nexgate/file-thunder/prod/minio
nexgate/file-thunder/prod/postgres
nexgate/notification-server/prod/rabbitmq
nexgate/shared/prod/cloudflare
 
 Why service/environment above component: that's the boundary policies cut along. One rule covers every component a service will ever have, including ones added next year. 
 Put the thing you cut permissions along highest in the path. 
 5.3 The KV v2 path gotcha 
 The single most common Vault mistake, and it will bite whoever writes the next policy. 
 What you type: vault kv get nexgate/ai/prod/postgres 
What goes over HTTP: GET /v1/nexgate/data/ai/prod/postgres 
 KV v2 splits operations into sub-paths — data/ , metadata/ , delete/ , undelete/ , destroy/ . The kv CLI inserts the right one for you. Policies have no wrapper. 
 Write the obvious-looking thing and you get permission denied with no hint why: 
 path "nexgate/ai/prod/*" { capabilities = ["read"] } # ← WRONG. Silently fails.
 
 5.4 Policies — one per service and environment 
 The v1 bug: 
 # services/ai/vault-policy.hcl ← BROKEN
path "nexgate/data/ai_staging" { capabilities = ["read"] }
path "nexgate/data/ai_prod" { capabilities = ["read"] }
 
 One policy granting both environments. If both ai_staging and ai_prod roles attach it, staging can read production secrets — defeating the isolation claimed two paragraphs above it in v1. 
 Correct — separate file per environment: 
 # services/ai/vault-policy-staging.hcl
path "nexgate/data/ai/staging/*" {
 capabilities = ["read"]
}
path "nexgate/metadata/ai/staging/*" {
 capabilities = ["list"]
}
 
 # services/ai/vault-policy-prod.hcl
path "nexgate/data/ai/prod/*" {
 capabilities = ["read"]
}
path "nexgate/metadata/ai/prod/*" {
 capabilities = ["list"]
}
 
 Read only. No create , no update , no delete . An app that reads its config has no business rewriting it. Deny-by-default means anything not listed is denied — no explicit deny rule needed. 
 Prefer generating these from a template so nobody hand-writes the data/ prefix wrong (§7 step 6). 
 5.5 AppRole per service per environment 
 # staging — looser, no CIDR binding if staging hosts vary
vault write auth/approle/role/ai-staging \
 token_policies="ai-staging" \
 token_ttl=4h \
 token_max_ttl=12h \
 secret_id_ttl=720h \
 secret_id_num_uses=0 \
 secret_id_bound_cidrs="<staging-host-ip>/32"

# prod — tightest
vault write auth/approle/role/ai-prod \
 token_policies="ai-prod" \
 token_ttl=1h \
 token_max_ttl=4h \
 secret_id_ttl=720h \
 secret_id_num_uses=0 \
 secret_id_bound_cidrs="<prod-host-ip>/32"
 
 token_max_ttl is your actual worst-case exposure window. Renewal extends but never resets — it's measured from creation. A stolen prod token is dead in 4 hours regardless of how often the thief renews it. 
 secret_id_bound_cidrs is free hardening. A stolen Role ID and Secret ID are useless from anywhere but that host. The attacker would have to run the attack from inside your infrastructure. 
 5.6 Where the two halves physically live 
 The rule: Role ID and Secret ID never in the same place. If one docker inspect yields both, the split has bought nothing. 
 Role ID → services/<svc>/config.env, committed to git
Secret ID → /etc/vault/<svc>-<env>-secret-id on the host
 chmod 400, chown root:root
 minted by Komodo at deploy, never in git
 
 Never do this: 
 environment:
 VAULT_ROLE_ID: "..."
 VAULT_SECRET_ID: "..." # ← docker inspect prints this in plaintext
 
 One command gives an attacker both halves. This is the single most common way AppRole deployments get compromised. 
 Do this — file mount: 
 services:
 vault-agent-ai:
 image: hashicorp/vault:1.20.3
 command: agent -config=/vault/agent.hcl
 environment:
 VAULT_ADDR: https://vault.qbitspark.com
 volumes:
 - ./services/ai/vault-agent.hcl:/vault/agent.hcl:ro
 - /etc/vault/ai-prod-role-id:/vault/role-id:ro
 - /etc/vault/ai-prod-secret-id:/vault/secret-id:ro
 - vault-secrets-ai:/run/secrets
 
 docker inspect now shows only mount paths. docker compose config shows nothing sensitive. Git holds only the Role ID. 
 5.7 Vault Agent — the app never touches Vault 
 This is why Agent is worth the extra container. The application never handles a Role ID, a Secret ID, or a Vault token. It reads a file. 
 # services/ai/vault-agent.hcl
vault {
 address = "https://vault.qbitspark.com"
}

auto_auth {
 method "approle" {
 config = {
 role_id_file_path = "/vault/role-id"
 secret_id_file_path = "/vault/secret-id"
 remove_secret_id_file_after_reading = false
 }
 }

 sink "file" {
 config = { path = "/run/secrets/token" }
 }
}

template {
 destination = "/run/secrets/application-secrets.properties"
 contents = <<EOT
{{- with secret "nexgate/data/ai/prod/postgres" }}
spring.datasource.username={{ .Data.data.username }}
spring.datasource.password={{ .Data.data.password }}
{{- end }}
{{- with secret "nexgate/data/ai/prod/openai" }}
openai.api-key={{ .Data.data.api_key }}
{{- end }}
EOT
}
 
 Agent logs in, renders the file, renews the token, and re-authenticates when token_max_ttl is hit. The app just reads /run/secrets/application-secrets.properties . 
 Note {{ .Data.data.username }} — the double .data is KV v2's envelope. Another place the data/ structure surfaces. 
 Alternative — Spring Cloud Vault , if you'd rather the app authenticate directly: 
 spring:
 cloud:
 vault:
 uri: https://vault.qbitspark.com
 authentication: APPROLE
 app-role:
 role-id: ${VAULT_ROLE_ID}
 secret-id: ${VAULT_SECRET_ID}
 kv:
 enabled: true
 backend: nexgate
 default-context: ai/prod
 
 Fewer containers, but the app handles the Secret ID. Prefer Agent — it keeps credentials out of application process memory and gives you one renewal implementation instead of one per language. 
 5.8 Two tiers, unchanged from v1 
 
 _infra tier — raw container credentials (Postgres/Redis/RabbitMQ/MinIO passwords) needed to boot the base containers 
 App tier — the service's own secrets (JWT, DB creds, third-party API keys) 
 
 5.9 Static vs dynamic — the migration target 
 Everything above stores static secrets in KV. That's the starting point, not the destination. 
 The limitation, concretely: if a static credential leaks, Vault cannot revoke it. Vault stores and guards it, but it can't kill something a third party issued. A leaked OpenAI key stays valid until someone revokes it at OpenAI. 
 Dynamic secrets invert this. Vault generates a real PostgreSQL user on demand with a TTL, then drops it: 
 vault read database/creds/ai-prod
# → v-approle-ai-prod-x7Kd92 / A1a-8sKdm3nQp (lease 1h)
vault read database/creds/ai-prod
# → v-approle-ai-prod-mN4vB1 / Z9z-2pLwq7Rt (lease 1h)
 
 Different every time. vault lease revoke runs DROP ROLE against Postgres directly. A stolen credential dies on its own. 
 Migration order: 
 
 PostgreSQL — all services, highest value 
 RabbitMQ — notification-server, backend 
 MinIO — via the AWS engine (S3-compatible STS) 
 PKI — internal CA for mTLS between services 
 Transit — encryption-as-a-service for sensitive payloads 
 
 What must stay static: third-party API keys (OpenAI, Cloudflare, payment providers). Vault didn't issue them, so it can't rotate them. These get quarterly manual rotation and versioned storage in KV. 
 Success measure: a leaked database credential is a non-event because it expired an hour ago. 
 
 6. The trust chain 
 
 v1 said "services authenticate via AppRole" and stopped there. That leaves the important question unanswered: who gives Komodo permission to mint Secret IDs, and who gives that to Komodo? 
 
 6.1 Komodo's Vault credentials 
 Komodo needs a Vault credential to mint per-service Secret IDs. That credential becomes the new secret zero, so it must be narrow . 
 # policies/komodo-deployer.hcl
path "auth/approle/role/+/secret-id" {
 capabilities = ["update"]
}
path "auth/approle/role/+/secret-id-accessor/destroy" {
 capabilities = ["update"]
}
 
 Can mint and destroy Secret IDs. Cannot read a single secret. Cannot change a policy. Cannot create a role. If it leaks, an attacker can generate Secret IDs — but must guess role names, and every generation appears in the audit log. 
 vault write auth/approle/role/komodo-deployer \
 token_policies="komodo-deployer" \
 token_ttl=15m \
 token_max_ttl=30m \
 secret_id_ttl=0 \
 secret_id_bound_cidrs="<host-ip>/32"
 
 15-minute tokens. Komodo logs in per deploy, uses the token, it dies. 
 6.2 Placing Komodo's credentials — the one manual step 
 ssh <host>
sudo mkdir -p /etc/vault

vault read -field=role_id auth/approle/role/komodo-deployer/role-id \
 | sudo tee /etc/vault/komodo-role-id > /dev/null
vault write -f -field=secret_id auth/approle/role/komodo-deployer/secret-id \
 | sudo tee /etc/vault/komodo-secret-id > /dev/null

sudo chmod 400 /etc/vault/komodo-*
sudo chown root:root /etc/vault/komodo-*
 
 This is the only credential a human ever places by hand, and only once per host. 
 6.3 Komodo's deploy hook 
 #!/usr/bin/env bash
# scripts/deploy-service.sh — run by Komodo on tag change
set -euo pipefail

SERVICE="$1" # e.g. ai
ENV="$2" # staging | prod
ROLE="${SERVICE}-${ENV}"

# 1. Komodo authenticates (15-minute token)
TOKEN=$(vault write -field=token auth/approle/login \
 role_id="$(cat /etc/vault/komodo-role-id)" \
 secret_id="$(cat /etc/vault/komodo-secret-id)")

# 2. Record existing Secret ID accessors so we can revoke them after
OLD_ACCESSORS=$(VAULT_TOKEN=$TOKEN vault list -format=json \
 "auth/approle/role/${ROLE}/secret-id" 2>/dev/null || echo '[]')

# 3. Mint a fresh Secret ID for this deploy
VAULT_TOKEN=$TOKEN vault write -f -field=secret_id \
 "auth/approle/role/${ROLE}/secret-id" \
 > "/etc/vault/${ROLE}-secret-id"
chmod 400 "/etc/vault/${ROLE}-secret-id"
chown root:root "/etc/vault/${ROLE}-secret-id"

# 4. Deploy
docker compose pull "$SERVICE"
docker compose up -d "$SERVICE"

# 5. Wait for health, THEN revoke the old Secret IDs
if ./scripts/wait-for-health.sh "$SERVICE" 120; then
 echo "$OLD_ACCESSORS" | jq -r '.[]' | while read -r acc; do
 VAULT_TOKEN=$TOKEN vault write \
 "auth/approle/role/${ROLE}/secret-id-accessor/destroy" \
 secret_id_accessor="$acc"
 done
 echo "✓ ${ROLE} deployed, old Secret IDs revoked"
else
 echo "✗ ${ROLE} health check failed — old Secret IDs left intact for rollback"
 exit 1
fi

# Komodo's own token expires in 15 minutes regardless
 
 Order matters. Revoke old Secret IDs after the new container is healthy, never before — otherwise a failed deploy leaves you unable to roll back. 
 6.4 The chain, complete 
 YOU ← SSH key. The actual root of trust.
 │
 └─ place Komodo's role-id + secret-id (once per host, by hand)
 │
 └─ KOMODO mints service Secret IDs (every deploy, 15-min tokens)
 │
 └─ VAULT AGENT logs in per service (1–4h tokens, auto-renewed)
 │
 └─ APP reads rendered secret files (never touches Vault)
 
 Four levels, each narrower than the one above. Only the top requires a human, and only once per host. 
 Your SSH key is the bottom of the chain — the real secret zero for the entire system. It deserves a passphrase, a hardware key if possible, and an entry in the break-glass documentation. 
 6.5 Rotating Komodo's own Secret ID 
 Manual, deliberately. 
 ssh <host>
vault write -f -field=secret_id auth/approle/role/komodo-deployer/secret-id \
 | sudo tee /etc/vault/komodo-secret-id > /dev/null
docker compose restart komodo
 
 It cannot be automated from Komodo — that would let a compromised Komodo renew its own access indefinitely. Keeping it manual bounds the chain. Quarterly, or on any suspicion. 
 6.6 Secret zero — the honest accounting 
 You cannot bootstrap trust from nothing. Every auth system has a first credential. The question is only what it is and how bad a leak would be . 
 Before (shared .env ): 
 DB_PASSWORD=real
JWT_PRIVATE_KEY=real
OPENAI_KEY=real
MINIO_SECRET=real
 
 One file. Everything. Never expires. No log of who read it. 
 After: 
 /etc/vault/komodo-secret-id: d4e5f6a7-b8c9-...
 
 One file. One UUID. 
 
 
 
 
 Old .env 
 Stolen Komodo Secret ID 
 
 
 
 
 What they get 
 Every credential immediately 
 Nothing — needs the Role ID too 
 
 
 With both halves 
 — 
 Can mint Secret IDs, cannot read secrets 
 
 
 Duration 
 Forever 
 Until rotated 
 
 
 IP-bindable 
 No 
 Yes 
 
 
 Do you find out 
 No 
 Yes — audit log 
 
 
 To revoke 
 Change everything everywhere 
 One command 
 
 
 
 Vault doesn't protect secrets by hiding them. It protects them by encrypting at rest, authenticating every request, authorising per path, expiring everything, logging every access, and revoking centrally. A .env file does none of these six. 
 
 7. Setup from scratch 
 Bootstrap order matters. 
 Step 1 — Provision hosts 
 Two Ubuntu VPS: staging and prod (separate boxes — §11). Docker Engine + Compose plugin. UFW allowing 22, 80, 443 only. 
 Also: confirm the Docker daemon API is not exposed. curl http://<host>:2375/version from outside must fail. An exposed Docker socket makes every other control here irrelevant. 
 Step 2 — Private networking (do it early) 
 WireGuard mesh (or provider private network) between hosts, even with one host each. Makes future splitting a one-line change instead of a migration. Services address each other by name / internal DNS, never hardcoded localhost . 
 Step 3 — Docker networks 
 docker network create proxy
docker network create nexgate-staging # on staging host
docker network create nexgate-prod # on prod host
 
 Step 4 — Deploy the base layer 
 Traefik (Let's Encrypt resolver le ), PostgreSQL, Redis, RabbitMQ, MinIO. MinIO always-on from day one. 
 Vault Agent comes after Vault is configured — step 6. 
 Step 5 — Vault: audit logging first 
 Before anything else. Non-disruptive, one command, and it means everything from here forward is attributable. 
 vault audit enable file file_path=/vault/logs/audit.log
vault audit list -detailed
 
 Vault refuses to serve requests if all audit devices fail to write — deliberate fail-closed design. Ship the log to Loki; sensitive values are HMAC'd while paths and field names stay plaintext, so it's safe to forward. 
 Also confirm, and write down offline: 
 
 Which storage backend is vault.qbitspark.com using? (determines backup strategy) 
 Where are the 5 unseal key shares, physically? 
 Are they split across locations , or all in one place? 
 Has a snapshot restore ever been tested? 
 
 
 Solo operator: Shamir's 3-of-5 assumes multiple people. Split across locations instead — bank box, trusted party, encrypted offsite, hidden at home. An untested recovery scheme is a story you tell yourself. 
 
 Step 6 — Vault: mount, auth, policies, roles 
 vault secrets enable -path=nexgate kv-v2
vault secrets tune -description="NexGate application secrets" nexgate/
vault auth enable approle
 
 Then run the bootstrap script (§7 step 8) which generates policies and roles from the service folders. 
 Retention worth setting per path: 
 vault kv metadata put -max-versions=10 nexgate/ai/prod/postgres
 
 Default is unlimited history — every rotation accumulates forever and old passwords stay readable. 
 Step 7 — Komodo 
 Install per host. Point at nexgate-infra . Staging = auto-sync, prod = manual-sync. 
 Place Komodo's Vault credentials by hand (§6.2). This is the one manual credential step. 
 Step 8 — Bootstrap script 
 #!/usr/bin/env bash
# scripts/vault-bootstrap.sh
# Generates policies and AppRoles from services/*/ folders.
# Idempotent — safe to re-run.
set -euo pipefail

STAGING_CIDR="${STAGING_CIDR:?set STAGING_CIDR}"
PROD_CIDR="${PROD_CIDR:?set PROD_CIDR}"

for dir in services/*/; do
 svc=$(basename "$dir")

 for env in staging prod; do
 name="${svc}-${env}"

 # policy — generated, so the data/ prefix is never hand-typed wrong
 cat > "policies/${name}.hcl" <<EOF
path "nexgate/data/${svc}/${env}/*" {
 capabilities = ["read"]
}
path "nexgate/metadata/${svc}/${env}/*" {
 capabilities = ["list"]
}
EOF
 vault policy write "$name" "policies/${name}.hcl"

 if [ "$env" = "prod" ]; then
 TTL=1h; MAXTTL=4h; CIDR="$PROD_CIDR"
 else
 TTL=4h; MAXTTL=12h; CIDR="$STAGING_CIDR"
 fi

 vault write "auth/approle/role/${name}" \
 token_policies="$name" \
 token_ttl="$TTL" \
 token_max_ttl="$MAXTTL" \
 secret_id_ttl=720h \
 secret_id_num_uses=0 \
 secret_id_bound_cidrs="$CIDR"

 # Role ID is not secret — emit for config.env
 echo "${name} role_id: $(vault read -field=role_id auth/approle/role/${name}/role-id)"
 done
done
 
 Step 9 — Verify isolation 
 Do not skip this. A policy tested only for success is untested. 
 #!/usr/bin/env bash
# scripts/verify-isolation.sh
set -uo pipefail
PASS=0; FAIL=0

check() {
 local role="$1" path="$2" expect="$3" rid sid tok result
 rid=$(vault read -field=role_id "auth/approle/role/${role}/role-id")
 sid=$(vault write -f -field=secret_id "auth/approle/role/${role}/secret-id")
 tok=$(vault write -field=token auth/approle/login role_id="$rid" secret_id="$sid")

 if VAULT_TOKEN="$tok" vault kv get "$path" >/dev/null 2>&1
 then result="ALLOW"; else result="DENY"; fi

 if [ "$result" = "$expect" ]; then
 echo " ✓ ${role} → ${path} = ${result}"; PASS=$((PASS+1))
 else
 echo " ✗ ${role} → ${path} = ${result} (expected ${expect})"; FAIL=$((FAIL+1))
 fi
}

echo "Positive (must ALLOW):"
check ai-prod nexgate/ai/prod/postgres ALLOW
check backend-staging nexgate/backend/staging/postgres ALLOW

echo "Negative (must DENY):"
check ai-prod nexgate/backend/prod/jwt-signing DENY
check ai-prod nexgate/ai/staging/postgres DENY # ← the v1 bug
check ai-staging nexgate/ai/prod/postgres DENY # ← the v1 bug
check backend-staging nexgate/backend/prod/postgres DENY
check file-thunder-prod nexgate/ai/prod/openai DENY

echo; echo "Passed: $PASS Failed: $FAIL"
[ "$FAIL" -eq 0 ] || exit 1
 
 The two marked lines are exactly what v1's policy would have failed. Run this after every policy change, forever. Wire it into CI on nexgate-infra . 
 Step 10 — service-template + reusable CI 
 Template repo with Dockerfile, workflow_call caller, /health + /metrics wired. One reusable GitHub Actions workflow in the org that all services call. 
 Step 11 — Onboard the first service 
 Run the §8 checklist end to end to validate the chain. 
 Step 12 — Notifications and observability 
 Telegram webhook in the reusable workflow. Komodo deploy alerts. Grafana stack on a dedicated ops VPS — §10. 
 
 8. Adding a new service 
 Assuming base, Komodo, template, and reusable workflow exist — onboarding ai : 
 
 Generate the repo — nexgate-hq/ai-service from service-template 
 Write the code — the only real work; everything below is config 
 First push to staging → CI builds ai:0.1.0 → GHCR (created) 
 Add one folder — services/ai/ with service.yml , config.env , vault-agent.hcl , tag.env 
 Write the secrets to nexgate/ai/staging/* and nexgate/ai/prod/* 
 Run vault-bootstrap.sh — generates both policies and both AppRoles 
 Run verify-isolation.sh — add positive and negative cases for the new service 
 Put the Role IDs in config.env (safe to commit) 
 Commit → Komodo syncs staging → mints a Secret ID → pulls ai:0.1.0 → live at ai.staging.nexgate.co (pulled #1) — validate 
 Promote to prod — PR bumping the prod tag. Merge → Komodo syncs prod → pulls the same image (pulled #2, identical) 
 
 The wiring (steps 4–8) is the entire cost — the old four-place scavenger hunt collapsed to one folder + one script run . 
 
 File Thunder note: additionally uncomment MinIO in base/ and declare its own Postgres (5433) and a ClamAV container in its service.yml . One-time, and entirely within File Thunder's folder + the base layer. 
 
 
 9. Deploying a change 
 
 Edit code, push to staging 
 CI builds <svc>:<version> , pushes to GHCR, bumps the staging tag (created) 
 Komodo mints a fresh Secret ID, pulls, restarts only that service in staging (pulled #1) . Automatic, no gate. 
 On health-check pass, Komodo revokes the previous Secret IDs 
 Validate in staging 
 Open a PR bumping the prod tag. The PR is the gate. 
 Merge → Komodo repeats for prod with the same image (pulled #2, identical) 
 
 Prod runs the byte-for-byte artifact you tested. 
 
 10. Operations 
 Rollback 
 Revert the tag-bump commit in nexgate-infra , or use Komodo's "redeploy previous." Images are immutable and promoted by reference, so rollback is re-pointing a tag at the last-good image. 
 Secret ID note: because old Secret IDs are only revoked after a health check passes, a failed deploy leaves the previous credential valid. Rollback works without re-minting. 
 Rough edges fixed by design 
 
 
 
 v1 problem 
 v2 fix 
 
 
 
 
 Shared .env clobber 
 Per-service tag.env 
 
 
 One VAULT_TOKEN for both envs 
 AppRole per service and environment 
 
 
 Policy granting staging + prod 
 Separate policy file per environment 
 
 
 Flat service_env path 
 Hierarchical service/env/component 
 
 
 Secret ID placement undefined 
 File on host, 0400, minted by Komodo 
 
 
 Komodo's Vault auth undefined 
 Narrow AppRole, §6 
 
 
 No isolation testing 
 verify-isolation.sh in CI 
 
 
 MinIO commented out 
 Always-on in base/ 
 
 
 
 Secret rotation 
 
 
 
 Credential 
 Frequency 
 Method 
 
 
 
 
 Service Secret IDs 
 Every deploy 
 Komodo, automatic 
 
 
 Service tokens 
 1–4h 
 Vault Agent, automatic 
 
 
 Komodo's Secret ID 
 Quarterly 
 Manual, by SSH (§6.5) 
 
 
 Third-party API keys 
 Quarterly + on incident 
 Manual, versioned in KV 
 
 
 DB credentials 
 1h 
 Dynamic engine (target state) 
 
 
 Vault encryption key 
 Quarterly 
 vault operator rotate 
 
 
 Unseal keys 
 Annually or on suspicion 
 vault operator rekey 
 
 
 Root token 
 Per use 
 Generate → use → revoke 
 
 
 
 Static secret rotation — order matters: 
 1. Create the new credential at the provider
2. vault kv put ... ← new version; old still readable
3. Redeploy / let Agent re-render
4. Verify healthy
5. ONLY THEN revoke the old one at the provider
6. Optionally vault kv destroy -versions=<n>
 
 Revoking before services pick up the new value causes an outage. KV v2 versioning is what makes step 5 safe — if step 4 fails, the previous version is one command away. 
 Backups 
 
 Postgres — scheduled pg_dump per database, shipped off-host 
 MinIO — bucket replication or scheduled sync off-site 
 Vault — vault operator raft snapshot save , encrypted, useless without unseal keys. Back up both, separately. Test the restore quarterly into a scratch instance. 
 nexgate-infra — it's git; already versioned 
 
 Monitoring — the Grafana stack 
 Runs on a dedicated ops VPS outside dev/staging/prod — never on a box it watches. One ops host watches every environment; targets labelled by env ( env=dev|staging|prod ), dashboards filter on that label. Joins the WireGuard mesh for internal IPs; hits public endpoints through Traefik. 
 
 
 
 Component 
 Question it answers 
 
 
 
 
 Prometheus 
 Metrics over time + alerting (scrapes /metrics ) 
 
 
 Loki 
 Log search (services log JSON to stdout) 
 
 
 Tempo 
 Distributed traces 
 
 
 Grafana 
 Dashboards, single view 
 
 
 Alertmanager 
 Routes alerts → Telegram / email 
 
 
 
 Collectors: node_exporter (host CPU/RAM/disk), cAdvisor (per-container resources), blackbox_exporter (uptime probes against /health — no separate Uptime Kuma needed), Grafana Alloy or Promtail (stdout JSON → Loki). 
 Vault-specific alerts — Vault exposes Prometheus metrics at /v1/sys/metrics?format=prometheus : 
 
 
 
 Condition 
 Severity 
 
 
 
 
 Vault sealed unexpectedly 
 Critical 
 
 
 Root token used 
 Critical 
 
 
 Audit device failure 
 Critical 
 
 
 Permission-denial spike 
 High — attack or misconfiguration 
 
 
 Token creation anomaly 
 High 
 
 
 Lease count growth 
 Medium — leaked unreleased leases 
 
 
 Secret/cert nearing expiry 
 Medium 
 
 
 
 Root token use should page you. In steady state it should never happen. 
 Because every service exposes /metrics and logs JSON to stdout (§12), adding a service to monitoring is a scrape target + a health probe, both env-labelled. 
 Komodo vs Grafana — you need both 
 
 
 
 
 Komodo 
 Grafana stack 
 
 
 
 
 Watches 
 Deployments (control plane) 
 Runtime health (observability) 
 
 
 Tells you 
 Did it deploy, what tag runs where 
 Is it up, slow, erroring; resource use; history 
 
 
 Lets you 
 Redeploy, roll back 
 Alert, investigate — not deploy 
 
 
 
 Komodo reports "deploy succeeded" the moment a container starts, even if it then crash-loops or serves 500s under load — Grafana catches that. Grafana can tell you a service is unhealthy but can't redeploy — that's Komodo. One tells you something's wrong ; the other lets you do something about it . 
 Komodo shows basic per-host CPU/RAM/disk, so it doubles as light resource monitoring — but no metrics history, no log search, no tracing. Run both on the ops VPS; both alert to Telegram. 
 Incident response 
 Suspected Vault compromise: 
 vault operator seal # everything slams shut, instantly
 
 Costs an outage; the attacker gets ciphertext. Recovery needs 3 of 5 unseal shares — which is why you must know where they are before the emergency. 
 Suspected service credential compromise: 
 vault list auth/approle/role/<svc>-<env>/secret-id
vault write auth/approle/role/<svc>-<env>/secret-id-accessor/destroy \
 secret_id_accessor="<accessor>"
vault lease revoke -prefix auth/approle/
 
 Redeploy to mint a fresh one. 
 Note the limit: revoking in Vault kills Vault access. It does not invalidate a third-party API key an attacker already read — that must be revoked at the provider. This is precisely why dynamic secrets matter (§5.9). 
 Upgrading base images 
 Postgres/Redis/etc. are pinned per service in service.yml (§11.4). Bump the pin, test in staging, promote. 
 Vault upgrades: read the changelog for every version between current and target. Snapshot first, always. Test on a scratch instance. Never skip major versions. 
 
 11. Scaling 
 11.1 Two kinds of scale 
 
 Many services (control plane) — scales freely. The 40th onboards like the 4th. 
 Heavy load (data plane) — single-host Compose has a ceiling. Climb the ladder without rebuilding the pattern. 
 
 11.2 The ladder 
 
 
 
 Rung 
 Setup 
 When 
 
 
 
 
 1 
 Single VPS, Compose + Komodo 
 Now 
 
 
 2 
 Bigger VPS 
 Vertical scale buys runway 
 
 
 3 
 Services split across hosts (Komodo multi-host) 
 When a service gets heavy 
 
 
 4 
 Swarm or K3s 
 Replicas + failover, much later 
 
 
 
 The repo and onboarding journey are identical at every rung . Moving up is a config change (which host a folder runs on), plus — at rung 3 — the private network from §7 step 2. 
 Secrets note: at rung 3, each host needs its own Komodo credentials placed by hand, and secret_id_bound_cidrs must list every host that runs a given service. At rung 4, switch to the Kubernetes auth method — pods authenticate via their service account and the Secret ID disappears entirely. 
 11.3 VPS topology 
 
 Split by environment (staging + prod VPS) — do this now. A safety baseline, not a scaling choice. It's also what makes secret_id_bound_cidrs meaningful: staging credentials physically cannot be used from the prod host and vice versa. 
 Split by service — a growth move. When File Thunder's transcoding starves the API, move it and its workers to their own host. 
 Every VPS runs everything — careful. Fine for stateless services behind Traefik. A trap for stateful ones: three MinIOs / Postgres / RabbitMQs are three diverging databases, not one system. Only works as true active-active with clustered datastores (Postgres replication, MinIO distributed mode, RabbitMQ quorum queues, Redis Sentinel) on Swarm/K3s — a distant milestone, never a casual switch. 
 
 Sequence: env-split today → service-split as load appears → all-have-all as a distant HA project. 
 11.4 Different versions per service 
 Postgres/Redis/MinIO are upstream public images pinned by tag — unrelated to "build once," which is only about your app image. Each service's datastore is a separate container in its own folder: 
 # services/file-thunder/service.yml
 ft-postgres:
 image: postgres:17
 volumes: [ft-pgdata:/var/lib/postgresql/data]
 
 # services/backend/service.yml
 backend-postgres:
 image: postgres:15
 volumes: [backend-pgdata:/var/lib/postgresql/data]
 
 Different names, volumes, versions — no collision. Shared vs dedicated is a per-service call: shared base Postgres saves RAM for light services; dedicated gives version/extension freedom and isolation (why File Thunder has its own). Never a base-layer constraint. 
 
 12. Conventions and invariants 
 Don't break these. 
 Deployment 
 
 Everything runs in a container — no exceptions. Every service and every base component. A static frontend ships as a container (nginx serving built assets), not a special case. This is what makes the model uniform. 
 One service = one repo = one folder in nexgate-infra 
 One deploy path for all service types. Backend, frontend, and AI share the same tail (push GHCR → bump tag → Komodo deploys). Only the build head differs per language — flavors of one reusable workflow, never separate deploy systems. 
 CI writes only services/<svc>/tag.env — never a shared file 
 Prod is promoted by reference , never rebuilt 
 Staging auto-deploys; prod is gated by a PR 
 The base layer is stable — deploying an app service never touches it 
 
 Secrets 
 
 No credential is ever committed to git. Role IDs are, because they're not credentials on their own. 
 Role ID and Secret ID never live in the same place 
 Secret IDs are files ( 0400 root ), never environment variables — docker inspect must not reveal one 
 One policy per service and environment. Never one policy spanning both. 
 Policies are generated, not hand-written — the data/ prefix is too easy to get wrong 
 Policies are read-only for application roles. No create , update , or delete . 
 verify-isolation.sh runs on every policy change , and includes negative tests 
 Applications never handle Vault credentials — Vault Agent renders files 
 Audit logging is always on. No exceptions, no "temporarily off to debug." 
 Root token is generated, used, and revoked. Never stored, never used for routine work. 
 
 Runtime 
 
 Services address each other by name / internal DNS , never hardcoded localhost 
 Every service exposes /health + /metrics and logs JSON to stdout (the service contract) 
 
 
 13. Glossary 
 
 Reconciler — Komodo. Watches nexgate-infra and applies changes on hosts. Replaces Jenkins' deploy role. Not code you write. 
 Desired state — what nexgate-infra declares should be running 
 Base layer — always-on platform containers 
 Service layer — application services deployed against the base layer 
 AppRole — Vault auth method for machines. Two halves: Role ID (identity, semi-public) + Secret ID (credential, sensitive). 
 Role ID — identifies which AppRole. Committed to git. Useless alone. 
 Secret ID — proves the identity. Never in git. File on host, 0400 , minted per deploy. 
 Vault Agent — sidecar that performs AppRole login, token renewal, and secret rendering so applications never handle Vault credentials 
 Secret zero — the first credential in any trust chain, which must be placed by a human. Here: Komodo's Secret ID, placed once per host over SSH. 
 Static secret — stored in Vault's KV engine. Vault guards it but cannot revoke it upstream. 
 Dynamic secret — generated by Vault on demand with a TTL. Vault issued it, so Vault can revoke it. 
 Promote by reference — moving the exact same image artifact from staging to prod, rather than rebuilding 
 Paved road — the standardized template + workflow + folder pattern that makes onboarding near-free 
 
 
 Migration checklist from v1 
 
 Split every combined policy into per-environment files — the v1 policy granted staging and prod together 
 Migrate paths from flat nexgate/<svc>_<env> to nexgate/<svc>/<env>/<component> 
 Enable audit logging — do this first, it's non-disruptive 
 Locate and verify the unseal key shares ; confirm they're split across locations 
 Test a Vault snapshot restore into a scratch instance 
 Create the komodo-deployer policy and AppRole 
 Place Komodo's credentials on each host by hand 
 Move Secret IDs out of any environment variables and into 0400 files 
 Add secret_id_bound_cidrs to every staging and prod role 
 Write and run verify-isolation.sh ; wire it into CI 
 Set max-versions on KV metadata paths 
 Add Vault alerts (root token use, denial spikes, seal events) to Alertmanager 
 Audit every running container: docker inspect <c> | grep -iE "secret|token|password|key" 
 Confirm the Docker daemon API is not exposed on any host 
 Plan the dynamic-secrets migration, starting with PostgreSQL 
 
 
 Versioned in nexgate-infra . Update whenever the architecture changes — it is the guide the whole organization relies on.

Vault Mastery Guide — v2 BOOK
From Root-Token Chaos to Production-Grade Secrets Management 
 Author: Kibuti — QBIT SPARK
 Environment: vault.qbitspark.com (HashiCorp Vault v1.20.3)
 Method: Local simulation first → master concepts → migrate production last
 Status: Lab Steps 1–8 complete. Steps 9–12 pending. 
 
 v2 changes: rewritten after completing the hands-on lab. Every analogy, error message, and "why doesn't this work" moment from the actual session is now captured. The mistakes sections are real, not hypothetical. 
 
 
 Table of Contents 
 PART 0 — Why This Exists 
 PART 1 — Foundations 
 PART 2 — Core Concepts (with the analogies that worked) 
 PART 3 — The Lab, Step by Step 
 PART 4 — Mistakes File 
 PART 5 — Humans, Machines & MFA 
 PART 6 — Rotation 
 PART 7 — Deployment Architecture 
 PART 8 — Production Migration 
 PART 9 — Operations 
 PART 10 — The Book 
 
 PART 0 — Why This Exists 
 0.1 The Incident 
 On 4–5 August 2026, an OpenAI API key ( sk-pro...CrQA ) was used by an unauthorized party to run ~129 automated LLM requests — a physics-grading workload (Feynman amplitude checkpoint scoring, some Chinese-language content) with no relation to any QBIT SPARK product. Roughly $7.95 consumed before detection. 
 Detection: OpenAI's abuse-detection email → manual log review → confirmed by reading request content. 
 Response: key revoked immediately. 
 Leak vector: still undetermined. Candidates under investigation: 
 
 Committed .env or hardcoded key in a Git repository 
 Baked into a published Docker image layer 
 Exposed Docker daemon API on the VPS 
 Application-level leak (debug endpoint, path traversal) 
 Client-side/frontend exposure 
 docker inspect on a container with the key in an env var 
 
 0.2 What the Incident Actually Exposed 
 The $7.95 was trivial. The structural problems it revealed were not: 
 
 
 
 Weakness 
 Risk 
 
 
 
 
 Root token used for routine Vault UI access 
 Full-scope master credential in daily circulation 
 
 
 Same root token bootstrapping local dev and production 
 Dev compromise = production compromise 
 
 
 No per-service scoped policies 
 Any leak potentially exposes all secrets 
 
 
 No audit logging 
 No forensic trail — couldn't answer "was this ever read from Vault?" 
 
 
 No MFA on human access 
 Single-factor compromise = total access 
 
 
 No rotation schedule 
 Leaked credentials valid indefinitely 
 
 
 No spending limits on downstream API keys 
 Financial blast radius unbounded 
 
 
 Unknown location of unseal key shares 
 Possible single point of catastrophic failure 
 
 
 
 The uncomfortable comparison: the same class of mistake applied to Vault's root token would expose every credential for JikoXpress, NexGate, Mauzodukani, GlueEmail and Textfy at once — including Snippe PSP payment integration, NexGate JWT signing keys, database passwords, and Vodacom SMPP credentials. 
 0.3 What "Mastery" Means Here 
 Not memorising commands. Being able to: 
 
 Explain every Vault component in your own words, without notes 
 Design a policy structure for a new service from scratch 
 Predict what will break before changing something 
 Recover from sealed / lost-key / corrupted-storage without panic 
 Teach it — which is why this ends in a book 
 
 Rule: do not proceed past a step until you can explain it aloud with the notes closed. 
 
 PART 1 — Foundations 
 1.1 What Is a Secret? 
 Any data that grants access or proves identity, and whose disclosure causes harm. 
 
 
 
 Type 
 Examples from this stack 
 
 
 
 
 Static credentials 
 PostgreSQL, RabbitMQ, MinIO passwords 
 
 
 API keys 
 OpenAI, Snippe PSP, Meta WhatsApp Business API, Vodacom SMPP 
 
 
 Cryptographic material 
 NexGate JWT RS256 signing keys, TLS private keys, OMEMO keys 
 
 
 Connection strings 
 DSNs embedding host + credentials 
 
 
 Certificates 
 Internal CA, client certs for mTLS 
 
 
 SSH keys 
 qbit-spark access 
 
 
 
 The problem: secrets must be available to apps at runtime, unreadable by anyone else, rotatable without downtime, and every access attributable. 
 .env files solve availability. Nothing else. 
 1.2 The Precision That Matters 
 Three words get conflated constantly. Get them separate: 
 vault kv put qbit/jikoxpress/prod/db username="jx_prod" password="s3cr3t"
 
 
 
 
 Part 
 Name 
 Sensitive? 
 
 
 
 
 qbit/jikoxpress/prod/db 
 path — the address 
 No 
 
 
 password 
 field name / key — the label 
 No 
 
 
 s3cr3t 
 field value 
 ← this is the secret 
 
 
 
 The house analogy: path is the address. Secret is the money inside. Token is the key in your pocket. 
 Why it matters practically: Vault's audit log records paths and field names in plaintext but HMACs the values. You can see that someone read qbit/jikoxpress/prod/db without the log itself leaking the password. 
 The secret/ confusion: in vault kv put secret/hello , the word "secret" is a mount path , not the data. Dev mode happens to mount a KV engine there. It could be called anything. This is why the lab moves to qbit/ — the word disappears and the ambiguity goes with it. 
 1.3 What Vault Is 
 
 An encrypted key-value store that requires authentication to read from, enforces fine-grained authorisation per request, logs every access, and can generate short-lived credentials on demand instead of merely storing long-lived ones. 
 
 Four jobs: 
 
 Secure storage — encrypted at rest; the storage backend never sees plaintext 
 Access control — every request authenticated and authorised per path 
 Audit — every request and response logged 
 Dynamic secrets — credentials with automatic expiry, so a leak dies on its own 
 
 What it is not: a human password manager, a config management system, or magic. A badly configured Vault is worse than none, because it creates false confidence. 
 1.4 The Mental Model 
 ┌─────────────────────────────────┐
 │ VAULT SERVER │
 You / Service │ ┌──────────┐ │
 │ │ │ AUTH │ ← "Prove who │
 ├──────────►│ │ METHOD │ you are" │
 │ │ └────┬─────┘ │
 │ │ │ issues │
 │ │ ▼ │
 │ │ ┌──────────┐ │
 │ │ │ TOKEN │ ← temporary badge │
 │ │ └────┬─────┘ │
 │ │ │ carries │
 │ │ ▼ │
 │ │ ┌──────────┐ │
 │ │ │ POLICIES │ ← "what doors │
 │ │ └────┬─────┘ does it open?" │
 │ │ ▼ │
 │ │ ┌──────────────────┐ │
 │ │ │ SECRETS ENGINES │ ← rooms │
 │ │ └────┬─────────────┘ │
 │ │ ▼ │
 │ │ ┌──────────────────┐ │
 │ │ │ STORAGE BACKEND │ ← encrypted
 │ │ └──────────────────┘ │
 │ │ ┌──────────────────┐ │
 │ │ │ AUDIT DEVICES │ ← the log │
 │ │ └──────────────────┘ │
 └───────────┴─────────────────────────────────┘
 
 Every Vault problem you will ever debug is a failure in one of those boxes. 
 1.5 The Alternatives Landscape 
 
 
 
 
 Vault 
 OpenBao 
 Infisical 
 SOPS 
 AWS SM 
 
 
 
 
 Licence 
 BUSL 1.1 
 MPL 2.0 
 MIT / paid 
 MPL 2.0 
 Proprietary 
 
 
 Self-host 
 Yes 
 Yes 
 Yes 
 N/A 
 No 
 
 
 Dynamic secrets 
 ✅ Extensive 
 ✅ (fork) 
 ⚠️ Limited 
 ❌ 
 ⚠️ Some 
 
 
 Auth methods 
 ✅ 20+ 
 ✅ 20+ 
 ⚠️ Fewer 
 ❌ 
 IAM only 
 
 
 Learning curve 
 Steep 
 Steep 
 Gentle 
 Gentle 
 Moderate 
 
 
 PKI / CA 
 ✅ 
 ✅ 
 ❌ 
 ❌ 
 ⚠️ Separate 
 
 
 Encryption-as-a-service 
 ✅ Transit 
 ✅ 
 ❌ 
 ❌ 
 ⚠️ KMS 
 
 
 Audit logging 
 ✅ Detailed 
 ✅ 
 ✅ Basic 
 ❌ 
 ✅ CloudTrail 
 
 
 Ops burden 
 High 
 High 
 Medium 
 Very low 
 None 
 
 
 Cost at this scale 
 Free 
 Free 
 Free tier 
 Free 
 ~$0.40/secret/mo 
 
 
 
 Categories, for the book: 
 
 A — env vars / config files. Trivial, no protection. What you're leaving. 
 B — encrypted files in Git (SOPS, git-crypt, Ansible Vault, Sealed Secrets). Genuinely good for GitOps. Worth pairing with Vault for Komodo deployment configs. 
 C — cloud native (AWS/Azure/GCP). Wrong topology for self-hosted VPS. 
 D — dedicated self-hosted (Vault, OpenBao, Conjur). Your category. 
 E — developer SaaS (Doppler, Infisical Cloud). Better DX, third party holds your secrets, data residency questions for Tanzanian regulatory context. 
 
 1.6 Why Vault — Decision Record 
 
 Already deployed at vault.qbitspark.com ; migration cost zero 
 Multi-product, multi-provider VPS topology — cloud-native can't span it 
 Dynamic secrets are the long-term win 
 PKI engine — internal CA for NexGate mTLS, and relevant to TCRA registrar work 
 Transit engine — encrypt JikoXpress financial data without app-held keys 
 Commercial credibility — "scoped AppRole auth with audit logging" answers a vendor questionnaire far better than ".env files." Already been through Vodacom BSR/OneTrust. 
 Transferable skill — appears in enterprise and East African fintech requirements 
 
 Hedge: track OpenBao. Migration is currently cheap. Revisit annually. 
 
 PART 2 — Core Concepts 
 2.1 Seal / Unseal — The Bank 
 Imagine a bank in Mbeya. 
 
 The building = the Vault server process 
 The steel vault door = the seal 
 The money inside = your secrets 
 
 When sealed: the building exists, the door is locked, the money is scrambled. Steal the entire building and you get a locked box full of ciphertext. 
 Vault starts sealed every time it boots. Deliberately. 
 The unseal keys = the combination, cut into 5 pieces 
 The combination isn't written anywhere. It was split into 5 shares given to 5 managers. Any 3 can open the door. One alone can't. Two can't. 
 Why 3-of-5 and not 5-of-5? Because if one manager dies, loses their piece, or travels, you'd never open the bank again. 3-of-5 survives losing 2. 
 The math (Shamir's Secret Sharing, 1979): two points define a line, three define a parabola. Hide the secret as a point on a curve; hand out other points. With enough points you reconstruct the curve. With fewer, infinitely many curves fit — you learn nothing , not "most of it." 
 Auto-unseal delegates to an external KMS so restarts don't need manual intervention. Convenient; shifts trust to that system. 
 If you lose enough shares, the data is permanently unrecoverable. There is no backdoor. 
 2.2 The Precise Chain 
 Common misunderstanding: unseal shares do not generate the master key. They reconstruct it — it already exists, split at initialization. 
 And the master key doesn't decrypt secrets directly: 
 3 unseal shares
 ↓ reconstruct
 Master Key
 ↓ decrypts
 Encryption Key ← this is what encrypts your secrets
 ↓
 Vault UNSEALED
 
 Why the extra layer? So vault operator rotate can change the encryption key without redistributing 5 new shares to 5 people. 
 2.3 Root Token vs Unseal Keys — The Rooms 
 Door open ≠ free access. Inside are many rooms — one per service. The root token is a master keycard opening every room. Scoped tokens open one. 
 
 
 
 
 Unseal keys 
 Root token 
 
 
 
 
 Opens 
 The vault door 
 The rooms inside 
 
 
 Count 
 5 shares, need 3 
 One 
 
 
 Used when 
 Vault restarts 
 Every request 
 
 
 Frequency 
 Rare 
 Should be near-never 
 
 
 
 You need both. Shares alone = open door, no keycard. Root alone = keycard, locked door. 
 Confirmed in the lab: the root token cannot unseal. That separation means two different attacks are required. 
 
 Steal the root token → useless against a sealed Vault 
 Steal the unseal keys → open door, but no token for any room 
 
 The reverse — the emergency button: root can seal. Sealing is easy, unsealing is hard, deliberately. 
 vault operator seal # everything slams shut, instantly
 
 If vault.qbitspark.com is ever suspected compromised, that's the fire alarm. Costs an outage; the attacker gets ciphertext. Then you need 3 of 5 shares to come back — which is why you must know where they are before the emergency. 
 2.4 Why Root Exists At All 
 Chicken-and-egg. A fresh Vault has no policies, no auth methods, nothing. Someone must create the first policy — but that needs permission. 
 Root is the bootstrap credential. It exists to create the system that replaces it. 
 Legitimate uses — the complete list: 
 
 Initial setup — enable auth methods, mount engines, write first policies 
 Emergency recovery — locked out, broken policy 
 A few root-protected operations — operator rekey , some seal management 
 
 Not on the list: daily work, application requests, UI login. 
 Where you store it: nowhere. You destroy it. 
 generate → use → vault token revoke → gone
 
 Need it next month? vault operator generate-root with 3 shares. 
 What you protect long-term is the shares , because those regenerate root. Root itself is disposable. 
 Where the shares go: 
 
 Split across different physical locations , never together 
 Encrypted USB in a safe; sealed envelope with a trusted person; encrypted offline backup elsewhere 
 Never on the same server as Vault 
 Never all in one folder, password manager, or laptop 
 
 Solo operator → can't split across people → split across places . If the house floods you still need 3. 
 Correction worth recording: root is not used to create app credentials day-to-day. Root builds the machinery once — enables AppRole, writes policies, creates roles — then is revoked. AppRole issues credentials forever afterward with no root involved. 
 
 Root builds the key-cutting machine. Then root leaves. The machine keeps cutting keys. 
 
 2.5 Storage Backends 
 Vault encrypts before writing, so a stolen backend yields ciphertext. 
 
 
 
 Backend 
 Use 
 
 
 
 
 Integrated (Raft) 
 Recommended default; HA capable 
 
 
 Consul 
 Legacy; more operational complexity 
 
 
 File 
 Single node; dev/test 
 
 
 In-memory 
 Dev only; vanishes on restart 
 
 
 PostgreSQL/MySQL 
 Supported, not recommended for new deployments 
 
 
 
 Action item: determine which backend vault.qbitspark.com uses. It determines your backup strategy. 
 Lab observation: Storage Type inmem means every restart is a brand-new Vault — new cluster ID, new unseal key, no data. On real Vault with persistent storage, the unseal key is generated once at initialization and never again. Lose it, lose everything. 
 2.6 Secrets Engines 
 ~20+ types , not four. Grouped by philosophy: 
 Static — you store, Vault guards 
 
 
 
 Engine 
 Purpose 
 
 
 
 
 kv-v2 
 Versioned key-value. The workhorse. 
 
 
 kv-v1 
 Unversioned, legacy 
 
 
 cubbyhole 
 Per-token private storage; dies with the token 
 
 
 
 Dynamic — Vault generates and expires 
 
 
 
 Engine 
 Generates 
 
 
 
 
 database 
 PostgreSQL, MySQL, MongoDB, Redis users 
 
 
 pki 
 X.509 certificates 
 
 
 ssh 
 Signed SSH certificates 
 
 
 aws / gcp / azure 
 Temporary cloud IAM credentials 
 
 
 rabbitmq / consul / nomad 
 Service-specific credentials 
 
 
 kubernetes 
 Service account tokens 
 
 
 
 Cryptographic services 
 
 
 
 Engine 
 Purpose 
 
 
 
 
 transit 
 Encrypt/decrypt without ever holding the key 
 
 
 transform 
 Format-preserving encryption, masking, tokenization 
 
 
 totp 
 Generate and validate 2FA codes 
 
 
 
 Infrastructure — identity/ and sys/ , always mounted, cannot be disabled. 
 Adoption order for QBIT SPARK: kv-v2 → database → pki → transit → ssh . 
 The insight: every engine answers "how do we handle this class of secret?" with a different strategy. KV says "store it carefully." Database says "don't store it, make a new one each time." Transit says "don't give it to the app at all — send the data here." 
 Rule of thumb: if Vault can generate it, don't store it. Store only what you're forced to — third-party API keys, mostly. 
 2.7 Mounts vs Paths vs Tenants 
 Two ways to split, easy to confuse: 
 Axis 1 — by engine type. Different engines are different machines. A KV engine can't issue certificates. 
 Axis 2 — by service. Possible, usually wrong: 
 ✅ One mount, path hierarchy: ❌ Mount per service:
qbit/jikoxpress/prod/db jikoxpress/prod/db
qbit/nexgate/prod/minio nexgate/prod/minio
 
 Both isolate fine via policy. Separate mounts add config, versioning settings, tuning, backup and audit surface with no extra security. 
 When separate mounts genuinely help — multi-tenancy. Relevant here: you hold a retainer with Wiki, deployed FinSpark for a Mbeya microfinance client, built PMK's site. If you host secrets on behalf of clients, mount per tenant: 
 qbit/ ← your products
wiki/ ← retainer client
kfz/ ← JikoXpress customer
 
 Why (and it's not isolation — policies do that either way): 
 
 
 
 Benefit 
 Why 
 
 
 
 
 Independent tuning 
 Different TTLs, audit settings per client 
 
 
 Clean deletion 
 Contract ends → vault secrets disable wiki → gone 
 
 
 Accident containment 
 A wildcard typo in a QBIT policy can't reach a client mount 
 
 
 Auditability 
 Log lines obviously client data 
 
 
 Handover 
 Export one mount, not a filtered subset 
 
 
 Vendor assessments 
 Structural separation is easier to evidence than path prefixes 
 
 
 
 Clean deletion is the strongest argument. With prefixes you hunt for paths and hope you missed nothing. 
 Caution: don't pre-mount tenants you don't have. 
 And note: qbit/ is already your org boundary. qbit/qbitspark/... says it twice. 
 2.8 The Model, Stated Plainly 
 
 Mount = the machine (what kind of engine)
 Path = the filing system inside it
 Policy = who may open which drawer 
 
 Policies are standalone objects , not nested under engines. Confirmed in the UI — Policies is its own menu. 
 A policy merely mentions a path as text. You can write a policy for a path that doesn't exist. Delete the mount and the policy sits there harmlessly pointing at nothing. 
 POLICY ─────────► names a path (as text)
 ▲
 │ referenced by name
AUTH METHOD ─────► issues tokens carrying that policy
 │
 ▼
TOKEN ─────────► request arrives
 ↓
 Vault checks: does any policy on this
 token allow this path + this verb?
 ↓
 SECRETS ENGINE serves or refuses
 
 Which is why editing a policy takes effect immediately , even on tokens issued yesterday. Nothing is baked into the token. 
 2.9 Policies 
 HCL. Deny by default — anything not granted is denied. No deny rule needed; absence is denial. 
 path "qbit/data/jikoxpress/prod/*" {
 capabilities = ["read"]
}
path "qbit/metadata/jikoxpress/prod/*" {
 capabilities = ["list"]
}
 
 Capabilities: 
 
 
 
 Capability 
 Verb 
 Meaning 
 
 
 
 
 create 
 POST/PUT 
 Create new 
 
 
 read 
 GET 
 Read existing 
 
 
 update 
 POST/PUT 
 Modify existing 
 
 
 delete 
 DELETE 
 Remove 
 
 
 list 
 LIST 
 Enumerate keys ( never values ) 
 
 
 sudo 
 — 
 Root-protected paths 
 
 
 deny 
 — 
 Explicit denial; overrides all grants 
 
 
 
 Policies are additive. A token gets the union of all its policies, plus default . To opt out of default , create the token with -no-default-policy . 
 token_policies="jikoxpress-prod,shared-readonly,monitoring"
 
 Except deny , which is absolute: 
 path "qbit/data/jikoxpress/prod/snippe-psp" {
 capabilities = ["deny"]
}
 
 Wildcards: 
 
 * — matches everything remaining. Only works at the end of a path. 
 + — matches exactly one segment. Works in the middle. 
 
 path "qbit/data/+/dev/*" { ... } # every service's dev
path "identity/oidc/provider/+/authorize" { ... } # from the default policy
 
 Design rule: write small, single-purpose policies and compose them — jikoxpress-prod-read , jikoxpress-prod-write , shared-read — rather than one giant policy per service. 
 Reading the built-in default policy taught four things: 
 
 Everything is self-scoped ( lookup-self , renew-self , revoke-self ) — no token can look up another 
 {{identity.entity.id}} templating resolves per-token at request time — one document, per-caller permissions 
 + demonstrated in the wild 
 cubbyhole/* full CRUD — every token gets private storage nobody else can read, not even root . This is the mechanism behind response wrapping. 
 
 2.10 The KV v2 Path Gotcha 
 The single most common Vault mistake. The CLI lies to you — helpfully. 
 What you type: 
 vault kv get qbit/jikoxpress/prod/postgres
 
 What goes over HTTP: 
 GET /v1/qbit/data/jikoxpress/prod/postgres
 
 Vault tells you this in every output: Secret Path: qbit/data/jikoxpress/prod/postgres . 
 Why the extra segment — KV v2 splits operations into sub-paths: 
 
 
 
 Real API path 
 Purpose 
 
 
 
 
 qbit/data/... 
 the values 
 
 
 qbit/metadata/... 
 version history, max_versions , delete-all 
 
 
 qbit/delete/... 
 soft-delete 
 
 
 qbit/undelete/... 
 restore 
 
 
 qbit/destroy/... 
 permanent wipe 
 
 
 
 All five appear in normal CLI output once you look: 
 Success! Data written to: qbit/undelete/jikoxpress/prod/postgres
Success! Data written to: qbit/destroy/jikoxpress/prod/postgres
Metadata Path: qbit/metadata/jikoxpress/prod/postgres
 
 The kv command inserts the right segment for you. Policies have no wrapper. 
 Write the obvious-looking thing and you get permission denied with no hint why: 
 path "qbit/jikoxpress/prod/*" { capabilities = ["read"] } # ← WRONG
 
 The useful consequence — separated paths let you grant operations independently: 
 path "qbit/data/jikoxpress/prod/*" { capabilities = ["read"] }
path "qbit/metadata/jikoxpress/prod/*" { capabilities = ["read", "list"] }
path "qbit/destroy/jikoxpress/prod/*" { capabilities = ["deny"] }
 
 An app that reads secrets has no business destroying them. 
 KV v1 has none of this — one path, no /data/ . Which is exactly why people who learned v1 get caught. 
 2.11 Path Design Is Policy Design 
 service/environment above component. Not negotiable. 
 Right — qbit/jikoxpress/prod/* : 
 path "qbit/data/jikoxpress/prod/*" { capabilities = ["read"] }
 
 One rule. Covers every component, including ones added next year, automatically. 
 Wrong — qbit/jikoxpress/db/prod : 
 path "qbit/data/jikoxpress/db/prod" { ... }
path "qbit/data/jikoxpress/snippe/prod" { ... }
path "qbit/data/jikoxpress/redis/prod" { ... }
...
 
 One rule per component, forever. 
 The dangerous failure mode: six months later you add a component, forget the policy line, and the app breaks at 2am. Under pressure you "fix" it with qbit/data/jikoxpress/* — which now grants dev and staging too . The isolation is gone and nothing tells you. 
 The rule: put the thing you cut permissions along highest in the path. 
 Adopted structure: 
 qbit/<service>/<environment>/<component>

qbit/jikoxpress/prod/postgres
qbit/jikoxpress/prod/snippe-psp
qbit/nexgate/staging/minio
qbit/nexgate/prod/jwt-signing
qbit/mauzodukani/prod/postgres
qbit/glueemail/prod/mailcow
qbit/textfy/prod/vodacom-smpp
qbit/shared/prod/openai ← belongs to no single product
qbit/shared/prod/cloudflare
 
 Directories aren't real. There's no mkdir . Write to a deep path and every intermediate level springs into existence, inferred. 
 2.12 Auth Methods 
 
 
 
 Method 
 For 
 Notes 
 
 
 
 
 Token 
 Bootstrap, break-glass 
 Always enabled, can't be disabled 
 
 
 AppRole 
 Machines 
 Role ID + Secret ID. Primary machine auth. 
 
 
 Userpass 
 Humans 
 Simple; pair with MFA 
 
 
 OIDC / JWT 
 Humans via SSO, CI/CD 
 Keycloak, Authentik, Google Workspace 
 
 
 LDAP 
 Humans via directory 
 If you run one 
 
 
 TLS certificates 
 Machines with mTLS 
 Strong, more setup 
 
 
 Kubernetes 
 Pods 
 If you move to K8s 
 
 
 GitHub 
 Humans / CI 
 Ties access to org membership 
 
 
 
 Vault does have users — userpass or oidc . Same machinery as AppRole, different door. 
 2.13 AppRole — The Hotel 
 The analogy that worked: 
 
 Policy = the rule sheet on the wall. "This kind of guest may open rooms 301–305." Not a key. 
 AppRole = the front desk. Checks ID, hands out keycards. 
 Role ID = your booking name. "I'm JikoXpress." Not secret. 
 Secret ID = your booking reference. "Here's proof." Secret. 
 Token = the keycard. Expires. This is what you actually carry and use. 
 
 Two ways to get a keycard: 
 Cheating (lab Step 6):
 You (root) → "make me a keycard" → token

Real (lab Step 8):
 App → role_id + secret_id → front desk checks → token
 
 Same keycard. Different route. An app can't use the first — it would need root, which is the problem being solved. 
 Who has what: 
 
 
 
 Thing 
 Lives where 
 Secret? 
 Expires? 
 
 
 
 
 Policy 
 In Vault 
 No 
 No 
 
 
 Role ID 
 App config, Git 
 Not really 
 No 
 
 
 Secret ID 
 Delivered to app 
 Yes 
 Yes 
 
 
 Token 
 App memory 
 Yes 
 Yes 
 
 
 
 Order of operations: 
 ONCE, by admin:
 1. write policy
 2. create approle role, attach policy by name

EVERY APP START:
 3. app sends role_id + secret_id
 4. Vault returns token
 5. app reads secrets

EVERY ~45 MIN:
 6. app renews token

AFTER token_max_ttl:
 7. back to step 3
 
 Steps 3–7 are what "automatic" means. Spring Cloud Vault or Vault Agent does them. You write config, not code. 
 Why split into two halves? Different security properties, different delivery paths. Neither alone works. Role ID can sit in Git; Secret ID is delivered at runtime and expires. The thing in your repo isn't sufficient to authenticate. 
 The Secret ID is not stored readably. vault write -f .../secret-id is a write — it generates one, stores a hash, returns plaintext once . No read gets it back. vault list .../secret-id shows accessors (for revocation), never the value. 
 2.14 Tokens 
 
 
 
 Type 
 Behaviour 
 
 
 
 
 Service 
 Default; server-side; renewable, revocable, supports children 
 
 
 Batch 
 Lightweight encrypted blobs; not stored; can't renew 
 
 
 Root 
 root policy; unlimited; no TTL 
 
 
 Periodic 
 Renews indefinitely within the period 
 
 
 Orphan 
 No parent; survives parent revocation 
 
 
 
 Tokens form a tree — revoking a parent revokes children. 
 Root token lookup, annotated (from the lab): 
 ttl 0s ← NOT expired. NO EXPIRY.
expire_time <nil> ← never
policies [root] ← everything
orphan true ← can't be killed via a parent
num_uses 0 ← unlimited
id root ← guessable (dev mode)
 
 AppRole token lookup, same fields: 
 ttl 56m16s ← counting down
expire_time 2026-08-06T11:02:24Z ← real
policies [default jikoxpress-prod] ← scoped
path auth/approle/login ← how it was obtained
meta map[role_name:jikoxpress-prod] ← attribution
entity_id bcb55ff4-... ← identity entity
 
 token_meta_role_name is the audit win. Not "root did it" — "jikoxpress-prod did it." If NexGate's role ever reads a JikoXpress secret, you see it. 
 2.15 Leases, TTLs, and the Renewal Trap 
 
 TTL — how long valid 
 Max TTL — hard ceiling; renewal cannot exceed it 
 Renewal — extending before expiry 
 Revocation — killing it early 
 
 Precedence: system default → mount tuning → role config → request-time. Each may only shorten . 
 768h = 32 days = the system default. A mount showing 768h isn't "configured" — it's "nobody set this." 
 The trap, discovered in the lab: every expiry you add is a renewal mechanism you must build. 
 token_ttl=1h → solved by renewal (Spring Cloud Vault does it in a background thread).
 token_max_ttl=4h → renewal can't exceed it; the app must be able to log in again .
 secret_id_ttl=24h → this is the one that kills services . When it expires, login itself fails. Renewal won't save you. 
 Four ways to handle Secret ID expiry: 
 
 Non-expiring — secret_id_ttl=0 , secret_id_num_uses=0 . Simplest, weakest. Still far better than root. 
 Vault Agent — sidecar handles login, renewal, re-auth. Writes a token to a file the app reads. The standard answer. 
 Long TTL, rotated on deploy — 720h , with Komodo issuing fresh on each deploy. 
 Response-wrapped at container start — most secure, most moving parts. 
 
 Recommended start: secret_id_ttl=0 . Get the flow working before fighting two problems at once. 
 2.16 Attacker Analysis 
 Can a thief renew a stolen token? Yes. Vault can't distinguish them. 
 But token_max_ttl is the wall. Renewal extends, never resets; max TTL is measured from creation . 
 token_ttl=1h, token_max_ttl=4h
Steal at hour 0 → renew, renew, renew → at hour 4: DEAD
 
 token_max_ttl is your actual worst-case exposure window. token_ttl is just how often renewal happens. 
 Hierarchy of theft: 
 
 
 
 Stolen 
 Exposure 
 
 
 
 
 Token only 
 Until token_max_ttl — hours 
 
 
 Token + Secret ID 
 Until secret_id_ttl — days 
 
 
 Role ID + Secret ID 
 Until you rotate 
 
 
 Root token 
 Forever, everything, silently 
 
 
 
 Which is why the two halves must not live together. If docker inspect yields both, the design bought you nothing. 
 Where each realistically leaks from: 
 
 
 
 Role ID 
 Secret ID 
 
 
 
 
 Committed compose/Komodo config 
 File on server 
 
 
 Baked into an image ( ENV ) 
 Env var → docker inspect 
 
 
 .env on the server 
 CI/CD secret store 
 
 
 Kubernetes ConfigMap 
 /proc/<pid>/environ 
 
 
 
 Shell history 
 
 
 
 App logs, if the app logs its config at startup 
 
 
 
 The attack, concretely: 
 docker inspect jikoxpress | grep -i vault
# VAULT_ROLE_ID=8c3f2a1b-...
# VAULT_SECRET_ID=d4e5f6a7-...
 
 Both halves, one command. 
 Free hardening not yet used — CIDR binding: 
 vault write auth/approle/role/jikoxpress-prod \
 secret_id_bound_cidrs="161.97.163.158/32" \
 token_bound_cidrs="161.97.163.158/32"
 
 Stolen credentials become useless anywhere but qbit-spark . The attacker would have to run the attack from your server . 
 Response wrapping — the strongest link: 
 vault write -f -wrap-ttl=60s auth/approle/role/jikoxpress-prod/secret-id
 
 Returns a wrapper token, not the Secret ID. Single use. If someone intercepts and unwraps first, your app's unwrap fails — and that failure is your intrusion alarm. Not just secrecy: detection . 
 Revocation: 
 vault token revoke <token>
vault write auth/approle/role/X/secret-id-accessor/destroy secret_id_accessor="<acc>"
vault write -f auth/approle/role/X/secret-id # rotate: kills existing
vault lease revoke -prefix auth/approle/
 
 The honest summary: you cannot prevent a stolen token from being used. You can only bound how long it works and detect that it's happening. Short max TTLs and audit logs are the two levers. 
 2.17 delete / undelete / destroy / revoke 
 Two different axes. Constantly conflated. 
 About the data (KV): 
 
 
 
 Command 
 Effect 
 Reversible 
 
 
 
 
 kv delete 
 Marks a version deleted; data still on disk 
 ✅ undelete 
 
 
 kv undelete 
 Clears the mark 
 — 
 
 
 kv destroy 
 Wipes that version's data; metadata tombstone remains 
 ❌ Never 
 
 
 kv metadata delete 
 Nukes the whole secret, all versions 
 ❌ Never 
 
 
 
 Recycle bin → shredder → burning the filing cabinet. 
 About access (tokens/leases): 
 vault token revoke <token>
vault lease revoke <lease_id>
vault lease revoke -prefix database/
 
 Revoking a dynamic secret lease doesn't just invalidate a record — Vault reaches into the backend and runs DROP ROLE . The user genuinely ceases to exist. 
 
 delete/destroy = removing the secret 
 revoke = removing access 
 
 Applied to the incident: the OpenAI key leaked. kv destroy removes it from Vault — the stolen key still works. vault token revoke is irrelevant — the attacker never had a Vault token. Only revoking at OpenAI's dashboard stopped it. 
 That's the limit of static secrets. Vault stores and guards them; it cannot revoke what a third party issued. 
 The contrast: a database engine credential → vault lease revoke drops the PostgreSQL user directly. Vault issued it, so Vault can kill it. That's the difference between storing secrets and owning them. 
 KV v2 metadata settings worth using: 
 vault kv metadata put -max-versions=10 qbit/jikoxpress/prod/postgres
 
 
 max_versions 0 = unlimited history. Every rotation accumulates forever; old passwords stay readable. 
 delete_version_after 0s = never auto-delete. Set 720h to self-expire. 
 cas_required false = Check-And-Set off. Turn on so writes must state which version they replace — prevents two processes clobbering each other during automated rotation. 
 
 2.18 Audit Devices 
 Log every request and response. Vault refuses to operate if all enabled audit devices fail to write — fail-closed by design: no audit trail, no service. 
 vault audit enable file file_path=/vault/logs/audit.log
 
 Sensitive values are HMAC'd; paths and field names are plaintext. Safe to ship to Grafana/Loki. You can compute the HMAC of a known value to search for it — forensics without exposure. 
 This is what was missing during the incident. With audit logging you could answer: was the OpenAI key ever read from Vault, by which role, from which IP, when? 
 2.19 Error Codes — Read Them Precisely 
 
 
 
 Error 
 Meaning 
 
 
 
 
 403 permission denied 
 Right verb, path exists, policy says no 
 
 
 404 / No value found 
 Right verb, nothing there 
 
 
 405 unsupported operation 
 Wrong verb — path exists but doesn't accept this method 
 
 
 
 In Step 5 you'll stare at 403s wondering whether the policy or the path is wrong. Knowing these apart is the difference between five minutes and an hour. 
 The debugging tool: 
 vault token capabilities <token> qbit/data/jikoxpress/prod/postgres
# → read
vault token capabilities <token> qbit/data/nexgate/prod/postgres
# → deny
 
 Ask Vault directly instead of guessing. 
 Deliberate ambiguity: the UI says "You do not have the required permissions or the directory does not exist." It refuses to say which — otherwise an attacker maps your infrastructure by probing. Same reason good login forms say "invalid username or password." 
 
 PART 3 — The Lab 
 Ground rules: nothing touches production. Don't proceed until the ✅ check passes. After each step, close your notes and explain it aloud. 
 Setup — Docker, not apt 
 # ~/vault-lab/docker-compose.yml
services:
 vault:
 image: hashicorp/vault:1.20.3
 container_name: vault-lab
 ports:
 - "8200:8200"
 cap_add:
 - IPC_LOCK
 environment:
 VAULT_DEV_ROOT_TOKEN_ID: "root"
 VAULT_DEV_LISTEN_ADDRESS: "0.0.0.0:8200"
 VAULT_ADDR: "http://127.0.0.1:8200"
 VAULT_TOKEN: "root"
 command: server -dev
 volumes:
 - ./policies:/policies
 
 Every line explained: 
 cap_add: IPC_LOCK — grants the mlock() capability. Vault holds decrypted secrets and the master key in RAM; if the kernel swaps that to disk, secrets get written to the swap file in plaintext . mlock() prevents it. Barely matters in dev mode, but correct from the start so the pattern carries. 
 VAULT_DEV_ROOT_TOKEN_ID: "root" — overrides the random dev token. Convenience only. Never anywhere real. 
 VAULT_DEV_LISTEN_ADDRESS: "0.0.0.0:8200" — which interface Vault binds to inside the container. 127.0.0.1 = container-only; Docker's port mapping would have nothing to forward to. 
 
 Security note: 0.0.0.0 inside + ports: mapping outside = exposed on every host interface. On a laptop behind a router, fine. On a VPS with no firewall, this exact pattern is how services get accidentally exposed to the internet. Possibly relevant to how the OpenAI key leaked. 
 
 command: server -dev — dev mode means: in-memory storage ( all data lost on restart ), auto-initialized and auto-unsealed, TLS disabled, KV v2 pre-mounted at secret/ , root token printed to logs. Dev mode removes every barrier to learning and every security guarantee. 
 VAULT_ADDR is not optional. Without it the CLI defaults to https://127.0.0.1:8200 , tries a TLS handshake against a plain-HTTP dev server, and fails with http: server gave HTTP response to HTTPS client . 
 Optional alias: 
 alias v='docker compose exec vault vault'
 
 
 Step 1 — Run it 
 docker compose up -d
sleep 3 # avoid the boot race
docker compose exec vault vault status
 
 ✅ Sealed false . 
 Read the boot logs. They print an unseal key and a root token in plaintext to stdout . In production that combination in a log file is game over. 
 The observation: the unseal key and cluster ID change on every restart . inmem = brand-new Vault each time. On persistent storage, the unseal key is generated once at initialization , never again. 
 Question: why would auto-unseal-with-the-key-in-the-logs be unacceptable on vault.qbitspark.com ? 
 Answer: both protections are nullified at once. Already unsealed → encryption gives you nothing, secrets are decrypted in memory now. Key in the logs → even if sealed, the key to reopen it is in docker compose logs . An attacker with log read access needs neither to break encryption nor gather shares. 
 
 The trap: Vault's protection is only as good as the separation between the encrypted data and the keys that decrypt it. Dev mode puts them together. So does printing unseal keys to logs. So does storing shares on the same server. So does one person holding all five shares in one folder. 
 
 
 Step 2 — Tokens and your first secret 
 docker compose exec vault vault token lookup
docker compose exec vault vault kv put secret/hello message="my first secret" owner="kibuti"
docker compose exec vault vault kv get secret/hello
docker compose exec vault vault kv get -field=message secret/hello
docker compose exec vault vault kv get -format=json secret/hello
 
 ✅ Explain: what is the token, what is the secret, which is more dangerous to leak? 
 The answer, refined: it depends on scope. A narrow token is less dangerous than a big secret. A root token is worse than any single secret, because it's the key to all of them. 
 
 
 
 Leaked 
 Damage 
 
 
 
 
 One secret 
 That one system. Bad, bounded. 
 
 
 Scoped token (1h) 
 What that role can read, for an hour 
 
 
 Root token 
 Everything, forever, silently 
 
 
 
 Which is why the fix isn't "protect tokens harder" — it's make tokens small and short-lived so leaking one barely matters. 
 -field output is designed for scripting: 
 DB_PASS=$(vault kv get -field=password qbit/jikoxpress/prod/db)
 
 
 Step 3 — Mount your own engine 
 docker compose exec vault vault secrets list # baseline first
docker compose exec vault vault secrets enable -path=qbit kv-v2
docker compose exec vault vault secrets tune -description="QBIT SPARK application secrets" qbit/
 
 Baseline on a dev server: cubbyhole/ , identity/ , secret/ , sys/ . On real Vault, secret/ wouldn't be there — three system mounts and nothing else. Everything useful, you mount yourself. 
 Explore the engine zoo (mount, look, remove): 
 docker compose exec vault vault secrets enable database
docker compose exec vault vault secrets enable pki
docker compose exec vault vault secrets enable transit
docker compose exec vault vault secrets enable ssh
docker compose exec vault vault secrets enable totp
docker compose exec vault vault secrets list -detailed
 
 Four things in the -detailed table: 
 
 Options = map[version:2] — the only place the table reveals KV v2 vs v1. Both show Plugin kv . 
 Default TTL / Max TTL = system — inherit the 768h default. On database/ these become the real ceiling. 
 Seal Wrap true — only on sys/ . Extra encryption layer using the seal mechanism, applied to Vault's own control plane. 
 Running Version — KV shows v0.24.0+builtin while others show v1.20.3+builtin.vault . The KV engine versions independently of Vault core — which is why plugin upgrades and Vault upgrades are different operations. 
 
 Dynamic engines are inert until configured: 
 docker compose exec vault vault list database/config # No value found
docker compose exec vault vault list transit/keys
 
 KV works the instant you mount it. Dynamic engines need to know what system to talk to and what to create there . 
 Cleanup — and note the danger: 
 docker compose exec vault vault secrets disable database
 
 disable destroys all data under that path. No confirmation prompt. On real Vault, vault secrets disable pki deletes your entire certificate authority. 
 Then build the tree: 
 docker compose exec vault vault kv put qbit/jikoxpress/prod/postgres username="jx" password="x"
docker compose exec vault vault kv put qbit/jikoxpress/prod/snippe-psp api_key="x" merchant_id="x"
docker compose exec vault vault kv put qbit/jikoxpress/staging/postgres username="jx" password="x"
docker compose exec vault vault kv put qbit/nexgate/prod/postgres username="ng" password="x"
docker compose exec vault vault kv put qbit/nexgate/prod/rabbitmq username="ng" password="x"
docker compose exec vault vault kv put qbit/nexgate/prod/jwt-signing private_key="x" public_key="x"
docker compose exec vault vault kv put qbit/nexgate/staging/minio access_key="x" secret_key="x"
docker compose exec vault vault kv put qbit/mauzodukani/prod/postgres username="md" password="x"
docker compose exec vault vault kv put qbit/glueemail/prod/mailcow admin_user="x" admin_pass="x"
docker compose exec vault vault kv put qbit/textfy/prod/vodacom-smpp system_id="x" password="x"
docker compose exec vault vault kv put qbit/shared/prod/openai api_key="sk-placeholder"
docker compose exec vault vault kv put qbit/shared/prod/cloudflare api_token="x"
 
 
 All values are x deliberately. Building the habit of never typing a real credential into a lab is worth more than the lab itself. 
 
 Walk the tree: 
 docker compose exec vault vault kv list qbit/
docker compose exec vault vault kv list qbit/jikoxpress/
docker compose exec vault vault kv list qbit/jikoxpress/prod/
 
 Trailing slashes mark directories: 
 jikoxpress/ ← has children
prod/ ← has children
postgres ← no slash: an actual secret
 
 list never shows values. Separate capability from read — you can let a junior dev see what exists without reading contents. 
 
 Step 4 — Versioning 
 docker compose exec vault vault kv put qbit/jikoxpress/prod/postgres username="jx" password="NEW-v2"
docker compose exec vault vault kv get qbit/jikoxpress/prod/postgres
docker compose exec vault vault kv get -version=1 qbit/jikoxpress/prod/postgres
docker compose exec vault vault kv metadata get qbit/jikoxpress/prod/postgres

docker compose exec vault vault kv delete qbit/jikoxpress/prod/postgres
docker compose exec vault vault kv undelete -versions=2 qbit/jikoxpress/prod/postgres
docker compose exec vault vault kv destroy -versions=1 qbit/jikoxpress/prod/postgres
 
 Gotcha found live: kv get without -version always fetches the latest . Undeleting v2 while v3 is deleted looks like the undelete failed. Use kv metadata get — it shows every version and its state at once. 
 ✅ Metadata should show: v1 destroyed true , v2 clean, v3 deletion_time set. 
 Why it matters for rotation: 
 1. Write new version ← old still live
2. Restart services
3. Verify healthy
4. THEN revoke at provider
5. Optionally destroy old version
 
 If step 3 fails, kv get -version=<previous> recovers instantly. 
 The trap: delete looks destructive but isn't. People think a secret is gone when it's fully recoverable by anyone with undelete . To actually remove a leaked credential from Vault, it's destroy . 
 
 Step 5 — First policy 
 mkdir -p ~/vault-lab/policies

cat > ~/vault-lab/policies/jikoxpress-prod.hcl <<'EOF'
path "qbit/data/jikoxpress/prod/*" {
 capabilities = ["read"]
}
path "qbit/metadata/jikoxpress/prod/*" {
 capabilities = ["list"]
}
EOF

docker compose exec vault vault policy write jikoxpress-prod /policies/jikoxpress-prod.hcl
docker compose exec vault vault policy read jikoxpress-prod
docker compose exec vault vault policy list
docker compose exec vault vault policy read default # read this one properly
 
 Anatomy of the command: 
 vault policy write jikoxpress-prod /policies/jikoxpress-prod.hcl
 │ │
 │ └─ file, INSIDE the container
 └─ policy name (unrelated to the filename)
 
 The file and the policy name are independent. Keep them matching by convention, not requirement. 
 /policies/ is the container path. Your volume maps ./policies → /policies . 
 Once written, the file no longer matters. Vault stores the policy internally; delete the .hcl and it keeps working. The file is source, not runtime — which is why policies belong in Git, giving you version history and review while Vault holds current state. 
 
 Step 6 — Prove the denial ⭐ 
 The most important step. A policy tested only for success is untested — you might still be root. 
 docker compose exec vault vault token create -policy=jikoxpress-prod -ttl=30m
TOK="<paste>"

docker compose exec -e VAULT_TOKEN=$TOK vault vault kv get qbit/jikoxpress/prod/postgres # ALLOW
docker compose exec -e VAULT_TOKEN=$TOK vault vault kv get qbit/jikoxpress/staging/postgres # DENY
docker compose exec -e VAULT_TOKEN=$TOK vault vault kv get qbit/nexgate/prod/postgres # DENY
docker compose exec -e VAULT_TOKEN=$TOK vault vault kv get qbit/shared/prod/openai # DENY
docker compose exec -e VAULT_TOKEN=$TOK vault vault kv put qbit/jikoxpress/prod/postgres password="hacked" # DENY
 
 ✅ Four 403 permission denied . Those errors are the proof. 
 The last denial is the subtle one: PUT blocked on a path the token can read. Same path, different verb, different answer. Least privilege at the operation level, not just the path level. 
 Note: token_policies ["default" "jikoxpress-prod"] — additive, as expected. 
 
 Step 7 — Enable AppRole 
 docker compose exec vault vault auth enable approle

docker compose exec vault vault write auth/approle/role/jikoxpress-prod \
 token_policies="jikoxpress-prod" \
 token_ttl=1h \
 token_max_ttl=4h \
 secret_id_ttl=0

docker compose exec vault vault read auth/approle/role/jikoxpress-prod
 
 Note token_policies="jikoxpress-prod" is just a name. The AppRole never mentions paths — the policy owns those. To change what a service can access, edit the policy; existing tokens pick it up on their next request. 
 Two empty fields worth filling in production: secret_id_bound_cidrs and token_bound_cidrs . 
 
 Lab warning: don't set CIDRs to your production IP in the lab — your requests come from inside a Docker container and you'll lock yourself out. 
 
 
 Step 8 — Log in as a service ⭐ 
 RID=$(docker compose exec -T vault vault read -field=role_id auth/approle/role/jikoxpress-prod/role-id)
SID=$(docker compose exec -T vault vault write -f -field=secret_id auth/approle/role/jikoxpress-prod/secret-id)

docker compose exec -e VAULT_TOKEN= -T vault vault write auth/approle/login \
 role_id="$RID" secret_id="$SID"
 
 -e VAULT_TOKEN= blanks the root token — proving login needs no prior credential. 
 APPTOK="<paste>"
docker compose exec -e VAULT_TOKEN=$APPTOK vault vault token lookup
docker compose exec -e VAULT_TOKEN=$APPTOK vault vault kv get qbit/jikoxpress/prod/postgres # ALLOW
docker compose exec -e VAULT_TOKEN=$APPTOK vault vault kv get qbit/nexgate/prod/postgres # DENY
docker compose exec -e VAULT_TOKEN=$APPTOK vault vault token renew # note: positional, not -self
 
 ✅ A production secret retrieved with zero root token involvement . 
 Where does the token come from at all? The CLI resolution order: 
 
 -address / explicit flag 
 VAULT_TOKEN environment variable ← what the compose file sets 
 ~/.vault-token (written by vault login ) 
 
 Being inside the container grants nothing. No implicit auth, no localhost exemption, no container trust. Prove it: 
 docker compose exec -e VAULT_TOKEN= vault vault kv get qbit/jikoxpress/prod/db # denied
 
 Vault is an HTTP API with a token header. The CLI is a wrapper: 
 curl -H "X-Vault-Token: root" http://127.0.0.1:8200/v1/qbit/data/jikoxpress/prod/db
 
 
 The uncomfortable parallel: "what token is being used?" — the answer was root, because it sat in a config file you edited and stopped thinking about. That's the shape of the production problem. 
 
 
 The UI Experiment ⭐ 
 Open http://127.0.0.1:8200 . 
 Sidebar structure — three separate things, three separate menus: 
 
 Secrets Engines — where secrets live 
 Access → Auth Methods — approle, token, userpass 
 Policies — standalone 
 
 Then: sign out, sign back in with the AppRole token. 
 What happens: the interface collapses. No Policies. No Access. No Seal Vault. Instead of a browsable tree, a box saying "Type the path of the secret you want to view." 
 Why: the policy grants list on qbit/metadata/jikoxpress/prod/* but not on qbit/ itself. The UI can't enumerate the top level, so it falls back to asking. 
 
 Type jikoxpress/prod/ → lists postgres , snippe-psp 
 Type nexgate/prod/ → "You do not have the required permissions or the directory does not exist." 
 
 This is what a compromised JikoXpress service looks like from the attacker's side. Two secrets. Not NexGate's JWT keys, not Cloudflare, not Vodacom. Compare to a stolen root token: everything, in a navigable tree. 
 A design decision it surfaces: apps don't browse — they fetch known paths, so read-only on exact paths is right. Humans with scoped policies find the inability to navigate genuinely annoying, so add list grants deliberately: 
 path "qbit/metadata" { capabilities = ["list"] }
path "qbit/metadata/jikoxpress" { capabilities = ["list"] }
 
 Also observed: a yellow banner — "We've stopped auto-renewing your token due to inactivity. It will expire in about 1 hour." with a Renew token button. The UI had been renewing in the background all along. That's vault token renew with a nicer face, and exactly what Spring Cloud Vault does in a thread. 
 And: Create secret + still renders even without create capability. Clicking fails with 403. UI affordances aren't permissions. 
 
 Seed script — because inmem wipes on restart 
 cat > ~/vault-lab/seed.sh <<'EOF'
#!/usr/bin/env bash
set -euo pipefail
V="docker compose exec -T vault vault"

$V secrets enable -path=qbit kv-v2 2>/dev/null || true
$V secrets tune -description="QBIT SPARK application secrets" qbit/
$V auth enable approle 2>/dev/null || true

$V kv put qbit/jikoxpress/prod/postgres username="jx" password="x"
$V kv put qbit/jikoxpress/prod/snippe-psp api_key="x" merchant_id="x"
$V kv put qbit/jikoxpress/staging/postgres username="jx" password="x"
$V kv put qbit/nexgate/prod/postgres username="ng" password="x"
$V kv put qbit/nexgate/prod/jwt-signing private_key="x" public_key="x"
$V kv put qbit/nexgate/staging/minio access_key="x" secret_key="x"
$V kv put qbit/mauzodukani/prod/postgres username="md" password="x"
$V kv put qbit/glueemail/prod/mailcow admin_user="x" admin_pass="x"
$V kv put qbit/textfy/prod/vodacom-smpp system_id="x" password="x"
$V kv put qbit/shared/prod/openai api_key="sk-placeholder"

$V policy write jikoxpress-prod /policies/jikoxpress-prod.hcl
$V write auth/approle/role/jikoxpress-prod \
 token_policies="jikoxpress-prod" token_ttl=1h token_max_ttl=4h secret_id_ttl=0

echo "✓ seeded"
EOF
chmod +x ~/vault-lab/seed.sh
 
 Wipe and rebuild freely: 
 docker compose down && docker compose up -d && sleep 3 && ./seed.sh
 
 
 Step 9 — All services (pending) 
 #!/usr/bin/env bash
set -euo pipefail
V="docker compose exec -T vault vault"

declare -A SERVICES=(
 [jikoxpress]="dev staging prod"
 [nexgate]="dev staging prod"
 [mauzodukani]="dev prod"
 [glueemail]="prod"
 [textfy]="dev prod"
 [shared]="prod"
)

for svc in "${!SERVICES[@]}"; do
 for env in ${SERVICES[$svc]}; do
 name="${svc}-${env}"

 cat > "policies/${name}.hcl" <<EOF
path "qbit/data/${svc}/${env}/*" {
 capabilities = ["read"]
}
path "qbit/metadata/${svc}/${env}/*" {
 capabilities = ["list"]
}
EOF
 $V policy write "$name" "/policies/${name}.hcl"

 if [ "$env" = "prod" ]; then TTL=1h; MAXTTL=4h; else TTL=4h; MAXTTL=12h; fi

 $V write "auth/approle/role/${name}" \
 token_policies="$name" token_ttl="$TTL" token_max_ttl="$MAXTTL" secret_id_ttl=0
 echo "✓ ${name} (ttl=${TTL})"
 done
done
 
 Production TTLs are shorter because blast radius matters more there. 
 Step 10 — Isolation verification (pending) 
 #!/usr/bin/env bash
set -uo pipefail
V="docker compose exec -T vault vault"
PASS=0; FAIL=0

check() {
 local role="$1" path="$2" expect="$3" rid sid tok result
 rid=$($V read -field=role_id "auth/approle/role/${role}/role-id")
 sid=$($V write -f -field=secret_id "auth/approle/role/${role}/secret-id")
 tok=$($V write -field=token auth/approle/login role_id="$rid" secret_id="$sid")

 if docker compose exec -T -e VAULT_TOKEN="$tok" vault vault kv get "$path" >/dev/null 2>&1
 then result="ALLOW"; else result="DENY"; fi

 if [ "$result" = "$expect" ]; then
 echo " ✓ ${role} → ${path} = ${result}"; PASS=$((PASS+1))
 else
 echo " ✗ ${role} → ${path} = ${result} (expected ${expect})"; FAIL=$((FAIL+1))
 fi
}

echo "Positive (must ALLOW):"
check jikoxpress-prod qbit/jikoxpress/prod/postgres ALLOW
check nexgate-staging qbit/nexgate/staging/minio ALLOW

echo "Negative (must DENY):"
check jikoxpress-prod qbit/nexgate/prod/jwt-signing DENY
check jikoxpress-prod qbit/jikoxpress/dev/postgres DENY
check nexgate-staging qbit/nexgate/prod/jwt-signing DENY
check textfy-prod qbit/jikoxpress/prod/postgres DENY
check jikoxpress-prod qbit/shared/prod/openai DENY

echo; echo "Passed: $PASS Failed: $FAIL"
[ "$FAIL" -eq 0 ] || exit 1
 
 Keep this script forever. Run after every policy change, in the lab and in production. 
 Step 11 — Audit logging (pending) 
 docker compose exec vault vault audit enable file file_path=/vault/logs/audit.log
docker compose exec vault vault audit list -detailed
docker compose exec vault vault kv get qbit/jikoxpress/prod/postgres
docker compose exec vault sh -c 'tail -n 5 /vault/logs/audit.log'
 
 ✅ Find your request. Confirm the value is HMAC'd but the path is plaintext. 
 Step 12 — Dynamic database credentials (pending) ⭐ 
 docker run -d --name vault-lab-pg \
 -e POSTGRES_PASSWORD=rootpass -e POSTGRES_DB=jikoxpress \
 -p 5433:5432 postgres:16

docker compose exec vault vault secrets enable database

docker compose exec vault vault write database/config/jikoxpress-pg \
 plugin_name=postgresql-database-plugin \
 allowed_roles="jikoxpress-app" \
 connection_url="postgresql://{{username}}:{{password}}@host.docker.internal:5433/jikoxpress?sslmode=disable" \
 username="postgres" password="rootpass"

docker compose exec vault vault write database/roles/jikoxpress-app \
 db_name=jikoxpress-pg \
 creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; \
 GRANT SELECT, INSERT, UPDATE, DELETE ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
 default_ttl="1h" max_ttl="24h"

docker compose exec vault vault read database/creds/jikoxpress-app
docker compose exec vault vault read database/creds/jikoxpress-app # different every time
 
 The contrast that makes it click: 
 Static (KV):
 vault kv get qbit/jikoxpress/prod/db
 → username: jx_prod, password: s3cr3t
 → same answer, every time, forever

Dynamic (database):
 vault read database/creds/jikoxpress-app
 → v-approle-jikoxpr-x7Kd92 / A1a-8sKdm3nQp (lease 1h)
 vault read database/creds/jikoxpress-app
 → v-approle-jikoxpr-mN4vB1 / Z9z-2pLwq7Rt (lease 1h)
 
 Vault runs CREATE ROLE live, then DROP ROLE an hour later. Verify with \du in psql — the users genuinely exist, then genuinely don't. 
 ✅ Generate twice, confirm both in \du , vault lease revoke one, confirm it's dropped. 
 Steal one of those and you've stolen something that dies on its own. That's the whole game. 
 
 PART 4 — Mistakes File 
 The book's differentiator. Real errors from the real session. 
 4.1 The invisible leading space 
 What happened: a \ line continuation with a trailing space made the shell pass mesage=... as one argument. Vault stored the key literally as " mesage" . 
 Why it was confusing: kv get displayed it as mesage — the table view strips the visual space. -field=mesage then said "not present in secret." 
 Diagnosis: 
 vault kv get -format=json secret/hello
# "data": { " mesage": "my first secret" }
 
 Lessons: 
 
 When a field "exists but isn't found," go straight to -format=json . The pretty output lies about whitespace. 
 Vault has no schema and no validation . It will happily store PASSWROD , api_ky , mesage . Then your app looks for password , finds nothing, and fails at runtime with a confusing error. Naming conventions are the only defence. 
 
 4.2 The boot race 
 Symptom: dial tcp 127.0.0.1:8200: connect: connection refused immediately after docker compose up -d . 
 Cause: the container reported "Started" ~1.5s before Vault bound the port. 
 Lesson: connection refused means nothing is listening — either not booted yet or crashed. Check logs before guessing. docker compose logs vault --tail=40 . Reading Vault's startup logs is a skill you'll use constantly. 
 Fix: sleep 3 . 
 4.3 VAULT_ADDR defaults to HTTPS 
 Symptom: Error checking seal status: http: server gave HTTP response to HTTPS client 
 Cause: the CLI defaults to https://127.0.0.1:8200 ; dev mode serves plain HTTP. 
 Lesson: VAULT_ADDR isn't optional. The CLI has no idea where your server is or what protocol it speaks. 
 4.4 405 ≠ 404 ≠ 403 
 Symptom: vault read database/config → 405 unsupported operation . Expected "not configured." 
 Cause: database/config only supports LIST . Wrong verb, not missing data. 
 Lesson: learn the three apart (see §2.19). It's the difference between five minutes and an hour when debugging policies. 
 4.5 kv get always fetches latest 
 Symptom: kv undelete -versions=2 succeeded, but kv get still showed a deleted secret. 
 Cause: kv get without -version fetches the latest version — v3, still deleted. v2 was correctly restored. 
 Lesson: use kv metadata get to see all versions and states at once. 
 4.6 token renew -self isn't a flag 
 Symptom: flag provided but not defined: -self 
 Fix: vault token renew — bare, renews the token in VAULT_TOKEN . 
 4.7 Missing exec vault 
 Symptom: unknown docker command: "compose vault" 
 Cause: docker compose vault ... instead of docker compose exec vault vault ... . The pattern is: first vault = service name, second = binary. 
 4.8 Expired token = working as designed 
 Symptom: UI login fails with invalid token . 
 Cause: token_ttl=1h elapsed. (Or the container restarted — inmem wipes tokens.) 
 Lesson — the mental shift: the instinct was "something is broken." Nothing broke. Tokens are supposed to stop working. Getting a new one is routine, not recovery. Apps do this automatically; you do it by hand because you're learning. 
 4.9 The word "secret" means two things 
 secret/ in a command is a mount path — just a folder name from dev mode. "A secret" in conversation is the sensitive data . Moving to qbit/ removes the ambiguity. 
 4.10 The /data/ insertion 
 See §2.10. Expect to hit it. Everyone does. 
 
 PART 5 — Humans, Machines & MFA 
 5.1 Two Populations 
 
 
 
 
 Humans 
 Machines 
 
 
 
 
 Auth method 
 Userpass, OIDC, LDAP 
 AppRole, TLS cert, K8s 
 
 
 MFA 
 Required 
 N/A 
 
 
 Token TTL 
 8h 
 1–4h, auto-renewed 
 
 
 Access 
 Broad but read-mostly, no prod 
 Narrow, exactly what's needed 
 
 
 Audit shows 
 Named individual 
 Named service 
 
 
 
 Current state: one root token serving both. That's the thing to fix. 
 5.2 Enabling Users 
 vault auth enable userpass

vault write auth/userpass/users/kibuti \
 password="..." token_policies="vault-admin" \
 token_ttl=8h token_max_ttl=12h

vault write auth/userpass/users/dev1 \
 password="..." token_policies="dev-engineer" token_ttl=8h
 
 Now the audit log says dev1 , not root . 
 5.3 Developer Policy 
 # dev-engineer.hcl
# full CRUD on any service's dev
path "qbit/data/+/dev/*" {
 capabilities = ["create", "read", "update", "delete", "list"]
}
path "qbit/metadata/+/dev/*" {
 capabilities = ["read", "list", "delete"]
}

# staging: read only
path "qbit/data/+/staging/*" {
 capabilities = ["read"]
}
path "qbit/metadata/+/staging/*" {
 capabilities = ["read", "list"]
}

# prod: no rule = denied
 
 Note the + — * only globs at the end of a path. 
 The hard rule: developers should not read production secrets at all. Not "read-only prod" — no prod. If a dev needs the production database password, something is wrong with your deployment process. Apps read prod secrets via AppRole; humans don't need to. 
 Who writes prod secrets? You, as admin. Or a CI pipeline with a narrow write-only policy. 
 5.4 The Policy Set 
 
 
 
 Policy 
 Who 
 Grants 
 
 
 
 
 vault-admin 
 You 
 Everything except root-only ops 
 
 
 dev-engineer 
 Devs 
 CRUD on +/dev/* , read +/staging/* 
 
 
 deploy-pipeline 
 Komodo 
 Generate Secret IDs only 
 
 
 <service>-<env> 
 Apps 
 Read one path 
 
 
 auditor 
 Vendor assessments 
 list on metadata, no read on data 
 
 
 
 The auditor policy is genuinely useful — someone verifies your secrets are properly organised without seeing a single value. 
 5.5 MFA 
 Vault Community has no native login MFA. That's Enterprise. Get it from the identity provider instead. 
 Option A — OIDC with an MFA-capable IdP (best long-term) 
 vault auth enable oidc
vault write auth/oidc/config \
 oidc_discovery_url="https://auth.qbitspark.com/realms/qbit" \
 oidc_client_id="vault" oidc_client_secret="..." default_role="engineer"

vault write auth/oidc/role/engineer \
 bound_audiences="vault" \
 allowed_redirect_uris="https://vault.qbitspark.com/ui/vault/auth/oidc/oidc/callback" \
 user_claim="email" token_policies="dev-engineer" token_ttl=8h
 
 Keycloak or Authentik self-hosted. Bonus: SSO across your other internal tools. 
 Option B — network-layer (fastest for you) 
Traefik + Authelia/Authentik forward-auth in front of the Vault UI. MFA happens before the request reaches Vault. You already run Traefik. 
 Option C — userpass + TOTP engine. Vault's TOTP engine manages codes but doesn't enforce them on login. Partial only. 
 Recommendation: B now, A later. 
 Machines don't get MFA — that's what AppRole + CIDR binding is for. 
 5.6 Break-Glass 
 # generate root (needs 3 shares)
vault operator generate-root -init
vault operator generate-root -nonce=<nonce>
vault operator generate-root -decode=<encoded> -otp=<otp>
vault token revoke <root-token> # IMMEDIATELY after use

# emergency seal
vault operator seal
 
 Document offline, before you need it: 
 
 Where are the 5 shares physically? 
 Who holds which? (currently: you alone — single point of failure) 
 Procedure if you're unavailable? 
 Is the recovery doc accessible without Vault? 
 
 
 Solo-operator note: Shamir assumes multiple people. Split across locations instead. Never all in one place, never all on one machine. Test the recovery once — an untested scheme is a story you tell yourself. 
 
 5.7 Shamir Beyond Vault 
 Same math, real-world uses: 
 
 Cryptocurrency inheritance — seed phrase split 3-of-5 across family, lawyer, safe deposit box. Casa and Unchained Capital are businesses built on this. 
 DNSSEC root key ceremony — seven people worldwide hold physical smartcards, meet in a locked facility on camera. Directly relevant to your TCRA .tz registrar work. 
 Certificate authority root keys — split among executives, stored in HSMs across locations. 
 Nuclear two-person rule — same principle. 
 Estate/banking arrangements — genuine legal instruments. 
 
 For personal property: 
 3-of-5:
 Share 1 → bank safe deposit box (Mbeya)
 Share 2 → family member, different city
 Share 3 → lawyer, sealed envelope with instructions
 Share 4 → encrypted, cloud storage (different provider than your infra)
 Share 5 → physical, hidden at home
 
 Survives fire, theft, one betrayal, one person unreachable. 
 sudo apt install ssss
echo "my-secret" | ssss-split -t 3 -n 5
ssss-combine -t 3
 
 Steel plates beat paper — paper burns, ink fades. Cryptosteel exists; a hardware store and letter punches do the same job cheaper. 
 Limitations, honestly: humans lose things; the instructions must survive too (a share is useless if nobody knows what it is); legal systems don't recognise "3-of-5" the way they recognise a notarised will — use it alongside proper instruments; and test it . 
 
 PART 6 — Rotation 
 6.1 Theory 
 Every credential's compromise probability rises with age and exposure. Rotation bounds the damage window. 
 Maturity ladder: 
 
 ❌ Never rotate — where you were 
 ⚠️ Manual scheduled — where you're going next 
 ✅ Automated rotation of static secrets 
 🏆 Dynamic secrets — rotation becomes irrelevant because nothing is long-lived 
 
 6.2 By Credential Type 
 Root token — generate → use → revoke . Never leave one alive. 
 Unseal keys 
 vault operator rekey -init -key-shares=5 -key-threshold=3
 
 Encryption key (transparent) 
 vault operator rotate
vault operator key-status
 
 AppRole Secret IDs 
 vault write -f auth/approle/role/X/secret-id # new
vault write auth/approle/role/X/secret-id-accessor/destroy \
 secret_id_accessor="<accessor>" # kill old
 
 KV static secrets — order matters 
 1. Create new credential at the provider
2. vault kv put ... ← new version; old still readable
3. Restart/reload services
4. Verify healthy
5. ONLY THEN revoke old at the provider
6. Optionally vault kv destroy -versions=<n>
 
 Revoking before services pick up the new value = outage. 
 Database credentials — move to the database engine; rotation becomes automatic. 
 6.3 Schedule 
 
 
 
 Credential 
 Frequency 
 Method 
 
 
 
 
 Root token 
 Per use 
 generate → use → revoke 
 
 
 Unseal keys 
 Annually or on suspicion 
 operator rekey 
 
 
 Encryption key 
 Quarterly 
 operator rotate 
 
 
 AppRole Secret IDs 
 Per deploy, or quarterly 
 Komodo, or manual 
 
 
 Service tokens 
 1–4h 
 Automatic 
 
 
 Third-party API keys 
 Quarterly + on incident 
 Manual, versioned in KV 
 
 
 DB credentials 
 1h 
 Dynamic engine 
 
 
 TLS certificates 
 90 days 
 PKI engine 
 
 
 Komodo's own Secret ID 
 Quarterly 
 Manual, by SSH 
 
 
 
 6.4 Dynamic Migration Order 
 
 PostgreSQL — all services; highest value 
 RabbitMQ — NexGate messaging 
 MinIO — via AWS engine (S3-compatible STS) 
 PKI — internal CA for NexGate mTLS 
 SSH — signed certs for qbit-spark 
 Transit — JikoXpress financial data 
 
 Success measure: a leaked credential is a non-event because it expired an hour ago. 
 
 PART 7 — Deployment Architecture 
 7.1 Where Credentials Live — Per Environment 
 The one rule: Role ID and Secret ID never in the same place. 
 Local (developer laptop) 
 Role ID → committed in repo (docker-compose.dev.yml)
Secret ID → .vault/secret-id, gitignored
 
 Each dev generates their own Secret ID. Never committed. 
 Staging 
 Role ID → docker-compose.yml on the staging box (Git fine)
Secret ID → /etc/vault/<service>-staging-secret-id
 chmod 400, chown root:root
 
 Production 
 Role ID → docker-compose.yml (Git fine)
Secret ID → /etc/vault/<service>-prod-secret-id
 chmod 400, chown root:root
 
 The secrets themselves 
 In Vault. Nowhere else. No .env , no config file, no baked image layer. Fetched at boot with the AppRole token. 
 7.2 Never Do This 
 environment:
 VAULT_ROLE_ID: "..."
 VAULT_SECRET_ID: "..." # ← docker inspect prints this
 
 One command yields both halves. This is likely how the OpenAI key walked out. 
 7.3 Do This 
 # on the server, as admin
sudo mkdir -p /etc/vault
vault write -f -field=secret_id auth/approle/role/bishamba-prod/secret-id \
 | sudo tee /etc/vault/bishamba-prod-secret-id > /dev/null
sudo chmod 400 /etc/vault/bishamba-prod-secret-id
sudo chown root:root /etc/vault/bishamba-prod-secret-id
 
 services:
 bishamba:
 environment:
 VAULT_ROLE_ID: "8c3f2a1b-4d5e-6f70-8192-a3b4c5d6e7f8"
 volumes:
 - /etc/vault/bishamba-prod-secret-id:/run/secrets/secret-id:ro
 
 #!/bin/sh
# entrypoint.sh
export VAULT_SECRET_ID=$(cat /run/secrets/secret-id)
exec java -jar app.jar
 
 Why file-mount beats env var: docker inspect shows only the mount path. docker compose config shows nothing sensitive. Git holds only the Role ID. 
 Honest limitation: the env var still exists inside the running process, so /proc/<pid>/environ shows it to root. You've removed it from the Docker metadata layer, compose file, and Git — which is where these things actually leak from. Vault Agent removes it entirely. 
 7.4 Spring Boot Integration 
 # application-prod.yml
spring:
 cloud:
 vault:
 uri: https://vault.qbitspark.com
 authentication: APPROLE
 app-role:
 role-id: ${VAULT_ROLE_ID}
 secret-id: ${VAULT_SECRET_ID}
 kv:
 enabled: true
 backend: qbit
 default-context: bishamba/prod
 
 @Value("${postgres.username}")
private String dbUser;

@Value("${postgres.password}")
private String dbPass;
 
 Spring Cloud Vault logs in at startup, fetches qbit/bishamba/prod/* , injects into properties, renews the token in a background thread. No credentials in code, no .env in production. 
 7.5 Three Environments, One Vault 
 Decision: connectivity isn't a constraint, so all three live on the central Vault. 
 for env in local staging prod; do
cat > policies/bishamba-${env}.hcl <<EOF
path "qbit/data/bishamba/${env}/*" {
 capabilities = ["read"]
}
path "qbit/metadata/bishamba/${env}/*" {
 capabilities = ["list"]
}
EOF
vault policy write bishamba-${env} policies/bishamba-${env}.hcl
done

# local — loose, no CIDR (devs move around)
vault write auth/approle/role/bishamba-local \
 token_policies="bishamba-local" \
 token_ttl=12h token_max_ttl=24h secret_id_ttl=0

# staging — bound to the staging box
vault write auth/approle/role/bishamba-staging \
 token_policies="bishamba-staging" \
 token_ttl=4h token_max_ttl=12h secret_id_ttl=720h \
 secret_id_bound_cidrs="<staging-ip>/32"

# prod — tightest
vault write auth/approle/role/bishamba-prod \
 token_policies="bishamba-prod" \
 token_ttl=1h token_max_ttl=4h secret_id_ttl=720h \
 secret_id_bound_cidrs="161.97.163.158/32"
 
 Only two lines of app config differ between environments — default-context , and in local, uri . 
 Two consequences of putting local on the shared Vault: 
 
 Local secrets must be fake. Everyone with a local role can read them. Enforce by convention and review. 
 You need per-developer revocation. Give each dev their own Secret ID , not a shared one. Then you revoke individually and the audit log tells you whose credential was used. 
 
 vault write -f auth/approle/role/bishamba-local/secret-id # rotate all
vault list auth/approle/role/bishamba-local/secret-id # accessors
vault write auth/approle/role/bishamba-local/secret-id-accessor/destroy \
 secret_id_accessor="<accessor>" # kill one
 
 Trade-off recorded (the case for a separate local Vault, should you revisit): 
 
 
 
 
 All three central 
 Local separate 
 
 
 
 
 Same code path 
 ✅ 
 ✅ 
 
 
 Dev needs prod-Vault access 
 ❌ Yes 
 ✅ No 
 
 
 Works offline 
 ❌ 
 ✅ 
 
 
 CIDR binding 
 staging/prod only 
 staging/prod only 
 
 
 Onboarding 
 Issue credentials 
 docker compose up 
 
 
 Dev leaves 
 Must revoke 
 Nothing to revoke 
 
 
 
 7.6 New Project Runbook 
 Phase 1 — you (admin) set up: decide paths, write policies per environment, create AppRoles, grant devs dev-engineer . 
 Phase 2 — devs work in dev: they write their own dev secrets, read via the dev AppRole, never touch prod . 
 Phase 3 — you write real production secrets. The devs who wrote the app never see these values. 
 Phase 4 — deployment: generate Role ID + Secret ID, place them separately (§7.3). 
 Phase 5 — app consumes via Spring Cloud Vault (§7.4). 
 
 The property this buys: a developer can build, test, and ship the entire application without ever knowing a single production credential. If their laptop is stolen, production is unaffected. 
 
 7.7 Komodo Integration 
 Komodo needs a Vault token to mint Secret IDs. That token becomes the new secret zero, so keep it narrow. 
 1. A policy that does one thing 
 # komodo-deployer.hcl
path "auth/approle/role/+/secret-id" {
 capabilities = ["update"]
}
 
 Can mint Secret IDs. Cannot read a single secret, cannot change policies, cannot create roles. 
 2. Komodo gets its own AppRole — same discipline 
 vault policy write komodo-deployer policies/komodo-deployer.hcl

vault write auth/approle/role/komodo-deployer \
 token_policies="komodo-deployer" \
 token_ttl=15m token_max_ttl=30m \
 secret_id_ttl=0 \
 secret_id_bound_cidrs="161.97.163.158/32"
 
 3. You place Komodo's credentials by hand, once, over SSH 
 ssh you@161.97.163.158
sudo mkdir -p /etc/vault

vault read -field=role_id auth/approle/role/komodo-deployer/role-id \
 | sudo tee /etc/vault/komodo-role-id > /dev/null
vault write -f -field=secret_id auth/approle/role/komodo-deployer/secret-id \
 | sudo tee /etc/vault/komodo-secret-id > /dev/null

sudo chmod 400 /etc/vault/komodo-*
sudo chown root:root /etc/vault/komodo-*
 
 services:
 komodo:
 volumes:
 - /etc/vault/komodo-role-id:/run/secrets/role-id:ro
 - /etc/vault/komodo-secret-id:/run/secrets/secret-id:ro
 
 4. Komodo's deploy script 
 #!/usr/bin/env bash
set -euo pipefail
SERVICE="$1" # e.g. bishamba-prod

TOKEN=$(vault write -field=token auth/approle/login \
 role_id="$(cat /run/secrets/role-id)" \
 secret_id="$(cat /run/secrets/secret-id)")

VAULT_TOKEN=$TOKEN vault write -f -field=secret_id \
 "auth/approle/role/${SERVICE}/secret-id" \
 > "/etc/vault/${SERVICE}-secret-id"
chmod 400 "/etc/vault/${SERVICE}-secret-id"

docker compose up -d "$SERVICE"

# after health check passes, revoke older Secret IDs
# Komodo's own token dies in 15 minutes regardless
 
 Revoke old Secret IDs after deploy — otherwise every deploy leaves a valid credential behind. Do the destroy after the new container is healthy, not before. 
 Rotating Komodo's own Secret ID is manual, deliberately: 
 ssh qbit-spark
vault write -f -field=secret_id auth/approle/role/komodo-deployer/secret-id \
 | sudo tee /etc/vault/komodo-secret-id > /dev/null
docker compose restart komodo
 
 It can't be automated from Komodo — that would let a compromised Komodo renew its own access forever. Keeping it manual bounds the chain. 
 7.8 Where the Chain Terminates 
 YOU ← SSH key. The actual root of trust.
 │
 └─ place Komodo's role-id + secret-id (once, by hand)
 │
 └─ KOMODO mints app Secret IDs (every deploy, 15min tokens)
 │
 └─ APPS log in, read their own secrets (every start, 1h tokens)
 
 Your SSH key is the bottom. That's the real secret zero for the whole system — which is why it deserves a passphrase, a hardware key if possible, and an entry in the break-glass documentation. 
 7.9 Secret Zero — The Honest Accounting 
 
 "To get a credential for Vault, you need a credential for Vault." 
 
 Correct. It never disappears. It shrinks. 
 Before: 
 .env on server:
 DB_PASSWORD=real
 SNIPPE_API_KEY=real
 JWT_PRIVATE_KEY=real
 OPENAI_KEY=real
 MINIO_SECRET=real
 
 One file. Everything. Never expires. No log of who read it. 
 After: 
 /etc/vault/prod-secret-id:
 d4e5f6a7-b8c9-...
 
 One file. One UUID. 
 
 
 
 
 Old .env 
 Stolen Secret ID 
 
 
 
 
 What they get 
 Every credential immediately 
 Nothing — need Role ID too 
 
 
 With both halves 
 — 
 One service, one environment 
 
 
 Duration 
 Forever 
 Until rotated 
 
 
 IP-bindable 
 No 
 Yes 
 
 
 Do you find out 
 No 
 Yes — audit log 
 
 
 To revoke 
 Change everything, everywhere 
 One command 
 
 
 
 The ladder of secret-zero strength: 
 
 
 
 Approach 
 Secret zero is... 
 
 
 
 
 Root token in app config 
 The root token. Worst case. 
 
 
 AppRole 
 Secret ID — scoped, expiring, revocable 
 
 
 Response wrapping 
 One-time wrapper — tamper-evident 
 
 
 TLS cert auth 
 A certificate the machine already has 
 
 
 Cloud/K8s identity 
 The platform vouches — no secret at all 
 
 
 
 Why "still in Vault" isn't circular: Vault doesn't protect secrets by hiding them. It protects them by (1) encrypting at rest, (2) authenticating every request, (3) authorising per path, (4) expiring everything, (5) logging every access, (6) revoking centrally. A .env file does none of these six. 
 The realistic goal was never "no secrets anywhere." It's: one small, scoped, expiring, revocable, logged, IP-bound credential instead of a plaintext file containing your entire business. 
 
 PART 8 — Production Migration 
 8.1 Pre-Migration Checklist 
 
 Lab Steps 1–12 complete and understood 
 Isolation verification script passes locally 
 Full backup of production Vault taken and restore tested 
 Storage backend of vault.qbitspark.com identified 
 Unseal key shares located, verified, distributed across locations 
 Audit logging enabled in production 
 Maintenance window agreed 
 Rollback plan written 
 OpenAI leak vector identified and closed 
 
 8.2 Cutover Order — Lowest Risk First 
 
 GlueEmail (smallest blast radius) 
 Textfy dev 
 Mauzodukani dev → prod 
 NexGate staging 
 JikoXpress dev → staging 
 NexGate prod 
 JikoXpress prod (customer payments — last, most carefully) 
 
 Per service: 
 1. Create policy in production Vault
2. Create AppRole
3. Write secrets to new paths (keep old paths intact)
4. Update ONE instance to use AppRole
5. Verify health, logs, functionality
6. Roll out to remaining instances
7. Verify again
8. Remove old path access from any broad policy
9. Delete old secret path
10. Run isolation verification
 
 Never delete the old path until the new one is proven in production. 
 8.3 Post-Migration Verification 
 
 Isolation script passes against production 
 Every service authenticates via AppRole, none via root 
 Root token revoked; none in any browser session or file 
 Human access via OIDC/proxy with MFA 
 Audit logs flowing to Grafana/Loki 
 Alerts: root token use, permission-denial spikes, seal events 
 Rotation schedule documented and calendared 
 Break-glass procedure written and stored offline 
 Backup automation running with tested restore 
 CIDR binding applied to staging and prod roles 
 
 
 PART 9 — Operations 
 9.1 Backup & DR 
 vault operator raft snapshot save vault-$(date +%F).snap
vault operator raft snapshot restore vault-2026-08-05.snap
 
 
 Snapshots are encrypted — useless without unseal keys. Back up both, separately . 
 A backup you haven't restored is not a backup. Test quarterly into a scratch instance. 
 Automate: daily snapshot → encrypted → offsite (MinIO on a different provider, plus one cold copy). 
 
 9.2 Monitoring 
 Prometheus metrics at /v1/sys/metrics?format=prometheus . 
 
 
 
 Condition 
 Severity 
 
 
 
 
 Vault sealed unexpectedly 
 Critical 
 
 
 Root token used 
 Critical 
 
 
 Audit device failure 
 Critical 
 
 
 Permission-denial spike 
 High — attack or misconfiguration 
 
 
 Token creation anomaly 
 High 
 
 
 Lease count growth 
 Medium — leaked unreleased leases 
 
 
 Secret/cert nearing expiry 
 Medium 
 
 
 
 Feed into the existing Grafana stack. 
 9.3 Upgrades 
 Read the changelog for every version between current and target. Snapshot before upgrading, always. Test on the lab instance first. Never skip major versions. Currently 1.20.3 — patch promptly for security fixes. 
 
 PART 10 — The Book 
 10.1 Concept 
 Working title: Vault From Zero to Production: Secrets Management for Small Teams and Solo Builders 
 The gap: HashiCorp's docs are reference material assuming an enterprise platform team. Existing books target large organisations with dedicated security staff. Nobody writes for the solo operator or three-person startup running real production on a handful of VPSes. 
 Unfair advantages: 
 
 Living the problem, not theorising — a real incident, a real recovery 
 African/emerging-market context: modest budgets, self-hosted VPS, no cloud-native luxury 
 Proven format and finishing discipline — EM/RF trilogy, and XMPP From Zero to Production in progress 
 Real multi-product stack (Spring Boot, PostgreSQL, RabbitMQ, MinIO, Docker, Traefik, Komodo), not toy examples 
 
 Running example: mirror the MaasaiChat approach. A fictional composite — "Zanzi Systems," a Tanzanian SaaS operator with three products and one engineer — whose secrets infrastructure is built across the book. Reader follows the arc from .env chaos to dynamic secrets. 
 Series consistency: XMPP From Zero to Production → Vault From Zero to Production . 
 10.2 Structure 
 Part I — The Problem (Ch. 1–4) 
 
 The $8 Lesson: anatomy of a credential leak 
 What Is a Secret, Really? (path vs field vs value) 
 Every Way Secrets Leak — and how each is discovered 
 The Landscape: choosing your tool 
 
 Part II — Fundamentals (Ch. 5–13)
5. Installing and Running Vault (Docker, and why IPC_LOCK )
6. Seal, Unseal, and the Bank in Mbeya
7. Shamir's Secret Sharing — the math, and its life beyond Vault
8. Storage Backends
9. Secrets Engines: static vs dynamic
10. Auth Methods: humans and machines
11. Policies: deny by default
12. The /data/ Gotcha and Other Path Traps
13. Tokens, Leases, and TTLs 
 Part III — Building It (Ch. 14–22)
14. Designing Your Path Hierarchy (why environment sits above component)
15. Writing Policies That Actually Work
16. Testing Denial — the step everyone skips
17. AppRole in Depth: the hotel, and secret zero
18. Integrating Spring Boot
19. Docker and Compose Integration
20. CI/CD with Komodo
21. Human Access, SSO, and MFA on Community Edition
22. Multi-Tenancy: hosting Vault for clients 
 Part IV — Going Dynamic (Ch. 23–28)
23. Dynamic Database Credentials
24. PKI: running your own CA
25. Transit: encryption as a service
26. SSH Certificates
27. Response Wrapping and Tamper-Evident Delivery
28. Rotation Strategy End-to-End 
 Part V — Production (Ch. 29–36)
29. Migrating an Existing System Without Downtime
30. High Availability
31. Backup and Disaster Recovery
32. Monitoring and Alerting
33. Upgrades
34. Incident Response
35. Compliance and Vendor Assessments
36. Cost and Capacity Planning 
 Part VI — Beyond Vault (Ch. 37–40)
37. OpenBao and the Licensing Question
38. SOPS and GitOps Secrets
39. When Vault Is the Wrong Answer
40. Security Culture as a Solo Operator 
 Appendices 
 
 A. Complete policy reference for a five-service stack 
 B. Verification and testing scripts 
 C. Emergency runbooks 
 D. The Mistakes File — every error, cause, and fix 
 E. CLI quick reference 
 F. Glossary 
 
 10.3 Workflow 
 The discipline: write each chapter as you complete the lab step , while the confusion is fresh. The book's value is remembering what was hard — that memory fades within weeks of mastery. 
 Per-chapter template: 
 
 The problem — a concrete scenario from Zanzi Systems 
 The concept — theory with a diagram 
 The lab — commands the reader runs 
 The verification — how to know it worked, including negative tests 
 What goes wrong — real errors and causes 
 Production notes — how this differs at scale 
 Exercises — solutions in the appendix 
 
 Cadence: ~4,000 words/chapter × 40 chapters ≈ 160,000 words. Two chapters/week → first draft in ~5 months. 
 vault-book/
├── manuscript/
│ ├── part-1/ ... part-6/
│ └── appendices/
├── labs/
│ ├── step-01/ ... step-12/
│ └── scripts/
├── diagrams/
├── mistakes.md ← the goldmine
└── outline.md
 
 Keep mistakes.md open during every lab session. Every error message, every confusion, every "why doesn't this work." That file is the differentiator — it's what the official docs will never contain. 
 
 Progress 
 Lab 
 
 Step 1 — Vault running (Docker) 
 Step 2 — Tokens and first secret 
 Step 3 — Own engine, path hierarchy, engine zoo 
 Step 4 — Versioning, delete/undelete/destroy 
 Step 5 — First policy 
 Step 6 — Denial proven (4× 403) 
 Step 7 — AppRole enabled 
 Step 8 — Service login, zero root token 
 UI experiment — scope made visible 
 Step 9 — All services scripted 
 Step 10 — Isolation verification script 
 Step 11 — Audit logging 
 Step 12 — Dynamic database credentials 
 
 Production homework 
 
 Find the OpenAI leak vector (repos, images, docker inspect , Docker API on 2375) 
 New OpenAI key with a spending limit 
 Enable audit logging on vault.qbitspark.com — one command, non-disruptive, do it now 
 Locate and verify unseal key shares 
 Identify the production storage backend 
 Test a snapshot restore 
 
 Book 
 
 Start mistakes.md (Part 4 above is the seed) 
 Draft chapters 1–4 
 Define Zanzi Systems 
 
 
 QBIT SPARK internal engineering document — v2, 6 August 2026