NexGate Deployment

NexGate Deployment Guide

NexGate Deployment Guide

NexGate Deployment Guide (New)

The paved road for shipping every NexGate service — architecture, flow, secrets, setup, and operations.

Owner: Kibuti · Organization: NexGate / nexgate-hq · Status: living document · v2

How to read this

Single source of truth for how services are built, deployed, secured, and operated. Five things:

  1. Architecture — what the pieces are and how they fit
  2. Flow — what happens when you push code
  3. Secrets — how credentials reach a running container without ever touching git
  4. Setup — standing it up from zero
  5. Operations — running it day to day, adding services, rolling back, scaling

If you remember one sentence:

You write code, drop one folder, and register one secret role. An image is built once when you push, and pulled once per environment by the reconciler. No credential ever enters git, and no human ever SSHes to deploy.

v2 changes: the secrets model is rebuilt from §5 onward. v1 described AppRole correctly at a high level but left the mechanics undefined — where Role IDs and Secret IDs physically live, how Komodo authenticates to Vault, how the trust chain terminates, and how any of it is verified. It also shipped a policy that granted one service both staging and prod. All fixed below.

Table of contents

  1. Core principles
  2. Architecture
  3. Repository layout
  4. Deploy flow
  5. Secrets model
  6. The trust chain
  7. Setup from scratch
  8. Adding a new service
  9. Deploying a change
  10. Operations
  11. Scaling
  12. Conventions and invariants
  13. Glossary

1. Core principles

The design turns on one shift away from the old Jenkins pipeline:

Stop editing shared state imperatively. Declare desired state, and let a reconciler make reality match it.

Everything follows:

2. Architecture

2.1 The four moving parts

Part What it is Who owns it
Service repos App code + a thin CI workflow (nexgate-hq/<service>) One repo per service
nexgate-infra Declarative desired state — what runs where, at which version One shared repo
Reconciler (Komodo) Watches nexgate-infra, applies changes on the host(s) Installed once per host
Vault Issues every credential any container ever uses Central, vault.qbitspark.com

There is no Jenkins. The reconciler is off-the-shelf — not code you write. It does what Jenkins did on the deploy side (docker compose pull + up -d), driven by git instead of a webhook.

2.2 Runtime layers

2.3 Component roles

3. Repository layout

nexgate-infra is the heart. Everything a service needs is in its own folder — the information that used to be crammed into two giant shared compose files, cut apart so each service owns its slice.

nexgate-infra/
├── base/                          # always-on platform layer
│   ├── traefik.yml
│   ├── vault-agent.yml
│   ├── postgres.yml               # optional shared Postgres for light services
│   ├── redis.yml
│   ├── rabbitmq.yml
│   └── minio.yml                  # always-on (File Thunder needs it live)
│
├── services/
│   ├── nexgate-backend/
│   │   ├── service.yml            # compose block: image, labels, depends_on
│   │   ├── config.env             # non-secret config ONLY
│   │   ├── vault-policy.hcl       # which Vault paths this service may read
│   │   ├── vault-agent.hcl        # secret rendering template
│   │   └── tag.env                # IMAGE_TAG=...  ← the one line CI edits
│   ├── notification-server/
│   ├── file-thunder/
│   └── ai/
│
├── policies/                      # generated from services/*/vault-policy.hcl
│   └── .gitkeep
│
├── scripts/
│   ├── vault-bootstrap.sh         # create policies + approles from service folders
│   └── verify-isolation.sh        # prove cross-service access is denied
│
├── staging/
│   └── stack.yml                  # include: base + services (staging tags)
└── prod/
    └── stack.yml                  # include: base + services (prod tags)

Nothing in a service folder is a program. service.yml is a compose snippet. tag.env is one line. vault-policy.hcl is a few lines of permissions. The layout guarantees one service's deploy is isolated to its own files.

Example service folder

# services/ai/service.yml
services:
  ai:
    image: ghcr.io/nexgate-hq/ai-service:${AI_TAG:-latest}
    env_file: [config.env, tag.env]
    networks: [nexgate-prod, proxy]
    depends_on: [postgres, redis, rabbitmq]
    volumes:
      # Vault Agent renders secrets here. The app reads a file.
      # It never sees a role_id or secret_id.
      - vault-secrets-ai:/run/secrets:ro
    labels:
      - traefik.enable=true
      - traefik.http.routers.ai.rule=Host(`ai.nexgate.co`)
      - traefik.http.routers.ai.tls.certresolver=le
    restart: unless-stopped

# services/ai/config.env      ← non-secret ONLY. Committed to git.
SPRING_PROFILES_ACTIVE=prod
LOG_LEVEL=INFO
VAULT_ADDR=https://vault.qbitspark.com

# services/ai/tag.env         ← CI rewrites ONLY this file
AI_TAG=0.1.0

Because tag.env is per-service, deploying ai cannot reset the backend's tag. The shared-.env clobber problem is structurally impossible.

4. Deploy flow

4.1 The one rule: built once, pulled per environment

Staging pulls right after the change. Prod pulls the same image after approval. Prod never rebuilds.

4.2 End-to-end

git push (staging/master)
   │
   ▼
CI (GitHub Actions, reusable workflow)
   • build image via Buildx
   • push to GHCR (version tag + latest)
   • bump services/<svc>/tag.env in nexgate-infra
   │
   ▼
nexgate-infra  ← declarative desired state
   │
   ▼  (prod only: approval gate — PR review)
   │
   ▼
Komodo reconciler on the VPS
   • detects the tag change
   • mints a fresh Secret ID for <svc>-<env>      ← §6
   • writes it to /etc/vault/<svc>-<env>-secret-id (0400 root)
   • docker compose pull <svc> && up -d <svc>     (only that service)
   • on health-check pass: destroys prior Secret IDs
   │
   ▼
Vault Agent authenticates via AppRole → renders secrets to /run/secrets
   │
   ▼
Service container starts, reads secret files
   │
   ▼
Traefik routes it over HTTPS (*.nexgate.co)

No SSH. No Jenkins job. No credential in git.

4.3 Approvals

The gate lives in git — stronger than a button because it is reviewable and permanent.

Optional hard gates:

Recommended: PR promotion + Komodo manual-sync for prod.

4.4 Notifications

Post-deploy runtime health comes from the Grafana stack — §10.

5. Secrets model

This is the section that was thin in v1. It is now the longest, because it's where the real risk lives.

5.1 The rule

No credential is ever in git. Only pointers to credentials.

Artifact In git? Why
config.env (non-secret config) ✅ Yes Log levels, profiles, Vault address
vault-policy.hcl ✅ Yes Permission rules, not credentials
Role ID ✅ Yes Identifies a role. Useless alone.
Secret ID ❌ Never Delivered at deploy, file on host only
Actual secrets ❌ Never Live in Vault, fetched at runtime

5.2 Path hierarchy

v1 used a flat nexgate/data/ai_staging — one segment for service_env. That blocks globbing, so every new component needs a new policy line, and eventually someone "fixes" it with a wildcard that grants too much.

v2 uses a proper hierarchy:

nexgate/<service>/<environment>/<component>

nexgate/ai/staging/postgres
nexgate/ai/prod/postgres
nexgate/ai/prod/openai
nexgate/backend/prod/postgres
nexgate/backend/prod/jwt-signing
nexgate/file-thunder/prod/minio
nexgate/file-thunder/prod/postgres
nexgate/notification-server/prod/rabbitmq
nexgate/shared/prod/cloudflare

Why service/environment above component: that's the boundary policies cut along. One rule covers every component a service will ever have, including ones added next year.

Put the thing you cut permissions along highest in the path.

5.3 The KV v2 path gotcha

The single most common Vault mistake, and it will bite whoever writes the next policy.

What you type: vault kv get nexgate/ai/prod/postgres What goes over HTTP: GET /v1/nexgate/data/ai/prod/postgres

KV v2 splits operations into sub-paths — data/, metadata/, delete/, undelete/, destroy/. The kv CLI inserts the right one for you. Policies have no wrapper.

Write the obvious-looking thing and you get permission denied with no hint why:

path "nexgate/ai/prod/*" { capabilities = ["read"] }   # ← WRONG. Silently fails.

5.4 Policies — one per service and environment

The v1 bug:

# services/ai/vault-policy.hcl  ← BROKEN
path "nexgate/data/ai_staging" { capabilities = ["read"] }
path "nexgate/data/ai_prod"    { capabilities = ["read"] }

One policy granting both environments. If both ai_staging and ai_prod roles attach it, staging can read production secrets — defeating the isolation claimed two paragraphs above it in v1.

Correct — separate file per environment:

# services/ai/vault-policy-staging.hcl
path "nexgate/data/ai/staging/*" {
  capabilities = ["read"]
}
path "nexgate/metadata/ai/staging/*" {
  capabilities = ["list"]
}

# services/ai/vault-policy-prod.hcl
path "nexgate/data/ai/prod/*" {
  capabilities = ["read"]
}
path "nexgate/metadata/ai/prod/*" {
  capabilities = ["list"]
}

Read only. No create, no update, no delete. An app that reads its config has no business rewriting it. Deny-by-default means anything not listed is denied — no explicit deny rule needed.

Prefer generating these from a template so nobody hand-writes the data/ prefix wrong (§7 step 6).

5.5 AppRole per service per environment

# staging — looser, no CIDR binding if staging hosts vary
vault write auth/approle/role/ai-staging \
    token_policies="ai-staging" \
    token_ttl=4h \
    token_max_ttl=12h \
    secret_id_ttl=720h \
    secret_id_num_uses=0 \
    secret_id_bound_cidrs="<staging-host-ip>/32"

# prod — tightest
vault write auth/approle/role/ai-prod \
    token_policies="ai-prod" \
    token_ttl=1h \
    token_max_ttl=4h \
    secret_id_ttl=720h \
    secret_id_num_uses=0 \
    secret_id_bound_cidrs="<prod-host-ip>/32"

token_max_ttl is your actual worst-case exposure window. Renewal extends but never resets — it's measured from creation. A stolen prod token is dead in 4 hours regardless of how often the thief renews it.

secret_id_bound_cidrs is free hardening. A stolen Role ID and Secret ID are useless from anywhere but that host. The attacker would have to run the attack from inside your infrastructure.

5.6 Where the two halves physically live

The rule: Role ID and Secret ID never in the same place. If one docker inspect yields both, the split has bought nothing.

Role ID    → services/<svc>/config.env, committed to git
Secret ID  → /etc/vault/<svc>-<env>-secret-id on the host
             chmod 400, chown root:root
             minted by Komodo at deploy, never in git

Never do this:

environment:
  VAULT_ROLE_ID: "..."
  VAULT_SECRET_ID: "..."   # ← docker inspect prints this in plaintext

One command gives an attacker both halves. This is the single most common way AppRole deployments get compromised.

Do this — file mount:

services:
  vault-agent-ai:
    image: hashicorp/vault:1.20.3
    command: agent -config=/vault/agent.hcl
    environment:
      VAULT_ADDR: https://vault.qbitspark.com
    volumes:
      - ./services/ai/vault-agent.hcl:/vault/agent.hcl:ro
      - /etc/vault/ai-prod-role-id:/vault/role-id:ro
      - /etc/vault/ai-prod-secret-id:/vault/secret-id:ro
      - vault-secrets-ai:/run/secrets

docker inspect now shows only mount paths. docker compose config shows nothing sensitive. Git holds only the Role ID.

5.7 Vault Agent — the app never touches Vault

This is why Agent is worth the extra container. The application never handles a Role ID, a Secret ID, or a Vault token. It reads a file.

# services/ai/vault-agent.hcl
vault {
  address = "https://vault.qbitspark.com"
}

auto_auth {
  method "approle" {
    config = {
      role_id_file_path                   = "/vault/role-id"
      secret_id_file_path                 = "/vault/secret-id"
      remove_secret_id_file_after_reading = false
    }
  }

  sink "file" {
    config = { path = "/run/secrets/token" }
  }
}

template {
  destination = "/run/secrets/application-secrets.properties"
  contents = <<EOT
{{- with secret "nexgate/data/ai/prod/postgres" }}
spring.datasource.username={{ .Data.data.username }}
spring.datasource.password={{ .Data.data.password }}
{{- end }}
{{- with secret "nexgate/data/ai/prod/openai" }}
openai.api-key={{ .Data.data.api_key }}
{{- end }}
EOT
}

Agent logs in, renders the file, renews the token, and re-authenticates when token_max_ttl is hit. The app just reads /run/secrets/application-secrets.properties.

Note {{ .Data.data.username }} — the double .data is KV v2's envelope. Another place the data/ structure surfaces.

Alternative — Spring Cloud Vault, if you'd rather the app authenticate directly:

spring:
  cloud:
    vault:
      uri: https://vault.qbitspark.com
      authentication: APPROLE
      app-role:
        role-id: ${VAULT_ROLE_ID}
        secret-id: ${VAULT_SECRET_ID}
      kv:
        enabled: true
        backend: nexgate
        default-context: ai/prod

Fewer containers, but the app handles the Secret ID. Prefer Agent — it keeps credentials out of application process memory and gives you one renewal implementation instead of one per language.

5.8 Two tiers, unchanged from v1

5.9 Static vs dynamic — the migration target

Everything above stores static secrets in KV. That's the starting point, not the destination.

The limitation, concretely: if a static credential leaks, Vault cannot revoke it. Vault stores and guards it, but it can't kill something a third party issued. A leaked OpenAI key stays valid until someone revokes it at OpenAI.

Dynamic secrets invert this. Vault generates a real PostgreSQL user on demand with a TTL, then drops it:

vault read database/creds/ai-prod
# → v-approle-ai-prod-x7Kd92 / A1a-8sKdm3nQp   (lease 1h)
vault read database/creds/ai-prod
# → v-approle-ai-prod-mN4vB1 / Z9z-2pLwq7Rt    (lease 1h)

Different every time. vault lease revoke runs DROP ROLE against Postgres directly. A stolen credential dies on its own.

Migration order:

  1. PostgreSQL — all services, highest value
  2. RabbitMQ — notification-server, backend
  3. MinIO — via the AWS engine (S3-compatible STS)
  4. PKI — internal CA for mTLS between services
  5. Transit — encryption-as-a-service for sensitive payloads

What must stay static: third-party API keys (OpenAI, Cloudflare, payment providers). Vault didn't issue them, so it can't rotate them. These get quarterly manual rotation and versioned storage in KV.

Success measure: a leaked database credential is a non-event because it expired an hour ago.

6. The trust chain

v1 said "services authenticate via AppRole" and stopped there. That leaves the important question unanswered: who gives Komodo permission to mint Secret IDs, and who gives that to Komodo?

6.1 Komodo's Vault credentials

Komodo needs a Vault credential to mint per-service Secret IDs. That credential becomes the new secret zero, so it must be narrow.

# policies/komodo-deployer.hcl
path "auth/approle/role/+/secret-id" {
  capabilities = ["update"]
}
path "auth/approle/role/+/secret-id-accessor/destroy" {
  capabilities = ["update"]
}

Can mint and destroy Secret IDs. Cannot read a single secret. Cannot change a policy. Cannot create a role. If it leaks, an attacker can generate Secret IDs — but must guess role names, and every generation appears in the audit log.

vault write auth/approle/role/komodo-deployer \
    token_policies="komodo-deployer" \
    token_ttl=15m \
    token_max_ttl=30m \
    secret_id_ttl=0 \
    secret_id_bound_cidrs="<host-ip>/32"

15-minute tokens. Komodo logs in per deploy, uses the token, it dies.

6.2 Placing Komodo's credentials — the one manual step

ssh <host>
sudo mkdir -p /etc/vault

vault read -field=role_id auth/approle/role/komodo-deployer/role-id \
  | sudo tee /etc/vault/komodo-role-id > /dev/null
vault write -f -field=secret_id auth/approle/role/komodo-deployer/secret-id \
  | sudo tee /etc/vault/komodo-secret-id > /dev/null

sudo chmod 400 /etc/vault/komodo-*
sudo chown root:root /etc/vault/komodo-*

This is the only credential a human ever places by hand, and only once per host.

6.3 Komodo's deploy hook

#!/usr/bin/env bash
# scripts/deploy-service.sh — run by Komodo on tag change
set -euo pipefail

SERVICE="$1"        # e.g. ai
ENV="$2"            # staging | prod
ROLE="${SERVICE}-${ENV}"

# 1. Komodo authenticates (15-minute token)
TOKEN=$(vault write -field=token auth/approle/login \
  role_id="$(cat /etc/vault/komodo-role-id)" \
  secret_id="$(cat /etc/vault/komodo-secret-id)")

# 2. Record existing Secret ID accessors so we can revoke them after
OLD_ACCESSORS=$(VAULT_TOKEN=$TOKEN vault list -format=json \
  "auth/approle/role/${ROLE}/secret-id" 2>/dev/null || echo '[]')

# 3. Mint a fresh Secret ID for this deploy
VAULT_TOKEN=$TOKEN vault write -f -field=secret_id \
  "auth/approle/role/${ROLE}/secret-id" \
  > "/etc/vault/${ROLE}-secret-id"
chmod 400 "/etc/vault/${ROLE}-secret-id"
chown root:root "/etc/vault/${ROLE}-secret-id"

# 4. Deploy
docker compose pull "$SERVICE"
docker compose up -d "$SERVICE"

# 5. Wait for health, THEN revoke the old Secret IDs
if ./scripts/wait-for-health.sh "$SERVICE" 120; then
  echo "$OLD_ACCESSORS" | jq -r '.[]' | while read -r acc; do
    VAULT_TOKEN=$TOKEN vault write \
      "auth/approle/role/${ROLE}/secret-id-accessor/destroy" \
      secret_id_accessor="$acc"
  done
  echo "✓ ${ROLE} deployed, old Secret IDs revoked"
else
  echo "✗ ${ROLE} health check failed — old Secret IDs left intact for rollback"
  exit 1
fi

# Komodo's own token expires in 15 minutes regardless

Order matters. Revoke old Secret IDs after the new container is healthy, never before — otherwise a failed deploy leaves you unable to roll back.

6.4 The chain, complete

YOU                          ← SSH key. The actual root of trust.
 │
 └─ place Komodo's role-id + secret-id     (once per host, by hand)
     │
     └─ KOMODO mints service Secret IDs    (every deploy, 15-min tokens)
         │
         └─ VAULT AGENT logs in per service (1–4h tokens, auto-renewed)
             │
             └─ APP reads rendered secret files (never touches Vault)

Four levels, each narrower than the one above. Only the top requires a human, and only once per host.

Your SSH key is the bottom of the chain — the real secret zero for the entire system. It deserves a passphrase, a hardware key if possible, and an entry in the break-glass documentation.

6.5 Rotating Komodo's own Secret ID

Manual, deliberately.

ssh <host>
vault write -f -field=secret_id auth/approle/role/komodo-deployer/secret-id \
  | sudo tee /etc/vault/komodo-secret-id > /dev/null
docker compose restart komodo

It cannot be automated from Komodo — that would let a compromised Komodo renew its own access indefinitely. Keeping it manual bounds the chain. Quarterly, or on any suspicion.

6.6 Secret zero — the honest accounting

You cannot bootstrap trust from nothing. Every auth system has a first credential. The question is only what it is and how bad a leak would be.

Before (shared .env):

DB_PASSWORD=real
JWT_PRIVATE_KEY=real
OPENAI_KEY=real
MINIO_SECRET=real

One file. Everything. Never expires. No log of who read it.

After:

/etc/vault/komodo-secret-id:  d4e5f6a7-b8c9-...

One file. One UUID.

Old .env Stolen Komodo Secret ID
What they get Every credential immediately Nothing — needs the Role ID too
With both halves — Can mint Secret IDs, cannot read secrets
Duration Forever Until rotated
IP-bindable No Yes
Do you find out No Yes — audit log
To revoke Change everything everywhere One command

Vault doesn't protect secrets by hiding them. It protects them by encrypting at rest, authenticating every request, authorising per path, expiring everything, logging every access, and revoking centrally. A .env file does none of these six.

7. Setup from scratch

Bootstrap order matters.

Step 1 — Provision hosts

Two Ubuntu VPS: staging and prod (separate boxes — §11). Docker Engine + Compose plugin. UFW allowing 22, 80, 443 only.

Also: confirm the Docker daemon API is not exposed. curl http://<host>:2375/version from outside must fail. An exposed Docker socket makes every other control here irrelevant.

Step 2 — Private networking (do it early)

WireGuard mesh (or provider private network) between hosts, even with one host each. Makes future splitting a one-line change instead of a migration. Services address each other by name / internal DNS, never hardcoded localhost.

Step 3 — Docker networks

docker network create proxy
docker network create nexgate-staging     # on staging host
docker network create nexgate-prod        # on prod host

Step 4 — Deploy the base layer

Traefik (Let's Encrypt resolver le), PostgreSQL, Redis, RabbitMQ, MinIO. MinIO always-on from day one.

Vault Agent comes after Vault is configured — step 6.

Step 5 — Vault: audit logging first

Before anything else. Non-disruptive, one command, and it means everything from here forward is attributable.

vault audit enable file file_path=/vault/logs/audit.log
vault audit list -detailed

Vault refuses to serve requests if all audit devices fail to write — deliberate fail-closed design. Ship the log to Loki; sensitive values are HMAC'd while paths and field names stay plaintext, so it's safe to forward.

Also confirm, and write down offline:

Solo operator: Shamir's 3-of-5 assumes multiple people. Split across locations instead — bank box, trusted party, encrypted offsite, hidden at home. An untested recovery scheme is a story you tell yourself.

Step 6 — Vault: mount, auth, policies, roles

vault secrets enable -path=nexgate kv-v2
vault secrets tune -description="NexGate application secrets" nexgate/
vault auth enable approle

Then run the bootstrap script (§7 step 8) which generates policies and roles from the service folders.

Retention worth setting per path:

vault kv metadata put -max-versions=10 nexgate/ai/prod/postgres

Default is unlimited history — every rotation accumulates forever and old passwords stay readable.

Step 7 — Komodo

Install per host. Point at nexgate-infra. Staging = auto-sync, prod = manual-sync.

Place Komodo's Vault credentials by hand (§6.2). This is the one manual credential step.

Step 8 — Bootstrap script

#!/usr/bin/env bash
# scripts/vault-bootstrap.sh
# Generates policies and AppRoles from services/*/ folders.
# Idempotent — safe to re-run.
set -euo pipefail

STAGING_CIDR="${STAGING_CIDR:?set STAGING_CIDR}"
PROD_CIDR="${PROD_CIDR:?set PROD_CIDR}"

for dir in services/*/; do
  svc=$(basename "$dir")

  for env in staging prod; do
    name="${svc}-${env}"

    # policy — generated, so the data/ prefix is never hand-typed wrong
    cat > "policies/${name}.hcl" <<EOF
path "nexgate/data/${svc}/${env}/*" {
  capabilities = ["read"]
}
path "nexgate/metadata/${svc}/${env}/*" {
  capabilities = ["list"]
}
EOF
    vault policy write "$name" "policies/${name}.hcl"

    if [ "$env" = "prod" ]; then
      TTL=1h; MAXTTL=4h; CIDR="$PROD_CIDR"
    else
      TTL=4h; MAXTTL=12h; CIDR="$STAGING_CIDR"
    fi

    vault write "auth/approle/role/${name}" \
      token_policies="$name" \
      token_ttl="$TTL" \
      token_max_ttl="$MAXTTL" \
      secret_id_ttl=720h \
      secret_id_num_uses=0 \
      secret_id_bound_cidrs="$CIDR"

    # Role ID is not secret — emit for config.env
    echo "${name} role_id: $(vault read -field=role_id auth/approle/role/${name}/role-id)"
  done
done

Step 9 — Verify isolation

Do not skip this. A policy tested only for success is untested.

#!/usr/bin/env bash
# scripts/verify-isolation.sh
set -uo pipefail
PASS=0; FAIL=0

check() {
  local role="$1" path="$2" expect="$3" rid sid tok result
  rid=$(vault read -field=role_id "auth/approle/role/${role}/role-id")
  sid=$(vault write -f -field=secret_id "auth/approle/role/${role}/secret-id")
  tok=$(vault write -field=token auth/approle/login role_id="$rid" secret_id="$sid")

  if VAULT_TOKEN="$tok" vault kv get "$path" >/dev/null 2>&1
  then result="ALLOW"; else result="DENY"; fi

  if [ "$result" = "$expect" ]; then
    echo "  ✓ ${role} → ${path} = ${result}"; PASS=$((PASS+1))
  else
    echo "  ✗ ${role} → ${path} = ${result} (expected ${expect})"; FAIL=$((FAIL+1))
  fi
}

echo "Positive (must ALLOW):"
check ai-prod          nexgate/ai/prod/postgres            ALLOW
check backend-staging  nexgate/backend/staging/postgres    ALLOW

echo "Negative (must DENY):"
check ai-prod          nexgate/backend/prod/jwt-signing    DENY
check ai-prod          nexgate/ai/staging/postgres         DENY   # ← the v1 bug
check ai-staging       nexgate/ai/prod/postgres            DENY   # ← the v1 bug
check backend-staging  nexgate/backend/prod/postgres       DENY
check file-thunder-prod nexgate/ai/prod/openai             DENY

echo; echo "Passed: $PASS  Failed: $FAIL"
[ "$FAIL" -eq 0 ] || exit 1

The two marked lines are exactly what v1's policy would have failed. Run this after every policy change, forever. Wire it into CI on nexgate-infra.

Step 10 — service-template + reusable CI

Template repo with Dockerfile, workflow_call caller, /health + /metrics wired. One reusable GitHub Actions workflow in the org that all services call.

Step 11 — Onboard the first service

Run the §8 checklist end to end to validate the chain.

Step 12 — Notifications and observability

Telegram webhook in the reusable workflow. Komodo deploy alerts. Grafana stack on a dedicated ops VPS — §10.

8. Adding a new service

Assuming base, Komodo, template, and reusable workflow exist — onboarding ai:

  1. Generate the repo — nexgate-hq/ai-service from service-template
  2. Write the code — the only real work; everything below is config
  3. First push to staging → CI builds ai:0.1.0 → GHCR (created)
  4. Add one folder — services/ai/ with service.yml, config.env, vault-agent.hcl, tag.env
  5. Write the secrets to nexgate/ai/staging/* and nexgate/ai/prod/*
  6. Run vault-bootstrap.sh — generates both policies and both AppRoles
  7. Run verify-isolation.sh — add positive and negative cases for the new service
  8. Put the Role IDs in config.env (safe to commit)
  9. Commit → Komodo syncs staging → mints a Secret ID → pulls ai:0.1.0 → live at ai.staging.nexgate.co (pulled #1) — validate
  10. Promote to prod — PR bumping the prod tag. Merge → Komodo syncs prod → pulls the same image (pulled #2, identical)

The wiring (steps 4–8) is the entire cost — the old four-place scavenger hunt collapsed to one folder + one script run.

File Thunder note: additionally uncomment MinIO in base/ and declare its own Postgres (5433) and a ClamAV container in its service.yml. One-time, and entirely within File Thunder's folder + the base layer.

9. Deploying a change

  1. Edit code, push to staging
  2. CI builds <svc>:<version>, pushes to GHCR, bumps the staging tag (created)
  3. Komodo mints a fresh Secret ID, pulls, restarts only that service in staging (pulled #1). Automatic, no gate.
  4. On health-check pass, Komodo revokes the previous Secret IDs
  5. Validate in staging
  6. Open a PR bumping the prod tag. The PR is the gate.
  7. Merge → Komodo repeats for prod with the same image (pulled #2, identical)

Prod runs the byte-for-byte artifact you tested.

10. Operations

Rollback

Revert the tag-bump commit in nexgate-infra, or use Komodo's "redeploy previous." Images are immutable and promoted by reference, so rollback is re-pointing a tag at the last-good image.

Secret ID note: because old Secret IDs are only revoked after a health check passes, a failed deploy leaves the previous credential valid. Rollback works without re-minting.

Rough edges fixed by design

v1 problem v2 fix
Shared .env clobber Per-service tag.env
One VAULT_TOKEN for both envs AppRole per service and environment
Policy granting staging + prod Separate policy file per environment
Flat service_env path Hierarchical service/env/component
Secret ID placement undefined File on host, 0400, minted by Komodo
Komodo's Vault auth undefined Narrow AppRole, §6
No isolation testing verify-isolation.sh in CI
MinIO commented out Always-on in base/

Secret rotation

Credential Frequency Method
Service Secret IDs Every deploy Komodo, automatic
Service tokens 1–4h Vault Agent, automatic
Komodo's Secret ID Quarterly Manual, by SSH (§6.5)
Third-party API keys Quarterly + on incident Manual, versioned in KV
DB credentials 1h Dynamic engine (target state)
Vault encryption key Quarterly vault operator rotate
Unseal keys Annually or on suspicion vault operator rekey
Root token Per use Generate → use → revoke

Static secret rotation — order matters:

1. Create the new credential at the provider
2. vault kv put ...              ← new version; old still readable
3. Redeploy / let Agent re-render
4. Verify healthy
5. ONLY THEN revoke the old one at the provider
6. Optionally vault kv destroy -versions=<n>

Revoking before services pick up the new value causes an outage. KV v2 versioning is what makes step 5 safe — if step 4 fails, the previous version is one command away.

Backups

Monitoring — the Grafana stack

Runs on a dedicated ops VPS outside dev/staging/prod — never on a box it watches. One ops host watches every environment; targets labelled by env (env=dev|staging|prod), dashboards filter on that label. Joins the WireGuard mesh for internal IPs; hits public endpoints through Traefik.

Component Question it answers
Prometheus Metrics over time + alerting (scrapes /metrics)
Loki Log search (services log JSON to stdout)
Tempo Distributed traces
Grafana Dashboards, single view
Alertmanager Routes alerts → Telegram / email

Collectors: node_exporter (host CPU/RAM/disk), cAdvisor (per-container resources), blackbox_exporter (uptime probes against /health — no separate Uptime Kuma needed), Grafana Alloy or Promtail (stdout JSON → Loki).

Vault-specific alerts — Vault exposes Prometheus metrics at /v1/sys/metrics?format=prometheus:

Condition Severity
Vault sealed unexpectedly Critical
Root token used Critical
Audit device failure Critical
Permission-denial spike High — attack or misconfiguration
Token creation anomaly High
Lease count growth Medium — leaked unreleased leases
Secret/cert nearing expiry Medium

Root token use should page you. In steady state it should never happen.

Because every service exposes /metrics and logs JSON to stdout (§12), adding a service to monitoring is a scrape target + a health probe, both env-labelled.

Komodo vs Grafana — you need both

Komodo Grafana stack
Watches Deployments (control plane) Runtime health (observability)
Tells you Did it deploy, what tag runs where Is it up, slow, erroring; resource use; history
Lets you Redeploy, roll back Alert, investigate — not deploy

Komodo reports "deploy succeeded" the moment a container starts, even if it then crash-loops or serves 500s under load — Grafana catches that. Grafana can tell you a service is unhealthy but can't redeploy — that's Komodo. One tells you something's wrong; the other lets you do something about it.

Komodo shows basic per-host CPU/RAM/disk, so it doubles as light resource monitoring — but no metrics history, no log search, no tracing. Run both on the ops VPS; both alert to Telegram.

Incident response

Suspected Vault compromise:

vault operator seal      # everything slams shut, instantly

Costs an outage; the attacker gets ciphertext. Recovery needs 3 of 5 unseal shares — which is why you must know where they are before the emergency.

Suspected service credential compromise:

vault list auth/approle/role/<svc>-<env>/secret-id
vault write auth/approle/role/<svc>-<env>/secret-id-accessor/destroy \
    secret_id_accessor="<accessor>"
vault lease revoke -prefix auth/approle/

Redeploy to mint a fresh one.

Note the limit: revoking in Vault kills Vault access. It does not invalidate a third-party API key an attacker already read — that must be revoked at the provider. This is precisely why dynamic secrets matter (§5.9).

Upgrading base images

Postgres/Redis/etc. are pinned per service in service.yml (§11.4). Bump the pin, test in staging, promote.

Vault upgrades: read the changelog for every version between current and target. Snapshot first, always. Test on a scratch instance. Never skip major versions.

11. Scaling

11.1 Two kinds of scale

11.2 The ladder

Rung Setup When
1 Single VPS, Compose + Komodo Now
2 Bigger VPS Vertical scale buys runway
3 Services split across hosts (Komodo multi-host) When a service gets heavy
4 Swarm or K3s Replicas + failover, much later

The repo and onboarding journey are identical at every rung. Moving up is a config change (which host a folder runs on), plus — at rung 3 — the private network from §7 step 2.

Secrets note: at rung 3, each host needs its own Komodo credentials placed by hand, and secret_id_bound_cidrs must list every host that runs a given service. At rung 4, switch to the Kubernetes auth method — pods authenticate via their service account and the Secret ID disappears entirely.

11.3 VPS topology

Sequence: env-split today → service-split as load appears → all-have-all as a distant HA project.

11.4 Different versions per service

Postgres/Redis/MinIO are upstream public images pinned by tag — unrelated to "build once," which is only about your app image. Each service's datastore is a separate container in its own folder:

# services/file-thunder/service.yml
  ft-postgres:
    image: postgres:17
    volumes: [ft-pgdata:/var/lib/postgresql/data]

# services/backend/service.yml
  backend-postgres:
    image: postgres:15
    volumes: [backend-pgdata:/var/lib/postgresql/data]

Different names, volumes, versions — no collision. Shared vs dedicated is a per-service call: shared base Postgres saves RAM for light services; dedicated gives version/extension freedom and isolation (why File Thunder has its own). Never a base-layer constraint.

12. Conventions and invariants

Don't break these.

Deployment

Secrets

Runtime

13. Glossary

Migration checklist from v1

Versioned in nexgate-infra. Update whenever the architecture changes — it is the guide the whole organization relies on.

NexGate Deployment Guide

Vault Mastery Guide — v2 BOOK

From Root-Token Chaos to Production-Grade Secrets Management

Author: Kibuti — QBIT SPARK Environment: vault.qbitspark.com (HashiCorp Vault v1.20.3) Method: Local simulation first → master concepts → migrate production last Status: Lab Steps 1–8 complete. Steps 9–12 pending.

v2 changes: rewritten after completing the hands-on lab. Every analogy, error message, and "why doesn't this work" moment from the actual session is now captured. The mistakes sections are real, not hypothetical.

Table of Contents

PART 0 — Why This Exists PART 1 — Foundations PART 2 — Core Concepts (with the analogies that worked) PART 3 — The Lab, Step by Step PART 4 — Mistakes File PART 5 — Humans, Machines & MFA PART 6 — Rotation PART 7 — Deployment Architecture PART 8 — Production Migration PART 9 — Operations PART 10 — The Book

PART 0 — Why This Exists

0.1 The Incident

On 4–5 August 2026, an OpenAI API key (sk-pro...CrQA) was used by an unauthorized party to run ~129 automated LLM requests — a physics-grading workload (Feynman amplitude checkpoint scoring, some Chinese-language content) with no relation to any QBIT SPARK product. Roughly $7.95 consumed before detection.

Detection: OpenAI's abuse-detection email → manual log review → confirmed by reading request content.

Response: key revoked immediately.

Leak vector: still undetermined. Candidates under investigation:

0.2 What the Incident Actually Exposed

The $7.95 was trivial. The structural problems it revealed were not:

Weakness Risk
Root token used for routine Vault UI access Full-scope master credential in daily circulation
Same root token bootstrapping local dev and production Dev compromise = production compromise
No per-service scoped policies Any leak potentially exposes all secrets
No audit logging No forensic trail — couldn't answer "was this ever read from Vault?"
No MFA on human access Single-factor compromise = total access
No rotation schedule Leaked credentials valid indefinitely
No spending limits on downstream API keys Financial blast radius unbounded
Unknown location of unseal key shares Possible single point of catastrophic failure

The uncomfortable comparison: the same class of mistake applied to Vault's root token would expose every credential for JikoXpress, NexGate, Mauzodukani, GlueEmail and Textfy at once — including Snippe PSP payment integration, NexGate JWT signing keys, database passwords, and Vodacom SMPP credentials.

0.3 What "Mastery" Means Here

Not memorising commands. Being able to:

  1. Explain every Vault component in your own words, without notes
  2. Design a policy structure for a new service from scratch
  3. Predict what will break before changing something
  4. Recover from sealed / lost-key / corrupted-storage without panic
  5. Teach it — which is why this ends in a book

Rule: do not proceed past a step until you can explain it aloud with the notes closed.

PART 1 — Foundations

1.1 What Is a Secret?

Any data that grants access or proves identity, and whose disclosure causes harm.

Type Examples from this stack
Static credentials PostgreSQL, RabbitMQ, MinIO passwords
API keys OpenAI, Snippe PSP, Meta WhatsApp Business API, Vodacom SMPP
Cryptographic material NexGate JWT RS256 signing keys, TLS private keys, OMEMO keys
Connection strings DSNs embedding host + credentials
Certificates Internal CA, client certs for mTLS
SSH keys qbit-spark access

The problem: secrets must be available to apps at runtime, unreadable by anyone else, rotatable without downtime, and every access attributable.

.env files solve availability. Nothing else.

1.2 The Precision That Matters

Three words get conflated constantly. Get them separate:

vault kv put qbit/jikoxpress/prod/db username="jx_prod" password="s3cr3t"
Part Name Sensitive?
qbit/jikoxpress/prod/db path — the address No
password field name / key — the label No
s3cr3t field value ← this is the secret

The house analogy: path is the address. Secret is the money inside. Token is the key in your pocket.

Why it matters practically: Vault's audit log records paths and field names in plaintext but HMACs the values. You can see that someone read qbit/jikoxpress/prod/db without the log itself leaking the password.

The secret/ confusion: in vault kv put secret/hello, the word "secret" is a mount path, not the data. Dev mode happens to mount a KV engine there. It could be called anything. This is why the lab moves to qbit/ — the word disappears and the ambiguity goes with it.

1.3 What Vault Is

An encrypted key-value store that requires authentication to read from, enforces fine-grained authorisation per request, logs every access, and can generate short-lived credentials on demand instead of merely storing long-lived ones.

Four jobs:

  1. Secure storage — encrypted at rest; the storage backend never sees plaintext
  2. Access control — every request authenticated and authorised per path
  3. Audit — every request and response logged
  4. Dynamic secrets — credentials with automatic expiry, so a leak dies on its own

What it is not: a human password manager, a config management system, or magic. A badly configured Vault is worse than none, because it creates false confidence.

1.4 The Mental Model

                    ┌─────────────────────────────────┐
                    │         VAULT SERVER            │
   You / Service    │  ┌──────────┐                   │
        │           │  │   AUTH   │ ← "Prove who      │
        ├──────────►│  │  METHOD  │    you are"       │
        │           │  └────┬─────┘                   │
        │           │       │ issues                  │
        │           │       ▼                         │
        │           │  ┌──────────┐                   │
        │           │  │  TOKEN   │ ← temporary badge │
        │           │  └────┬─────┘                   │
        │           │       │ carries                 │
        │           │       ▼                         │
        │           │  ┌──────────┐                   │
        │           │  │ POLICIES │ ← "what doors     │
        │           │  └────┬─────┘    does it open?" │
        │           │       ▼                         │
        │           │  ┌──────────────────┐           │
        │           │  │ SECRETS ENGINES  │ ← rooms   │
        │           │  └────┬─────────────┘           │
        │           │       ▼                         │
        │           │  ┌──────────────────┐           │
        │           │  │ STORAGE BACKEND  │ ← encrypted
        │           │  └──────────────────┘           │
        │           │  ┌──────────────────┐           │
        │           │  │  AUDIT DEVICES   │ ← the log │
        │           │  └──────────────────┘           │
        └───────────┴─────────────────────────────────┘

Every Vault problem you will ever debug is a failure in one of those boxes.

1.5 The Alternatives Landscape

Vault OpenBao Infisical SOPS AWS SM
Licence BUSL 1.1 MPL 2.0 MIT / paid MPL 2.0 Proprietary
Self-host Yes Yes Yes N/A No
Dynamic secrets ✅ Extensive ✅ (fork) ⚠️ Limited ❌ ⚠️ Some
Auth methods ✅ 20+ ✅ 20+ ⚠️ Fewer ❌ IAM only
Learning curve Steep Steep Gentle Gentle Moderate
PKI / CA ✅ ✅ ❌ ❌ ⚠️ Separate
Encryption-as-a-service ✅ Transit ✅ ❌ ❌ ⚠️ KMS
Audit logging ✅ Detailed ✅ ✅ Basic ❌ ✅ CloudTrail
Ops burden High High Medium Very low None
Cost at this scale Free Free Free tier Free ~$0.40/secret/mo

Categories, for the book:

1.6 Why Vault — Decision Record

  1. Already deployed at vault.qbitspark.com; migration cost zero
  2. Multi-product, multi-provider VPS topology — cloud-native can't span it
  3. Dynamic secrets are the long-term win
  4. PKI engine — internal CA for NexGate mTLS, and relevant to TCRA registrar work
  5. Transit engine — encrypt JikoXpress financial data without app-held keys
  6. Commercial credibility — "scoped AppRole auth with audit logging" answers a vendor questionnaire far better than ".env files." Already been through Vodacom BSR/OneTrust.
  7. Transferable skill — appears in enterprise and East African fintech requirements

Hedge: track OpenBao. Migration is currently cheap. Revisit annually.

PART 2 — Core Concepts

2.1 Seal / Unseal — The Bank

Imagine a bank in Mbeya.

When sealed: the building exists, the door is locked, the money is scrambled. Steal the entire building and you get a locked box full of ciphertext.

Vault starts sealed every time it boots. Deliberately.

The unseal keys = the combination, cut into 5 pieces

The combination isn't written anywhere. It was split into 5 shares given to 5 managers. Any 3 can open the door. One alone can't. Two can't.

Why 3-of-5 and not 5-of-5? Because if one manager dies, loses their piece, or travels, you'd never open the bank again. 3-of-5 survives losing 2.

The math (Shamir's Secret Sharing, 1979): two points define a line, three define a parabola. Hide the secret as a point on a curve; hand out other points. With enough points you reconstruct the curve. With fewer, infinitely many curves fit — you learn nothing, not "most of it."

Auto-unseal delegates to an external KMS so restarts don't need manual intervention. Convenient; shifts trust to that system.

If you lose enough shares, the data is permanently unrecoverable. There is no backdoor.

2.2 The Precise Chain

Common misunderstanding: unseal shares do not generate the master key. They reconstruct it — it already exists, split at initialization.

And the master key doesn't decrypt secrets directly:

3 unseal shares
      ↓ reconstruct
  Master Key
      ↓ decrypts
 Encryption Key   ← this is what encrypts your secrets
      ↓
   Vault UNSEALED

Why the extra layer? So vault operator rotate can change the encryption key without redistributing 5 new shares to 5 people.

2.3 Root Token vs Unseal Keys — The Rooms

Door open ≠ free access. Inside are many rooms — one per service. The root token is a master keycard opening every room. Scoped tokens open one.

Unseal keys Root token
Opens The vault door The rooms inside
Count 5 shares, need 3 One
Used when Vault restarts Every request
Frequency Rare Should be near-never

You need both. Shares alone = open door, no keycard. Root alone = keycard, locked door.

Confirmed in the lab: the root token cannot unseal. That separation means two different attacks are required.

The reverse — the emergency button: root can seal. Sealing is easy, unsealing is hard, deliberately.

vault operator seal    # everything slams shut, instantly

If vault.qbitspark.com is ever suspected compromised, that's the fire alarm. Costs an outage; the attacker gets ciphertext. Then you need 3 of 5 shares to come back — which is why you must know where they are before the emergency.

2.4 Why Root Exists At All

Chicken-and-egg. A fresh Vault has no policies, no auth methods, nothing. Someone must create the first policy — but that needs permission.

Root is the bootstrap credential. It exists to create the system that replaces it.

Legitimate uses — the complete list:

  1. Initial setup — enable auth methods, mount engines, write first policies
  2. Emergency recovery — locked out, broken policy
  3. A few root-protected operations — operator rekey, some seal management

Not on the list: daily work, application requests, UI login.

Where you store it: nowhere. You destroy it.

generate → use → vault token revoke → gone

Need it next month? vault operator generate-root with 3 shares.

What you protect long-term is the shares, because those regenerate root. Root itself is disposable.

Where the shares go:

Solo operator → can't split across people → split across places. If the house floods you still need 3.

Correction worth recording: root is not used to create app credentials day-to-day. Root builds the machinery once — enables AppRole, writes policies, creates roles — then is revoked. AppRole issues credentials forever afterward with no root involved.

Root builds the key-cutting machine. Then root leaves. The machine keeps cutting keys.

2.5 Storage Backends

Vault encrypts before writing, so a stolen backend yields ciphertext.

Backend Use
Integrated (Raft) Recommended default; HA capable
Consul Legacy; more operational complexity
File Single node; dev/test
In-memory Dev only; vanishes on restart
PostgreSQL/MySQL Supported, not recommended for new deployments

Action item: determine which backend vault.qbitspark.com uses. It determines your backup strategy.

Lab observation: Storage Type inmem means every restart is a brand-new Vault — new cluster ID, new unseal key, no data. On real Vault with persistent storage, the unseal key is generated once at initialization and never again. Lose it, lose everything.

2.6 Secrets Engines

~20+ types, not four. Grouped by philosophy:

Static — you store, Vault guards

Engine Purpose
kv-v2 Versioned key-value. The workhorse.
kv-v1 Unversioned, legacy
cubbyhole Per-token private storage; dies with the token

Dynamic — Vault generates and expires

Engine Generates
database PostgreSQL, MySQL, MongoDB, Redis users
pki X.509 certificates
ssh Signed SSH certificates
aws/gcp/azure Temporary cloud IAM credentials
rabbitmq/consul/nomad Service-specific credentials
kubernetes Service account tokens

Cryptographic services

Engine Purpose
transit Encrypt/decrypt without ever holding the key
transform Format-preserving encryption, masking, tokenization
totp Generate and validate 2FA codes

Infrastructure — identity/ and sys/, always mounted, cannot be disabled.

Adoption order for QBIT SPARK: kv-v2 → database → pki → transit → ssh.

The insight: every engine answers "how do we handle this class of secret?" with a different strategy. KV says "store it carefully." Database says "don't store it, make a new one each time." Transit says "don't give it to the app at all — send the data here."

Rule of thumb: if Vault can generate it, don't store it. Store only what you're forced to — third-party API keys, mostly.

2.7 Mounts vs Paths vs Tenants

Two ways to split, easy to confuse:

Axis 1 — by engine type. Different engines are different machines. A KV engine can't issue certificates.

Axis 2 — by service. Possible, usually wrong:

✅ One mount, path hierarchy:      ❌ Mount per service:
qbit/jikoxpress/prod/db            jikoxpress/prod/db
qbit/nexgate/prod/minio            nexgate/prod/minio

Both isolate fine via policy. Separate mounts add config, versioning settings, tuning, backup and audit surface with no extra security.

When separate mounts genuinely help — multi-tenancy. Relevant here: you hold a retainer with Wiki, deployed FinSpark for a Mbeya microfinance client, built PMK's site. If you host secrets on behalf of clients, mount per tenant:

qbit/     ← your products
wiki/     ← retainer client
kfz/      ← JikoXpress customer

Why (and it's not isolation — policies do that either way):

Benefit Why
Independent tuning Different TTLs, audit settings per client
Clean deletion Contract ends → vault secrets disable wiki → gone
Accident containment A wildcard typo in a QBIT policy can't reach a client mount
Auditability Log lines obviously client data
Handover Export one mount, not a filtered subset
Vendor assessments Structural separation is easier to evidence than path prefixes

Clean deletion is the strongest argument. With prefixes you hunt for paths and hope you missed nothing.

Caution: don't pre-mount tenants you don't have.

And note: qbit/ is already your org boundary. qbit/qbitspark/... says it twice.

2.8 The Model, Stated Plainly

Mount = the machine (what kind of engine) Path = the filing system inside it Policy = who may open which drawer

Policies are standalone objects, not nested under engines. Confirmed in the UI — Policies is its own menu.

A policy merely mentions a path as text. You can write a policy for a path that doesn't exist. Delete the mount and the policy sits there harmlessly pointing at nothing.

POLICY  ─────────► names a path (as text)
   ▲
   │ referenced by name
AUTH METHOD ─────► issues tokens carrying that policy
   │
   ▼
TOKEN   ─────────► request arrives
                        ↓
                  Vault checks: does any policy on this
                  token allow this path + this verb?
                        ↓
                  SECRETS ENGINE serves or refuses

Which is why editing a policy takes effect immediately, even on tokens issued yesterday. Nothing is baked into the token.

2.9 Policies

HCL. Deny by default — anything not granted is denied. No deny rule needed; absence is denial.

path "qbit/data/jikoxpress/prod/*" {
  capabilities = ["read"]
}
path "qbit/metadata/jikoxpress/prod/*" {
  capabilities = ["list"]
}

Capabilities:

Capability Verb Meaning
create POST/PUT Create new
read GET Read existing
update POST/PUT Modify existing
delete DELETE Remove
list LIST Enumerate keys (never values)
sudo — Root-protected paths
deny — Explicit denial; overrides all grants

Policies are additive. A token gets the union of all its policies, plus default. To opt out of default, create the token with -no-default-policy.

token_policies="jikoxpress-prod,shared-readonly,monitoring"

Except deny, which is absolute:

path "qbit/data/jikoxpress/prod/snippe-psp" {
  capabilities = ["deny"]
}

Wildcards:

path "qbit/data/+/dev/*" { ... }        # every service's dev
path "identity/oidc/provider/+/authorize" { ... }   # from the default policy

Design rule: write small, single-purpose policies and compose them — jikoxpress-prod-read, jikoxpress-prod-write, shared-read — rather than one giant policy per service.

Reading the built-in default policy taught four things:

  1. Everything is self-scoped (lookup-self, renew-self, revoke-self) — no token can look up another
  2. {{identity.entity.id}} templating resolves per-token at request time — one document, per-caller permissions
  3. + demonstrated in the wild
  4. cubbyhole/* full CRUD — every token gets private storage nobody else can read, not even root. This is the mechanism behind response wrapping.

2.10 The KV v2 Path Gotcha

The single most common Vault mistake. The CLI lies to you — helpfully.

What you type:

vault kv get qbit/jikoxpress/prod/postgres

What goes over HTTP:

GET /v1/qbit/data/jikoxpress/prod/postgres

Vault tells you this in every output: Secret Path: qbit/data/jikoxpress/prod/postgres.

Why the extra segment — KV v2 splits operations into sub-paths:

Real API path Purpose
qbit/data/... the values
qbit/metadata/... version history, max_versions, delete-all
qbit/delete/... soft-delete
qbit/undelete/... restore
qbit/destroy/... permanent wipe

All five appear in normal CLI output once you look:

Success! Data written to: qbit/undelete/jikoxpress/prod/postgres
Success! Data written to: qbit/destroy/jikoxpress/prod/postgres
Metadata Path: qbit/metadata/jikoxpress/prod/postgres

The kv command inserts the right segment for you. Policies have no wrapper.

Write the obvious-looking thing and you get permission denied with no hint why:

path "qbit/jikoxpress/prod/*" { capabilities = ["read"] }   # ← WRONG

The useful consequence — separated paths let you grant operations independently:

path "qbit/data/jikoxpress/prod/*"     { capabilities = ["read"] }
path "qbit/metadata/jikoxpress/prod/*" { capabilities = ["read", "list"] }
path "qbit/destroy/jikoxpress/prod/*"  { capabilities = ["deny"] }

An app that reads secrets has no business destroying them.

KV v1 has none of this — one path, no /data/. Which is exactly why people who learned v1 get caught.

2.11 Path Design Is Policy Design

service/environment above component. Not negotiable.

Right — qbit/jikoxpress/prod/*:

path "qbit/data/jikoxpress/prod/*" { capabilities = ["read"] }

One rule. Covers every component, including ones added next year, automatically.

Wrong — qbit/jikoxpress/db/prod:

path "qbit/data/jikoxpress/db/prod"       { ... }
path "qbit/data/jikoxpress/snippe/prod"   { ... }
path "qbit/data/jikoxpress/redis/prod"    { ... }
...

One rule per component, forever.

The dangerous failure mode: six months later you add a component, forget the policy line, and the app breaks at 2am. Under pressure you "fix" it with qbit/data/jikoxpress/* — which now grants dev and staging too. The isolation is gone and nothing tells you.

The rule: put the thing you cut permissions along highest in the path.

Adopted structure:

qbit/<service>/<environment>/<component>

qbit/jikoxpress/prod/postgres
qbit/jikoxpress/prod/snippe-psp
qbit/nexgate/staging/minio
qbit/nexgate/prod/jwt-signing
qbit/mauzodukani/prod/postgres
qbit/glueemail/prod/mailcow
qbit/textfy/prod/vodacom-smpp
qbit/shared/prod/openai          ← belongs to no single product
qbit/shared/prod/cloudflare

Directories aren't real. There's no mkdir. Write to a deep path and every intermediate level springs into existence, inferred.

2.12 Auth Methods

Method For Notes
Token Bootstrap, break-glass Always enabled, can't be disabled
AppRole Machines Role ID + Secret ID. Primary machine auth.
Userpass Humans Simple; pair with MFA
OIDC / JWT Humans via SSO, CI/CD Keycloak, Authentik, Google Workspace
LDAP Humans via directory If you run one
TLS certificates Machines with mTLS Strong, more setup
Kubernetes Pods If you move to K8s
GitHub Humans / CI Ties access to org membership

Vault does have users — userpass or oidc. Same machinery as AppRole, different door.

2.13 AppRole — The Hotel

The analogy that worked:

Two ways to get a keycard:

Cheating (lab Step 6):
  You (root) → "make me a keycard" → token

Real (lab Step 8):
  App → role_id + secret_id → front desk checks → token

Same keycard. Different route. An app can't use the first — it would need root, which is the problem being solved.

Who has what:

Thing Lives where Secret? Expires?
Policy In Vault No No
Role ID App config, Git Not really No
Secret ID Delivered to app Yes Yes
Token App memory Yes Yes

Order of operations:

ONCE, by admin:
  1. write policy
  2. create approle role, attach policy by name

EVERY APP START:
  3. app sends role_id + secret_id
  4. Vault returns token
  5. app reads secrets

EVERY ~45 MIN:
  6. app renews token

AFTER token_max_ttl:
  7. back to step 3

Steps 3–7 are what "automatic" means. Spring Cloud Vault or Vault Agent does them. You write config, not code.

Why split into two halves? Different security properties, different delivery paths. Neither alone works. Role ID can sit in Git; Secret ID is delivered at runtime and expires. The thing in your repo isn't sufficient to authenticate.

The Secret ID is not stored readably. vault write -f .../secret-id is a write — it generates one, stores a hash, returns plaintext once. No read gets it back. vault list .../secret-id shows accessors (for revocation), never the value.

2.14 Tokens

Type Behaviour
Service Default; server-side; renewable, revocable, supports children
Batch Lightweight encrypted blobs; not stored; can't renew
Root root policy; unlimited; no TTL
Periodic Renews indefinitely within the period
Orphan No parent; survives parent revocation

Tokens form a tree — revoking a parent revokes children.

Root token lookup, annotated (from the lab):

ttl                 0s          ← NOT expired. NO EXPIRY.
expire_time         <nil>       ← never
policies            [root]      ← everything
orphan              true        ← can't be killed via a parent
num_uses            0           ← unlimited
id                  root        ← guessable (dev mode)

AppRole token lookup, same fields:

ttl                 56m16s                    ← counting down
expire_time         2026-08-06T11:02:24Z      ← real
policies            [default jikoxpress-prod] ← scoped
path                auth/approle/login        ← how it was obtained
meta                map[role_name:jikoxpress-prod]  ← attribution
entity_id           bcb55ff4-...              ← identity entity

token_meta_role_name is the audit win. Not "root did it" — "jikoxpress-prod did it." If NexGate's role ever reads a JikoXpress secret, you see it.

2.15 Leases, TTLs, and the Renewal Trap

Precedence: system default → mount tuning → role config → request-time. Each may only shorten.

768h = 32 days = the system default. A mount showing 768h isn't "configured" — it's "nobody set this."

The trap, discovered in the lab: every expiry you add is a renewal mechanism you must build.

token_ttl=1h → solved by renewal (Spring Cloud Vault does it in a background thread). token_max_ttl=4h → renewal can't exceed it; the app must be able to log in again. secret_id_ttl=24h → this is the one that kills services. When it expires, login itself fails. Renewal won't save you.

Four ways to handle Secret ID expiry:

  1. Non-expiring — secret_id_ttl=0, secret_id_num_uses=0. Simplest, weakest. Still far better than root.
  2. Vault Agent — sidecar handles login, renewal, re-auth. Writes a token to a file the app reads. The standard answer.
  3. Long TTL, rotated on deploy — 720h, with Komodo issuing fresh on each deploy.
  4. Response-wrapped at container start — most secure, most moving parts.

2.16 Attacker Analysis

Can a thief renew a stolen token? Yes. Vault can't distinguish them.

But token_max_ttl is the wall. Renewal extends, never resets; max TTL is measured from creation.

token_ttl=1h, token_max_ttl=4h
Steal at hour 0 → renew, renew, renew → at hour 4: DEAD

token_max_ttl is your actual worst-case exposure window. token_ttl is just how often renewal happens.

Hierarchy of theft:

Stolen Exposure
Token only Until token_max_ttl — hours
Token + Secret ID Until secret_id_ttl — days
Role ID + Secret ID Until you rotate
Root token Forever, everything, silently

Which is why the two halves must not live together. If docker inspect yields both, the design bought you nothing.

Where each realistically leaks from:

Role ID Secret ID
Committed compose/Komodo config File on server
Baked into an image (ENV) Env var → docker inspect
.env on the server CI/CD secret store
Kubernetes ConfigMap /proc/<pid>/environ
Shell history
App logs, if the app logs its config at startup

The attack, concretely:

docker inspect jikoxpress | grep -i vault
# VAULT_ROLE_ID=8c3f2a1b-...
# VAULT_SECRET_ID=d4e5f6a7-...

Both halves, one command.

Free hardening not yet used — CIDR binding:

vault write auth/approle/role/jikoxpress-prod \
    secret_id_bound_cidrs="161.97.163.158/32" \
    token_bound_cidrs="161.97.163.158/32"

Stolen credentials become useless anywhere but qbit-spark. The attacker would have to run the attack from your server.

Response wrapping — the strongest link:

vault write -f -wrap-ttl=60s auth/approle/role/jikoxpress-prod/secret-id

Returns a wrapper token, not the Secret ID. Single use. If someone intercepts and unwraps first, your app's unwrap fails — and that failure is your intrusion alarm. Not just secrecy: detection.

Revocation:

vault token revoke <token>
vault write auth/approle/role/X/secret-id-accessor/destroy secret_id_accessor="<acc>"
vault write -f auth/approle/role/X/secret-id      # rotate: kills existing
vault lease revoke -prefix auth/approle/

The honest summary: you cannot prevent a stolen token from being used. You can only bound how long it works and detect that it's happening. Short max TTLs and audit logs are the two levers.

2.17 delete / undelete / destroy / revoke

Two different axes. Constantly conflated.

About the data (KV):

Command Effect Reversible
kv delete Marks a version deleted; data still on disk ✅ undelete
kv undelete Clears the mark —
kv destroy Wipes that version's data; metadata tombstone remains ❌ Never
kv metadata delete Nukes the whole secret, all versions ❌ Never

Recycle bin → shredder → burning the filing cabinet.

About access (tokens/leases):

vault token revoke <token>
vault lease revoke <lease_id>
vault lease revoke -prefix database/

Revoking a dynamic secret lease doesn't just invalidate a record — Vault reaches into the backend and runs DROP ROLE. The user genuinely ceases to exist.

delete/destroy = removing the secret revoke = removing access

Applied to the incident: the OpenAI key leaked. kv destroy removes it from Vault — the stolen key still works. vault token revoke is irrelevant — the attacker never had a Vault token. Only revoking at OpenAI's dashboard stopped it.

That's the limit of static secrets. Vault stores and guards them; it cannot revoke what a third party issued.

The contrast: a database engine credential → vault lease revoke drops the PostgreSQL user directly. Vault issued it, so Vault can kill it. That's the difference between storing secrets and owning them.

KV v2 metadata settings worth using:

vault kv metadata put -max-versions=10 qbit/jikoxpress/prod/postgres

2.18 Audit Devices

Log every request and response. Vault refuses to operate if all enabled audit devices fail to write — fail-closed by design: no audit trail, no service.

vault audit enable file file_path=/vault/logs/audit.log

Sensitive values are HMAC'd; paths and field names are plaintext. Safe to ship to Grafana/Loki. You can compute the HMAC of a known value to search for it — forensics without exposure.

This is what was missing during the incident. With audit logging you could answer: was the OpenAI key ever read from Vault, by which role, from which IP, when?

2.19 Error Codes — Read Them Precisely

Error Meaning
403 permission denied Right verb, path exists, policy says no
404 / No value found Right verb, nothing there
405 unsupported operation Wrong verb — path exists but doesn't accept this method

In Step 5 you'll stare at 403s wondering whether the policy or the path is wrong. Knowing these apart is the difference between five minutes and an hour.

The debugging tool:

vault token capabilities <token> qbit/data/jikoxpress/prod/postgres
# → read
vault token capabilities <token> qbit/data/nexgate/prod/postgres
# → deny

Ask Vault directly instead of guessing.

Deliberate ambiguity: the UI says "You do not have the required permissions or the directory does not exist." It refuses to say which — otherwise an attacker maps your infrastructure by probing. Same reason good login forms say "invalid username or password."

PART 3 — The Lab

Ground rules: nothing touches production. Don't proceed until the ✅ check passes. After each step, close your notes and explain it aloud.

Setup — Docker, not apt

# ~/vault-lab/docker-compose.yml
services:
  vault:
    image: hashicorp/vault:1.20.3
    container_name: vault-lab
    ports:
      - "8200:8200"
    cap_add:
      - IPC_LOCK
    environment:
      VAULT_DEV_ROOT_TOKEN_ID: "root"
      VAULT_DEV_LISTEN_ADDRESS: "0.0.0.0:8200"
      VAULT_ADDR: "http://127.0.0.1:8200"
      VAULT_TOKEN: "root"
    command: server -dev
    volumes:
      - ./policies:/policies

Every line explained:

cap_add: IPC_LOCK — grants the mlock() capability. Vault holds decrypted secrets and the master key in RAM; if the kernel swaps that to disk, secrets get written to the swap file in plaintext. mlock() prevents it. Barely matters in dev mode, but correct from the start so the pattern carries.

VAULT_DEV_ROOT_TOKEN_ID: "root" — overrides the random dev token. Convenience only. Never anywhere real.

VAULT_DEV_LISTEN_ADDRESS: "0.0.0.0:8200" — which interface Vault binds to inside the container. 127.0.0.1 = container-only; Docker's port mapping would have nothing to forward to.

Security note: 0.0.0.0 inside + ports: mapping outside = exposed on every host interface. On a laptop behind a router, fine. On a VPS with no firewall, this exact pattern is how services get accidentally exposed to the internet. Possibly relevant to how the OpenAI key leaked.

command: server -dev — dev mode means: in-memory storage (all data lost on restart), auto-initialized and auto-unsealed, TLS disabled, KV v2 pre-mounted at secret/, root token printed to logs. Dev mode removes every barrier to learning and every security guarantee.

VAULT_ADDR is not optional. Without it the CLI defaults to https://127.0.0.1:8200, tries a TLS handshake against a plain-HTTP dev server, and fails with http: server gave HTTP response to HTTPS client.

Optional alias:

alias v='docker compose exec vault vault'

Step 1 — Run it

docker compose up -d
sleep 3                      # avoid the boot race
docker compose exec vault vault status

✅ Sealed false.

Read the boot logs. They print an unseal key and a root token in plaintext to stdout. In production that combination in a log file is game over.

The observation: the unseal key and cluster ID change on every restart. inmem = brand-new Vault each time. On persistent storage, the unseal key is generated once at initialization, never again.

Question: why would auto-unseal-with-the-key-in-the-logs be unacceptable on vault.qbitspark.com?

Answer: both protections are nullified at once. Already unsealed → encryption gives you nothing, secrets are decrypted in memory now. Key in the logs → even if sealed, the key to reopen it is in docker compose logs. An attacker with log read access needs neither to break encryption nor gather shares.

The trap: Vault's protection is only as good as the separation between the encrypted data and the keys that decrypt it. Dev mode puts them together. So does printing unseal keys to logs. So does storing shares on the same server. So does one person holding all five shares in one folder.

Step 2 — Tokens and your first secret

docker compose exec vault vault token lookup
docker compose exec vault vault kv put secret/hello message="my first secret" owner="kibuti"
docker compose exec vault vault kv get secret/hello
docker compose exec vault vault kv get -field=message secret/hello
docker compose exec vault vault kv get -format=json secret/hello

✅ Explain: what is the token, what is the secret, which is more dangerous to leak?

The answer, refined: it depends on scope. A narrow token is less dangerous than a big secret. A root token is worse than any single secret, because it's the key to all of them.

Leaked Damage
One secret That one system. Bad, bounded.
Scoped token (1h) What that role can read, for an hour
Root token Everything, forever, silently

Which is why the fix isn't "protect tokens harder" — it's make tokens small and short-lived so leaking one barely matters.

-field output is designed for scripting:

DB_PASS=$(vault kv get -field=password qbit/jikoxpress/prod/db)

Step 3 — Mount your own engine

docker compose exec vault vault secrets list        # baseline first
docker compose exec vault vault secrets enable -path=qbit kv-v2
docker compose exec vault vault secrets tune -description="QBIT SPARK application secrets" qbit/

Baseline on a dev server: cubbyhole/, identity/, secret/, sys/. On real Vault, secret/ wouldn't be there — three system mounts and nothing else. Everything useful, you mount yourself.

Explore the engine zoo (mount, look, remove):

docker compose exec vault vault secrets enable database
docker compose exec vault vault secrets enable pki
docker compose exec vault vault secrets enable transit
docker compose exec vault vault secrets enable ssh
docker compose exec vault vault secrets enable totp
docker compose exec vault vault secrets list -detailed

Four things in the -detailed table:

  1. Options = map[version:2] — the only place the table reveals KV v2 vs v1. Both show Plugin kv.
  2. Default TTL/Max TTL = system — inherit the 768h default. On database/ these become the real ceiling.
  3. Seal Wrap true — only on sys/. Extra encryption layer using the seal mechanism, applied to Vault's own control plane.
  4. Running Version — KV shows v0.24.0+builtin while others show v1.20.3+builtin.vault. The KV engine versions independently of Vault core — which is why plugin upgrades and Vault upgrades are different operations.

Dynamic engines are inert until configured:

docker compose exec vault vault list database/config    # No value found
docker compose exec vault vault list transit/keys

KV works the instant you mount it. Dynamic engines need to know what system to talk to and what to create there.

Cleanup — and note the danger:

docker compose exec vault vault secrets disable database

disable destroys all data under that path. No confirmation prompt. On real Vault, vault secrets disable pki deletes your entire certificate authority.

Then build the tree:

docker compose exec vault vault kv put qbit/jikoxpress/prod/postgres username="jx" password="x"
docker compose exec vault vault kv put qbit/jikoxpress/prod/snippe-psp api_key="x" merchant_id="x"
docker compose exec vault vault kv put qbit/jikoxpress/staging/postgres username="jx" password="x"
docker compose exec vault vault kv put qbit/nexgate/prod/postgres username="ng" password="x"
docker compose exec vault vault kv put qbit/nexgate/prod/rabbitmq username="ng" password="x"
docker compose exec vault vault kv put qbit/nexgate/prod/jwt-signing private_key="x" public_key="x"
docker compose exec vault vault kv put qbit/nexgate/staging/minio access_key="x" secret_key="x"
docker compose exec vault vault kv put qbit/mauzodukani/prod/postgres username="md" password="x"
docker compose exec vault vault kv put qbit/glueemail/prod/mailcow admin_user="x" admin_pass="x"
docker compose exec vault vault kv put qbit/textfy/prod/vodacom-smpp system_id="x" password="x"
docker compose exec vault vault kv put qbit/shared/prod/openai api_key="sk-placeholder"
docker compose exec vault vault kv put qbit/shared/prod/cloudflare api_token="x"

All values are x deliberately. Building the habit of never typing a real credential into a lab is worth more than the lab itself.

Walk the tree:

docker compose exec vault vault kv list qbit/
docker compose exec vault vault kv list qbit/jikoxpress/
docker compose exec vault vault kv list qbit/jikoxpress/prod/

Trailing slashes mark directories:

jikoxpress/    ← has children
prod/          ← has children
postgres       ← no slash: an actual secret

list never shows values. Separate capability from read — you can let a junior dev see what exists without reading contents.

Step 4 — Versioning

docker compose exec vault vault kv put qbit/jikoxpress/prod/postgres username="jx" password="NEW-v2"
docker compose exec vault vault kv get qbit/jikoxpress/prod/postgres
docker compose exec vault vault kv get -version=1 qbit/jikoxpress/prod/postgres
docker compose exec vault vault kv metadata get qbit/jikoxpress/prod/postgres

docker compose exec vault vault kv delete qbit/jikoxpress/prod/postgres
docker compose exec vault vault kv undelete -versions=2 qbit/jikoxpress/prod/postgres
docker compose exec vault vault kv destroy -versions=1 qbit/jikoxpress/prod/postgres

Gotcha found live: kv get without -version always fetches the latest. Undeleting v2 while v3 is deleted looks like the undelete failed. Use kv metadata get — it shows every version and its state at once.

✅ Metadata should show: v1 destroyed true, v2 clean, v3 deletion_time set.

Why it matters for rotation:

1. Write new version        ← old still live
2. Restart services
3. Verify healthy
4. THEN revoke at provider
5. Optionally destroy old version

If step 3 fails, kv get -version=<previous> recovers instantly.

The trap: delete looks destructive but isn't. People think a secret is gone when it's fully recoverable by anyone with undelete. To actually remove a leaked credential from Vault, it's destroy.

Step 5 — First policy

mkdir -p ~/vault-lab/policies

cat > ~/vault-lab/policies/jikoxpress-prod.hcl <<'EOF'
path "qbit/data/jikoxpress/prod/*" {
  capabilities = ["read"]
}
path "qbit/metadata/jikoxpress/prod/*" {
  capabilities = ["list"]
}
EOF

docker compose exec vault vault policy write jikoxpress-prod /policies/jikoxpress-prod.hcl
docker compose exec vault vault policy read jikoxpress-prod
docker compose exec vault vault policy list
docker compose exec vault vault policy read default    # read this one properly

Anatomy of the command:

vault policy write jikoxpress-prod /policies/jikoxpress-prod.hcl
                   │                │
                   │                └─ file, INSIDE the container
                   └─ policy name (unrelated to the filename)

The file and the policy name are independent. Keep them matching by convention, not requirement.

/policies/ is the container path. Your volume maps ./policies → /policies.

Once written, the file no longer matters. Vault stores the policy internally; delete the .hcl and it keeps working. The file is source, not runtime — which is why policies belong in Git, giving you version history and review while Vault holds current state.

Step 6 — Prove the denial ⭐

The most important step. A policy tested only for success is untested — you might still be root.

docker compose exec vault vault token create -policy=jikoxpress-prod -ttl=30m
TOK="<paste>"

docker compose exec -e VAULT_TOKEN=$TOK vault vault kv get qbit/jikoxpress/prod/postgres      # ALLOW
docker compose exec -e VAULT_TOKEN=$TOK vault vault kv get qbit/jikoxpress/staging/postgres   # DENY
docker compose exec -e VAULT_TOKEN=$TOK vault vault kv get qbit/nexgate/prod/postgres         # DENY
docker compose exec -e VAULT_TOKEN=$TOK vault vault kv get qbit/shared/prod/openai            # DENY
docker compose exec -e VAULT_TOKEN=$TOK vault vault kv put qbit/jikoxpress/prod/postgres password="hacked"  # DENY

✅ Four 403 permission denied. Those errors are the proof.

The last denial is the subtle one: PUT blocked on a path the token can read. Same path, different verb, different answer. Least privilege at the operation level, not just the path level.

Note: token_policies ["default" "jikoxpress-prod"] — additive, as expected.

Step 7 — Enable AppRole

docker compose exec vault vault auth enable approle

docker compose exec vault vault write auth/approle/role/jikoxpress-prod \
    token_policies="jikoxpress-prod" \
    token_ttl=1h \
    token_max_ttl=4h \
    secret_id_ttl=0

docker compose exec vault vault read auth/approle/role/jikoxpress-prod

Note token_policies="jikoxpress-prod" is just a name. The AppRole never mentions paths — the policy owns those. To change what a service can access, edit the policy; existing tokens pick it up on their next request.

Two empty fields worth filling in production: secret_id_bound_cidrs and token_bound_cidrs.

Lab warning: don't set CIDRs to your production IP in the lab — your requests come from inside a Docker container and you'll lock yourself out.

Step 8 — Log in as a service ⭐

RID=$(docker compose exec -T vault vault read -field=role_id auth/approle/role/jikoxpress-prod/role-id)
SID=$(docker compose exec -T vault vault write -f -field=secret_id auth/approle/role/jikoxpress-prod/secret-id)

docker compose exec -e VAULT_TOKEN= -T vault vault write auth/approle/login \
    role_id="$RID" secret_id="$SID"

-e VAULT_TOKEN= blanks the root token — proving login needs no prior credential.

APPTOK="<paste>"
docker compose exec -e VAULT_TOKEN=$APPTOK vault vault token lookup
docker compose exec -e VAULT_TOKEN=$APPTOK vault vault kv get qbit/jikoxpress/prod/postgres   # ALLOW
docker compose exec -e VAULT_TOKEN=$APPTOK vault vault kv get qbit/nexgate/prod/postgres      # DENY
docker compose exec -e VAULT_TOKEN=$APPTOK vault vault token renew                            # note: positional, not -self

✅ A production secret retrieved with zero root token involvement.

Where does the token come from at all? The CLI resolution order:

  1. -address / explicit flag
  2. VAULT_TOKEN environment variable ← what the compose file sets
  3. ~/.vault-token (written by vault login)

Being inside the container grants nothing. No implicit auth, no localhost exemption, no container trust. Prove it:

docker compose exec -e VAULT_TOKEN= vault vault kv get qbit/jikoxpress/prod/db   # denied

Vault is an HTTP API with a token header. The CLI is a wrapper:

curl -H "X-Vault-Token: root" http://127.0.0.1:8200/v1/qbit/data/jikoxpress/prod/db

The uncomfortable parallel: "what token is being used?" — the answer was root, because it sat in a config file you edited and stopped thinking about. That's the shape of the production problem.

The UI Experiment ⭐

Open http://127.0.0.1:8200.

Sidebar structure — three separate things, three separate menus:

Then: sign out, sign back in with the AppRole token.

What happens: the interface collapses. No Policies. No Access. No Seal Vault. Instead of a browsable tree, a box saying "Type the path of the secret you want to view."

Why: the policy grants list on qbit/metadata/jikoxpress/prod/* but not on qbit/ itself. The UI can't enumerate the top level, so it falls back to asking.

This is what a compromised JikoXpress service looks like from the attacker's side. Two secrets. Not NexGate's JWT keys, not Cloudflare, not Vodacom. Compare to a stolen root token: everything, in a navigable tree.

A design decision it surfaces: apps don't browse — they fetch known paths, so read-only on exact paths is right. Humans with scoped policies find the inability to navigate genuinely annoying, so add list grants deliberately:

path "qbit/metadata"            { capabilities = ["list"] }
path "qbit/metadata/jikoxpress" { capabilities = ["list"] }

Also observed: a yellow banner — "We've stopped auto-renewing your token due to inactivity. It will expire in about 1 hour." with a Renew token button. The UI had been renewing in the background all along. That's vault token renew with a nicer face, and exactly what Spring Cloud Vault does in a thread.

And: Create secret + still renders even without create capability. Clicking fails with 403. UI affordances aren't permissions.

Seed script — because inmem wipes on restart

cat > ~/vault-lab/seed.sh <<'EOF'
#!/usr/bin/env bash
set -euo pipefail
V="docker compose exec -T vault vault"

$V secrets enable -path=qbit kv-v2 2>/dev/null || true
$V secrets tune -description="QBIT SPARK application secrets" qbit/
$V auth enable approle 2>/dev/null || true

$V kv put qbit/jikoxpress/prod/postgres username="jx" password="x"
$V kv put qbit/jikoxpress/prod/snippe-psp api_key="x" merchant_id="x"
$V kv put qbit/jikoxpress/staging/postgres username="jx" password="x"
$V kv put qbit/nexgate/prod/postgres username="ng" password="x"
$V kv put qbit/nexgate/prod/jwt-signing private_key="x" public_key="x"
$V kv put qbit/nexgate/staging/minio access_key="x" secret_key="x"
$V kv put qbit/mauzodukani/prod/postgres username="md" password="x"
$V kv put qbit/glueemail/prod/mailcow admin_user="x" admin_pass="x"
$V kv put qbit/textfy/prod/vodacom-smpp system_id="x" password="x"
$V kv put qbit/shared/prod/openai api_key="sk-placeholder"

$V policy write jikoxpress-prod /policies/jikoxpress-prod.hcl
$V write auth/approle/role/jikoxpress-prod \
    token_policies="jikoxpress-prod" token_ttl=1h token_max_ttl=4h secret_id_ttl=0

echo "✓ seeded"
EOF
chmod +x ~/vault-lab/seed.sh

Wipe and rebuild freely:

docker compose down && docker compose up -d && sleep 3 && ./seed.sh

Step 9 — All services (pending)

#!/usr/bin/env bash
set -euo pipefail
V="docker compose exec -T vault vault"

declare -A SERVICES=(
  [jikoxpress]="dev staging prod"
  [nexgate]="dev staging prod"
  [mauzodukani]="dev prod"
  [glueemail]="prod"
  [textfy]="dev prod"
  [shared]="prod"
)

for svc in "${!SERVICES[@]}"; do
  for env in ${SERVICES[$svc]}; do
    name="${svc}-${env}"

    cat > "policies/${name}.hcl" <<EOF
path "qbit/data/${svc}/${env}/*" {
  capabilities = ["read"]
}
path "qbit/metadata/${svc}/${env}/*" {
  capabilities = ["list"]
}
EOF
    $V policy write "$name" "/policies/${name}.hcl"

    if [ "$env" = "prod" ]; then TTL=1h; MAXTTL=4h; else TTL=4h; MAXTTL=12h; fi

    $V write "auth/approle/role/${name}" \
        token_policies="$name" token_ttl="$TTL" token_max_ttl="$MAXTTL" secret_id_ttl=0
    echo "✓ ${name} (ttl=${TTL})"
  done
done

Production TTLs are shorter because blast radius matters more there.

Step 10 — Isolation verification (pending)

#!/usr/bin/env bash
set -uo pipefail
V="docker compose exec -T vault vault"
PASS=0; FAIL=0

check() {
  local role="$1" path="$2" expect="$3" rid sid tok result
  rid=$($V read -field=role_id "auth/approle/role/${role}/role-id")
  sid=$($V write -f -field=secret_id "auth/approle/role/${role}/secret-id")
  tok=$($V write -field=token auth/approle/login role_id="$rid" secret_id="$sid")

  if docker compose exec -T -e VAULT_TOKEN="$tok" vault vault kv get "$path" >/dev/null 2>&1
  then result="ALLOW"; else result="DENY"; fi

  if [ "$result" = "$expect" ]; then
    echo "  ✓ ${role} → ${path} = ${result}"; PASS=$((PASS+1))
  else
    echo "  ✗ ${role} → ${path} = ${result} (expected ${expect})"; FAIL=$((FAIL+1))
  fi
}

echo "Positive (must ALLOW):"
check jikoxpress-prod  qbit/jikoxpress/prod/postgres  ALLOW
check nexgate-staging  qbit/nexgate/staging/minio     ALLOW

echo "Negative (must DENY):"
check jikoxpress-prod  qbit/nexgate/prod/jwt-signing  DENY
check jikoxpress-prod  qbit/jikoxpress/dev/postgres   DENY
check nexgate-staging  qbit/nexgate/prod/jwt-signing  DENY
check textfy-prod      qbit/jikoxpress/prod/postgres  DENY
check jikoxpress-prod  qbit/shared/prod/openai        DENY

echo; echo "Passed: $PASS  Failed: $FAIL"
[ "$FAIL" -eq 0 ] || exit 1

Keep this script forever. Run after every policy change, in the lab and in production.

Step 11 — Audit logging (pending)

docker compose exec vault vault audit enable file file_path=/vault/logs/audit.log
docker compose exec vault vault audit list -detailed
docker compose exec vault vault kv get qbit/jikoxpress/prod/postgres
docker compose exec vault sh -c 'tail -n 5 /vault/logs/audit.log'

✅ Find your request. Confirm the value is HMAC'd but the path is plaintext.

Step 12 — Dynamic database credentials (pending) ⭐

docker run -d --name vault-lab-pg \
  -e POSTGRES_PASSWORD=rootpass -e POSTGRES_DB=jikoxpress \
  -p 5433:5432 postgres:16

docker compose exec vault vault secrets enable database

docker compose exec vault vault write database/config/jikoxpress-pg \
    plugin_name=postgresql-database-plugin \
    allowed_roles="jikoxpress-app" \
    connection_url="postgresql://{{username}}:{{password}}@host.docker.internal:5433/jikoxpress?sslmode=disable" \
    username="postgres" password="rootpass"

docker compose exec vault vault write database/roles/jikoxpress-app \
    db_name=jikoxpress-pg \
    creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; \
        GRANT SELECT, INSERT, UPDATE, DELETE ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
    default_ttl="1h" max_ttl="24h"

docker compose exec vault vault read database/creds/jikoxpress-app
docker compose exec vault vault read database/creds/jikoxpress-app    # different every time

The contrast that makes it click:

Static (KV):
  vault kv get qbit/jikoxpress/prod/db
  → username: jx_prod, password: s3cr3t
  → same answer, every time, forever

Dynamic (database):
  vault read database/creds/jikoxpress-app
  → v-approle-jikoxpr-x7Kd92 / A1a-8sKdm3nQp   (lease 1h)
  vault read database/creds/jikoxpress-app
  → v-approle-jikoxpr-mN4vB1 / Z9z-2pLwq7Rt    (lease 1h)

Vault runs CREATE ROLE live, then DROP ROLE an hour later. Verify with \du in psql — the users genuinely exist, then genuinely don't.

✅ Generate twice, confirm both in \du, vault lease revoke one, confirm it's dropped.

Steal one of those and you've stolen something that dies on its own. That's the whole game.

PART 4 — Mistakes File

The book's differentiator. Real errors from the real session.

4.1 The invisible leading space

What happened: a \ line continuation with a trailing space made the shell pass mesage=... as one argument. Vault stored the key literally as " mesage".

Why it was confusing: kv get displayed it as mesage — the table view strips the visual space. -field=mesage then said "not present in secret."

Diagnosis:

vault kv get -format=json secret/hello
# "data": { " mesage": "my first secret" }

Lessons:

  1. When a field "exists but isn't found," go straight to -format=json. The pretty output lies about whitespace.
  2. Vault has no schema and no validation. It will happily store PASSWROD, api_ky, mesage. Then your app looks for password, finds nothing, and fails at runtime with a confusing error. Naming conventions are the only defence.

4.2 The boot race

Symptom: dial tcp 127.0.0.1:8200: connect: connection refused immediately after docker compose up -d.

Cause: the container reported "Started" ~1.5s before Vault bound the port.

Lesson: connection refused means nothing is listening — either not booted yet or crashed. Check logs before guessing. docker compose logs vault --tail=40. Reading Vault's startup logs is a skill you'll use constantly.

Fix: sleep 3.

4.3 VAULT_ADDR defaults to HTTPS

Symptom: Error checking seal status: http: server gave HTTP response to HTTPS client

Cause: the CLI defaults to https://127.0.0.1:8200; dev mode serves plain HTTP.

Lesson: VAULT_ADDR isn't optional. The CLI has no idea where your server is or what protocol it speaks.

4.4 405 ≠ 404 ≠ 403

Symptom: vault read database/config → 405 unsupported operation. Expected "not configured."

Cause: database/config only supports LIST. Wrong verb, not missing data.

Lesson: learn the three apart (see §2.19). It's the difference between five minutes and an hour when debugging policies.

4.5 kv get always fetches latest

Symptom: kv undelete -versions=2 succeeded, but kv get still showed a deleted secret.

Cause: kv get without -version fetches the latest version — v3, still deleted. v2 was correctly restored.

Lesson: use kv metadata get to see all versions and states at once.

4.6 token renew -self isn't a flag

Symptom: flag provided but not defined: -self

Fix: vault token renew — bare, renews the token in VAULT_TOKEN.

4.7 Missing exec vault

Symptom: unknown docker command: "compose vault"

Cause: docker compose vault ... instead of docker compose exec vault vault .... The pattern is: first vault = service name, second = binary.

4.8 Expired token = working as designed

Symptom: UI login fails with invalid token.

Cause: token_ttl=1h elapsed. (Or the container restarted — inmem wipes tokens.)

Lesson — the mental shift: the instinct was "something is broken." Nothing broke. Tokens are supposed to stop working. Getting a new one is routine, not recovery. Apps do this automatically; you do it by hand because you're learning.

4.9 The word "secret" means two things

secret/ in a command is a mount path — just a folder name from dev mode. "A secret" in conversation is the sensitive data. Moving to qbit/ removes the ambiguity.

4.10 The /data/ insertion

See §2.10. Expect to hit it. Everyone does.

PART 5 — Humans, Machines & MFA

5.1 Two Populations

Humans Machines
Auth method Userpass, OIDC, LDAP AppRole, TLS cert, K8s
MFA Required N/A
Token TTL 8h 1–4h, auto-renewed
Access Broad but read-mostly, no prod Narrow, exactly what's needed
Audit shows Named individual Named service

Current state: one root token serving both. That's the thing to fix.

5.2 Enabling Users

vault auth enable userpass

vault write auth/userpass/users/kibuti \
    password="..." token_policies="vault-admin" \
    token_ttl=8h token_max_ttl=12h

vault write auth/userpass/users/dev1 \
    password="..." token_policies="dev-engineer" token_ttl=8h

Now the audit log says dev1, not root.

5.3 Developer Policy

# dev-engineer.hcl
# full CRUD on any service's dev
path "qbit/data/+/dev/*" {
  capabilities = ["create", "read", "update", "delete", "list"]
}
path "qbit/metadata/+/dev/*" {
  capabilities = ["read", "list", "delete"]
}

# staging: read only
path "qbit/data/+/staging/*" {
  capabilities = ["read"]
}
path "qbit/metadata/+/staging/*" {
  capabilities = ["read", "list"]
}

# prod: no rule = denied

Note the + — * only globs at the end of a path.

The hard rule: developers should not read production secrets at all. Not "read-only prod" — no prod. If a dev needs the production database password, something is wrong with your deployment process. Apps read prod secrets via AppRole; humans don't need to.

Who writes prod secrets? You, as admin. Or a CI pipeline with a narrow write-only policy.

5.4 The Policy Set

Policy Who Grants
vault-admin You Everything except root-only ops
dev-engineer Devs CRUD on +/dev/*, read +/staging/*
deploy-pipeline Komodo Generate Secret IDs only
<service>-<env> Apps Read one path
auditor Vendor assessments list on metadata, no read on data

The auditor policy is genuinely useful — someone verifies your secrets are properly organised without seeing a single value.

5.5 MFA

Vault Community has no native login MFA. That's Enterprise. Get it from the identity provider instead.

Option A — OIDC with an MFA-capable IdP (best long-term)

vault auth enable oidc
vault write auth/oidc/config \
    oidc_discovery_url="https://auth.qbitspark.com/realms/qbit" \
    oidc_client_id="vault" oidc_client_secret="..." default_role="engineer"

vault write auth/oidc/role/engineer \
    bound_audiences="vault" \
    allowed_redirect_uris="https://vault.qbitspark.com/ui/vault/auth/oidc/oidc/callback" \
    user_claim="email" token_policies="dev-engineer" token_ttl=8h

Keycloak or Authentik self-hosted. Bonus: SSO across your other internal tools.

Option B — network-layer (fastest for you) Traefik + Authelia/Authentik forward-auth in front of the Vault UI. MFA happens before the request reaches Vault. You already run Traefik.

Option C — userpass + TOTP engine. Vault's TOTP engine manages codes but doesn't enforce them on login. Partial only.

Recommendation: B now, A later.

Machines don't get MFA — that's what AppRole + CIDR binding is for.

5.6 Break-Glass

# generate root (needs 3 shares)
vault operator generate-root -init
vault operator generate-root -nonce=<nonce>
vault operator generate-root -decode=<encoded> -otp=<otp>
vault token revoke <root-token>      # IMMEDIATELY after use

# emergency seal
vault operator seal

Document offline, before you need it:

Solo-operator note: Shamir assumes multiple people. Split across locations instead. Never all in one place, never all on one machine. Test the recovery once — an untested scheme is a story you tell yourself.

5.7 Shamir Beyond Vault

Same math, real-world uses:

For personal property:

3-of-5:
  Share 1 → bank safe deposit box (Mbeya)
  Share 2 → family member, different city
  Share 3 → lawyer, sealed envelope with instructions
  Share 4 → encrypted, cloud storage (different provider than your infra)
  Share 5 → physical, hidden at home

Survives fire, theft, one betrayal, one person unreachable.

sudo apt install ssss
echo "my-secret" | ssss-split -t 3 -n 5
ssss-combine -t 3

Steel plates beat paper — paper burns, ink fades. Cryptosteel exists; a hardware store and letter punches do the same job cheaper.

Limitations, honestly: humans lose things; the instructions must survive too (a share is useless if nobody knows what it is); legal systems don't recognise "3-of-5" the way they recognise a notarised will — use it alongside proper instruments; and test it.

PART 6 — Rotation

6.1 Theory

Every credential's compromise probability rises with age and exposure. Rotation bounds the damage window.

Maturity ladder:

  1. ❌ Never rotate — where you were
  2. ⚠️ Manual scheduled — where you're going next
  3. ✅ Automated rotation of static secrets
  4. 🏆 Dynamic secrets — rotation becomes irrelevant because nothing is long-lived

6.2 By Credential Type

Root token — generate → use → revoke. Never leave one alive.

Unseal keys

vault operator rekey -init -key-shares=5 -key-threshold=3

Encryption key (transparent)

vault operator rotate
vault operator key-status

AppRole Secret IDs

vault write -f auth/approle/role/X/secret-id                    # new
vault write auth/approle/role/X/secret-id-accessor/destroy \
    secret_id_accessor="<accessor>"                             # kill old

KV static secrets — order matters

1. Create new credential at the provider
2. vault kv put ...            ← new version; old still readable
3. Restart/reload services
4. Verify healthy
5. ONLY THEN revoke old at the provider
6. Optionally vault kv destroy -versions=<n>

Revoking before services pick up the new value = outage.

Database credentials — move to the database engine; rotation becomes automatic.

6.3 Schedule

Credential Frequency Method
Root token Per use generate → use → revoke
Unseal keys Annually or on suspicion operator rekey
Encryption key Quarterly operator rotate
AppRole Secret IDs Per deploy, or quarterly Komodo, or manual
Service tokens 1–4h Automatic
Third-party API keys Quarterly + on incident Manual, versioned in KV
DB credentials 1h Dynamic engine
TLS certificates 90 days PKI engine
Komodo's own Secret ID Quarterly Manual, by SSH

6.4 Dynamic Migration Order

  1. PostgreSQL — all services; highest value
  2. RabbitMQ — NexGate messaging
  3. MinIO — via AWS engine (S3-compatible STS)
  4. PKI — internal CA for NexGate mTLS
  5. SSH — signed certs for qbit-spark
  6. Transit — JikoXpress financial data

Success measure: a leaked credential is a non-event because it expired an hour ago.

PART 7 — Deployment Architecture

7.1 Where Credentials Live — Per Environment

The one rule: Role ID and Secret ID never in the same place.

Local (developer laptop)

Role ID    → committed in repo (docker-compose.dev.yml)
Secret ID  → .vault/secret-id, gitignored

Each dev generates their own Secret ID. Never committed.

Staging

Role ID    → docker-compose.yml on the staging box (Git fine)
Secret ID  → /etc/vault/<service>-staging-secret-id
             chmod 400, chown root:root

Production

Role ID    → docker-compose.yml (Git fine)
Secret ID  → /etc/vault/<service>-prod-secret-id
             chmod 400, chown root:root

The secrets themselves

In Vault. Nowhere else. No .env, no config file, no baked image layer. Fetched at boot with the AppRole token.

7.2 Never Do This

environment:
  VAULT_ROLE_ID: "..."
  VAULT_SECRET_ID: "..."   # ← docker inspect prints this

One command yields both halves. This is likely how the OpenAI key walked out.

7.3 Do This

# on the server, as admin
sudo mkdir -p /etc/vault
vault write -f -field=secret_id auth/approle/role/bishamba-prod/secret-id \
  | sudo tee /etc/vault/bishamba-prod-secret-id > /dev/null
sudo chmod 400 /etc/vault/bishamba-prod-secret-id
sudo chown root:root /etc/vault/bishamba-prod-secret-id

services:
  bishamba:
    environment:
      VAULT_ROLE_ID: "8c3f2a1b-4d5e-6f70-8192-a3b4c5d6e7f8"
    volumes:
      - /etc/vault/bishamba-prod-secret-id:/run/secrets/secret-id:ro

#!/bin/sh
# entrypoint.sh
export VAULT_SECRET_ID=$(cat /run/secrets/secret-id)
exec java -jar app.jar

Why file-mount beats env var: docker inspect shows only the mount path. docker compose config shows nothing sensitive. Git holds only the Role ID.

Honest limitation: the env var still exists inside the running process, so /proc/<pid>/environ shows it to root. You've removed it from the Docker metadata layer, compose file, and Git — which is where these things actually leak from. Vault Agent removes it entirely.

7.4 Spring Boot Integration

# application-prod.yml
spring:
  cloud:
    vault:
      uri: https://vault.qbitspark.com
      authentication: APPROLE
      app-role:
        role-id: ${VAULT_ROLE_ID}
        secret-id: ${VAULT_SECRET_ID}
      kv:
        enabled: true
        backend: qbit
        default-context: bishamba/prod

@Value("${postgres.username}")
private String dbUser;

@Value("${postgres.password}")
private String dbPass;

Spring Cloud Vault logs in at startup, fetches qbit/bishamba/prod/*, injects into properties, renews the token in a background thread. No credentials in code, no .env in production.

7.5 Three Environments, One Vault

Decision: connectivity isn't a constraint, so all three live on the central Vault.

for env in local staging prod; do
cat > policies/bishamba-${env}.hcl <<EOF
path "qbit/data/bishamba/${env}/*" {
  capabilities = ["read"]
}
path "qbit/metadata/bishamba/${env}/*" {
  capabilities = ["list"]
}
EOF
vault policy write bishamba-${env} policies/bishamba-${env}.hcl
done

# local — loose, no CIDR (devs move around)
vault write auth/approle/role/bishamba-local \
    token_policies="bishamba-local" \
    token_ttl=12h token_max_ttl=24h secret_id_ttl=0

# staging — bound to the staging box
vault write auth/approle/role/bishamba-staging \
    token_policies="bishamba-staging" \
    token_ttl=4h token_max_ttl=12h secret_id_ttl=720h \
    secret_id_bound_cidrs="<staging-ip>/32"

# prod — tightest
vault write auth/approle/role/bishamba-prod \
    token_policies="bishamba-prod" \
    token_ttl=1h token_max_ttl=4h secret_id_ttl=720h \
    secret_id_bound_cidrs="161.97.163.158/32"

Only two lines of app config differ between environments — default-context, and in local, uri.

Two consequences of putting local on the shared Vault:

  1. Local secrets must be fake. Everyone with a local role can read them. Enforce by convention and review.
  2. You need per-developer revocation. Give each dev their own Secret ID, not a shared one. Then you revoke individually and the audit log tells you whose credential was used.
vault write -f auth/approle/role/bishamba-local/secret-id       # rotate all
vault list auth/approle/role/bishamba-local/secret-id           # accessors
vault write auth/approle/role/bishamba-local/secret-id-accessor/destroy \
    secret_id_accessor="<accessor>"                             # kill one

Trade-off recorded (the case for a separate local Vault, should you revisit):

All three central Local separate
Same code path ✅ ✅
Dev needs prod-Vault access ❌ Yes ✅ No
Works offline ❌ ✅
CIDR binding staging/prod only staging/prod only
Onboarding Issue credentials docker compose up
Dev leaves Must revoke Nothing to revoke

7.6 New Project Runbook

Phase 1 — you (admin) set up: decide paths, write policies per environment, create AppRoles, grant devs dev-engineer.

Phase 2 — devs work in dev: they write their own dev secrets, read via the dev AppRole, never touch prod.

Phase 3 — you write real production secrets. The devs who wrote the app never see these values.

Phase 4 — deployment: generate Role ID + Secret ID, place them separately (§7.3).

Phase 5 — app consumes via Spring Cloud Vault (§7.4).

The property this buys: a developer can build, test, and ship the entire application without ever knowing a single production credential. If their laptop is stolen, production is unaffected.

7.7 Komodo Integration

Komodo needs a Vault token to mint Secret IDs. That token becomes the new secret zero, so keep it narrow.

1. A policy that does one thing

# komodo-deployer.hcl
path "auth/approle/role/+/secret-id" {
  capabilities = ["update"]
}

Can mint Secret IDs. Cannot read a single secret, cannot change policies, cannot create roles.

2. Komodo gets its own AppRole — same discipline

vault policy write komodo-deployer policies/komodo-deployer.hcl

vault write auth/approle/role/komodo-deployer \
    token_policies="komodo-deployer" \
    token_ttl=15m token_max_ttl=30m \
    secret_id_ttl=0 \
    secret_id_bound_cidrs="161.97.163.158/32"

3. You place Komodo's credentials by hand, once, over SSH

ssh you@161.97.163.158
sudo mkdir -p /etc/vault

vault read -field=role_id auth/approle/role/komodo-deployer/role-id \
  | sudo tee /etc/vault/komodo-role-id > /dev/null
vault write -f -field=secret_id auth/approle/role/komodo-deployer/secret-id \
  | sudo tee /etc/vault/komodo-secret-id > /dev/null

sudo chmod 400 /etc/vault/komodo-*
sudo chown root:root /etc/vault/komodo-*

services:
  komodo:
    volumes:
      - /etc/vault/komodo-role-id:/run/secrets/role-id:ro
      - /etc/vault/komodo-secret-id:/run/secrets/secret-id:ro

4. Komodo's deploy script

#!/usr/bin/env bash
set -euo pipefail
SERVICE="$1"   # e.g. bishamba-prod

TOKEN=$(vault write -field=token auth/approle/login \
  role_id="$(cat /run/secrets/role-id)" \
  secret_id="$(cat /run/secrets/secret-id)")

VAULT_TOKEN=$TOKEN vault write -f -field=secret_id \
  "auth/approle/role/${SERVICE}/secret-id" \
  > "/etc/vault/${SERVICE}-secret-id"
chmod 400 "/etc/vault/${SERVICE}-secret-id"

docker compose up -d "$SERVICE"

# after health check passes, revoke older Secret IDs
# Komodo's own token dies in 15 minutes regardless

Revoke old Secret IDs after deploy — otherwise every deploy leaves a valid credential behind. Do the destroy after the new container is healthy, not before.

Rotating Komodo's own Secret ID is manual, deliberately:

ssh qbit-spark
vault write -f -field=secret_id auth/approle/role/komodo-deployer/secret-id \
  | sudo tee /etc/vault/komodo-secret-id > /dev/null
docker compose restart komodo

It can't be automated from Komodo — that would let a compromised Komodo renew its own access forever. Keeping it manual bounds the chain.

7.8 Where the Chain Terminates

YOU                    ← SSH key. The actual root of trust.
 │
 └─ place Komodo's role-id + secret-id  (once, by hand)
     │
     └─ KOMODO mints app Secret IDs      (every deploy, 15min tokens)
         │
         └─ APPS log in, read their own secrets  (every start, 1h tokens)

Your SSH key is the bottom. That's the real secret zero for the whole system — which is why it deserves a passphrase, a hardware key if possible, and an entry in the break-glass documentation.

7.9 Secret Zero — The Honest Accounting

"To get a credential for Vault, you need a credential for Vault."

Correct. It never disappears. It shrinks.

Before:

.env on server:
  DB_PASSWORD=real
  SNIPPE_API_KEY=real
  JWT_PRIVATE_KEY=real
  OPENAI_KEY=real
  MINIO_SECRET=real

One file. Everything. Never expires. No log of who read it.

After:

/etc/vault/prod-secret-id:
  d4e5f6a7-b8c9-...

One file. One UUID.

Old .env Stolen Secret ID
What they get Every credential immediately Nothing — need Role ID too
With both halves — One service, one environment
Duration Forever Until rotated
IP-bindable No Yes
Do you find out No Yes — audit log
To revoke Change everything, everywhere One command

The ladder of secret-zero strength:

Approach Secret zero is...
Root token in app config The root token. Worst case.
AppRole Secret ID — scoped, expiring, revocable
Response wrapping One-time wrapper — tamper-evident
TLS cert auth A certificate the machine already has
Cloud/K8s identity The platform vouches — no secret at all

Why "still in Vault" isn't circular: Vault doesn't protect secrets by hiding them. It protects them by (1) encrypting at rest, (2) authenticating every request, (3) authorising per path, (4) expiring everything, (5) logging every access, (6) revoking centrally. A .env file does none of these six.

The realistic goal was never "no secrets anywhere." It's: one small, scoped, expiring, revocable, logged, IP-bound credential instead of a plaintext file containing your entire business.

PART 8 — Production Migration

8.1 Pre-Migration Checklist

8.2 Cutover Order — Lowest Risk First

  1. GlueEmail (smallest blast radius)
  2. Textfy dev
  3. Mauzodukani dev → prod
  4. NexGate staging
  5. JikoXpress dev → staging
  6. NexGate prod
  7. JikoXpress prod (customer payments — last, most carefully)

Per service:

1. Create policy in production Vault
2. Create AppRole
3. Write secrets to new paths (keep old paths intact)
4. Update ONE instance to use AppRole
5. Verify health, logs, functionality
6. Roll out to remaining instances
7. Verify again
8. Remove old path access from any broad policy
9. Delete old secret path
10. Run isolation verification

Never delete the old path until the new one is proven in production.

8.3 Post-Migration Verification

PART 9 — Operations

9.1 Backup & DR

vault operator raft snapshot save vault-$(date +%F).snap
vault operator raft snapshot restore vault-2026-08-05.snap

9.2 Monitoring

Prometheus metrics at /v1/sys/metrics?format=prometheus.

Condition Severity
Vault sealed unexpectedly Critical
Root token used Critical
Audit device failure Critical
Permission-denial spike High — attack or misconfiguration
Token creation anomaly High
Lease count growth Medium — leaked unreleased leases
Secret/cert nearing expiry Medium

Feed into the existing Grafana stack.

9.3 Upgrades

Read the changelog for every version between current and target. Snapshot before upgrading, always. Test on the lab instance first. Never skip major versions. Currently 1.20.3 — patch promptly for security fixes.

PART 10 — The Book

10.1 Concept

Working title: Vault From Zero to Production: Secrets Management for Small Teams and Solo Builders

The gap: HashiCorp's docs are reference material assuming an enterprise platform team. Existing books target large organisations with dedicated security staff. Nobody writes for the solo operator or three-person startup running real production on a handful of VPSes.

Unfair advantages:

  1. Living the problem, not theorising — a real incident, a real recovery
  2. African/emerging-market context: modest budgets, self-hosted VPS, no cloud-native luxury
  3. Proven format and finishing discipline — EM/RF trilogy, and XMPP From Zero to Production in progress
  4. Real multi-product stack (Spring Boot, PostgreSQL, RabbitMQ, MinIO, Docker, Traefik, Komodo), not toy examples

Running example: mirror the MaasaiChat approach. A fictional composite — "Zanzi Systems," a Tanzanian SaaS operator with three products and one engineer — whose secrets infrastructure is built across the book. Reader follows the arc from .env chaos to dynamic secrets.

Series consistency: XMPP From Zero to Production → Vault From Zero to Production.

10.2 Structure

Part I — The Problem (Ch. 1–4)

  1. The $8 Lesson: anatomy of a credential leak
  2. What Is a Secret, Really? (path vs field vs value)
  3. Every Way Secrets Leak — and how each is discovered
  4. The Landscape: choosing your tool

Part II — Fundamentals (Ch. 5–13) 5. Installing and Running Vault (Docker, and why IPC_LOCK) 6. Seal, Unseal, and the Bank in Mbeya 7. Shamir's Secret Sharing — the math, and its life beyond Vault 8. Storage Backends 9. Secrets Engines: static vs dynamic 10. Auth Methods: humans and machines 11. Policies: deny by default 12. The /data/ Gotcha and Other Path Traps 13. Tokens, Leases, and TTLs

Part III — Building It (Ch. 14–22) 14. Designing Your Path Hierarchy (why environment sits above component) 15. Writing Policies That Actually Work 16. Testing Denial — the step everyone skips 17. AppRole in Depth: the hotel, and secret zero 18. Integrating Spring Boot 19. Docker and Compose Integration 20. CI/CD with Komodo 21. Human Access, SSO, and MFA on Community Edition 22. Multi-Tenancy: hosting Vault for clients

Part IV — Going Dynamic (Ch. 23–28) 23. Dynamic Database Credentials 24. PKI: running your own CA 25. Transit: encryption as a service 26. SSH Certificates 27. Response Wrapping and Tamper-Evident Delivery 28. Rotation Strategy End-to-End

Part V — Production (Ch. 29–36) 29. Migrating an Existing System Without Downtime 30. High Availability 31. Backup and Disaster Recovery 32. Monitoring and Alerting 33. Upgrades 34. Incident Response 35. Compliance and Vendor Assessments 36. Cost and Capacity Planning

Part VI — Beyond Vault (Ch. 37–40) 37. OpenBao and the Licensing Question 38. SOPS and GitOps Secrets 39. When Vault Is the Wrong Answer 40. Security Culture as a Solo Operator

Appendices

10.3 Workflow

The discipline: write each chapter as you complete the lab step, while the confusion is fresh. The book's value is remembering what was hard — that memory fades within weeks of mastery.

Per-chapter template:

  1. The problem — a concrete scenario from Zanzi Systems
  2. The concept — theory with a diagram
  3. The lab — commands the reader runs
  4. The verification — how to know it worked, including negative tests
  5. What goes wrong — real errors and causes
  6. Production notes — how this differs at scale
  7. Exercises — solutions in the appendix

Cadence: ~4,000 words/chapter × 40 chapters ≈ 160,000 words. Two chapters/week → first draft in ~5 months.

vault-book/
├── manuscript/
│   ├── part-1/ ... part-6/
│   └── appendices/
├── labs/
│   ├── step-01/ ... step-12/
│   └── scripts/
├── diagrams/
├── mistakes.md          ← the goldmine
└── outline.md

Keep mistakes.md open during every lab session. Every error message, every confusion, every "why doesn't this work." That file is the differentiator — it's what the official docs will never contain.

Progress

Lab

Production homework

Book

QBIT SPARK internal engineering document — v2, 6 August 2026