# Vault Mastery Guide — v2 BOOK # Vault Mastery Guide — v2 ### From Root-Token Chaos to Production-Grade Secrets Management **Author:** Kibuti — QBIT SPARK **Environment:** `vault.qbitspark.com` (HashiCorp Vault v1.20.3) **Method:** Local simulation first → master concepts → migrate production last **Status:** Lab Steps 1–8 complete. Steps 9–12 pending. > **v2 changes:** rewritten after completing the hands-on lab. Every analogy, error message, and "why doesn't this work" moment from the actual session is now captured. The `mistakes` sections are real, not hypothetical. --- ## Table of Contents **PART 0 — Why This Exists** **PART 1 — Foundations** **PART 2 — Core Concepts (with the analogies that worked)** **PART 3 — The Lab, Step by Step** **PART 4 — Mistakes File** **PART 5 — Humans, Machines & MFA** **PART 6 — Rotation** **PART 7 — Deployment Architecture** **PART 8 — Production Migration** **PART 9 — Operations** **PART 10 — The Book** --- # PART 0 — Why This Exists ## 0.1 The Incident On 4–5 August 2026, an OpenAI API key (`sk-pro...CrQA`) was used by an unauthorized party to run ~129 automated LLM requests — a physics-grading workload (Feynman amplitude checkpoint scoring, some Chinese-language content) with no relation to any QBIT SPARK product. Roughly **$7.95** consumed before detection. **Detection:** OpenAI's abuse-detection email → manual log review → confirmed by reading request content. **Response:** key revoked immediately. **Leak vector:** still undetermined. Candidates under investigation: - Committed `.env` or hardcoded key in a Git repository - Baked into a published Docker image layer - Exposed Docker daemon API on the VPS - Application-level leak (debug endpoint, path traversal) - Client-side/frontend exposure - `docker inspect` on a container with the key in an env var ## 0.2 What the Incident Actually Exposed The $7.95 was trivial. The structural problems it revealed were not: | Weakness | Risk | |---|---| | Root token used for routine Vault UI access | Full-scope master credential in daily circulation | | Same root token bootstrapping local dev and production | Dev compromise = production compromise | | No per-service scoped policies | Any leak potentially exposes all secrets | | No audit logging | No forensic trail — couldn't answer "was this ever read from Vault?" | | No MFA on human access | Single-factor compromise = total access | | No rotation schedule | Leaked credentials valid indefinitely | | No spending limits on downstream API keys | Financial blast radius unbounded | | Unknown location of unseal key shares | Possible single point of catastrophic failure | **The uncomfortable comparison:** the same class of mistake applied to Vault's root token would expose every credential for JikoXpress, NexGate, Mauzodukani, GlueEmail and Textfy at once — including Snippe PSP payment integration, NexGate JWT signing keys, database passwords, and Vodacom SMPP credentials. ## 0.3 What "Mastery" Means Here Not memorising commands. Being able to: 1. Explain every Vault component in your own words, without notes 2. Design a policy structure for a new service from scratch 3. Predict what will break before changing something 4. Recover from sealed / lost-key / corrupted-storage without panic 5. Teach it — which is why this ends in a book **Rule:** do not proceed past a step until you can explain it aloud with the notes closed. --- # PART 1 — Foundations ## 1.1 What Is a Secret? Any data that grants access or proves identity, and whose disclosure causes harm. | Type | Examples from this stack | |---|---| | Static credentials | PostgreSQL, RabbitMQ, MinIO passwords | | API keys | OpenAI, Snippe PSP, Meta WhatsApp Business API, Vodacom SMPP | | Cryptographic material | NexGate JWT RS256 signing keys, TLS private keys, OMEMO keys | | Connection strings | DSNs embedding host + credentials | | Certificates | Internal CA, client certs for mTLS | | SSH keys | `qbit-spark` access | **The problem:** secrets must be available to apps at runtime, unreadable by anyone else, rotatable without downtime, and every access attributable. `.env` files solve availability. Nothing else. ## 1.2 The Precision That Matters Three words get conflated constantly. Get them separate: ```bash vault kv put qbit/jikoxpress/prod/db username="jx_prod" password="s3cr3t" ``` | Part | Name | Sensitive? | |---|---|---| | `qbit/jikoxpress/prod/db` | **path** — the address | No | | `password` | **field name / key** — the label | No | | `s3cr3t` | **field value** | **← this is the secret** | **The house analogy:** path is the address. Secret is the money inside. Token is the key in your pocket. **Why it matters practically:** Vault's audit log records paths and field names in plaintext but HMACs the values. You can see *that* someone read `qbit/jikoxpress/prod/db` without the log itself leaking the password. **The `secret/` confusion:** in `vault kv put secret/hello`, the word "secret" is a **mount path**, not the data. Dev mode happens to mount a KV engine there. It could be called anything. This is why the lab moves to `qbit/` — the word disappears and the ambiguity goes with it. ## 1.3 What Vault Is > An encrypted key-value store that requires authentication to read from, enforces fine-grained authorisation per request, logs every access, and can *generate* short-lived credentials on demand instead of merely storing long-lived ones. **Four jobs:** 1. **Secure storage** — encrypted at rest; the storage backend never sees plaintext 2. **Access control** — every request authenticated and authorised per path 3. **Audit** — every request and response logged 4. **Dynamic secrets** — credentials with automatic expiry, so a leak dies on its own **What it is not:** a human password manager, a config management system, or magic. A badly configured Vault is worse than none, because it creates false confidence. ## 1.4 The Mental Model ``` ┌─────────────────────────────────┐ │ VAULT SERVER │ You / Service │ ┌──────────┐ │ │ │ │ AUTH │ ← "Prove who │ ├──────────►│ │ METHOD │ you are" │ │ │ └────┬─────┘ │ │ │ │ issues │ │ │ ▼ │ │ │ ┌──────────┐ │ │ │ │ TOKEN │ ← temporary badge │ │ │ └────┬─────┘ │ │ │ │ carries │ │ │ ▼ │ │ │ ┌──────────┐ │ │ │ │ POLICIES │ ← "what doors │ │ │ └────┬─────┘ does it open?" │ │ │ ▼ │ │ │ ┌──────────────────┐ │ │ │ │ SECRETS ENGINES │ ← rooms │ │ │ └────┬─────────────┘ │ │ │ ▼ │ │ │ ┌──────────────────┐ │ │ │ │ STORAGE BACKEND │ ← encrypted │ │ └──────────────────┘ │ │ │ ┌──────────────────┐ │ │ │ │ AUDIT DEVICES │ ← the log │ │ │ └──────────────────┘ │ └───────────┴─────────────────────────────────┘ ``` Every Vault problem you will ever debug is a failure in one of those boxes. ## 1.5 The Alternatives Landscape | | **Vault** | **OpenBao** | **Infisical** | **SOPS** | **AWS SM** | |---|---|---|---|---|---| | Licence | BUSL 1.1 | MPL 2.0 | MIT / paid | MPL 2.0 | Proprietary | | Self-host | Yes | Yes | Yes | N/A | No | | Dynamic secrets | ✅ Extensive | ✅ (fork) | ⚠️ Limited | ❌ | ⚠️ Some | | Auth methods | ✅ 20+ | ✅ 20+ | ⚠️ Fewer | ❌ | IAM only | | Learning curve | Steep | Steep | Gentle | Gentle | Moderate | | PKI / CA | ✅ | ✅ | ❌ | ❌ | ⚠️ Separate | | Encryption-as-a-service | ✅ Transit | ✅ | ❌ | ❌ | ⚠️ KMS | | Audit logging | ✅ Detailed | ✅ | ✅ Basic | ❌ | ✅ CloudTrail | | Ops burden | High | High | Medium | Very low | None | | Cost at this scale | Free | Free | Free tier | Free | ~$0.40/secret/mo | **Categories, for the book:** - **A — env vars / config files.** Trivial, no protection. What you're leaving. - **B — encrypted files in Git** (SOPS, git-crypt, Ansible Vault, Sealed Secrets). Genuinely good for GitOps. Worth pairing with Vault for Komodo deployment configs. - **C — cloud native** (AWS/Azure/GCP). Wrong topology for self-hosted VPS. - **D — dedicated self-hosted** (Vault, OpenBao, Conjur). Your category. - **E — developer SaaS** (Doppler, Infisical Cloud). Better DX, third party holds your secrets, data residency questions for Tanzanian regulatory context. ## 1.6 Why Vault — Decision Record 1. Already deployed at `vault.qbitspark.com`; migration cost zero 2. Multi-product, multi-provider VPS topology — cloud-native can't span it 3. Dynamic secrets are the long-term win 4. PKI engine — internal CA for NexGate mTLS, and relevant to TCRA registrar work 5. Transit engine — encrypt JikoXpress financial data without app-held keys 6. Commercial credibility — "scoped AppRole auth with audit logging" answers a vendor questionnaire far better than ".env files." Already been through Vodacom BSR/OneTrust. 7. Transferable skill — appears in enterprise and East African fintech requirements **Hedge:** track OpenBao. Migration is currently cheap. Revisit annually. --- # PART 2 — Core Concepts ## 2.1 Seal / Unseal — The Bank Imagine a bank in Mbeya. - **The building** = the Vault server process - **The steel vault door** = the seal - **The money inside** = your secrets **When sealed:** the building exists, the door is locked, the money is scrambled. Steal the entire building and you get a locked box full of ciphertext. **Vault starts sealed every time it boots.** Deliberately. **The unseal keys = the combination, cut into 5 pieces** The combination isn't written anywhere. It was split into 5 shares given to 5 managers. **Any 3 can open the door.** One alone can't. Two can't. **Why 3-of-5 and not 5-of-5?** Because if one manager dies, loses their piece, or travels, you'd never open the bank again. 3-of-5 survives losing 2. **The math (Shamir's Secret Sharing, 1979):** two points define a line, three define a parabola. Hide the secret as a point on a curve; hand out other points. With enough points you reconstruct the curve. With fewer, **infinitely many curves fit** — you learn *nothing*, not "most of it." **Auto-unseal** delegates to an external KMS so restarts don't need manual intervention. Convenient; shifts trust to that system. **If you lose enough shares, the data is permanently unrecoverable.** There is no backdoor. ## 2.2 The Precise Chain Common misunderstanding: unseal shares do **not** *generate* the master key. They **reconstruct** it — it already exists, split at initialization. And the master key doesn't decrypt secrets directly: ``` 3 unseal shares ↓ reconstruct Master Key ↓ decrypts Encryption Key ← this is what encrypts your secrets ↓ Vault UNSEALED ``` **Why the extra layer?** So `vault operator rotate` can change the encryption key without redistributing 5 new shares to 5 people. ## 2.3 Root Token vs Unseal Keys — The Rooms Door open ≠ free access. Inside are many rooms — one per service. The **root token** is a master keycard opening every room. Scoped tokens open one. | | Unseal keys | Root token | |---|---|---| | Opens | The vault **door** | The **rooms** inside | | Count | 5 shares, need 3 | One | | Used when | Vault restarts | Every request | | Frequency | Rare | Should be near-never | **You need both.** Shares alone = open door, no keycard. Root alone = keycard, locked door. **Confirmed in the lab:** the root token **cannot** unseal. That separation means two different attacks are required. - Steal the root token → useless against a sealed Vault - Steal the unseal keys → open door, but no token for any room **The reverse — the emergency button:** root *can* seal. Sealing is easy, unsealing is hard, deliberately. ```bash vault operator seal # everything slams shut, instantly ``` If `vault.qbitspark.com` is ever suspected compromised, that's the fire alarm. Costs an outage; the attacker gets ciphertext. Then you need 3 of 5 shares to come back — which is why you must know where they are *before* the emergency. ## 2.4 Why Root Exists At All **Chicken-and-egg.** A fresh Vault has no policies, no auth methods, nothing. Someone must create the first policy — but that needs permission. Root is the bootstrap credential. **It exists to create the system that replaces it.** **Legitimate uses — the complete list:** 1. Initial setup — enable auth methods, mount engines, write first policies 2. Emergency recovery — locked out, broken policy 3. A few root-protected operations — `operator rekey`, some seal management Not on the list: daily work, application requests, UI login. **Where you store it: nowhere.** You destroy it. ``` generate → use → vault token revoke → gone ``` Need it next month? `vault operator generate-root` with 3 shares. **What you protect long-term is the shares**, because those regenerate root. Root itself is disposable. **Where the shares go:** - Split across **different physical locations**, never together - Encrypted USB in a safe; sealed envelope with a trusted person; encrypted offline backup elsewhere - **Never** on the same server as Vault - **Never** all in one folder, password manager, or laptop Solo operator → can't split across people → **split across places**. If the house floods you still need 3. **Correction worth recording:** root is **not** used to create app credentials day-to-day. Root builds the machinery once — enables AppRole, writes policies, creates roles — then is revoked. AppRole issues credentials forever afterward with no root involved. > Root builds the key-cutting machine. Then root leaves. The machine keeps cutting keys. ## 2.5 Storage Backends Vault encrypts *before* writing, so a stolen backend yields ciphertext. | Backend | Use | |---|---| | Integrated (Raft) | Recommended default; HA capable | | Consul | Legacy; more operational complexity | | File | Single node; dev/test | | In-memory | Dev only; vanishes on restart | | PostgreSQL/MySQL | Supported, not recommended for new deployments | **Action item:** determine which backend `vault.qbitspark.com` uses. It determines your backup strategy. **Lab observation:** `Storage Type inmem` means every restart is a brand-new Vault — new cluster ID, new unseal key, no data. On real Vault with persistent storage, the unseal key is generated **once at initialization** and never again. Lose it, lose everything. ## 2.6 Secrets Engines **~20+ types**, not four. Grouped by philosophy: **Static — you store, Vault guards** | Engine | Purpose | |---|---| | `kv-v2` | Versioned key-value. The workhorse. | | `kv-v1` | Unversioned, legacy | | `cubbyhole` | Per-token private storage; dies with the token | **Dynamic — Vault generates and expires** | Engine | Generates | |---|---| | `database` | PostgreSQL, MySQL, MongoDB, Redis users | | `pki` | X.509 certificates | | `ssh` | Signed SSH certificates | | `aws`/`gcp`/`azure` | Temporary cloud IAM credentials | | `rabbitmq`/`consul`/`nomad` | Service-specific credentials | | `kubernetes` | Service account tokens | **Cryptographic services** | Engine | Purpose | |---|---| | `transit` | Encrypt/decrypt without ever holding the key | | `transform` | Format-preserving encryption, masking, tokenization | | `totp` | Generate and validate 2FA codes | **Infrastructure** — `identity/` and `sys/`, always mounted, cannot be disabled. **Adoption order for QBIT SPARK:** `kv-v2` → `database` → `pki` → `transit` → `ssh`. **The insight:** every engine answers *"how do we handle this class of secret?"* with a different strategy. KV says "store it carefully." Database says "don't store it, make a new one each time." Transit says "don't give it to the app at all — send the data here." **Rule of thumb:** if Vault can generate it, don't store it. Store only what you're forced to — third-party API keys, mostly. ## 2.7 Mounts vs Paths vs Tenants Two ways to split, easy to confuse: **Axis 1 — by engine type.** Different engines are different machines. A KV engine can't issue certificates. **Axis 2 — by service.** Possible, usually wrong: ``` ✅ One mount, path hierarchy: ❌ Mount per service: qbit/jikoxpress/prod/db jikoxpress/prod/db qbit/nexgate/prod/minio nexgate/prod/minio ``` Both isolate fine via policy. Separate mounts add config, versioning settings, tuning, backup and audit surface with no extra security. **When separate mounts genuinely help — multi-tenancy.** Relevant here: you hold a retainer with Wiki, deployed FinSpark for a Mbeya microfinance client, built PMK's site. If you host secrets on behalf of clients, mount per tenant: ``` qbit/ ← your products wiki/ ← retainer client kfz/ ← JikoXpress customer ``` **Why (and it's not isolation — policies do that either way):** | Benefit | Why | |---|---| | Independent tuning | Different TTLs, audit settings per client | | **Clean deletion** | Contract ends → `vault secrets disable wiki` → gone | | Accident containment | A wildcard typo in a QBIT policy can't reach a client mount | | Auditability | Log lines obviously client data | | Handover | Export one mount, not a filtered subset | | Vendor assessments | Structural separation is easier to evidence than path prefixes | **Clean deletion is the strongest argument.** With prefixes you hunt for paths and hope you missed nothing. **Caution:** don't pre-mount tenants you don't have. **And note:** `qbit/` *is already* your org boundary. `qbit/qbitspark/...` says it twice. ## 2.8 The Model, Stated Plainly > **Mount** = the machine (what kind of engine) > **Path** = the filing system inside it > **Policy** = who may open which drawer Policies are **standalone objects**, not nested under engines. Confirmed in the UI — Policies is its own menu. A policy merely *mentions* a path as text. You can write a policy for a path that doesn't exist. Delete the mount and the policy sits there harmlessly pointing at nothing. ``` POLICY ─────────► names a path (as text) ▲ │ referenced by name AUTH METHOD ─────► issues tokens carrying that policy │ ▼ TOKEN ─────────► request arrives ↓ Vault checks: does any policy on this token allow this path + this verb? ↓ SECRETS ENGINE serves or refuses ``` **Which is why editing a policy takes effect immediately**, even on tokens issued yesterday. Nothing is baked into the token. ## 2.9 Policies HCL. **Deny by default** — anything not granted is denied. No `deny` rule needed; absence is denial. ```hcl path "qbit/data/jikoxpress/prod/*" { capabilities = ["read"] } path "qbit/metadata/jikoxpress/prod/*" { capabilities = ["list"] } ``` **Capabilities:** | Capability | Verb | Meaning | |---|---|---| | `create` | POST/PUT | Create new | | `read` | GET | Read existing | | `update` | POST/PUT | Modify existing | | `delete` | DELETE | Remove | | `list` | LIST | Enumerate keys (**never values**) | | `sudo` | — | Root-protected paths | | `deny` | — | Explicit denial; **overrides all grants** | **Policies are additive.** A token gets the **union** of all its policies, plus `default`. To opt out of `default`, create the token with `-no-default-policy`. ```bash token_policies="jikoxpress-prod,shared-readonly,monitoring" ``` **Except `deny`, which is absolute:** ```hcl path "qbit/data/jikoxpress/prod/snippe-psp" { capabilities = ["deny"] } ``` **Wildcards:** - `*` — matches everything remaining. **Only works at the end of a path.** - `+` — matches exactly **one** segment. Works in the middle. ```hcl path "qbit/data/+/dev/*" { ... } # every service's dev path "identity/oidc/provider/+/authorize" { ... } # from the default policy ``` **Design rule:** write **small, single-purpose policies** and compose them — `jikoxpress-prod-read`, `jikoxpress-prod-write`, `shared-read` — rather than one giant policy per service. **Reading the built-in `default` policy taught four things:** 1. Everything is self-scoped (`lookup-self`, `renew-self`, `revoke-self`) — no token can look up another 2. `{{identity.entity.id}}` templating resolves per-token at request time — one document, per-caller permissions 3. `+` demonstrated in the wild 4. `cubbyhole/*` full CRUD — every token gets private storage nobody else can read, **not even root**. This is the mechanism behind response wrapping. ## 2.10 The KV v2 Path Gotcha **The single most common Vault mistake.** The CLI lies to you — helpfully. **What you type:** ```bash vault kv get qbit/jikoxpress/prod/postgres ``` **What goes over HTTP:** ``` GET /v1/qbit/data/jikoxpress/prod/postgres ``` Vault tells you this in every output: `Secret Path: qbit/data/jikoxpress/prod/postgres`. **Why the extra segment** — KV v2 splits operations into sub-paths: | Real API path | Purpose | |---|---| | `qbit/data/...` | the values | | `qbit/metadata/...` | version history, `max_versions`, delete-all | | `qbit/delete/...` | soft-delete | | `qbit/undelete/...` | restore | | `qbit/destroy/...` | permanent wipe | All five appear in normal CLI output once you look: ``` Success! Data written to: qbit/undelete/jikoxpress/prod/postgres Success! Data written to: qbit/destroy/jikoxpress/prod/postgres Metadata Path: qbit/metadata/jikoxpress/prod/postgres ``` **The `kv` command inserts the right segment for you. Policies have no wrapper.** Write the obvious-looking thing and you get `permission denied` with no hint why: ```hcl path "qbit/jikoxpress/prod/*" { capabilities = ["read"] } # ← WRONG ``` **The useful consequence** — separated paths let you grant operations independently: ```hcl path "qbit/data/jikoxpress/prod/*" { capabilities = ["read"] } path "qbit/metadata/jikoxpress/prod/*" { capabilities = ["read", "list"] } path "qbit/destroy/jikoxpress/prod/*" { capabilities = ["deny"] } ``` An app that reads secrets has no business destroying them. **KV v1 has none of this** — one path, no `/data/`. Which is exactly why people who learned v1 get caught. ## 2.11 Path Design Is Policy Design **`service/environment` above component.** Not negotiable. **Right — `qbit/jikoxpress/prod/*`:** ```hcl path "qbit/data/jikoxpress/prod/*" { capabilities = ["read"] } ``` One rule. Covers every component, including ones added next year, automatically. **Wrong — `qbit/jikoxpress/db/prod`:** ```hcl path "qbit/data/jikoxpress/db/prod" { ... } path "qbit/data/jikoxpress/snippe/prod" { ... } path "qbit/data/jikoxpress/redis/prod" { ... } ... ``` One rule per component, forever. **The dangerous failure mode:** six months later you add a component, forget the policy line, and the app breaks at 2am. Under pressure you "fix" it with `qbit/data/jikoxpress/*` — which now grants **dev and staging too**. The isolation is gone and nothing tells you. **The rule:** put the thing you cut permissions along **highest** in the path. **Adopted structure:** ``` qbit/// qbit/jikoxpress/prod/postgres qbit/jikoxpress/prod/snippe-psp qbit/nexgate/staging/minio qbit/nexgate/prod/jwt-signing qbit/mauzodukani/prod/postgres qbit/glueemail/prod/mailcow qbit/textfy/prod/vodacom-smpp qbit/shared/prod/openai ← belongs to no single product qbit/shared/prod/cloudflare ``` **Directories aren't real.** There's no `mkdir`. Write to a deep path and every intermediate level springs into existence, inferred. ## 2.12 Auth Methods | Method | For | Notes | |---|---|---| | Token | Bootstrap, break-glass | Always enabled, can't be disabled | | **AppRole** | Machines | Role ID + Secret ID. Primary machine auth. | | Userpass | Humans | Simple; pair with MFA | | OIDC / JWT | Humans via SSO, CI/CD | Keycloak, Authentik, Google Workspace | | LDAP | Humans via directory | If you run one | | TLS certificates | Machines with mTLS | Strong, more setup | | Kubernetes | Pods | If you move to K8s | | GitHub | Humans / CI | Ties access to org membership | **Vault does have users** — `userpass` or `oidc`. Same machinery as AppRole, different door. ## 2.13 AppRole — The Hotel The analogy that worked: - **Policy** = the rule sheet on the wall. *"This kind of guest may open rooms 301–305."* Not a key. - **AppRole** = the front desk. Checks ID, hands out keycards. - **Role ID** = your booking name. *"I'm JikoXpress."* Not secret. - **Secret ID** = your booking reference. *"Here's proof."* Secret. - **Token** = the keycard. Expires. **This is what you actually carry and use.** **Two ways to get a keycard:** ``` Cheating (lab Step 6): You (root) → "make me a keycard" → token Real (lab Step 8): App → role_id + secret_id → front desk checks → token ``` Same keycard. Different route. An app can't use the first — it would need root, which is the problem being solved. **Who has what:** | Thing | Lives where | Secret? | Expires? | |---|---|---|---| | Policy | In Vault | No | No | | Role ID | App config, Git | Not really | No | | Secret ID | Delivered to app | **Yes** | Yes | | Token | App memory | **Yes** | Yes | **Order of operations:** ``` ONCE, by admin: 1. write policy 2. create approle role, attach policy by name EVERY APP START: 3. app sends role_id + secret_id 4. Vault returns token 5. app reads secrets EVERY ~45 MIN: 6. app renews token AFTER token_max_ttl: 7. back to step 3 ``` Steps 3–7 are what "automatic" means. Spring Cloud Vault or Vault Agent does them. You write config, not code. **Why split into two halves?** Different security properties, different delivery paths. Neither alone works. Role ID can sit in Git; Secret ID is delivered at runtime and expires. **The thing in your repo isn't sufficient to authenticate.** **The Secret ID is not stored readably.** `vault write -f .../secret-id` is a **write** — it generates one, stores a hash, returns plaintext **once**. No read gets it back. `vault list .../secret-id` shows *accessors* (for revocation), never the value. ## 2.14 Tokens | Type | Behaviour | |---|---| | Service | Default; server-side; renewable, revocable, supports children | | Batch | Lightweight encrypted blobs; not stored; can't renew | | Root | `root` policy; unlimited; **no TTL** | | Periodic | Renews indefinitely within the period | | Orphan | No parent; survives parent revocation | Tokens form a tree — revoking a parent revokes children. **Root token lookup, annotated** (from the lab): ``` ttl 0s ← NOT expired. NO EXPIRY. expire_time ← never policies [root] ← everything orphan true ← can't be killed via a parent num_uses 0 ← unlimited id root ← guessable (dev mode) ``` **AppRole token lookup, same fields:** ``` ttl 56m16s ← counting down expire_time 2026-08-06T11:02:24Z ← real policies [default jikoxpress-prod] ← scoped path auth/approle/login ← how it was obtained meta map[role_name:jikoxpress-prod] ← attribution entity_id bcb55ff4-... ← identity entity ``` **`token_meta_role_name` is the audit win.** Not "root did it" — "jikoxpress-prod did it." If NexGate's role ever reads a JikoXpress secret, you see it. ## 2.15 Leases, TTLs, and the Renewal Trap - **TTL** — how long valid - **Max TTL** — hard ceiling; renewal cannot exceed it - **Renewal** — extending before expiry - **Revocation** — killing it early Precedence: system default → mount tuning → role config → request-time. Each may only *shorten*. **`768h` = 32 days = the system default.** A mount showing 768h isn't "configured" — it's "nobody set this." **The trap, discovered in the lab:** every expiry you add is a renewal mechanism you must build. `token_ttl=1h` → solved by renewal (Spring Cloud Vault does it in a background thread). `token_max_ttl=4h` → renewal can't exceed it; the app must be able to **log in again**. `secret_id_ttl=24h` → **this is the one that kills services**. When it expires, login itself fails. Renewal won't save you. **Four ways to handle Secret ID expiry:** 1. **Non-expiring** — `secret_id_ttl=0`, `secret_id_num_uses=0`. Simplest, weakest. Still far better than root. 2. **Vault Agent** — sidecar handles login, renewal, re-auth. Writes a token to a file the app reads. The standard answer. 3. **Long TTL, rotated on deploy** — `720h`, with Komodo issuing fresh on each deploy. 4. **Response-wrapped at container start** — most secure, most moving parts. **Recommended start:** `secret_id_ttl=0`. Get the flow working before fighting two problems at once. ## 2.16 Attacker Analysis **Can a thief renew a stolen token?** Yes. Vault can't distinguish them. **But `token_max_ttl` is the wall.** Renewal extends, never resets; max TTL is measured from **creation**. ``` token_ttl=1h, token_max_ttl=4h Steal at hour 0 → renew, renew, renew → at hour 4: DEAD ``` `token_max_ttl` is your actual worst-case exposure window. `token_ttl` is just how often renewal happens. **Hierarchy of theft:** | Stolen | Exposure | |---|---| | Token only | Until `token_max_ttl` — hours | | Token + Secret ID | Until `secret_id_ttl` — days | | Role ID + Secret ID | Until you rotate | | **Root token** | **Forever, everything, silently** | **Which is why the two halves must not live together.** If `docker inspect` yields both, the design bought you nothing. **Where each realistically leaks from:** | Role ID | Secret ID | |---|---| | Committed compose/Komodo config | File on server | | Baked into an image (`ENV`) | Env var → `docker inspect` | | `.env` on the server | CI/CD secret store | | Kubernetes ConfigMap | `/proc//environ` | | | Shell history | | | **App logs, if the app logs its config at startup** | **The attack, concretely:** ```bash docker inspect jikoxpress | grep -i vault # VAULT_ROLE_ID=8c3f2a1b-... # VAULT_SECRET_ID=d4e5f6a7-... ``` Both halves, one command. **Free hardening not yet used — CIDR binding:** ```bash vault write auth/approle/role/jikoxpress-prod \ secret_id_bound_cidrs="161.97.163.158/32" \ token_bound_cidrs="161.97.163.158/32" ``` Stolen credentials become useless anywhere but `qbit-spark`. The attacker would have to run the attack *from your server*. **Response wrapping — the strongest link:** ```bash vault write -f -wrap-ttl=60s auth/approle/role/jikoxpress-prod/secret-id ``` Returns a wrapper token, not the Secret ID. Single use. If someone intercepts and unwraps first, **your app's unwrap fails** — and that failure is your intrusion alarm. Not just secrecy: **detection**. **Revocation:** ```bash vault token revoke vault write auth/approle/role/X/secret-id-accessor/destroy secret_id_accessor="" vault write -f auth/approle/role/X/secret-id # rotate: kills existing vault lease revoke -prefix auth/approle/ ``` **The honest summary:** you cannot prevent a stolen token from being used. You can only bound how long it works and detect that it's happening. Short max TTLs and audit logs are the two levers. ## 2.17 delete / undelete / destroy / revoke Two different axes. Constantly conflated. **About the data (KV):** | Command | Effect | Reversible | |---|---|---| | `kv delete` | Marks a version deleted; data still on disk | ✅ `undelete` | | `kv undelete` | Clears the mark | — | | `kv destroy` | Wipes that version's data; metadata tombstone remains | ❌ Never | | `kv metadata delete` | Nukes the whole secret, all versions | ❌ Never | Recycle bin → shredder → burning the filing cabinet. **About access (tokens/leases):** ```bash vault token revoke vault lease revoke vault lease revoke -prefix database/ ``` Revoking a **dynamic secret lease** doesn't just invalidate a record — Vault reaches into the backend and runs `DROP ROLE`. The user genuinely ceases to exist. > **delete/destroy** = removing the *secret* > **revoke** = removing *access* **Applied to the incident:** the OpenAI key leaked. `kv destroy` removes it from Vault — **the stolen key still works.** `vault token revoke` is irrelevant — the attacker never had a Vault token. Only revoking at OpenAI's dashboard stopped it. **That's the limit of static secrets.** Vault stores and guards them; it cannot revoke what a third party issued. **The contrast:** a `database` engine credential → `vault lease revoke` drops the PostgreSQL user directly. Vault issued it, so Vault can kill it. **That's the difference between storing secrets and owning them.** **KV v2 metadata settings worth using:** ```bash vault kv metadata put -max-versions=10 qbit/jikoxpress/prod/postgres ``` - `max_versions 0` = unlimited history. Every rotation accumulates forever; old passwords stay readable. - `delete_version_after 0s` = never auto-delete. Set `720h` to self-expire. - `cas_required false` = Check-And-Set off. Turn on so writes must state which version they replace — prevents two processes clobbering each other during automated rotation. ## 2.18 Audit Devices Log every request and response. **Vault refuses to operate if all enabled audit devices fail to write** — fail-closed by design: no audit trail, no service. ```bash vault audit enable file file_path=/vault/logs/audit.log ``` Sensitive **values** are HMAC'd; paths and field names are plaintext. Safe to ship to Grafana/Loki. You can compute the HMAC of a known value to search for it — forensics without exposure. **This is what was missing during the incident.** With audit logging you could answer: was the OpenAI key ever read from Vault, by which role, from which IP, when? ## 2.19 Error Codes — Read Them Precisely | Error | Meaning | |---|---| | `403 permission denied` | Right verb, path exists, policy says no | | `404` / `No value found` | Right verb, nothing there | | `405 unsupported operation` | **Wrong verb** — path exists but doesn't accept this method | In Step 5 you'll stare at 403s wondering whether the policy or the path is wrong. Knowing these apart is the difference between five minutes and an hour. **The debugging tool:** ```bash vault token capabilities qbit/data/jikoxpress/prod/postgres # → read vault token capabilities qbit/data/nexgate/prod/postgres # → deny ``` Ask Vault directly instead of guessing. **Deliberate ambiguity:** the UI says *"You do not have the required permissions **or** the directory does not exist."* It refuses to say which — otherwise an attacker maps your infrastructure by probing. Same reason good login forms say "invalid username or password." --- # PART 3 — The Lab **Ground rules:** nothing touches production. Don't proceed until the ✅ check passes. After each step, close your notes and explain it aloud. ## Setup — Docker, not apt ```yaml # ~/vault-lab/docker-compose.yml services: vault: image: hashicorp/vault:1.20.3 container_name: vault-lab ports: - "8200:8200" cap_add: - IPC_LOCK environment: VAULT_DEV_ROOT_TOKEN_ID: "root" VAULT_DEV_LISTEN_ADDRESS: "0.0.0.0:8200" VAULT_ADDR: "http://127.0.0.1:8200" VAULT_TOKEN: "root" command: server -dev volumes: - ./policies:/policies ``` **Every line explained:** **`cap_add: IPC_LOCK`** — grants the `mlock()` capability. Vault holds decrypted secrets and the master key in RAM; if the kernel swaps that to disk, **secrets get written to the swap file in plaintext**. `mlock()` prevents it. Barely matters in dev mode, but correct from the start so the pattern carries. **`VAULT_DEV_ROOT_TOKEN_ID: "root"`** — overrides the random dev token. Convenience only. Never anywhere real. **`VAULT_DEV_LISTEN_ADDRESS: "0.0.0.0:8200"`** — which interface Vault binds to *inside* the container. `127.0.0.1` = container-only; Docker's port mapping would have nothing to forward to. > **Security note:** `0.0.0.0` inside + `ports:` mapping outside = exposed on every host interface. On a laptop behind a router, fine. **On a VPS with no firewall, this exact pattern is how services get accidentally exposed to the internet.** Possibly relevant to how the OpenAI key leaked. **`command: server -dev`** — dev mode means: in-memory storage (**all data lost on restart**), auto-initialized and auto-unsealed, TLS disabled, KV v2 pre-mounted at `secret/`, root token printed to logs. Dev mode removes every barrier to learning and every security guarantee. **`VAULT_ADDR` is not optional.** Without it the CLI defaults to `https://127.0.0.1:8200`, tries a TLS handshake against a plain-HTTP dev server, and fails with `http: server gave HTTP response to HTTPS client`. Optional alias: ```bash alias v='docker compose exec vault vault' ``` --- ### Step 1 — Run it ```bash docker compose up -d sleep 3 # avoid the boot race docker compose exec vault vault status ``` ✅ `Sealed false`. **Read the boot logs.** They print an **unseal key and a root token in plaintext to stdout**. In production that combination in a log file is game over. **The observation:** the unseal key and cluster ID **change on every restart**. `inmem` = brand-new Vault each time. On persistent storage, the unseal key is generated **once at initialization**, never again. **Question:** why would auto-unseal-with-the-key-in-the-logs be unacceptable on `vault.qbitspark.com`? **Answer:** both protections are nullified at once. Already unsealed → encryption gives you nothing, secrets are decrypted in memory now. Key in the logs → even if sealed, the key to reopen it is in `docker compose logs`. An attacker with log read access needs neither to break encryption nor gather shares. > **The trap:** Vault's protection is only as good as the separation between the encrypted data and the keys that decrypt it. Dev mode puts them together. So does printing unseal keys to logs. So does storing shares on the same server. **So does one person holding all five shares in one folder.** --- ### Step 2 — Tokens and your first secret ```bash docker compose exec vault vault token lookup docker compose exec vault vault kv put secret/hello message="my first secret" owner="kibuti" docker compose exec vault vault kv get secret/hello docker compose exec vault vault kv get -field=message secret/hello docker compose exec vault vault kv get -format=json secret/hello ``` ✅ Explain: what is the token, what is the secret, which is more dangerous to leak? **The answer, refined:** it depends on scope. A *narrow* token is less dangerous than a big secret. A *root* token is worse than any single secret, because it's the key to all of them. | Leaked | Damage | |---|---| | One secret | That one system. Bad, bounded. | | Scoped token (1h) | What that role can read, for an hour | | **Root token** | **Everything, forever, silently** | **Which is why the fix isn't "protect tokens harder" — it's make tokens small and short-lived so leaking one barely matters.** **`-field` output is designed for scripting:** ```bash DB_PASS=$(vault kv get -field=password qbit/jikoxpress/prod/db) ``` --- ### Step 3 — Mount your own engine ```bash docker compose exec vault vault secrets list # baseline first docker compose exec vault vault secrets enable -path=qbit kv-v2 docker compose exec vault vault secrets tune -description="QBIT SPARK application secrets" qbit/ ``` **Baseline on a dev server:** `cubbyhole/`, `identity/`, `secret/`, `sys/`. On **real** Vault, `secret/` wouldn't be there — three system mounts and nothing else. Everything useful, you mount yourself. **Explore the engine zoo** (mount, look, remove): ```bash docker compose exec vault vault secrets enable database docker compose exec vault vault secrets enable pki docker compose exec vault vault secrets enable transit docker compose exec vault vault secrets enable ssh docker compose exec vault vault secrets enable totp docker compose exec vault vault secrets list -detailed ``` **Four things in the `-detailed` table:** 1. **`Options` = `map[version:2]`** — the *only* place the table reveals KV **v2** vs v1. Both show `Plugin kv`. 2. **`Default TTL`/`Max TTL` = `system`** — inherit the 768h default. On `database/` these become the real ceiling. 3. **`Seal Wrap true` — only on `sys/`.** Extra encryption layer using the seal mechanism, applied to Vault's own control plane. 4. **`Running Version`** — KV shows `v0.24.0+builtin` while others show `v1.20.3+builtin.vault`. The KV engine versions **independently of Vault core** — which is why plugin upgrades and Vault upgrades are different operations. **Dynamic engines are inert until configured:** ```bash docker compose exec vault vault list database/config # No value found docker compose exec vault vault list transit/keys ``` KV works the instant you mount it. Dynamic engines need to know *what system to talk to* and *what to create there*. **Cleanup — and note the danger:** ```bash docker compose exec vault vault secrets disable database ``` `disable` **destroys all data under that path.** No confirmation prompt. On real Vault, `vault secrets disable pki` deletes your entire certificate authority. **Then build the tree:** ```bash docker compose exec vault vault kv put qbit/jikoxpress/prod/postgres username="jx" password="x" docker compose exec vault vault kv put qbit/jikoxpress/prod/snippe-psp api_key="x" merchant_id="x" docker compose exec vault vault kv put qbit/jikoxpress/staging/postgres username="jx" password="x" docker compose exec vault vault kv put qbit/nexgate/prod/postgres username="ng" password="x" docker compose exec vault vault kv put qbit/nexgate/prod/rabbitmq username="ng" password="x" docker compose exec vault vault kv put qbit/nexgate/prod/jwt-signing private_key="x" public_key="x" docker compose exec vault vault kv put qbit/nexgate/staging/minio access_key="x" secret_key="x" docker compose exec vault vault kv put qbit/mauzodukani/prod/postgres username="md" password="x" docker compose exec vault vault kv put qbit/glueemail/prod/mailcow admin_user="x" admin_pass="x" docker compose exec vault vault kv put qbit/textfy/prod/vodacom-smpp system_id="x" password="x" docker compose exec vault vault kv put qbit/shared/prod/openai api_key="sk-placeholder" docker compose exec vault vault kv put qbit/shared/prod/cloudflare api_token="x" ``` > **All values are `x` deliberately.** Building the habit of never typing a real credential into a lab is worth more than the lab itself. **Walk the tree:** ```bash docker compose exec vault vault kv list qbit/ docker compose exec vault vault kv list qbit/jikoxpress/ docker compose exec vault vault kv list qbit/jikoxpress/prod/ ``` **Trailing slashes mark directories:** ``` jikoxpress/ ← has children prod/ ← has children postgres ← no slash: an actual secret ``` `list` **never shows values.** Separate capability from `read` — you can let a junior dev see *what* exists without reading contents. --- ### Step 4 — Versioning ```bash docker compose exec vault vault kv put qbit/jikoxpress/prod/postgres username="jx" password="NEW-v2" docker compose exec vault vault kv get qbit/jikoxpress/prod/postgres docker compose exec vault vault kv get -version=1 qbit/jikoxpress/prod/postgres docker compose exec vault vault kv metadata get qbit/jikoxpress/prod/postgres docker compose exec vault vault kv delete qbit/jikoxpress/prod/postgres docker compose exec vault vault kv undelete -versions=2 qbit/jikoxpress/prod/postgres docker compose exec vault vault kv destroy -versions=1 qbit/jikoxpress/prod/postgres ``` **Gotcha found live:** `kv get` without `-version` always fetches the **latest**. Undeleting v2 while v3 is deleted looks like the undelete failed. Use `kv metadata get` — it shows every version and its state at once. ✅ Metadata should show: v1 `destroyed true`, v2 clean, v3 `deletion_time` set. **Why it matters for rotation:** ``` 1. Write new version ← old still live 2. Restart services 3. Verify healthy 4. THEN revoke at provider 5. Optionally destroy old version ``` If step 3 fails, `kv get -version=` recovers instantly. **The trap:** `delete` looks destructive but isn't. People think a secret is gone when it's fully recoverable by anyone with `undelete`. To actually remove a leaked credential from Vault, it's `destroy`. --- ### Step 5 — First policy ```bash mkdir -p ~/vault-lab/policies cat > ~/vault-lab/policies/jikoxpress-prod.hcl <<'EOF' path "qbit/data/jikoxpress/prod/*" { capabilities = ["read"] } path "qbit/metadata/jikoxpress/prod/*" { capabilities = ["list"] } EOF docker compose exec vault vault policy write jikoxpress-prod /policies/jikoxpress-prod.hcl docker compose exec vault vault policy read jikoxpress-prod docker compose exec vault vault policy list docker compose exec vault vault policy read default # read this one properly ``` **Anatomy of the command:** ``` vault policy write jikoxpress-prod /policies/jikoxpress-prod.hcl │ │ │ └─ file, INSIDE the container └─ policy name (unrelated to the filename) ``` The file and the policy name are independent. Keep them matching by convention, not requirement. **`/policies/` is the container path.** Your volume maps `./policies` → `/policies`. **Once written, the file no longer matters.** Vault stores the policy internally; delete the `.hcl` and it keeps working. The file is source, not runtime — which is why policies belong in Git, giving you version history and review while Vault holds current state. --- ### Step 6 — Prove the denial ⭐ **The most important step.** A policy tested only for success is untested — you might still be root. ```bash docker compose exec vault vault token create -policy=jikoxpress-prod -ttl=30m TOK="" docker compose exec -e VAULT_TOKEN=$TOK vault vault kv get qbit/jikoxpress/prod/postgres # ALLOW docker compose exec -e VAULT_TOKEN=$TOK vault vault kv get qbit/jikoxpress/staging/postgres # DENY docker compose exec -e VAULT_TOKEN=$TOK vault vault kv get qbit/nexgate/prod/postgres # DENY docker compose exec -e VAULT_TOKEN=$TOK vault vault kv get qbit/shared/prod/openai # DENY docker compose exec -e VAULT_TOKEN=$TOK vault vault kv put qbit/jikoxpress/prod/postgres password="hacked" # DENY ``` ✅ **Four `403 permission denied`.** Those errors are the proof. **The last denial is the subtle one:** `PUT` blocked on a path the token *can* read. Same path, different verb, different answer. Least privilege at the **operation** level, not just the path level. **Note:** `token_policies ["default" "jikoxpress-prod"]` — additive, as expected. --- ### Step 7 — Enable AppRole ```bash docker compose exec vault vault auth enable approle docker compose exec vault vault write auth/approle/role/jikoxpress-prod \ token_policies="jikoxpress-prod" \ token_ttl=1h \ token_max_ttl=4h \ secret_id_ttl=0 docker compose exec vault vault read auth/approle/role/jikoxpress-prod ``` **Note `token_policies="jikoxpress-prod"` is just a name.** The AppRole never mentions paths — the policy owns those. To change what a service can access, edit the policy; existing tokens pick it up on their next request. **Two empty fields worth filling in production:** `secret_id_bound_cidrs` and `token_bound_cidrs`. > **Lab warning:** don't set CIDRs to your production IP in the lab — your requests come from inside a Docker container and you'll lock yourself out. --- ### Step 8 — Log in as a service ⭐ ```bash RID=$(docker compose exec -T vault vault read -field=role_id auth/approle/role/jikoxpress-prod/role-id) SID=$(docker compose exec -T vault vault write -f -field=secret_id auth/approle/role/jikoxpress-prod/secret-id) docker compose exec -e VAULT_TOKEN= -T vault vault write auth/approle/login \ role_id="$RID" secret_id="$SID" ``` **`-e VAULT_TOKEN=` blanks the root token** — proving login needs no prior credential. ```bash APPTOK="" docker compose exec -e VAULT_TOKEN=$APPTOK vault vault token lookup docker compose exec -e VAULT_TOKEN=$APPTOK vault vault kv get qbit/jikoxpress/prod/postgres # ALLOW docker compose exec -e VAULT_TOKEN=$APPTOK vault vault kv get qbit/nexgate/prod/postgres # DENY docker compose exec -e VAULT_TOKEN=$APPTOK vault vault token renew # note: positional, not -self ``` ✅ A production secret retrieved with **zero root token involvement**. **Where does the token come from at all?** The CLI resolution order: 1. `-address` / explicit flag 2. `VAULT_TOKEN` environment variable ← what the compose file sets 3. `~/.vault-token` (written by `vault login`) **Being inside the container grants nothing.** No implicit auth, no localhost exemption, no container trust. Prove it: ```bash docker compose exec -e VAULT_TOKEN= vault vault kv get qbit/jikoxpress/prod/db # denied ``` **Vault is an HTTP API with a token header.** The CLI is a wrapper: ```bash curl -H "X-Vault-Token: root" http://127.0.0.1:8200/v1/qbit/data/jikoxpress/prod/db ``` > **The uncomfortable parallel:** "what token is being used?" — the answer was root, because it sat in a config file you edited and stopped thinking about. That's the shape of the production problem. --- ### The UI Experiment ⭐ Open `http://127.0.0.1:8200`. **Sidebar structure — three separate things, three separate menus:** - **Secrets Engines** — where secrets live - **Access → Auth Methods** — approle, token, userpass - **Policies** — standalone **Then: sign out, sign back in with the AppRole token.** **What happens:** the interface collapses. No Policies. No Access. No Seal Vault. Instead of a browsable tree, a box saying *"Type the path of the secret you want to view."* **Why:** the policy grants `list` on `qbit/metadata/jikoxpress/prod/*` but **not** on `qbit/` itself. The UI can't enumerate the top level, so it falls back to asking. - Type `jikoxpress/prod/` → lists `postgres`, `snippe-psp` - Type `nexgate/prod/` → *"You do not have the required permissions or the directory does not exist."* **This is what a compromised JikoXpress service looks like from the attacker's side.** Two secrets. Not NexGate's JWT keys, not Cloudflare, not Vodacom. Compare to a stolen root token: everything, in a navigable tree. **A design decision it surfaces:** apps don't browse — they fetch known paths, so read-only on exact paths is right. **Humans** with scoped policies find the inability to navigate genuinely annoying, so add list grants deliberately: ```hcl path "qbit/metadata" { capabilities = ["list"] } path "qbit/metadata/jikoxpress" { capabilities = ["list"] } ``` **Also observed:** a yellow banner — *"We've stopped auto-renewing your token due to inactivity. It will expire in about 1 hour."* with a **Renew token** button. The UI had been renewing in the background all along. That's `vault token renew` with a nicer face, and exactly what Spring Cloud Vault does in a thread. **And:** `Create secret +` still renders even without `create` capability. Clicking fails with 403. **UI affordances aren't permissions.** --- ### Seed script — because `inmem` wipes on restart ```bash cat > ~/vault-lab/seed.sh <<'EOF' #!/usr/bin/env bash set -euo pipefail V="docker compose exec -T vault vault" $V secrets enable -path=qbit kv-v2 2>/dev/null || true $V secrets tune -description="QBIT SPARK application secrets" qbit/ $V auth enable approle 2>/dev/null || true $V kv put qbit/jikoxpress/prod/postgres username="jx" password="x" $V kv put qbit/jikoxpress/prod/snippe-psp api_key="x" merchant_id="x" $V kv put qbit/jikoxpress/staging/postgres username="jx" password="x" $V kv put qbit/nexgate/prod/postgres username="ng" password="x" $V kv put qbit/nexgate/prod/jwt-signing private_key="x" public_key="x" $V kv put qbit/nexgate/staging/minio access_key="x" secret_key="x" $V kv put qbit/mauzodukani/prod/postgres username="md" password="x" $V kv put qbit/glueemail/prod/mailcow admin_user="x" admin_pass="x" $V kv put qbit/textfy/prod/vodacom-smpp system_id="x" password="x" $V kv put qbit/shared/prod/openai api_key="sk-placeholder" $V policy write jikoxpress-prod /policies/jikoxpress-prod.hcl $V write auth/approle/role/jikoxpress-prod \ token_policies="jikoxpress-prod" token_ttl=1h token_max_ttl=4h secret_id_ttl=0 echo "✓ seeded" EOF chmod +x ~/vault-lab/seed.sh ``` Wipe and rebuild freely: ```bash docker compose down && docker compose up -d && sleep 3 && ./seed.sh ``` --- ### Step 9 — All services (pending) ```bash #!/usr/bin/env bash set -euo pipefail V="docker compose exec -T vault vault" declare -A SERVICES=( [jikoxpress]="dev staging prod" [nexgate]="dev staging prod" [mauzodukani]="dev prod" [glueemail]="prod" [textfy]="dev prod" [shared]="prod" ) for svc in "${!SERVICES[@]}"; do for env in ${SERVICES[$svc]}; do name="${svc}-${env}" cat > "policies/${name}.hcl" </dev/null 2>&1 then result="ALLOW"; else result="DENY"; fi if [ "$result" = "$expect" ]; then echo " ✓ ${role} → ${path} = ${result}"; PASS=$((PASS+1)) else echo " ✗ ${role} → ${path} = ${result} (expected ${expect})"; FAIL=$((FAIL+1)) fi } echo "Positive (must ALLOW):" check jikoxpress-prod qbit/jikoxpress/prod/postgres ALLOW check nexgate-staging qbit/nexgate/staging/minio ALLOW echo "Negative (must DENY):" check jikoxpress-prod qbit/nexgate/prod/jwt-signing DENY check jikoxpress-prod qbit/jikoxpress/dev/postgres DENY check nexgate-staging qbit/nexgate/prod/jwt-signing DENY check textfy-prod qbit/jikoxpress/prod/postgres DENY check jikoxpress-prod qbit/shared/prod/openai DENY echo; echo "Passed: $PASS Failed: $FAIL" [ "$FAIL" -eq 0 ] || exit 1 ``` **Keep this script forever.** Run after every policy change, in the lab and in production. ### Step 11 — Audit logging (pending) ```bash docker compose exec vault vault audit enable file file_path=/vault/logs/audit.log docker compose exec vault vault audit list -detailed docker compose exec vault vault kv get qbit/jikoxpress/prod/postgres docker compose exec vault sh -c 'tail -n 5 /vault/logs/audit.log' ``` ✅ Find your request. Confirm the **value** is HMAC'd but the **path** is plaintext. ### Step 12 — Dynamic database credentials (pending) ⭐ ```bash docker run -d --name vault-lab-pg \ -e POSTGRES_PASSWORD=rootpass -e POSTGRES_DB=jikoxpress \ -p 5433:5432 postgres:16 docker compose exec vault vault secrets enable database docker compose exec vault vault write database/config/jikoxpress-pg \ plugin_name=postgresql-database-plugin \ allowed_roles="jikoxpress-app" \ connection_url="postgresql://{{username}}:{{password}}@host.docker.internal:5433/jikoxpress?sslmode=disable" \ username="postgres" password="rootpass" docker compose exec vault vault write database/roles/jikoxpress-app \ db_name=jikoxpress-pg \ creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; \ GRANT SELECT, INSERT, UPDATE, DELETE ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \ default_ttl="1h" max_ttl="24h" docker compose exec vault vault read database/creds/jikoxpress-app docker compose exec vault vault read database/creds/jikoxpress-app # different every time ``` **The contrast that makes it click:** ``` Static (KV): vault kv get qbit/jikoxpress/prod/db → username: jx_prod, password: s3cr3t → same answer, every time, forever Dynamic (database): vault read database/creds/jikoxpress-app → v-approle-jikoxpr-x7Kd92 / A1a-8sKdm3nQp (lease 1h) vault read database/creds/jikoxpress-app → v-approle-jikoxpr-mN4vB1 / Z9z-2pLwq7Rt (lease 1h) ``` Vault runs `CREATE ROLE` live, then `DROP ROLE` an hour later. Verify with `\du` in psql — the users genuinely exist, then genuinely don't. ✅ Generate twice, confirm both in `\du`, `vault lease revoke` one, confirm it's dropped. **Steal one of those and you've stolen something that dies on its own.** That's the whole game. --- # PART 4 — Mistakes File *The book's differentiator. Real errors from the real session.* ### 4.1 The invisible leading space **What happened:** a `\` line continuation with a trailing space made the shell pass ` mesage=...` as one argument. Vault stored the key literally as `" mesage"`. **Why it was confusing:** `kv get` displayed it as `mesage` — the table view strips the visual space. `-field=mesage` then said "not present in secret." **Diagnosis:** ```bash vault kv get -format=json secret/hello # "data": { " mesage": "my first secret" } ``` **Lessons:** 1. When a field "exists but isn't found," go straight to `-format=json`. **The pretty output lies about whitespace.** 2. Vault has **no schema and no validation**. It will happily store `PASSWROD`, `api_ky`, `mesage`. Then your app looks for `password`, finds nothing, and fails at runtime with a confusing error. Naming conventions are the only defence. ### 4.2 The boot race **Symptom:** `dial tcp 127.0.0.1:8200: connect: connection refused` immediately after `docker compose up -d`. **Cause:** the container reported "Started" ~1.5s before Vault bound the port. **Lesson:** `connection refused` means *nothing is listening* — either not booted yet or crashed. **Check logs before guessing.** `docker compose logs vault --tail=40`. Reading Vault's startup logs is a skill you'll use constantly. Fix: `sleep 3`. ### 4.3 VAULT_ADDR defaults to HTTPS **Symptom:** `Error checking seal status: http: server gave HTTP response to HTTPS client` **Cause:** the CLI defaults to `https://127.0.0.1:8200`; dev mode serves plain HTTP. **Lesson:** `VAULT_ADDR` isn't optional. The CLI has no idea where your server is or what protocol it speaks. ### 4.4 405 ≠ 404 ≠ 403 **Symptom:** `vault read database/config` → `405 unsupported operation`. Expected "not configured." **Cause:** `database/config` only supports `LIST`. Wrong verb, not missing data. **Lesson:** learn the three apart (see §2.19). It's the difference between five minutes and an hour when debugging policies. ### 4.5 `kv get` always fetches latest **Symptom:** `kv undelete -versions=2` succeeded, but `kv get` still showed a deleted secret. **Cause:** `kv get` without `-version` fetches the **latest** version — v3, still deleted. v2 was correctly restored. **Lesson:** use `kv metadata get` to see all versions and states at once. ### 4.6 `token renew -self` isn't a flag **Symptom:** `flag provided but not defined: -self` **Fix:** `vault token renew` — bare, renews the token in `VAULT_TOKEN`. ### 4.7 Missing `exec vault` **Symptom:** `unknown docker command: "compose vault"` **Cause:** `docker compose vault ...` instead of `docker compose exec vault vault ...`. The pattern is: first `vault` = service name, second = binary. ### 4.8 Expired token = working as designed **Symptom:** UI login fails with `invalid token`. **Cause:** `token_ttl=1h` elapsed. (Or the container restarted — `inmem` wipes tokens.) **Lesson — the mental shift:** the instinct was "something is broken." Nothing broke. **Tokens are supposed to stop working.** Getting a new one is routine, not recovery. Apps do this automatically; you do it by hand because you're learning. ### 4.9 The word "secret" means two things `secret/` in a command is a **mount path** — just a folder name from dev mode. "A secret" in conversation is the **sensitive data**. Moving to `qbit/` removes the ambiguity. ### 4.10 The `/data/` insertion See §2.10. Expect to hit it. Everyone does. --- # PART 5 — Humans, Machines & MFA ## 5.1 Two Populations | | **Humans** | **Machines** | |---|---|---| | Auth method | Userpass, OIDC, LDAP | AppRole, TLS cert, K8s | | MFA | **Required** | N/A | | Token TTL | 8h | 1–4h, auto-renewed | | Access | Broad but read-mostly, **no prod** | Narrow, exactly what's needed | | Audit shows | Named individual | Named service | **Current state:** one root token serving both. That's the thing to fix. ## 5.2 Enabling Users ```bash vault auth enable userpass vault write auth/userpass/users/kibuti \ password="..." token_policies="vault-admin" \ token_ttl=8h token_max_ttl=12h vault write auth/userpass/users/dev1 \ password="..." token_policies="dev-engineer" token_ttl=8h ``` Now the audit log says `dev1`, not `root`. ## 5.3 Developer Policy ```hcl # dev-engineer.hcl # full CRUD on any service's dev path "qbit/data/+/dev/*" { capabilities = ["create", "read", "update", "delete", "list"] } path "qbit/metadata/+/dev/*" { capabilities = ["read", "list", "delete"] } # staging: read only path "qbit/data/+/staging/*" { capabilities = ["read"] } path "qbit/metadata/+/staging/*" { capabilities = ["read", "list"] } # prod: no rule = denied ``` **Note the `+`** — `*` only globs at the end of a path. **The hard rule: developers should not read production secrets at all.** Not "read-only prod" — *no* prod. If a dev needs the production database password, something is wrong with your deployment process. Apps read prod secrets via AppRole; humans don't need to. **Who writes prod secrets?** You, as admin. Or a CI pipeline with a narrow write-only policy. ## 5.4 The Policy Set | Policy | Who | Grants | |---|---|---| | `vault-admin` | You | Everything except root-only ops | | `dev-engineer` | Devs | CRUD on `+/dev/*`, read `+/staging/*` | | `deploy-pipeline` | Komodo | Generate Secret IDs only | | `-` | Apps | Read one path | | `auditor` | Vendor assessments | `list` on metadata, **no read on data** | The `auditor` policy is genuinely useful — someone verifies your secrets are properly organised without seeing a single value. ## 5.5 MFA **Vault Community has no native login MFA.** That's Enterprise. Get it from the identity provider instead. **Option A — OIDC with an MFA-capable IdP (best long-term)** ```bash vault auth enable oidc vault write auth/oidc/config \ oidc_discovery_url="https://auth.qbitspark.com/realms/qbit" \ oidc_client_id="vault" oidc_client_secret="..." default_role="engineer" vault write auth/oidc/role/engineer \ bound_audiences="vault" \ allowed_redirect_uris="https://vault.qbitspark.com/ui/vault/auth/oidc/oidc/callback" \ user_claim="email" token_policies="dev-engineer" token_ttl=8h ``` Keycloak or Authentik self-hosted. Bonus: SSO across your other internal tools. **Option B — network-layer (fastest for you)** Traefik + Authelia/Authentik forward-auth in front of the Vault UI. MFA happens before the request reaches Vault. **You already run Traefik.** **Option C — userpass + TOTP engine.** Vault's TOTP engine *manages* codes but doesn't enforce them on login. Partial only. **Recommendation:** B now, A later. Machines don't get MFA — that's what AppRole + CIDR binding is for. ## 5.6 Break-Glass ```bash # generate root (needs 3 shares) vault operator generate-root -init vault operator generate-root -nonce= vault operator generate-root -decode= -otp= vault token revoke # IMMEDIATELY after use # emergency seal vault operator seal ``` **Document offline, before you need it:** - [ ] Where are the 5 shares physically? - [ ] Who holds which? (currently: you alone — single point of failure) - [ ] Procedure if you're unavailable? - [ ] Is the recovery doc accessible **without** Vault? > **Solo-operator note:** Shamir assumes multiple people. Split across **locations** instead. Never all in one place, never all on one machine. **Test the recovery once** — an untested scheme is a story you tell yourself. ## 5.7 Shamir Beyond Vault Same math, real-world uses: - **Cryptocurrency inheritance** — seed phrase split 3-of-5 across family, lawyer, safe deposit box. Casa and Unchained Capital are businesses built on this. - **DNSSEC root key ceremony** — seven people worldwide hold physical smartcards, meet in a locked facility on camera. **Directly relevant to your TCRA `.tz` registrar work.** - **Certificate authority root keys** — split among executives, stored in HSMs across locations. - **Nuclear two-person rule** — same principle. - **Estate/banking arrangements** — genuine legal instruments. **For personal property:** ``` 3-of-5: Share 1 → bank safe deposit box (Mbeya) Share 2 → family member, different city Share 3 → lawyer, sealed envelope with instructions Share 4 → encrypted, cloud storage (different provider than your infra) Share 5 → physical, hidden at home ``` Survives fire, theft, one betrayal, one person unreachable. ```bash sudo apt install ssss echo "my-secret" | ssss-split -t 3 -n 5 ssss-combine -t 3 ``` Steel plates beat paper — paper burns, ink fades. Cryptosteel exists; a hardware store and letter punches do the same job cheaper. **Limitations, honestly:** humans lose things; **the instructions must survive too** (a share is useless if nobody knows what it is); legal systems don't recognise "3-of-5" the way they recognise a notarised will — use it *alongside* proper instruments; **and test it**. --- # PART 6 — Rotation ## 6.1 Theory Every credential's compromise probability rises with age and exposure. Rotation bounds the damage window. **Maturity ladder:** 1. ❌ Never rotate — where you were 2. ⚠️ Manual scheduled — where you're going next 3. ✅ Automated rotation of static secrets 4. 🏆 **Dynamic secrets** — rotation becomes irrelevant because nothing is long-lived ## 6.2 By Credential Type **Root token** — `generate → use → revoke`. Never leave one alive. **Unseal keys** ```bash vault operator rekey -init -key-shares=5 -key-threshold=3 ``` **Encryption key (transparent)** ```bash vault operator rotate vault operator key-status ``` **AppRole Secret IDs** ```bash vault write -f auth/approle/role/X/secret-id # new vault write auth/approle/role/X/secret-id-accessor/destroy \ secret_id_accessor="" # kill old ``` **KV static secrets — order matters** ``` 1. Create new credential at the provider 2. vault kv put ... ← new version; old still readable 3. Restart/reload services 4. Verify healthy 5. ONLY THEN revoke old at the provider 6. Optionally vault kv destroy -versions= ``` Revoking before services pick up the new value = outage. **Database credentials** — move to the `database` engine; rotation becomes automatic. ## 6.3 Schedule | Credential | Frequency | Method | |---|---|---| | Root token | Per use | generate → use → revoke | | Unseal keys | Annually or on suspicion | `operator rekey` | | Encryption key | Quarterly | `operator rotate` | | AppRole Secret IDs | Per deploy, or quarterly | Komodo, or manual | | Service tokens | 1–4h | Automatic | | Third-party API keys | Quarterly + on incident | Manual, versioned in KV | | DB credentials | 1h | Dynamic engine | | TLS certificates | 90 days | PKI engine | | Komodo's own Secret ID | Quarterly | **Manual, by SSH** | ## 6.4 Dynamic Migration Order 1. **PostgreSQL** — all services; highest value 2. **RabbitMQ** — NexGate messaging 3. **MinIO** — via AWS engine (S3-compatible STS) 4. **PKI** — internal CA for NexGate mTLS 5. **SSH** — signed certs for `qbit-spark` 6. **Transit** — JikoXpress financial data **Success measure:** a leaked credential is a non-event because it expired an hour ago. --- # PART 7 — Deployment Architecture ## 7.1 Where Credentials Live — Per Environment **The one rule: Role ID and Secret ID never in the same place.** ### Local (developer laptop) ``` Role ID → committed in repo (docker-compose.dev.yml) Secret ID → .vault/secret-id, gitignored ``` Each dev generates their own Secret ID. Never committed. ### Staging ``` Role ID → docker-compose.yml on the staging box (Git fine) Secret ID → /etc/vault/-staging-secret-id chmod 400, chown root:root ``` ### Production ``` Role ID → docker-compose.yml (Git fine) Secret ID → /etc/vault/-prod-secret-id chmod 400, chown root:root ``` ### The secrets themselves **In Vault.** Nowhere else. No `.env`, no config file, no baked image layer. Fetched at boot with the AppRole token. ## 7.2 Never Do This ```yaml environment: VAULT_ROLE_ID: "..." VAULT_SECRET_ID: "..." # ← docker inspect prints this ``` One command yields both halves. **This is likely how the OpenAI key walked out.** ## 7.3 Do This ```bash # on the server, as admin sudo mkdir -p /etc/vault vault write -f -field=secret_id auth/approle/role/bishamba-prod/secret-id \ | sudo tee /etc/vault/bishamba-prod-secret-id > /dev/null sudo chmod 400 /etc/vault/bishamba-prod-secret-id sudo chown root:root /etc/vault/bishamba-prod-secret-id ``` ```yaml services: bishamba: environment: VAULT_ROLE_ID: "8c3f2a1b-4d5e-6f70-8192-a3b4c5d6e7f8" volumes: - /etc/vault/bishamba-prod-secret-id:/run/secrets/secret-id:ro ``` ```bash #!/bin/sh # entrypoint.sh export VAULT_SECRET_ID=$(cat /run/secrets/secret-id) exec java -jar app.jar ``` **Why file-mount beats env var:** `docker inspect` shows only the mount path. `docker compose config` shows nothing sensitive. Git holds only the Role ID. **Honest limitation:** the env var still exists inside the running process, so `/proc//environ` shows it to root. You've removed it from the Docker metadata layer, compose file, and Git — which is where these things actually leak from. **Vault Agent removes it entirely.** ## 7.4 Spring Boot Integration ```yaml # application-prod.yml spring: cloud: vault: uri: https://vault.qbitspark.com authentication: APPROLE app-role: role-id: ${VAULT_ROLE_ID} secret-id: ${VAULT_SECRET_ID} kv: enabled: true backend: qbit default-context: bishamba/prod ``` ```java @Value("${postgres.username}") private String dbUser; @Value("${postgres.password}") private String dbPass; ``` Spring Cloud Vault logs in at startup, fetches `qbit/bishamba/prod/*`, injects into properties, renews the token in a background thread. **No credentials in code, no `.env` in production.** ## 7.5 Three Environments, One Vault Decision: connectivity isn't a constraint, so all three live on the central Vault. ```bash for env in local staging prod; do cat > policies/bishamba-${env}.hcl < **The property this buys:** a developer can build, test, and ship the entire application without ever knowing a single production credential. If their laptop is stolen, production is unaffected. ## 7.7 Komodo Integration **Komodo needs a Vault token to mint Secret IDs.** That token becomes the new secret zero, so keep it narrow. **1. A policy that does one thing** ```hcl # komodo-deployer.hcl path "auth/approle/role/+/secret-id" { capabilities = ["update"] } ``` Can mint Secret IDs. **Cannot read a single secret, cannot change policies, cannot create roles.** **2. Komodo gets its own AppRole — same discipline** ```bash vault policy write komodo-deployer policies/komodo-deployer.hcl vault write auth/approle/role/komodo-deployer \ token_policies="komodo-deployer" \ token_ttl=15m token_max_ttl=30m \ secret_id_ttl=0 \ secret_id_bound_cidrs="161.97.163.158/32" ``` **3. You place Komodo's credentials by hand, once, over SSH** ```bash ssh you@161.97.163.158 sudo mkdir -p /etc/vault vault read -field=role_id auth/approle/role/komodo-deployer/role-id \ | sudo tee /etc/vault/komodo-role-id > /dev/null vault write -f -field=secret_id auth/approle/role/komodo-deployer/secret-id \ | sudo tee /etc/vault/komodo-secret-id > /dev/null sudo chmod 400 /etc/vault/komodo-* sudo chown root:root /etc/vault/komodo-* ``` ```yaml services: komodo: volumes: - /etc/vault/komodo-role-id:/run/secrets/role-id:ro - /etc/vault/komodo-secret-id:/run/secrets/secret-id:ro ``` **4. Komodo's deploy script** ```bash #!/usr/bin/env bash set -euo pipefail SERVICE="$1" # e.g. bishamba-prod TOKEN=$(vault write -field=token auth/approle/login \ role_id="$(cat /run/secrets/role-id)" \ secret_id="$(cat /run/secrets/secret-id)") VAULT_TOKEN=$TOKEN vault write -f -field=secret_id \ "auth/approle/role/${SERVICE}/secret-id" \ > "/etc/vault/${SERVICE}-secret-id" chmod 400 "/etc/vault/${SERVICE}-secret-id" docker compose up -d "$SERVICE" # after health check passes, revoke older Secret IDs # Komodo's own token dies in 15 minutes regardless ``` **Revoke old Secret IDs after deploy** — otherwise every deploy leaves a valid credential behind. Do the destroy **after** the new container is healthy, not before. **Rotating Komodo's own Secret ID is manual, deliberately:** ```bash ssh qbit-spark vault write -f -field=secret_id auth/approle/role/komodo-deployer/secret-id \ | sudo tee /etc/vault/komodo-secret-id > /dev/null docker compose restart komodo ``` It can't be automated *from Komodo* — that would let a compromised Komodo renew its own access forever. Keeping it manual bounds the chain. ## 7.8 Where the Chain Terminates ``` YOU ← SSH key. The actual root of trust. │ └─ place Komodo's role-id + secret-id (once, by hand) │ └─ KOMODO mints app Secret IDs (every deploy, 15min tokens) │ └─ APPS log in, read their own secrets (every start, 1h tokens) ``` **Your SSH key is the bottom.** That's the real secret zero for the whole system — which is why it deserves a passphrase, a hardware key if possible, and an entry in the break-glass documentation. ## 7.9 Secret Zero — The Honest Accounting > "To get a credential for Vault, you need a credential for Vault." **Correct. It never disappears. It shrinks.** **Before:** ``` .env on server: DB_PASSWORD=real SNIPPE_API_KEY=real JWT_PRIVATE_KEY=real OPENAI_KEY=real MINIO_SECRET=real ``` One file. Everything. Never expires. No log of who read it. **After:** ``` /etc/vault/prod-secret-id: d4e5f6a7-b8c9-... ``` One file. One UUID. | | Old `.env` | Stolen Secret ID | |---|---|---| | What they get | Every credential immediately | Nothing — need Role ID too | | With both halves | — | One service, one environment | | Duration | Forever | Until rotated | | IP-bindable | No | **Yes** | | Do you find out | No | **Yes — audit log** | | To revoke | Change everything, everywhere | One command | **The ladder of secret-zero strength:** | Approach | Secret zero is... | |---|---| | Root token in app config | The root token. Worst case. | | **AppRole** | Secret ID — scoped, expiring, revocable | | Response wrapping | One-time wrapper — **tamper-evident** | | TLS cert auth | A certificate the machine already has | | Cloud/K8s identity | The platform vouches — no secret at all | **Why "still in Vault" isn't circular:** Vault doesn't protect secrets by hiding them. It protects them by (1) encrypting at rest, (2) authenticating every request, (3) authorising per path, (4) expiring everything, (5) logging every access, (6) revoking centrally. **A `.env` file does none of these six.** **The realistic goal was never "no secrets anywhere."** It's: *one small, scoped, expiring, revocable, logged, IP-bound credential instead of a plaintext file containing your entire business.* --- # PART 8 — Production Migration ## 8.1 Pre-Migration Checklist - [ ] Lab Steps 1–12 complete and understood - [ ] Isolation verification script passes locally - [ ] Full backup of production Vault taken and **restore tested** - [ ] Storage backend of `vault.qbitspark.com` identified - [ ] Unseal key shares located, verified, distributed across locations - [ ] Audit logging enabled in production - [ ] Maintenance window agreed - [ ] Rollback plan written - [ ] **OpenAI leak vector identified and closed** ## 8.2 Cutover Order — Lowest Risk First 1. GlueEmail (smallest blast radius) 2. Textfy dev 3. Mauzodukani dev → prod 4. NexGate staging 5. JikoXpress dev → staging 6. NexGate prod 7. **JikoXpress prod** (customer payments — last, most carefully) **Per service:** ``` 1. Create policy in production Vault 2. Create AppRole 3. Write secrets to new paths (keep old paths intact) 4. Update ONE instance to use AppRole 5. Verify health, logs, functionality 6. Roll out to remaining instances 7. Verify again 8. Remove old path access from any broad policy 9. Delete old secret path 10. Run isolation verification ``` **Never delete the old path until the new one is proven in production.** ## 8.3 Post-Migration Verification - [ ] Isolation script passes against production - [ ] Every service authenticates via AppRole, none via root - [ ] Root token revoked; none in any browser session or file - [ ] Human access via OIDC/proxy with MFA - [ ] Audit logs flowing to Grafana/Loki - [ ] Alerts: root token use, permission-denial spikes, seal events - [ ] Rotation schedule documented and calendared - [ ] Break-glass procedure written and stored **offline** - [ ] Backup automation running with tested restore - [ ] CIDR binding applied to staging and prod roles --- # PART 9 — Operations ## 9.1 Backup & DR ```bash vault operator raft snapshot save vault-$(date +%F).snap vault operator raft snapshot restore vault-2026-08-05.snap ``` - Snapshots are **encrypted** — useless without unseal keys. Back up both, **separately**. - **A backup you haven't restored is not a backup.** Test quarterly into a scratch instance. - Automate: daily snapshot → encrypted → offsite (MinIO on a different provider, plus one cold copy). ## 9.2 Monitoring Prometheus metrics at `/v1/sys/metrics?format=prometheus`. | Condition | Severity | |---|---| | Vault sealed unexpectedly | Critical | | **Root token used** | Critical | | Audit device failure | Critical | | Permission-denial spike | High — attack or misconfiguration | | Token creation anomaly | High | | Lease count growth | Medium — leaked unreleased leases | | Secret/cert nearing expiry | Medium | Feed into the existing Grafana stack. ## 9.3 Upgrades Read the changelog for **every** version between current and target. Snapshot before upgrading, always. Test on the lab instance first. Never skip major versions. Currently 1.20.3 — patch promptly for security fixes. --- # PART 10 — The Book ## 10.1 Concept **Working title:** *Vault From Zero to Production: Secrets Management for Small Teams and Solo Builders* **The gap:** HashiCorp's docs are reference material assuming an enterprise platform team. Existing books target large organisations with dedicated security staff. Nobody writes for the solo operator or three-person startup running real production on a handful of VPSes. **Unfair advantages:** 1. Living the problem, not theorising — a real incident, a real recovery 2. African/emerging-market context: modest budgets, self-hosted VPS, no cloud-native luxury 3. Proven format and finishing discipline — EM/RF trilogy, and *XMPP From Zero to Production* in progress 4. Real multi-product stack (Spring Boot, PostgreSQL, RabbitMQ, MinIO, Docker, Traefik, Komodo), not toy examples **Running example:** mirror the MaasaiChat approach. A fictional composite — **"Zanzi Systems,"** a Tanzanian SaaS operator with three products and one engineer — whose secrets infrastructure is built across the book. Reader follows the arc from `.env` chaos to dynamic secrets. **Series consistency:** *XMPP From Zero to Production* → *Vault From Zero to Production*. ## 10.2 Structure **Part I — The Problem** (Ch. 1–4) 1. The $8 Lesson: anatomy of a credential leak 2. What Is a Secret, Really? (path vs field vs value) 3. Every Way Secrets Leak — and how each is discovered 4. The Landscape: choosing your tool **Part II — Fundamentals** (Ch. 5–13) 5. Installing and Running Vault (Docker, and why `IPC_LOCK`) 6. Seal, Unseal, and the Bank in Mbeya 7. Shamir's Secret Sharing — the math, and its life beyond Vault 8. Storage Backends 9. Secrets Engines: static vs dynamic 10. Auth Methods: humans and machines 11. Policies: deny by default 12. The `/data/` Gotcha and Other Path Traps 13. Tokens, Leases, and TTLs **Part III — Building It** (Ch. 14–22) 14. Designing Your Path Hierarchy (why environment sits above component) 15. Writing Policies That Actually Work 16. Testing Denial — the step everyone skips 17. AppRole in Depth: the hotel, and secret zero 18. Integrating Spring Boot 19. Docker and Compose Integration 20. CI/CD with Komodo 21. Human Access, SSO, and MFA on Community Edition 22. Multi-Tenancy: hosting Vault for clients **Part IV — Going Dynamic** (Ch. 23–28) 23. Dynamic Database Credentials 24. PKI: running your own CA 25. Transit: encryption as a service 26. SSH Certificates 27. Response Wrapping and Tamper-Evident Delivery 28. Rotation Strategy End-to-End **Part V — Production** (Ch. 29–36) 29. Migrating an Existing System Without Downtime 30. High Availability 31. Backup and Disaster Recovery 32. Monitoring and Alerting 33. Upgrades 34. Incident Response 35. Compliance and Vendor Assessments 36. Cost and Capacity Planning **Part VI — Beyond Vault** (Ch. 37–40) 37. OpenBao and the Licensing Question 38. SOPS and GitOps Secrets 39. When Vault Is the Wrong Answer 40. Security Culture as a Solo Operator **Appendices** - A. Complete policy reference for a five-service stack - B. Verification and testing scripts - C. Emergency runbooks - D. **The Mistakes File** — every error, cause, and fix - E. CLI quick reference - F. Glossary ## 10.3 Workflow **The discipline:** write each chapter *as you complete the lab step*, while the confusion is fresh. The book's value is remembering what was hard — that memory fades within weeks of mastery. **Per-chapter template:** 1. **The problem** — a concrete scenario from Zanzi Systems 2. **The concept** — theory with a diagram 3. **The lab** — commands the reader runs 4. **The verification** — how to know it worked, **including negative tests** 5. **What goes wrong** — real errors and causes 6. **Production notes** — how this differs at scale 7. **Exercises** — solutions in the appendix **Cadence:** ~4,000 words/chapter × 40 chapters ≈ 160,000 words. Two chapters/week → first draft in ~5 months. ``` vault-book/ ├── manuscript/ │ ├── part-1/ ... part-6/ │ └── appendices/ ├── labs/ │ ├── step-01/ ... step-12/ │ └── scripts/ ├── diagrams/ ├── mistakes.md ← the goldmine └── outline.md ``` **Keep `mistakes.md` open during every lab session.** Every error message, every confusion, every "why doesn't this work." That file is the differentiator — it's what the official docs will never contain. --- ## Progress **Lab** - [x] Step 1 — Vault running (Docker) - [x] Step 2 — Tokens and first secret - [x] Step 3 — Own engine, path hierarchy, engine zoo - [x] Step 4 — Versioning, delete/undelete/destroy - [x] Step 5 — First policy - [x] Step 6 — **Denial proven (4× 403)** - [x] Step 7 — AppRole enabled - [x] Step 8 — **Service login, zero root token** - [x] UI experiment — scope made visible - [ ] Step 9 — All services scripted - [ ] Step 10 — Isolation verification script - [ ] Step 11 — Audit logging - [ ] Step 12 — Dynamic database credentials **Production homework** - [ ] Find the OpenAI leak vector (repos, images, `docker inspect`, Docker API on 2375) - [ ] New OpenAI key **with a spending limit** - [ ] **Enable audit logging on `vault.qbitspark.com`** — one command, non-disruptive, do it now - [ ] Locate and verify unseal key shares - [ ] Identify the production storage backend - [ ] Test a snapshot restore **Book** - [ ] Start `mistakes.md` (Part 4 above is the seed) - [ ] Draft chapters 1–4 - [ ] Define Zanzi Systems --- *QBIT SPARK internal engineering document — v2, 6 August 2026*