Vault Mastery Guide — v2 BOOK

Vault Mastery Guide — v2

From Root-Token Chaos to Production-Grade Secrets Management

Author: Kibuti — QBIT SPARK Environment: vault.qbitspark.com (HashiCorp Vault v1.20.3) Method: Local simulation first → master concepts → migrate production last Status: Lab Steps 1–8 complete. Steps 9–12 pending.

v2 changes: rewritten after completing the hands-on lab. Every analogy, error message, and "why doesn't this work" moment from the actual session is now captured. The mistakes sections are real, not hypothetical.

Table of Contents

PART 0 — Why This Exists PART 1 — Foundations PART 2 — Core Concepts (with the analogies that worked) PART 3 — The Lab, Step by Step PART 4 — Mistakes File PART 5 — Humans, Machines & MFA PART 6 — Rotation PART 7 — Deployment Architecture PART 8 — Production Migration PART 9 — Operations PART 10 — The Book

PART 0 — Why This Exists

0.1 The Incident

On 4–5 August 2026, an OpenAI API key (sk-pro...CrQA) was used by an unauthorized party to run ~129 automated LLM requests — a physics-grading workload (Feynman amplitude checkpoint scoring, some Chinese-language content) with no relation to any QBIT SPARK product. Roughly $7.95 consumed before detection.

Detection: OpenAI's abuse-detection email → manual log review → confirmed by reading request content.

Response: key revoked immediately.

Leak vector: still undetermined. Candidates under investigation:

0.2 What the Incident Actually Exposed

The $7.95 was trivial. The structural problems it revealed were not:

Weakness Risk
Root token used for routine Vault UI access Full-scope master credential in daily circulation
Same root token bootstrapping local dev and production Dev compromise = production compromise
No per-service scoped policies Any leak potentially exposes all secrets
No audit logging No forensic trail — couldn't answer "was this ever read from Vault?"
No MFA on human access Single-factor compromise = total access
No rotation schedule Leaked credentials valid indefinitely
No spending limits on downstream API keys Financial blast radius unbounded
Unknown location of unseal key shares Possible single point of catastrophic failure

The uncomfortable comparison: the same class of mistake applied to Vault's root token would expose every credential for JikoXpress, NexGate, Mauzodukani, GlueEmail and Textfy at once — including Snippe PSP payment integration, NexGate JWT signing keys, database passwords, and Vodacom SMPP credentials.

0.3 What "Mastery" Means Here

Not memorising commands. Being able to:

  1. Explain every Vault component in your own words, without notes
  2. Design a policy structure for a new service from scratch
  3. Predict what will break before changing something
  4. Recover from sealed / lost-key / corrupted-storage without panic
  5. Teach it — which is why this ends in a book

Rule: do not proceed past a step until you can explain it aloud with the notes closed.

PART 1 — Foundations

1.1 What Is a Secret?

Any data that grants access or proves identity, and whose disclosure causes harm.

Type Examples from this stack
Static credentials PostgreSQL, RabbitMQ, MinIO passwords
API keys OpenAI, Snippe PSP, Meta WhatsApp Business API, Vodacom SMPP
Cryptographic material NexGate JWT RS256 signing keys, TLS private keys, OMEMO keys
Connection strings DSNs embedding host + credentials
Certificates Internal CA, client certs for mTLS
SSH keys qbit-spark access

The problem: secrets must be available to apps at runtime, unreadable by anyone else, rotatable without downtime, and every access attributable.

.env files solve availability. Nothing else.

1.2 The Precision That Matters

Three words get conflated constantly. Get them separate:

vault kv put qbit/jikoxpress/prod/db username="jx_prod" password="s3cr3t"
Part Name Sensitive?
qbit/jikoxpress/prod/db path — the address No
password field name / key — the label No
s3cr3t field value ← this is the secret

The house analogy: path is the address. Secret is the money inside. Token is the key in your pocket.

Why it matters practically: Vault's audit log records paths and field names in plaintext but HMACs the values. You can see that someone read qbit/jikoxpress/prod/db without the log itself leaking the password.

The secret/ confusion: in vault kv put secret/hello, the word "secret" is a mount path, not the data. Dev mode happens to mount a KV engine there. It could be called anything. This is why the lab moves to qbit/ — the word disappears and the ambiguity goes with it.

1.3 What Vault Is

An encrypted key-value store that requires authentication to read from, enforces fine-grained authorisation per request, logs every access, and can generate short-lived credentials on demand instead of merely storing long-lived ones.

Four jobs:

  1. Secure storage — encrypted at rest; the storage backend never sees plaintext
  2. Access control — every request authenticated and authorised per path
  3. Audit — every request and response logged
  4. Dynamic secrets — credentials with automatic expiry, so a leak dies on its own

What it is not: a human password manager, a config management system, or magic. A badly configured Vault is worse than none, because it creates false confidence.

1.4 The Mental Model

                    ┌─────────────────────────────────┐
                    │         VAULT SERVER            │
   You / Service    │  ┌──────────┐                   │
        │           │  │   AUTH   │ ← "Prove who      │
        ├──────────►│  │  METHOD  │    you are"       │
        │           │  └────┬─────┘                   │
        │           │       │ issues                  │
        │           │       ▼                         │
        │           │  ┌──────────┐                   │
        │           │  │  TOKEN   │ ← temporary badge │
        │           │  └────┬─────┘                   │
        │           │       │ carries                 │
        │           │       ▼                         │
        │           │  ┌──────────┐                   │
        │           │  │ POLICIES │ ← "what doors     │
        │           │  └────┬─────┘    does it open?" │
        │           │       ▼                         │
        │           │  ┌──────────────────┐           │
        │           │  │ SECRETS ENGINES  │ ← rooms   │
        │           │  └────┬─────────────┘           │
        │           │       ▼                         │
        │           │  ┌──────────────────┐           │
        │           │  │ STORAGE BACKEND  │ ← encrypted
        │           │  └──────────────────┘           │
        │           │  ┌──────────────────┐           │
        │           │  │  AUDIT DEVICES   │ ← the log │
        │           │  └──────────────────┘           │
        └───────────┴─────────────────────────────────┘

Every Vault problem you will ever debug is a failure in one of those boxes.

1.5 The Alternatives Landscape

Vault OpenBao Infisical SOPS AWS SM
Licence BUSL 1.1 MPL 2.0 MIT / paid MPL 2.0 Proprietary
Self-host Yes Yes Yes N/A No
Dynamic secrets ✅ Extensive ✅ (fork) ⚠️ Limited ❌ ⚠️ Some
Auth methods ✅ 20+ ✅ 20+ ⚠️ Fewer ❌ IAM only
Learning curve Steep Steep Gentle Gentle Moderate
PKI / CA ✅ ✅ ❌ ❌ ⚠️ Separate
Encryption-as-a-service ✅ Transit ✅ ❌ ❌ ⚠️ KMS
Audit logging ✅ Detailed ✅ ✅ Basic ❌ ✅ CloudTrail
Ops burden High High Medium Very low None
Cost at this scale Free Free Free tier Free ~$0.40/secret/mo

Categories, for the book:

1.6 Why Vault — Decision Record

  1. Already deployed at vault.qbitspark.com; migration cost zero
  2. Multi-product, multi-provider VPS topology — cloud-native can't span it
  3. Dynamic secrets are the long-term win
  4. PKI engine — internal CA for NexGate mTLS, and relevant to TCRA registrar work
  5. Transit engine — encrypt JikoXpress financial data without app-held keys
  6. Commercial credibility — "scoped AppRole auth with audit logging" answers a vendor questionnaire far better than ".env files." Already been through Vodacom BSR/OneTrust.
  7. Transferable skill — appears in enterprise and East African fintech requirements

Hedge: track OpenBao. Migration is currently cheap. Revisit annually.

PART 2 — Core Concepts

2.1 Seal / Unseal — The Bank

Imagine a bank in Mbeya.

When sealed: the building exists, the door is locked, the money is scrambled. Steal the entire building and you get a locked box full of ciphertext.

Vault starts sealed every time it boots. Deliberately.

The unseal keys = the combination, cut into 5 pieces

The combination isn't written anywhere. It was split into 5 shares given to 5 managers. Any 3 can open the door. One alone can't. Two can't.

Why 3-of-5 and not 5-of-5? Because if one manager dies, loses their piece, or travels, you'd never open the bank again. 3-of-5 survives losing 2.

The math (Shamir's Secret Sharing, 1979): two points define a line, three define a parabola. Hide the secret as a point on a curve; hand out other points. With enough points you reconstruct the curve. With fewer, infinitely many curves fit — you learn nothing, not "most of it."

Auto-unseal delegates to an external KMS so restarts don't need manual intervention. Convenient; shifts trust to that system.

If you lose enough shares, the data is permanently unrecoverable. There is no backdoor.

2.2 The Precise Chain

Common misunderstanding: unseal shares do not generate the master key. They reconstruct it — it already exists, split at initialization.

And the master key doesn't decrypt secrets directly:

3 unseal shares
      ↓ reconstruct
  Master Key
      ↓ decrypts
 Encryption Key   ← this is what encrypts your secrets
      ↓
   Vault UNSEALED

Why the extra layer? So vault operator rotate can change the encryption key without redistributing 5 new shares to 5 people.

2.3 Root Token vs Unseal Keys — The Rooms

Door open ≠ free access. Inside are many rooms — one per service. The root token is a master keycard opening every room. Scoped tokens open one.

Unseal keys Root token
Opens The vault door The rooms inside
Count 5 shares, need 3 One
Used when Vault restarts Every request
Frequency Rare Should be near-never

You need both. Shares alone = open door, no keycard. Root alone = keycard, locked door.

Confirmed in the lab: the root token cannot unseal. That separation means two different attacks are required.

The reverse — the emergency button: root can seal. Sealing is easy, unsealing is hard, deliberately.

vault operator seal    # everything slams shut, instantly

If vault.qbitspark.com is ever suspected compromised, that's the fire alarm. Costs an outage; the attacker gets ciphertext. Then you need 3 of 5 shares to come back — which is why you must know where they are before the emergency.

2.4 Why Root Exists At All

Chicken-and-egg. A fresh Vault has no policies, no auth methods, nothing. Someone must create the first policy — but that needs permission.

Root is the bootstrap credential. It exists to create the system that replaces it.

Legitimate uses — the complete list:

  1. Initial setup — enable auth methods, mount engines, write first policies
  2. Emergency recovery — locked out, broken policy
  3. A few root-protected operations — operator rekey, some seal management

Not on the list: daily work, application requests, UI login.

Where you store it: nowhere. You destroy it.

generate → use → vault token revoke → gone

Need it next month? vault operator generate-root with 3 shares.

What you protect long-term is the shares, because those regenerate root. Root itself is disposable.

Where the shares go:

Solo operator → can't split across people → split across places. If the house floods you still need 3.

Correction worth recording: root is not used to create app credentials day-to-day. Root builds the machinery once — enables AppRole, writes policies, creates roles — then is revoked. AppRole issues credentials forever afterward with no root involved.

Root builds the key-cutting machine. Then root leaves. The machine keeps cutting keys.

2.5 Storage Backends

Vault encrypts before writing, so a stolen backend yields ciphertext.

Backend Use
Integrated (Raft) Recommended default; HA capable
Consul Legacy; more operational complexity
File Single node; dev/test
In-memory Dev only; vanishes on restart
PostgreSQL/MySQL Supported, not recommended for new deployments

Action item: determine which backend vault.qbitspark.com uses. It determines your backup strategy.

Lab observation: Storage Type inmem means every restart is a brand-new Vault — new cluster ID, new unseal key, no data. On real Vault with persistent storage, the unseal key is generated once at initialization and never again. Lose it, lose everything.

2.6 Secrets Engines

~20+ types, not four. Grouped by philosophy:

Static — you store, Vault guards

Engine Purpose
kv-v2 Versioned key-value. The workhorse.
kv-v1 Unversioned, legacy
cubbyhole Per-token private storage; dies with the token

Dynamic — Vault generates and expires

Engine Generates
database PostgreSQL, MySQL, MongoDB, Redis users
pki X.509 certificates
ssh Signed SSH certificates
aws/gcp/azure Temporary cloud IAM credentials
rabbitmq/consul/nomad Service-specific credentials
kubernetes Service account tokens

Cryptographic services

Engine Purpose
transit Encrypt/decrypt without ever holding the key
transform Format-preserving encryption, masking, tokenization
totp Generate and validate 2FA codes

Infrastructure — identity/ and sys/, always mounted, cannot be disabled.

Adoption order for QBIT SPARK: kv-v2 → database → pki → transit → ssh.

The insight: every engine answers "how do we handle this class of secret?" with a different strategy. KV says "store it carefully." Database says "don't store it, make a new one each time." Transit says "don't give it to the app at all — send the data here."

Rule of thumb: if Vault can generate it, don't store it. Store only what you're forced to — third-party API keys, mostly.

2.7 Mounts vs Paths vs Tenants

Two ways to split, easy to confuse:

Axis 1 — by engine type. Different engines are different machines. A KV engine can't issue certificates.

Axis 2 — by service. Possible, usually wrong:

✅ One mount, path hierarchy:      ❌ Mount per service:
qbit/jikoxpress/prod/db            jikoxpress/prod/db
qbit/nexgate/prod/minio            nexgate/prod/minio

Both isolate fine via policy. Separate mounts add config, versioning settings, tuning, backup and audit surface with no extra security.

When separate mounts genuinely help — multi-tenancy. Relevant here: you hold a retainer with Wiki, deployed FinSpark for a Mbeya microfinance client, built PMK's site. If you host secrets on behalf of clients, mount per tenant:

qbit/     ← your products
wiki/     ← retainer client
kfz/      ← JikoXpress customer

Why (and it's not isolation — policies do that either way):

Benefit Why
Independent tuning Different TTLs, audit settings per client
Clean deletion Contract ends → vault secrets disable wiki → gone
Accident containment A wildcard typo in a QBIT policy can't reach a client mount
Auditability Log lines obviously client data
Handover Export one mount, not a filtered subset
Vendor assessments Structural separation is easier to evidence than path prefixes

Clean deletion is the strongest argument. With prefixes you hunt for paths and hope you missed nothing.

Caution: don't pre-mount tenants you don't have.

And note: qbit/ is already your org boundary. qbit/qbitspark/... says it twice.

2.8 The Model, Stated Plainly

Mount = the machine (what kind of engine) Path = the filing system inside it Policy = who may open which drawer

Policies are standalone objects, not nested under engines. Confirmed in the UI — Policies is its own menu.

A policy merely mentions a path as text. You can write a policy for a path that doesn't exist. Delete the mount and the policy sits there harmlessly pointing at nothing.

POLICY  ─────────► names a path (as text)
   ▲
   │ referenced by name
AUTH METHOD ─────► issues tokens carrying that policy
   │
   ▼
TOKEN   ─────────► request arrives
                        ↓
                  Vault checks: does any policy on this
                  token allow this path + this verb?
                        ↓
                  SECRETS ENGINE serves or refuses

Which is why editing a policy takes effect immediately, even on tokens issued yesterday. Nothing is baked into the token.

2.9 Policies

HCL. Deny by default — anything not granted is denied. No deny rule needed; absence is denial.

path "qbit/data/jikoxpress/prod/*" {
  capabilities = ["read"]
}
path "qbit/metadata/jikoxpress/prod/*" {
  capabilities = ["list"]
}

Capabilities:

Capability Verb Meaning
create POST/PUT Create new
read GET Read existing
update POST/PUT Modify existing
delete DELETE Remove
list LIST Enumerate keys (never values)
sudo — Root-protected paths
deny — Explicit denial; overrides all grants

Policies are additive. A token gets the union of all its policies, plus default. To opt out of default, create the token with -no-default-policy.

token_policies="jikoxpress-prod,shared-readonly,monitoring"

Except deny, which is absolute:

path "qbit/data/jikoxpress/prod/snippe-psp" {
  capabilities = ["deny"]
}

Wildcards:

path "qbit/data/+/dev/*" { ... }        # every service's dev
path "identity/oidc/provider/+/authorize" { ... }   # from the default policy

Design rule: write small, single-purpose policies and compose them — jikoxpress-prod-read, jikoxpress-prod-write, shared-read — rather than one giant policy per service.

Reading the built-in default policy taught four things:

  1. Everything is self-scoped (lookup-self, renew-self, revoke-self) — no token can look up another
  2. {{identity.entity.id}} templating resolves per-token at request time — one document, per-caller permissions
  3. + demonstrated in the wild
  4. cubbyhole/* full CRUD — every token gets private storage nobody else can read, not even root. This is the mechanism behind response wrapping.

2.10 The KV v2 Path Gotcha

The single most common Vault mistake. The CLI lies to you — helpfully.

What you type:

vault kv get qbit/jikoxpress/prod/postgres

What goes over HTTP:

GET /v1/qbit/data/jikoxpress/prod/postgres

Vault tells you this in every output: Secret Path: qbit/data/jikoxpress/prod/postgres.

Why the extra segment — KV v2 splits operations into sub-paths:

Real API path Purpose
qbit/data/... the values
qbit/metadata/... version history, max_versions, delete-all
qbit/delete/... soft-delete
qbit/undelete/... restore
qbit/destroy/... permanent wipe

All five appear in normal CLI output once you look:

Success! Data written to: qbit/undelete/jikoxpress/prod/postgres
Success! Data written to: qbit/destroy/jikoxpress/prod/postgres
Metadata Path: qbit/metadata/jikoxpress/prod/postgres

The kv command inserts the right segment for you. Policies have no wrapper.

Write the obvious-looking thing and you get permission denied with no hint why:

path "qbit/jikoxpress/prod/*" { capabilities = ["read"] }   # ← WRONG

The useful consequence — separated paths let you grant operations independently:

path "qbit/data/jikoxpress/prod/*"     { capabilities = ["read"] }
path "qbit/metadata/jikoxpress/prod/*" { capabilities = ["read", "list"] }
path "qbit/destroy/jikoxpress/prod/*"  { capabilities = ["deny"] }

An app that reads secrets has no business destroying them.

KV v1 has none of this — one path, no /data/. Which is exactly why people who learned v1 get caught.

2.11 Path Design Is Policy Design

service/environment above component. Not negotiable.

Right — qbit/jikoxpress/prod/*:

path "qbit/data/jikoxpress/prod/*" { capabilities = ["read"] }

One rule. Covers every component, including ones added next year, automatically.

Wrong — qbit/jikoxpress/db/prod:

path "qbit/data/jikoxpress/db/prod"       { ... }
path "qbit/data/jikoxpress/snippe/prod"   { ... }
path "qbit/data/jikoxpress/redis/prod"    { ... }
...

One rule per component, forever.

The dangerous failure mode: six months later you add a component, forget the policy line, and the app breaks at 2am. Under pressure you "fix" it with qbit/data/jikoxpress/* — which now grants dev and staging too. The isolation is gone and nothing tells you.

The rule: put the thing you cut permissions along highest in the path.

Adopted structure:

qbit/<service>/<environment>/<component>

qbit/jikoxpress/prod/postgres
qbit/jikoxpress/prod/snippe-psp
qbit/nexgate/staging/minio
qbit/nexgate/prod/jwt-signing
qbit/mauzodukani/prod/postgres
qbit/glueemail/prod/mailcow
qbit/textfy/prod/vodacom-smpp
qbit/shared/prod/openai          ← belongs to no single product
qbit/shared/prod/cloudflare

Directories aren't real. There's no mkdir. Write to a deep path and every intermediate level springs into existence, inferred.

2.12 Auth Methods

Method For Notes
Token Bootstrap, break-glass Always enabled, can't be disabled
AppRole Machines Role ID + Secret ID. Primary machine auth.
Userpass Humans Simple; pair with MFA
OIDC / JWT Humans via SSO, CI/CD Keycloak, Authentik, Google Workspace
LDAP Humans via directory If you run one
TLS certificates Machines with mTLS Strong, more setup
Kubernetes Pods If you move to K8s
GitHub Humans / CI Ties access to org membership

Vault does have users — userpass or oidc. Same machinery as AppRole, different door.

2.13 AppRole — The Hotel

The analogy that worked:

Two ways to get a keycard:

Cheating (lab Step 6):
  You (root) → "make me a keycard" → token

Real (lab Step 8):
  App → role_id + secret_id → front desk checks → token

Same keycard. Different route. An app can't use the first — it would need root, which is the problem being solved.

Who has what:

Thing Lives where Secret? Expires?
Policy In Vault No No
Role ID App config, Git Not really No
Secret ID Delivered to app Yes Yes
Token App memory Yes Yes

Order of operations:

ONCE, by admin:
  1. write policy
  2. create approle role, attach policy by name

EVERY APP START:
  3. app sends role_id + secret_id
  4. Vault returns token
  5. app reads secrets

EVERY ~45 MIN:
  6. app renews token

AFTER token_max_ttl:
  7. back to step 3

Steps 3–7 are what "automatic" means. Spring Cloud Vault or Vault Agent does them. You write config, not code.

Why split into two halves? Different security properties, different delivery paths. Neither alone works. Role ID can sit in Git; Secret ID is delivered at runtime and expires. The thing in your repo isn't sufficient to authenticate.

The Secret ID is not stored readably. vault write -f .../secret-id is a write — it generates one, stores a hash, returns plaintext once. No read gets it back. vault list .../secret-id shows accessors (for revocation), never the value.

2.14 Tokens

Type Behaviour
Service Default; server-side; renewable, revocable, supports children
Batch Lightweight encrypted blobs; not stored; can't renew
Root root policy; unlimited; no TTL
Periodic Renews indefinitely within the period
Orphan No parent; survives parent revocation

Tokens form a tree — revoking a parent revokes children.

Root token lookup, annotated (from the lab):

ttl                 0s          ← NOT expired. NO EXPIRY.
expire_time         <nil>       ← never
policies            [root]      ← everything
orphan              true        ← can't be killed via a parent
num_uses            0           ← unlimited
id                  root        ← guessable (dev mode)

AppRole token lookup, same fields:

ttl                 56m16s                    ← counting down
expire_time         2026-08-06T11:02:24Z      ← real
policies            [default jikoxpress-prod] ← scoped
path                auth/approle/login        ← how it was obtained
meta                map[role_name:jikoxpress-prod]  ← attribution
entity_id           bcb55ff4-...              ← identity entity

token_meta_role_name is the audit win. Not "root did it" — "jikoxpress-prod did it." If NexGate's role ever reads a JikoXpress secret, you see it.

2.15 Leases, TTLs, and the Renewal Trap

Precedence: system default → mount tuning → role config → request-time. Each may only shorten.

768h = 32 days = the system default. A mount showing 768h isn't "configured" — it's "nobody set this."

The trap, discovered in the lab: every expiry you add is a renewal mechanism you must build.

token_ttl=1h → solved by renewal (Spring Cloud Vault does it in a background thread). token_max_ttl=4h → renewal can't exceed it; the app must be able to log in again. secret_id_ttl=24h → this is the one that kills services. When it expires, login itself fails. Renewal won't save you.

Four ways to handle Secret ID expiry:

  1. Non-expiring — secret_id_ttl=0, secret_id_num_uses=0. Simplest, weakest. Still far better than root.
  2. Vault Agent — sidecar handles login, renewal, re-auth. Writes a token to a file the app reads. The standard answer.
  3. Long TTL, rotated on deploy — 720h, with Komodo issuing fresh on each deploy.
  4. Response-wrapped at container start — most secure, most moving parts.

2.16 Attacker Analysis

Can a thief renew a stolen token? Yes. Vault can't distinguish them.

But token_max_ttl is the wall. Renewal extends, never resets; max TTL is measured from creation.

token_ttl=1h, token_max_ttl=4h
Steal at hour 0 → renew, renew, renew → at hour 4: DEAD

token_max_ttl is your actual worst-case exposure window. token_ttl is just how often renewal happens.

Hierarchy of theft:

Stolen Exposure
Token only Until token_max_ttl — hours
Token + Secret ID Until secret_id_ttl — days
Role ID + Secret ID Until you rotate
Root token Forever, everything, silently

Which is why the two halves must not live together. If docker inspect yields both, the design bought you nothing.

Where each realistically leaks from:

Role ID Secret ID
Committed compose/Komodo config File on server
Baked into an image (ENV) Env var → docker inspect
.env on the server CI/CD secret store
Kubernetes ConfigMap /proc/<pid>/environ
Shell history
App logs, if the app logs its config at startup

The attack, concretely:

docker inspect jikoxpress | grep -i vault
# VAULT_ROLE_ID=8c3f2a1b-...
# VAULT_SECRET_ID=d4e5f6a7-...

Both halves, one command.

Free hardening not yet used — CIDR binding:

vault write auth/approle/role/jikoxpress-prod \
    secret_id_bound_cidrs="161.97.163.158/32" \
    token_bound_cidrs="161.97.163.158/32"

Stolen credentials become useless anywhere but qbit-spark. The attacker would have to run the attack from your server.

Response wrapping — the strongest link:

vault write -f -wrap-ttl=60s auth/approle/role/jikoxpress-prod/secret-id

Returns a wrapper token, not the Secret ID. Single use. If someone intercepts and unwraps first, your app's unwrap fails — and that failure is your intrusion alarm. Not just secrecy: detection.

Revocation:

vault token revoke <token>
vault write auth/approle/role/X/secret-id-accessor/destroy secret_id_accessor="<acc>"
vault write -f auth/approle/role/X/secret-id      # rotate: kills existing
vault lease revoke -prefix auth/approle/

The honest summary: you cannot prevent a stolen token from being used. You can only bound how long it works and detect that it's happening. Short max TTLs and audit logs are the two levers.

2.17 delete / undelete / destroy / revoke

Two different axes. Constantly conflated.

About the data (KV):

Command Effect Reversible
kv delete Marks a version deleted; data still on disk ✅ undelete
kv undelete Clears the mark —
kv destroy Wipes that version's data; metadata tombstone remains ❌ Never
kv metadata delete Nukes the whole secret, all versions ❌ Never

Recycle bin → shredder → burning the filing cabinet.

About access (tokens/leases):

vault token revoke <token>
vault lease revoke <lease_id>
vault lease revoke -prefix database/

Revoking a dynamic secret lease doesn't just invalidate a record — Vault reaches into the backend and runs DROP ROLE. The user genuinely ceases to exist.

delete/destroy = removing the secret revoke = removing access

Applied to the incident: the OpenAI key leaked. kv destroy removes it from Vault — the stolen key still works. vault token revoke is irrelevant — the attacker never had a Vault token. Only revoking at OpenAI's dashboard stopped it.

That's the limit of static secrets. Vault stores and guards them; it cannot revoke what a third party issued.

The contrast: a database engine credential → vault lease revoke drops the PostgreSQL user directly. Vault issued it, so Vault can kill it. That's the difference between storing secrets and owning them.

KV v2 metadata settings worth using:

vault kv metadata put -max-versions=10 qbit/jikoxpress/prod/postgres

2.18 Audit Devices

Log every request and response. Vault refuses to operate if all enabled audit devices fail to write — fail-closed by design: no audit trail, no service.

vault audit enable file file_path=/vault/logs/audit.log

Sensitive values are HMAC'd; paths and field names are plaintext. Safe to ship to Grafana/Loki. You can compute the HMAC of a known value to search for it — forensics without exposure.

This is what was missing during the incident. With audit logging you could answer: was the OpenAI key ever read from Vault, by which role, from which IP, when?

2.19 Error Codes — Read Them Precisely

Error Meaning
403 permission denied Right verb, path exists, policy says no
404 / No value found Right verb, nothing there
405 unsupported operation Wrong verb — path exists but doesn't accept this method

In Step 5 you'll stare at 403s wondering whether the policy or the path is wrong. Knowing these apart is the difference between five minutes and an hour.

The debugging tool:

vault token capabilities <token> qbit/data/jikoxpress/prod/postgres
# → read
vault token capabilities <token> qbit/data/nexgate/prod/postgres
# → deny

Ask Vault directly instead of guessing.

Deliberate ambiguity: the UI says "You do not have the required permissions or the directory does not exist." It refuses to say which — otherwise an attacker maps your infrastructure by probing. Same reason good login forms say "invalid username or password."

PART 3 — The Lab

Ground rules: nothing touches production. Don't proceed until the ✅ check passes. After each step, close your notes and explain it aloud.

Setup — Docker, not apt

# ~/vault-lab/docker-compose.yml
services:
  vault:
    image: hashicorp/vault:1.20.3
    container_name: vault-lab
    ports:
      - "8200:8200"
    cap_add:
      - IPC_LOCK
    environment:
      VAULT_DEV_ROOT_TOKEN_ID: "root"
      VAULT_DEV_LISTEN_ADDRESS: "0.0.0.0:8200"
      VAULT_ADDR: "http://127.0.0.1:8200"
      VAULT_TOKEN: "root"
    command: server -dev
    volumes:
      - ./policies:/policies

Every line explained:

cap_add: IPC_LOCK — grants the mlock() capability. Vault holds decrypted secrets and the master key in RAM; if the kernel swaps that to disk, secrets get written to the swap file in plaintext. mlock() prevents it. Barely matters in dev mode, but correct from the start so the pattern carries.

VAULT_DEV_ROOT_TOKEN_ID: "root" — overrides the random dev token. Convenience only. Never anywhere real.

VAULT_DEV_LISTEN_ADDRESS: "0.0.0.0:8200" — which interface Vault binds to inside the container. 127.0.0.1 = container-only; Docker's port mapping would have nothing to forward to.

Security note: 0.0.0.0 inside + ports: mapping outside = exposed on every host interface. On a laptop behind a router, fine. On a VPS with no firewall, this exact pattern is how services get accidentally exposed to the internet. Possibly relevant to how the OpenAI key leaked.

command: server -dev — dev mode means: in-memory storage (all data lost on restart), auto-initialized and auto-unsealed, TLS disabled, KV v2 pre-mounted at secret/, root token printed to logs. Dev mode removes every barrier to learning and every security guarantee.

VAULT_ADDR is not optional. Without it the CLI defaults to https://127.0.0.1:8200, tries a TLS handshake against a plain-HTTP dev server, and fails with http: server gave HTTP response to HTTPS client.

Optional alias:

alias v='docker compose exec vault vault'

Step 1 — Run it

docker compose up -d
sleep 3                      # avoid the boot race
docker compose exec vault vault status

✅ Sealed false.

Read the boot logs. They print an unseal key and a root token in plaintext to stdout. In production that combination in a log file is game over.

The observation: the unseal key and cluster ID change on every restart. inmem = brand-new Vault each time. On persistent storage, the unseal key is generated once at initialization, never again.

Question: why would auto-unseal-with-the-key-in-the-logs be unacceptable on vault.qbitspark.com?

Answer: both protections are nullified at once. Already unsealed → encryption gives you nothing, secrets are decrypted in memory now. Key in the logs → even if sealed, the key to reopen it is in docker compose logs. An attacker with log read access needs neither to break encryption nor gather shares.

The trap: Vault's protection is only as good as the separation between the encrypted data and the keys that decrypt it. Dev mode puts them together. So does printing unseal keys to logs. So does storing shares on the same server. So does one person holding all five shares in one folder.

Step 2 — Tokens and your first secret

docker compose exec vault vault token lookup
docker compose exec vault vault kv put secret/hello message="my first secret" owner="kibuti"
docker compose exec vault vault kv get secret/hello
docker compose exec vault vault kv get -field=message secret/hello
docker compose exec vault vault kv get -format=json secret/hello

✅ Explain: what is the token, what is the secret, which is more dangerous to leak?

The answer, refined: it depends on scope. A narrow token is less dangerous than a big secret. A root token is worse than any single secret, because it's the key to all of them.

Leaked Damage
One secret That one system. Bad, bounded.
Scoped token (1h) What that role can read, for an hour
Root token Everything, forever, silently

Which is why the fix isn't "protect tokens harder" — it's make tokens small and short-lived so leaking one barely matters.

-field output is designed for scripting:

DB_PASS=$(vault kv get -field=password qbit/jikoxpress/prod/db)

Step 3 — Mount your own engine

docker compose exec vault vault secrets list        # baseline first
docker compose exec vault vault secrets enable -path=qbit kv-v2
docker compose exec vault vault secrets tune -description="QBIT SPARK application secrets" qbit/

Baseline on a dev server: cubbyhole/, identity/, secret/, sys/. On real Vault, secret/ wouldn't be there — three system mounts and nothing else. Everything useful, you mount yourself.

Explore the engine zoo (mount, look, remove):

docker compose exec vault vault secrets enable database
docker compose exec vault vault secrets enable pki
docker compose exec vault vault secrets enable transit
docker compose exec vault vault secrets enable ssh
docker compose exec vault vault secrets enable totp
docker compose exec vault vault secrets list -detailed

Four things in the -detailed table:

  1. Options = map[version:2] — the only place the table reveals KV v2 vs v1. Both show Plugin kv.
  2. Default TTL/Max TTL = system — inherit the 768h default. On database/ these become the real ceiling.
  3. Seal Wrap true — only on sys/. Extra encryption layer using the seal mechanism, applied to Vault's own control plane.
  4. Running Version — KV shows v0.24.0+builtin while others show v1.20.3+builtin.vault. The KV engine versions independently of Vault core — which is why plugin upgrades and Vault upgrades are different operations.

Dynamic engines are inert until configured:

docker compose exec vault vault list database/config    # No value found
docker compose exec vault vault list transit/keys

KV works the instant you mount it. Dynamic engines need to know what system to talk to and what to create there.

Cleanup — and note the danger:

docker compose exec vault vault secrets disable database

disable destroys all data under that path. No confirmation prompt. On real Vault, vault secrets disable pki deletes your entire certificate authority.

Then build the tree:

docker compose exec vault vault kv put qbit/jikoxpress/prod/postgres username="jx" password="x"
docker compose exec vault vault kv put qbit/jikoxpress/prod/snippe-psp api_key="x" merchant_id="x"
docker compose exec vault vault kv put qbit/jikoxpress/staging/postgres username="jx" password="x"
docker compose exec vault vault kv put qbit/nexgate/prod/postgres username="ng" password="x"
docker compose exec vault vault kv put qbit/nexgate/prod/rabbitmq username="ng" password="x"
docker compose exec vault vault kv put qbit/nexgate/prod/jwt-signing private_key="x" public_key="x"
docker compose exec vault vault kv put qbit/nexgate/staging/minio access_key="x" secret_key="x"
docker compose exec vault vault kv put qbit/mauzodukani/prod/postgres username="md" password="x"
docker compose exec vault vault kv put qbit/glueemail/prod/mailcow admin_user="x" admin_pass="x"
docker compose exec vault vault kv put qbit/textfy/prod/vodacom-smpp system_id="x" password="x"
docker compose exec vault vault kv put qbit/shared/prod/openai api_key="sk-placeholder"
docker compose exec vault vault kv put qbit/shared/prod/cloudflare api_token="x"

All values are x deliberately. Building the habit of never typing a real credential into a lab is worth more than the lab itself.

Walk the tree:

docker compose exec vault vault kv list qbit/
docker compose exec vault vault kv list qbit/jikoxpress/
docker compose exec vault vault kv list qbit/jikoxpress/prod/

Trailing slashes mark directories:

jikoxpress/    ← has children
prod/          ← has children
postgres       ← no slash: an actual secret

list never shows values. Separate capability from read — you can let a junior dev see what exists without reading contents.

Step 4 — Versioning

docker compose exec vault vault kv put qbit/jikoxpress/prod/postgres username="jx" password="NEW-v2"
docker compose exec vault vault kv get qbit/jikoxpress/prod/postgres
docker compose exec vault vault kv get -version=1 qbit/jikoxpress/prod/postgres
docker compose exec vault vault kv metadata get qbit/jikoxpress/prod/postgres

docker compose exec vault vault kv delete qbit/jikoxpress/prod/postgres
docker compose exec vault vault kv undelete -versions=2 qbit/jikoxpress/prod/postgres
docker compose exec vault vault kv destroy -versions=1 qbit/jikoxpress/prod/postgres

Gotcha found live: kv get without -version always fetches the latest. Undeleting v2 while v3 is deleted looks like the undelete failed. Use kv metadata get — it shows every version and its state at once.

✅ Metadata should show: v1 destroyed true, v2 clean, v3 deletion_time set.

Why it matters for rotation:

1. Write new version        ← old still live
2. Restart services
3. Verify healthy
4. THEN revoke at provider
5. Optionally destroy old version

If step 3 fails, kv get -version=<previous> recovers instantly.

The trap: delete looks destructive but isn't. People think a secret is gone when it's fully recoverable by anyone with undelete. To actually remove a leaked credential from Vault, it's destroy.

Step 5 — First policy

mkdir -p ~/vault-lab/policies

cat > ~/vault-lab/policies/jikoxpress-prod.hcl <<'EOF'
path "qbit/data/jikoxpress/prod/*" {
  capabilities = ["read"]
}
path "qbit/metadata/jikoxpress/prod/*" {
  capabilities = ["list"]
}
EOF

docker compose exec vault vault policy write jikoxpress-prod /policies/jikoxpress-prod.hcl
docker compose exec vault vault policy read jikoxpress-prod
docker compose exec vault vault policy list
docker compose exec vault vault policy read default    # read this one properly

Anatomy of the command:

vault policy write jikoxpress-prod /policies/jikoxpress-prod.hcl
                   │                │
                   │                └─ file, INSIDE the container
                   └─ policy name (unrelated to the filename)

The file and the policy name are independent. Keep them matching by convention, not requirement.

/policies/ is the container path. Your volume maps ./policies → /policies.

Once written, the file no longer matters. Vault stores the policy internally; delete the .hcl and it keeps working. The file is source, not runtime — which is why policies belong in Git, giving you version history and review while Vault holds current state.

Step 6 — Prove the denial ⭐

The most important step. A policy tested only for success is untested — you might still be root.

docker compose exec vault vault token create -policy=jikoxpress-prod -ttl=30m
TOK="<paste>"

docker compose exec -e VAULT_TOKEN=$TOK vault vault kv get qbit/jikoxpress/prod/postgres      # ALLOW
docker compose exec -e VAULT_TOKEN=$TOK vault vault kv get qbit/jikoxpress/staging/postgres   # DENY
docker compose exec -e VAULT_TOKEN=$TOK vault vault kv get qbit/nexgate/prod/postgres         # DENY
docker compose exec -e VAULT_TOKEN=$TOK vault vault kv get qbit/shared/prod/openai            # DENY
docker compose exec -e VAULT_TOKEN=$TOK vault vault kv put qbit/jikoxpress/prod/postgres password="hacked"  # DENY

✅ Four 403 permission denied. Those errors are the proof.

The last denial is the subtle one: PUT blocked on a path the token can read. Same path, different verb, different answer. Least privilege at the operation level, not just the path level.

Note: token_policies ["default" "jikoxpress-prod"] — additive, as expected.

Step 7 — Enable AppRole

docker compose exec vault vault auth enable approle

docker compose exec vault vault write auth/approle/role/jikoxpress-prod \
    token_policies="jikoxpress-prod" \
    token_ttl=1h \
    token_max_ttl=4h \
    secret_id_ttl=0

docker compose exec vault vault read auth/approle/role/jikoxpress-prod

Note token_policies="jikoxpress-prod" is just a name. The AppRole never mentions paths — the policy owns those. To change what a service can access, edit the policy; existing tokens pick it up on their next request.

Two empty fields worth filling in production: secret_id_bound_cidrs and token_bound_cidrs.

Lab warning: don't set CIDRs to your production IP in the lab — your requests come from inside a Docker container and you'll lock yourself out.

Step 8 — Log in as a service ⭐

RID=$(docker compose exec -T vault vault read -field=role_id auth/approle/role/jikoxpress-prod/role-id)
SID=$(docker compose exec -T vault vault write -f -field=secret_id auth/approle/role/jikoxpress-prod/secret-id)

docker compose exec -e VAULT_TOKEN= -T vault vault write auth/approle/login \
    role_id="$RID" secret_id="$SID"

-e VAULT_TOKEN= blanks the root token — proving login needs no prior credential.

APPTOK="<paste>"
docker compose exec -e VAULT_TOKEN=$APPTOK vault vault token lookup
docker compose exec -e VAULT_TOKEN=$APPTOK vault vault kv get qbit/jikoxpress/prod/postgres   # ALLOW
docker compose exec -e VAULT_TOKEN=$APPTOK vault vault kv get qbit/nexgate/prod/postgres      # DENY
docker compose exec -e VAULT_TOKEN=$APPTOK vault vault token renew                            # note: positional, not -self

✅ A production secret retrieved with zero root token involvement.

Where does the token come from at all? The CLI resolution order:

  1. -address / explicit flag
  2. VAULT_TOKEN environment variable ← what the compose file sets
  3. ~/.vault-token (written by vault login)

Being inside the container grants nothing. No implicit auth, no localhost exemption, no container trust. Prove it:

docker compose exec -e VAULT_TOKEN= vault vault kv get qbit/jikoxpress/prod/db   # denied

Vault is an HTTP API with a token header. The CLI is a wrapper:

curl -H "X-Vault-Token: root" http://127.0.0.1:8200/v1/qbit/data/jikoxpress/prod/db

The uncomfortable parallel: "what token is being used?" — the answer was root, because it sat in a config file you edited and stopped thinking about. That's the shape of the production problem.

The UI Experiment ⭐

Open http://127.0.0.1:8200.

Sidebar structure — three separate things, three separate menus:

Then: sign out, sign back in with the AppRole token.

What happens: the interface collapses. No Policies. No Access. No Seal Vault. Instead of a browsable tree, a box saying "Type the path of the secret you want to view."

Why: the policy grants list on qbit/metadata/jikoxpress/prod/* but not on qbit/ itself. The UI can't enumerate the top level, so it falls back to asking.

This is what a compromised JikoXpress service looks like from the attacker's side. Two secrets. Not NexGate's JWT keys, not Cloudflare, not Vodacom. Compare to a stolen root token: everything, in a navigable tree.

A design decision it surfaces: apps don't browse — they fetch known paths, so read-only on exact paths is right. Humans with scoped policies find the inability to navigate genuinely annoying, so add list grants deliberately:

path "qbit/metadata"            { capabilities = ["list"] }
path "qbit/metadata/jikoxpress" { capabilities = ["list"] }

Also observed: a yellow banner — "We've stopped auto-renewing your token due to inactivity. It will expire in about 1 hour." with a Renew token button. The UI had been renewing in the background all along. That's vault token renew with a nicer face, and exactly what Spring Cloud Vault does in a thread.

And: Create secret + still renders even without create capability. Clicking fails with 403. UI affordances aren't permissions.

Seed script — because inmem wipes on restart

cat > ~/vault-lab/seed.sh <<'EOF'
#!/usr/bin/env bash
set -euo pipefail
V="docker compose exec -T vault vault"

$V secrets enable -path=qbit kv-v2 2>/dev/null || true
$V secrets tune -description="QBIT SPARK application secrets" qbit/
$V auth enable approle 2>/dev/null || true

$V kv put qbit/jikoxpress/prod/postgres username="jx" password="x"
$V kv put qbit/jikoxpress/prod/snippe-psp api_key="x" merchant_id="x"
$V kv put qbit/jikoxpress/staging/postgres username="jx" password="x"
$V kv put qbit/nexgate/prod/postgres username="ng" password="x"
$V kv put qbit/nexgate/prod/jwt-signing private_key="x" public_key="x"
$V kv put qbit/nexgate/staging/minio access_key="x" secret_key="x"
$V kv put qbit/mauzodukani/prod/postgres username="md" password="x"
$V kv put qbit/glueemail/prod/mailcow admin_user="x" admin_pass="x"
$V kv put qbit/textfy/prod/vodacom-smpp system_id="x" password="x"
$V kv put qbit/shared/prod/openai api_key="sk-placeholder"

$V policy write jikoxpress-prod /policies/jikoxpress-prod.hcl
$V write auth/approle/role/jikoxpress-prod \
    token_policies="jikoxpress-prod" token_ttl=1h token_max_ttl=4h secret_id_ttl=0

echo "✓ seeded"
EOF
chmod +x ~/vault-lab/seed.sh

Wipe and rebuild freely:

docker compose down && docker compose up -d && sleep 3 && ./seed.sh

Step 9 — All services (pending)

#!/usr/bin/env bash
set -euo pipefail
V="docker compose exec -T vault vault"

declare -A SERVICES=(
  [jikoxpress]="dev staging prod"
  [nexgate]="dev staging prod"
  [mauzodukani]="dev prod"
  [glueemail]="prod"
  [textfy]="dev prod"
  [shared]="prod"
)

for svc in "${!SERVICES[@]}"; do
  for env in ${SERVICES[$svc]}; do
    name="${svc}-${env}"

    cat > "policies/${name}.hcl" <<EOF
path "qbit/data/${svc}/${env}/*" {
  capabilities = ["read"]
}
path "qbit/metadata/${svc}/${env}/*" {
  capabilities = ["list"]
}
EOF
    $V policy write "$name" "/policies/${name}.hcl"

    if [ "$env" = "prod" ]; then TTL=1h; MAXTTL=4h; else TTL=4h; MAXTTL=12h; fi

    $V write "auth/approle/role/${name}" \
        token_policies="$name" token_ttl="$TTL" token_max_ttl="$MAXTTL" secret_id_ttl=0
    echo "✓ ${name} (ttl=${TTL})"
  done
done

Production TTLs are shorter because blast radius matters more there.

Step 10 — Isolation verification (pending)

#!/usr/bin/env bash
set -uo pipefail
V="docker compose exec -T vault vault"
PASS=0; FAIL=0

check() {
  local role="$1" path="$2" expect="$3" rid sid tok result
  rid=$($V read -field=role_id "auth/approle/role/${role}/role-id")
  sid=$($V write -f -field=secret_id "auth/approle/role/${role}/secret-id")
  tok=$($V write -field=token auth/approle/login role_id="$rid" secret_id="$sid")

  if docker compose exec -T -e VAULT_TOKEN="$tok" vault vault kv get "$path" >/dev/null 2>&1
  then result="ALLOW"; else result="DENY"; fi

  if [ "$result" = "$expect" ]; then
    echo "  ✓ ${role} → ${path} = ${result}"; PASS=$((PASS+1))
  else
    echo "  ✗ ${role} → ${path} = ${result} (expected ${expect})"; FAIL=$((FAIL+1))
  fi
}

echo "Positive (must ALLOW):"
check jikoxpress-prod  qbit/jikoxpress/prod/postgres  ALLOW
check nexgate-staging  qbit/nexgate/staging/minio     ALLOW

echo "Negative (must DENY):"
check jikoxpress-prod  qbit/nexgate/prod/jwt-signing  DENY
check jikoxpress-prod  qbit/jikoxpress/dev/postgres   DENY
check nexgate-staging  qbit/nexgate/prod/jwt-signing  DENY
check textfy-prod      qbit/jikoxpress/prod/postgres  DENY
check jikoxpress-prod  qbit/shared/prod/openai        DENY

echo; echo "Passed: $PASS  Failed: $FAIL"
[ "$FAIL" -eq 0 ] || exit 1

Keep this script forever. Run after every policy change, in the lab and in production.

Step 11 — Audit logging (pending)

docker compose exec vault vault audit enable file file_path=/vault/logs/audit.log
docker compose exec vault vault audit list -detailed
docker compose exec vault vault kv get qbit/jikoxpress/prod/postgres
docker compose exec vault sh -c 'tail -n 5 /vault/logs/audit.log'

✅ Find your request. Confirm the value is HMAC'd but the path is plaintext.

Step 12 — Dynamic database credentials (pending) ⭐

docker run -d --name vault-lab-pg \
  -e POSTGRES_PASSWORD=rootpass -e POSTGRES_DB=jikoxpress \
  -p 5433:5432 postgres:16

docker compose exec vault vault secrets enable database

docker compose exec vault vault write database/config/jikoxpress-pg \
    plugin_name=postgresql-database-plugin \
    allowed_roles="jikoxpress-app" \
    connection_url="postgresql://{{username}}:{{password}}@host.docker.internal:5433/jikoxpress?sslmode=disable" \
    username="postgres" password="rootpass"

docker compose exec vault vault write database/roles/jikoxpress-app \
    db_name=jikoxpress-pg \
    creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; \
        GRANT SELECT, INSERT, UPDATE, DELETE ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
    default_ttl="1h" max_ttl="24h"

docker compose exec vault vault read database/creds/jikoxpress-app
docker compose exec vault vault read database/creds/jikoxpress-app    # different every time

The contrast that makes it click:

Static (KV):
  vault kv get qbit/jikoxpress/prod/db
  → username: jx_prod, password: s3cr3t
  → same answer, every time, forever

Dynamic (database):
  vault read database/creds/jikoxpress-app
  → v-approle-jikoxpr-x7Kd92 / A1a-8sKdm3nQp   (lease 1h)
  vault read database/creds/jikoxpress-app
  → v-approle-jikoxpr-mN4vB1 / Z9z-2pLwq7Rt    (lease 1h)

Vault runs CREATE ROLE live, then DROP ROLE an hour later. Verify with \du in psql — the users genuinely exist, then genuinely don't.

✅ Generate twice, confirm both in \du, vault lease revoke one, confirm it's dropped.

Steal one of those and you've stolen something that dies on its own. That's the whole game.

PART 4 — Mistakes File

The book's differentiator. Real errors from the real session.

4.1 The invisible leading space

What happened: a \ line continuation with a trailing space made the shell pass mesage=... as one argument. Vault stored the key literally as " mesage".

Why it was confusing: kv get displayed it as mesage — the table view strips the visual space. -field=mesage then said "not present in secret."

Diagnosis:

vault kv get -format=json secret/hello
# "data": { " mesage": "my first secret" }

Lessons:

  1. When a field "exists but isn't found," go straight to -format=json. The pretty output lies about whitespace.
  2. Vault has no schema and no validation. It will happily store PASSWROD, api_ky, mesage. Then your app looks for password, finds nothing, and fails at runtime with a confusing error. Naming conventions are the only defence.

4.2 The boot race

Symptom: dial tcp 127.0.0.1:8200: connect: connection refused immediately after docker compose up -d.

Cause: the container reported "Started" ~1.5s before Vault bound the port.

Lesson: connection refused means nothing is listening — either not booted yet or crashed. Check logs before guessing. docker compose logs vault --tail=40. Reading Vault's startup logs is a skill you'll use constantly.

Fix: sleep 3.

4.3 VAULT_ADDR defaults to HTTPS

Symptom: Error checking seal status: http: server gave HTTP response to HTTPS client

Cause: the CLI defaults to https://127.0.0.1:8200; dev mode serves plain HTTP.

Lesson: VAULT_ADDR isn't optional. The CLI has no idea where your server is or what protocol it speaks.

4.4 405 ≠ 404 ≠ 403

Symptom: vault read database/config → 405 unsupported operation. Expected "not configured."

Cause: database/config only supports LIST. Wrong verb, not missing data.

Lesson: learn the three apart (see §2.19). It's the difference between five minutes and an hour when debugging policies.

4.5 kv get always fetches latest

Symptom: kv undelete -versions=2 succeeded, but kv get still showed a deleted secret.

Cause: kv get without -version fetches the latest version — v3, still deleted. v2 was correctly restored.

Lesson: use kv metadata get to see all versions and states at once.

4.6 token renew -self isn't a flag

Symptom: flag provided but not defined: -self

Fix: vault token renew — bare, renews the token in VAULT_TOKEN.

4.7 Missing exec vault

Symptom: unknown docker command: "compose vault"

Cause: docker compose vault ... instead of docker compose exec vault vault .... The pattern is: first vault = service name, second = binary.

4.8 Expired token = working as designed

Symptom: UI login fails with invalid token.

Cause: token_ttl=1h elapsed. (Or the container restarted — inmem wipes tokens.)

Lesson — the mental shift: the instinct was "something is broken." Nothing broke. Tokens are supposed to stop working. Getting a new one is routine, not recovery. Apps do this automatically; you do it by hand because you're learning.

4.9 The word "secret" means two things

secret/ in a command is a mount path — just a folder name from dev mode. "A secret" in conversation is the sensitive data. Moving to qbit/ removes the ambiguity.

4.10 The /data/ insertion

See §2.10. Expect to hit it. Everyone does.

PART 5 — Humans, Machines & MFA

5.1 Two Populations

Humans Machines
Auth method Userpass, OIDC, LDAP AppRole, TLS cert, K8s
MFA Required N/A
Token TTL 8h 1–4h, auto-renewed
Access Broad but read-mostly, no prod Narrow, exactly what's needed
Audit shows Named individual Named service

Current state: one root token serving both. That's the thing to fix.

5.2 Enabling Users

vault auth enable userpass

vault write auth/userpass/users/kibuti \
    password="..." token_policies="vault-admin" \
    token_ttl=8h token_max_ttl=12h

vault write auth/userpass/users/dev1 \
    password="..." token_policies="dev-engineer" token_ttl=8h

Now the audit log says dev1, not root.

5.3 Developer Policy

# dev-engineer.hcl
# full CRUD on any service's dev
path "qbit/data/+/dev/*" {
  capabilities = ["create", "read", "update", "delete", "list"]
}
path "qbit/metadata/+/dev/*" {
  capabilities = ["read", "list", "delete"]
}

# staging: read only
path "qbit/data/+/staging/*" {
  capabilities = ["read"]
}
path "qbit/metadata/+/staging/*" {
  capabilities = ["read", "list"]
}

# prod: no rule = denied

Note the + — * only globs at the end of a path.

The hard rule: developers should not read production secrets at all. Not "read-only prod" — no prod. If a dev needs the production database password, something is wrong with your deployment process. Apps read prod secrets via AppRole; humans don't need to.

Who writes prod secrets? You, as admin. Or a CI pipeline with a narrow write-only policy.

5.4 The Policy Set

Policy Who Grants
vault-admin You Everything except root-only ops
dev-engineer Devs CRUD on +/dev/*, read +/staging/*
deploy-pipeline Komodo Generate Secret IDs only
<service>-<env> Apps Read one path
auditor Vendor assessments list on metadata, no read on data

The auditor policy is genuinely useful — someone verifies your secrets are properly organised without seeing a single value.

5.5 MFA

Vault Community has no native login MFA. That's Enterprise. Get it from the identity provider instead.

Option A — OIDC with an MFA-capable IdP (best long-term)

vault auth enable oidc
vault write auth/oidc/config \
    oidc_discovery_url="https://auth.qbitspark.com/realms/qbit" \
    oidc_client_id="vault" oidc_client_secret="..." default_role="engineer"

vault write auth/oidc/role/engineer \
    bound_audiences="vault" \
    allowed_redirect_uris="https://vault.qbitspark.com/ui/vault/auth/oidc/oidc/callback" \
    user_claim="email" token_policies="dev-engineer" token_ttl=8h

Keycloak or Authentik self-hosted. Bonus: SSO across your other internal tools.

Option B — network-layer (fastest for you) Traefik + Authelia/Authentik forward-auth in front of the Vault UI. MFA happens before the request reaches Vault. You already run Traefik.

Option C — userpass + TOTP engine. Vault's TOTP engine manages codes but doesn't enforce them on login. Partial only.

Recommendation: B now, A later.

Machines don't get MFA — that's what AppRole + CIDR binding is for.

5.6 Break-Glass

# generate root (needs 3 shares)
vault operator generate-root -init
vault operator generate-root -nonce=<nonce>
vault operator generate-root -decode=<encoded> -otp=<otp>
vault token revoke <root-token>      # IMMEDIATELY after use

# emergency seal
vault operator seal

Document offline, before you need it:

Solo-operator note: Shamir assumes multiple people. Split across locations instead. Never all in one place, never all on one machine. Test the recovery once — an untested scheme is a story you tell yourself.

5.7 Shamir Beyond Vault

Same math, real-world uses:

For personal property:

3-of-5:
  Share 1 → bank safe deposit box (Mbeya)
  Share 2 → family member, different city
  Share 3 → lawyer, sealed envelope with instructions
  Share 4 → encrypted, cloud storage (different provider than your infra)
  Share 5 → physical, hidden at home

Survives fire, theft, one betrayal, one person unreachable.

sudo apt install ssss
echo "my-secret" | ssss-split -t 3 -n 5
ssss-combine -t 3

Steel plates beat paper — paper burns, ink fades. Cryptosteel exists; a hardware store and letter punches do the same job cheaper.

Limitations, honestly: humans lose things; the instructions must survive too (a share is useless if nobody knows what it is); legal systems don't recognise "3-of-5" the way they recognise a notarised will — use it alongside proper instruments; and test it.

PART 6 — Rotation

6.1 Theory

Every credential's compromise probability rises with age and exposure. Rotation bounds the damage window.

Maturity ladder:

  1. ❌ Never rotate — where you were
  2. ⚠️ Manual scheduled — where you're going next
  3. ✅ Automated rotation of static secrets
  4. 🏆 Dynamic secrets — rotation becomes irrelevant because nothing is long-lived

6.2 By Credential Type

Root token — generate → use → revoke. Never leave one alive.

Unseal keys

vault operator rekey -init -key-shares=5 -key-threshold=3

Encryption key (transparent)

vault operator rotate
vault operator key-status

AppRole Secret IDs

vault write -f auth/approle/role/X/secret-id                    # new
vault write auth/approle/role/X/secret-id-accessor/destroy \
    secret_id_accessor="<accessor>"                             # kill old

KV static secrets — order matters

1. Create new credential at the provider
2. vault kv put ...            ← new version; old still readable
3. Restart/reload services
4. Verify healthy
5. ONLY THEN revoke old at the provider
6. Optionally vault kv destroy -versions=<n>

Revoking before services pick up the new value = outage.

Database credentials — move to the database engine; rotation becomes automatic.

6.3 Schedule

Credential Frequency Method
Root token Per use generate → use → revoke
Unseal keys Annually or on suspicion operator rekey
Encryption key Quarterly operator rotate
AppRole Secret IDs Per deploy, or quarterly Komodo, or manual
Service tokens 1–4h Automatic
Third-party API keys Quarterly + on incident Manual, versioned in KV
DB credentials 1h Dynamic engine
TLS certificates 90 days PKI engine
Komodo's own Secret ID Quarterly Manual, by SSH

6.4 Dynamic Migration Order

  1. PostgreSQL — all services; highest value
  2. RabbitMQ — NexGate messaging
  3. MinIO — via AWS engine (S3-compatible STS)
  4. PKI — internal CA for NexGate mTLS
  5. SSH — signed certs for qbit-spark
  6. Transit — JikoXpress financial data

Success measure: a leaked credential is a non-event because it expired an hour ago.

PART 7 — Deployment Architecture

7.1 Where Credentials Live — Per Environment

The one rule: Role ID and Secret ID never in the same place.

Local (developer laptop)

Role ID    → committed in repo (docker-compose.dev.yml)
Secret ID  → .vault/secret-id, gitignored

Each dev generates their own Secret ID. Never committed.

Staging

Role ID    → docker-compose.yml on the staging box (Git fine)
Secret ID  → /etc/vault/<service>-staging-secret-id
             chmod 400, chown root:root

Production

Role ID    → docker-compose.yml (Git fine)
Secret ID  → /etc/vault/<service>-prod-secret-id
             chmod 400, chown root:root

The secrets themselves

In Vault. Nowhere else. No .env, no config file, no baked image layer. Fetched at boot with the AppRole token.

7.2 Never Do This

environment:
  VAULT_ROLE_ID: "..."
  VAULT_SECRET_ID: "..."   # ← docker inspect prints this

One command yields both halves. This is likely how the OpenAI key walked out.

7.3 Do This

# on the server, as admin
sudo mkdir -p /etc/vault
vault write -f -field=secret_id auth/approle/role/bishamba-prod/secret-id \
  | sudo tee /etc/vault/bishamba-prod-secret-id > /dev/null
sudo chmod 400 /etc/vault/bishamba-prod-secret-id
sudo chown root:root /etc/vault/bishamba-prod-secret-id

services:
  bishamba:
    environment:
      VAULT_ROLE_ID: "8c3f2a1b-4d5e-6f70-8192-a3b4c5d6e7f8"
    volumes:
      - /etc/vault/bishamba-prod-secret-id:/run/secrets/secret-id:ro

#!/bin/sh
# entrypoint.sh
export VAULT_SECRET_ID=$(cat /run/secrets/secret-id)
exec java -jar app.jar

Why file-mount beats env var: docker inspect shows only the mount path. docker compose config shows nothing sensitive. Git holds only the Role ID.

Honest limitation: the env var still exists inside the running process, so /proc/<pid>/environ shows it to root. You've removed it from the Docker metadata layer, compose file, and Git — which is where these things actually leak from. Vault Agent removes it entirely.

7.4 Spring Boot Integration

# application-prod.yml
spring:
  cloud:
    vault:
      uri: https://vault.qbitspark.com
      authentication: APPROLE
      app-role:
        role-id: ${VAULT_ROLE_ID}
        secret-id: ${VAULT_SECRET_ID}
      kv:
        enabled: true
        backend: qbit
        default-context: bishamba/prod

@Value("${postgres.username}")
private String dbUser;

@Value("${postgres.password}")
private String dbPass;

Spring Cloud Vault logs in at startup, fetches qbit/bishamba/prod/*, injects into properties, renews the token in a background thread. No credentials in code, no .env in production.

7.5 Three Environments, One Vault

Decision: connectivity isn't a constraint, so all three live on the central Vault.

for env in local staging prod; do
cat > policies/bishamba-${env}.hcl <<EOF
path "qbit/data/bishamba/${env}/*" {
  capabilities = ["read"]
}
path "qbit/metadata/bishamba/${env}/*" {
  capabilities = ["list"]
}
EOF
vault policy write bishamba-${env} policies/bishamba-${env}.hcl
done

# local — loose, no CIDR (devs move around)
vault write auth/approle/role/bishamba-local \
    token_policies="bishamba-local" \
    token_ttl=12h token_max_ttl=24h secret_id_ttl=0

# staging — bound to the staging box
vault write auth/approle/role/bishamba-staging \
    token_policies="bishamba-staging" \
    token_ttl=4h token_max_ttl=12h secret_id_ttl=720h \
    secret_id_bound_cidrs="<staging-ip>/32"

# prod — tightest
vault write auth/approle/role/bishamba-prod \
    token_policies="bishamba-prod" \
    token_ttl=1h token_max_ttl=4h secret_id_ttl=720h \
    secret_id_bound_cidrs="161.97.163.158/32"

Only two lines of app config differ between environments — default-context, and in local, uri.

Two consequences of putting local on the shared Vault:

  1. Local secrets must be fake. Everyone with a local role can read them. Enforce by convention and review.
  2. You need per-developer revocation. Give each dev their own Secret ID, not a shared one. Then you revoke individually and the audit log tells you whose credential was used.
vault write -f auth/approle/role/bishamba-local/secret-id       # rotate all
vault list auth/approle/role/bishamba-local/secret-id           # accessors
vault write auth/approle/role/bishamba-local/secret-id-accessor/destroy \
    secret_id_accessor="<accessor>"                             # kill one

Trade-off recorded (the case for a separate local Vault, should you revisit):

All three central Local separate
Same code path ✅ ✅
Dev needs prod-Vault access ❌ Yes ✅ No
Works offline ❌ ✅
CIDR binding staging/prod only staging/prod only
Onboarding Issue credentials docker compose up
Dev leaves Must revoke Nothing to revoke

7.6 New Project Runbook

Phase 1 — you (admin) set up: decide paths, write policies per environment, create AppRoles, grant devs dev-engineer.

Phase 2 — devs work in dev: they write their own dev secrets, read via the dev AppRole, never touch prod.

Phase 3 — you write real production secrets. The devs who wrote the app never see these values.

Phase 4 — deployment: generate Role ID + Secret ID, place them separately (§7.3).

Phase 5 — app consumes via Spring Cloud Vault (§7.4).

The property this buys: a developer can build, test, and ship the entire application without ever knowing a single production credential. If their laptop is stolen, production is unaffected.

7.7 Komodo Integration

Komodo needs a Vault token to mint Secret IDs. That token becomes the new secret zero, so keep it narrow.

1. A policy that does one thing

# komodo-deployer.hcl
path "auth/approle/role/+/secret-id" {
  capabilities = ["update"]
}

Can mint Secret IDs. Cannot read a single secret, cannot change policies, cannot create roles.

2. Komodo gets its own AppRole — same discipline

vault policy write komodo-deployer policies/komodo-deployer.hcl

vault write auth/approle/role/komodo-deployer \
    token_policies="komodo-deployer" \
    token_ttl=15m token_max_ttl=30m \
    secret_id_ttl=0 \
    secret_id_bound_cidrs="161.97.163.158/32"

3. You place Komodo's credentials by hand, once, over SSH

ssh you@161.97.163.158
sudo mkdir -p /etc/vault

vault read -field=role_id auth/approle/role/komodo-deployer/role-id \
  | sudo tee /etc/vault/komodo-role-id > /dev/null
vault write -f -field=secret_id auth/approle/role/komodo-deployer/secret-id \
  | sudo tee /etc/vault/komodo-secret-id > /dev/null

sudo chmod 400 /etc/vault/komodo-*
sudo chown root:root /etc/vault/komodo-*

services:
  komodo:
    volumes:
      - /etc/vault/komodo-role-id:/run/secrets/role-id:ro
      - /etc/vault/komodo-secret-id:/run/secrets/secret-id:ro

4. Komodo's deploy script

#!/usr/bin/env bash
set -euo pipefail
SERVICE="$1"   # e.g. bishamba-prod

TOKEN=$(vault write -field=token auth/approle/login \
  role_id="$(cat /run/secrets/role-id)" \
  secret_id="$(cat /run/secrets/secret-id)")

VAULT_TOKEN=$TOKEN vault write -f -field=secret_id \
  "auth/approle/role/${SERVICE}/secret-id" \
  > "/etc/vault/${SERVICE}-secret-id"
chmod 400 "/etc/vault/${SERVICE}-secret-id"

docker compose up -d "$SERVICE"

# after health check passes, revoke older Secret IDs
# Komodo's own token dies in 15 minutes regardless

Revoke old Secret IDs after deploy — otherwise every deploy leaves a valid credential behind. Do the destroy after the new container is healthy, not before.

Rotating Komodo's own Secret ID is manual, deliberately:

ssh qbit-spark
vault write -f -field=secret_id auth/approle/role/komodo-deployer/secret-id \
  | sudo tee /etc/vault/komodo-secret-id > /dev/null
docker compose restart komodo

It can't be automated from Komodo — that would let a compromised Komodo renew its own access forever. Keeping it manual bounds the chain.

7.8 Where the Chain Terminates

YOU                    ← SSH key. The actual root of trust.
 │
 └─ place Komodo's role-id + secret-id  (once, by hand)
     │
     └─ KOMODO mints app Secret IDs      (every deploy, 15min tokens)
         │
         └─ APPS log in, read their own secrets  (every start, 1h tokens)

Your SSH key is the bottom. That's the real secret zero for the whole system — which is why it deserves a passphrase, a hardware key if possible, and an entry in the break-glass documentation.

7.9 Secret Zero — The Honest Accounting

"To get a credential for Vault, you need a credential for Vault."

Correct. It never disappears. It shrinks.

Before:

.env on server:
  DB_PASSWORD=real
  SNIPPE_API_KEY=real
  JWT_PRIVATE_KEY=real
  OPENAI_KEY=real
  MINIO_SECRET=real

One file. Everything. Never expires. No log of who read it.

After:

/etc/vault/prod-secret-id:
  d4e5f6a7-b8c9-...

One file. One UUID.

Old .env Stolen Secret ID
What they get Every credential immediately Nothing — need Role ID too
With both halves — One service, one environment
Duration Forever Until rotated
IP-bindable No Yes
Do you find out No Yes — audit log
To revoke Change everything, everywhere One command

The ladder of secret-zero strength:

Approach Secret zero is...
Root token in app config The root token. Worst case.
AppRole Secret ID — scoped, expiring, revocable
Response wrapping One-time wrapper — tamper-evident
TLS cert auth A certificate the machine already has
Cloud/K8s identity The platform vouches — no secret at all

Why "still in Vault" isn't circular: Vault doesn't protect secrets by hiding them. It protects them by (1) encrypting at rest, (2) authenticating every request, (3) authorising per path, (4) expiring everything, (5) logging every access, (6) revoking centrally. A .env file does none of these six.

The realistic goal was never "no secrets anywhere." It's: one small, scoped, expiring, revocable, logged, IP-bound credential instead of a plaintext file containing your entire business.

PART 8 — Production Migration

8.1 Pre-Migration Checklist

8.2 Cutover Order — Lowest Risk First

  1. GlueEmail (smallest blast radius)
  2. Textfy dev
  3. Mauzodukani dev → prod
  4. NexGate staging
  5. JikoXpress dev → staging
  6. NexGate prod
  7. JikoXpress prod (customer payments — last, most carefully)

Per service:

1. Create policy in production Vault
2. Create AppRole
3. Write secrets to new paths (keep old paths intact)
4. Update ONE instance to use AppRole
5. Verify health, logs, functionality
6. Roll out to remaining instances
7. Verify again
8. Remove old path access from any broad policy
9. Delete old secret path
10. Run isolation verification

Never delete the old path until the new one is proven in production.

8.3 Post-Migration Verification

PART 9 — Operations

9.1 Backup & DR

vault operator raft snapshot save vault-$(date +%F).snap
vault operator raft snapshot restore vault-2026-08-05.snap

9.2 Monitoring

Prometheus metrics at /v1/sys/metrics?format=prometheus.

Condition Severity
Vault sealed unexpectedly Critical
Root token used Critical
Audit device failure Critical
Permission-denial spike High — attack or misconfiguration
Token creation anomaly High
Lease count growth Medium — leaked unreleased leases
Secret/cert nearing expiry Medium

Feed into the existing Grafana stack.

9.3 Upgrades

Read the changelog for every version between current and target. Snapshot before upgrading, always. Test on the lab instance first. Never skip major versions. Currently 1.20.3 — patch promptly for security fixes.

PART 10 — The Book

10.1 Concept

Working title: Vault From Zero to Production: Secrets Management for Small Teams and Solo Builders

The gap: HashiCorp's docs are reference material assuming an enterprise platform team. Existing books target large organisations with dedicated security staff. Nobody writes for the solo operator or three-person startup running real production on a handful of VPSes.

Unfair advantages:

  1. Living the problem, not theorising — a real incident, a real recovery
  2. African/emerging-market context: modest budgets, self-hosted VPS, no cloud-native luxury
  3. Proven format and finishing discipline — EM/RF trilogy, and XMPP From Zero to Production in progress
  4. Real multi-product stack (Spring Boot, PostgreSQL, RabbitMQ, MinIO, Docker, Traefik, Komodo), not toy examples

Running example: mirror the MaasaiChat approach. A fictional composite — "Zanzi Systems," a Tanzanian SaaS operator with three products and one engineer — whose secrets infrastructure is built across the book. Reader follows the arc from .env chaos to dynamic secrets.

Series consistency: XMPP From Zero to Production → Vault From Zero to Production.

10.2 Structure

Part I — The Problem (Ch. 1–4)

  1. The $8 Lesson: anatomy of a credential leak
  2. What Is a Secret, Really? (path vs field vs value)
  3. Every Way Secrets Leak — and how each is discovered
  4. The Landscape: choosing your tool

Part II — Fundamentals (Ch. 5–13) 5. Installing and Running Vault (Docker, and why IPC_LOCK) 6. Seal, Unseal, and the Bank in Mbeya 7. Shamir's Secret Sharing — the math, and its life beyond Vault 8. Storage Backends 9. Secrets Engines: static vs dynamic 10. Auth Methods: humans and machines 11. Policies: deny by default 12. The /data/ Gotcha and Other Path Traps 13. Tokens, Leases, and TTLs

Part III — Building It (Ch. 14–22) 14. Designing Your Path Hierarchy (why environment sits above component) 15. Writing Policies That Actually Work 16. Testing Denial — the step everyone skips 17. AppRole in Depth: the hotel, and secret zero 18. Integrating Spring Boot 19. Docker and Compose Integration 20. CI/CD with Komodo 21. Human Access, SSO, and MFA on Community Edition 22. Multi-Tenancy: hosting Vault for clients

Part IV — Going Dynamic (Ch. 23–28) 23. Dynamic Database Credentials 24. PKI: running your own CA 25. Transit: encryption as a service 26. SSH Certificates 27. Response Wrapping and Tamper-Evident Delivery 28. Rotation Strategy End-to-End

Part V — Production (Ch. 29–36) 29. Migrating an Existing System Without Downtime 30. High Availability 31. Backup and Disaster Recovery 32. Monitoring and Alerting 33. Upgrades 34. Incident Response 35. Compliance and Vendor Assessments 36. Cost and Capacity Planning

Part VI — Beyond Vault (Ch. 37–40) 37. OpenBao and the Licensing Question 38. SOPS and GitOps Secrets 39. When Vault Is the Wrong Answer 40. Security Culture as a Solo Operator

Appendices

10.3 Workflow

The discipline: write each chapter as you complete the lab step, while the confusion is fresh. The book's value is remembering what was hard — that memory fades within weeks of mastery.

Per-chapter template:

  1. The problem — a concrete scenario from Zanzi Systems
  2. The concept — theory with a diagram
  3. The lab — commands the reader runs
  4. The verification — how to know it worked, including negative tests
  5. What goes wrong — real errors and causes
  6. Production notes — how this differs at scale
  7. Exercises — solutions in the appendix

Cadence: ~4,000 words/chapter × 40 chapters ≈ 160,000 words. Two chapters/week → first draft in ~5 months.

vault-book/
├── manuscript/
│   ├── part-1/ ... part-6/
│   └── appendices/
├── labs/
│   ├── step-01/ ... step-12/
│   └── scripts/
├── diagrams/
├── mistakes.md          ← the goldmine
└── outline.md

Keep mistakes.md open during every lab session. Every error message, every confusion, every "why doesn't this work." That file is the differentiator — it's what the official docs will never contain.

Progress

Lab

Production homework

Book

QBIT SPARK internal engineering document — v2, 6 August 2026

Revision #1
Created 6 August 2026 16:57:29 by Admin Qbit
Updated 6 August 2026 16:58:14 by Admin Qbit