Skip to main content

NexGate Deployment Guide (New)

The paved road for shipping every NexGate service — architecture, flow, secrets, setup, and operations.

Owner: Kibuti · Organization: NexGate / nexgate-hq · Status: living document · v2

How to read this

Single source of truth for how services are built, deployed, secured, and operated. Five things:

  1. Architecture — what the pieces are and how they fit
  2. Flow — what happens when you push code
  3. Secrets — how credentials reach a running container without ever touching git
  4. Setup — standing it up from zero
  5. Operations — running it day to day, adding services, rolling back, scaling

If you remember one sentence:

You write code, drop one folder, and register one secret role. An image is built once when you push, and pulled once per environment by the reconciler. No credential ever enters git, and no human ever SSHes to deploy.

v2 changes: the secrets model is rebuilt from §5 onward. v1 described AppRole correctly at a high level but left the mechanics undefined — where Role IDs and Secret IDs physically live, how Komodo authenticates to Vault, how the trust chain terminates, and how any of it is verified. It also shipped a policy that granted one service both staging and prod. All fixed below.

Table of contents

  1. Core principles
  2. Architecture
  3. Repository layout
  4. Deploy flow
  5. Secrets model
  6. The trust chain
  7. Setup from scratch
  8. Adding a new service
  9. Deploying a change
  10. Operations
  11. Scaling
  12. Conventions and invariants
  13. Glossary

1. Core principles

The design turns on one shift away from the old Jenkins pipeline:

Stop editing shared state imperatively. Declare desired state, and let a reconciler make reality match it.

Everything follows:

  • Every service has the same shape — onboarding the 40th is no harder than the 4th
  • Each service is self-contained — its own folder describes containers, config, and secret permissions. One service's deploy can never touch another's.
  • Build once, promote by reference — the image tested in staging is the exact artifact in production. Prod never rebuilds.
  • Git is the control plane — approvals are pull requests, history is the audit log, rollback is a revert
  • No credential is ever in git — only pointers to credentials
  • Every access is attributable — audit logs name the service and environment, never "root"
  • Substrate-agnostic — the same repo and flow run on one VPS today and many hosts later

2. Architecture

2.1 The four moving parts

Part What it is Who owns it
Service repos App code + a thin CI workflow (nexgate-hq/<service>) One repo per service
nexgate-infra Declarative desired state — what runs where, at which version One shared repo
Reconciler (Komodo) Watches nexgate-infra, applies changes on the host(s) Installed once per host
Vault Issues every credential any container ever uses Central, vault.qbitspark.com

There is no Jenkins. The reconciler is off-the-shelf — not code you write. It does what Jenkins did on the deploy side (docker compose pull + up -d), driven by git instead of a webhook.

2.2 Runtime layers

  • Base layer — always-on platform: Traefik, Vault Agent, PostgreSQL, Redis, RabbitMQ, MinIO. Rarely redeployed. Deploying an app service never touches it.
  • Service layer — application services (backend, notifications, File Thunder, ai, …), each deployed independently against the base layer.

2.3 Component roles

  • GHCR (ghcr.io/nexgate-hq/*) — image registry. CI pushes; the reconciler pulls.
  • Komodo — the reconciler. Watches nexgate-infra, applies stack changes, dashboard for logs/redeploy/rollback, emits deploy notifications. Also mints per-service Vault Secret IDs at deploy time (§6).
  • Vault (vault.qbitspark.com) — secrets. Services authenticate via AppRole, scoped per service and per environment.
  • Vault Agent — sidecar per host. Handles AppRole login, token renewal, and secret rendering so application containers never handle Vault credentials directly.
  • Traefik — edge router. Auto-discovers by container labels, serves *.nexgate.co over HTTPS (Let's Encrypt).

3. Repository layout

nexgate-infra is the heart. Everything a service needs is in its own folder — the information that used to be crammed into two giant shared compose files, cut apart so each service owns its slice.

nexgate-infra/
├── base/                          # always-on platform layer
│   ├── traefik.yml
│   ├── vault-agent.yml
│   ├── postgres.yml               # optional shared Postgres for light services
│   ├── redis.yml
│   ├── rabbitmq.yml
│   └── minio.yml                  # always-on (File Thunder needs it live)
│
├── services/
│   ├── nexgate-backend/
│   │   ├── service.yml            # compose block: image, labels, depends_on
│   │   ├── config.env             # non-secret config ONLY
│   │   ├── vault-policy.hcl       # which Vault paths this service may read
│   │   ├── vault-agent.hcl        # secret rendering template
│   │   └── tag.env                # IMAGE_TAG=...  ← the one line CI edits
│   ├── notification-server/
│   ├── file-thunder/
│   └── ai/
│
├── policies/                      # generated from services/*/vault-policy.hcl
│   └── .gitkeep
│
├── scripts/
│   ├── vault-bootstrap.sh         # create policies + approles from service folders
│   └── verify-isolation.sh        # prove cross-service access is denied
│
├── staging/
│   └── stack.yml                  # include: base + services (staging tags)
└── prod/
    └── stack.yml                  # include: base + services (prod tags)

Nothing in a service folder is a program. service.yml is a compose snippet. tag.env is one line. vault-policy.hcl is a few lines of permissions. The layout guarantees one service's deploy is isolated to its own files.

Example service folder

# services/ai/service.yml
services:
  ai:
    image: ghcr.io/nexgate-hq/ai-service:${AI_TAG:-latest}
    env_file: [config.env, tag.env]
    networks: [nexgate-prod, proxy]
    depends_on: [postgres, redis, rabbitmq]
    volumes:
      # Vault Agent renders secrets here. The app reads a file.
      # It never sees a role_id or secret_id.
      - vault-secrets-ai:/run/secrets:ro
    labels:
      - traefik.enable=true
      - traefik.http.routers.ai.rule=Host(`ai.nexgate.co`)
      - traefik.http.routers.ai.tls.certresolver=le
    restart: unless-stopped

# services/ai/config.env      ← non-secret ONLY. Committed to git.
SPRING_PROFILES_ACTIVE=prod
LOG_LEVEL=INFO
VAULT_ADDR=https://vault.qbitspark.com

# services/ai/tag.env         ← CI rewrites ONLY this file
AI_TAG=0.1.0

Because tag.env is per-service, deploying ai cannot reset the backend's tag. The shared-.env clobber problem is structurally impossible.

4. Deploy flow

4.1 The one rule: built once, pulled per environment

  • Created (built + pushed to GHCR): once, by CI, when you push. Tied to the code, not the environment.
  • Pulled: by the reconciler, once per environment, when that environment's tag changes.

Staging pulls right after the change. Prod pulls the same image after approval. Prod never rebuilds.

4.2 End-to-end

git push (staging/master)
   │
   ▼
CI (GitHub Actions, reusable workflow)
   • build image via Buildx
   • push to GHCR (version tag + latest)
   • bump services/<svc>/tag.env in nexgate-infra
   │
   ▼
nexgate-infra  ← declarative desired state
   │
   ▼  (prod only: approval gate — PR review)
   │
   ▼
Komodo reconciler on the VPS
   • detects the tag change
   • mints a fresh Secret ID for <svc>-<env>      ← §6
   • writes it to /etc/vault/<svc>-<env>-secret-id (0400 root)
   • docker compose pull <svc> && up -d <svc>     (only that service)
   • on health-check pass: destroys prior Secret IDs
   │
   ▼
Vault Agent authenticates via AppRole → renders secrets to /run/secrets
   │
   ▼
Service container starts, reads secret files
   │
   ▼
Traefik routes it over HTTPS (*.nexgate.co)

No SSH. No Jenkins job. No credential in git.

4.3 Approvals

The gate lives in git — stronger than a button because it is reviewable and permanent.

  • Staging — no gate. Merge to staging, auto-syncs. Fast feedback.
  • Prod — promotion is a pull request bumping the prod tag. The PR is the approval; you see exactly which service and tag are moving. Merge = deploy.

Optional hard gates:

  • A GitHub production environment with a required reviewer (may need a paid plan for private repos; branch protection requiring review before merge to master gives the same gate free)
  • Komodo manual-sync for prod — auto-sync staging, hold prod for a manual click. Closest one-to-one replacement for the old Jenkins button.

Recommended: PR promotion + Komodo manual-sync for prod.

4.4 Notifications

  • CI → webhook posts build + tag-bump status to Telegram
  • Komodo → deploy success/failure to the same channel, reported after a health check rather than "compose exited 0"
  • GitHub → free email/mobile pings on workflow failures and pending approvals
  • Vault audit → alert on root token use, permission-denial spikes, seal events (§10)

Post-deploy runtime health comes from the Grafana stack — §10.

5. Secrets model

This is the section that was thin in v1. It is now the longest, because it's where the real risk lives.

5.1 The rule

No credential is ever in git. Only pointers to credentials.

Artifact In git? Why
config.env (non-secret config) ✅ Yes Log levels, profiles, Vault address
vault-policy.hcl ✅ Yes Permission rules, not credentials
Role ID ✅ Yes Identifies a role. Useless alone.
Secret ID ❌ Never Delivered at deploy, file on host only
Actual secrets ❌ Never Live in Vault, fetched at runtime

5.2 Path hierarchy

v1 used a flat nexgate/data/ai_staging — one segment for service_env. That blocks globbing, so every new component needs a new policy line, and eventually someone "fixes" it with a wildcard that grants too much.

v2 uses a proper hierarchy:

nexgate/<service>/<environment>/<component>

nexgate/ai/staging/postgres
nexgate/ai/prod/postgres
nexgate/ai/prod/openai
nexgate/backend/prod/postgres
nexgate/backend/prod/jwt-signing
nexgate/file-thunder/prod/minio
nexgate/file-thunder/prod/postgres
nexgate/notification-server/prod/rabbitmq
nexgate/shared/prod/cloudflare

Why service/environment above component: that's the boundary policies cut along. One rule covers every component a service will ever have, including ones added next year.

Put the thing you cut permissions along highest in the path.

5.3 The KV v2 path gotcha

The single most common Vault mistake, and it will bite whoever writes the next policy.

What you type: vault kv get nexgate/ai/prod/postgres What goes over HTTP: GET /v1/nexgate/data/ai/prod/postgres

KV v2 splits operations into sub-paths — data/, metadata/, delete/, undelete/, destroy/. The kv CLI inserts the right one for you. Policies have no wrapper.

Write the obvious-looking thing and you get permission denied with no hint why:

path "nexgate/ai/prod/*" { capabilities = ["read"] }   # ← WRONG. Silently fails.

5.4 Policies — one per service and environment

The v1 bug:

# services/ai/vault-policy.hcl  ← BROKEN
path "nexgate/data/ai_staging" { capabilities = ["read"] }
path "nexgate/data/ai_prod"    { capabilities = ["read"] }

One policy granting both environments. If both ai_staging and ai_prod roles attach it, staging can read production secrets — defeating the isolation claimed two paragraphs above it in v1.

Correct — separate file per environment:

# services/ai/vault-policy-staging.hcl
path "nexgate/data/ai/staging/*" {
  capabilities = ["read"]
}
path "nexgate/metadata/ai/staging/*" {
  capabilities = ["list"]
}

# services/ai/vault-policy-prod.hcl
path "nexgate/data/ai/prod/*" {
  capabilities = ["read"]
}
path "nexgate/metadata/ai/prod/*" {
  capabilities = ["list"]
}

Read only. No create, no update, no delete. An app that reads its config has no business rewriting it. Deny-by-default means anything not listed is denied — no explicit deny rule needed.

Prefer generating these from a template so nobody hand-writes the data/ prefix wrong (§7 step 6).

5.5 AppRole per service per environment

# staging — looser, no CIDR binding if staging hosts vary
vault write auth/approle/role/ai-staging \
    token_policies="ai-staging" \
    token_ttl=4h \
    token_max_ttl=12h \
    secret_id_ttl=720h \
    secret_id_num_uses=0 \
    secret_id_bound_cidrs="<staging-host-ip>/32"

# prod — tightest
vault write auth/approle/role/ai-prod \
    token_policies="ai-prod" \
    token_ttl=1h \
    token_max_ttl=4h \
    secret_id_ttl=720h \
    secret_id_num_uses=0 \
    secret_id_bound_cidrs="<prod-host-ip>/32"

token_max_ttl is your actual worst-case exposure window. Renewal extends but never resets — it's measured from creation. A stolen prod token is dead in 4 hours regardless of how often the thief renews it.

secret_id_bound_cidrs is free hardening. A stolen Role ID and Secret ID are useless from anywhere but that host. The attacker would have to run the attack from inside your infrastructure.

5.6 Where the two halves physically live

The rule: Role ID and Secret ID never in the same place. If one docker inspect yields both, the split has bought nothing.

Role ID    → services/<svc>/config.env, committed to git
Secret ID  → /etc/vault/<svc>-<env>-secret-id on the host
             chmod 400, chown root:root
             minted by Komodo at deploy, never in git

Never do this:

environment:
  VAULT_ROLE_ID: "..."
  VAULT_SECRET_ID: "..."   # ← docker inspect prints this in plaintext

One command gives an attacker both halves. This is the single most common way AppRole deployments get compromised.

Do this — file mount:

services:
  vault-agent-ai:
    image: hashicorp/vault:1.20.3
    command: agent -config=/vault/agent.hcl
    environment:
      VAULT_ADDR: https://vault.qbitspark.com
    volumes:
      - ./services/ai/vault-agent.hcl:/vault/agent.hcl:ro
      - /etc/vault/ai-prod-role-id:/vault/role-id:ro
      - /etc/vault/ai-prod-secret-id:/vault/secret-id:ro
      - vault-secrets-ai:/run/secrets

docker inspect now shows only mount paths. docker compose config shows nothing sensitive. Git holds only the Role ID.

5.7 Vault Agent — the app never touches Vault

This is why Agent is worth the extra container. The application never handles a Role ID, a Secret ID, or a Vault token. It reads a file.

# services/ai/vault-agent.hcl
vault {
  address = "https://vault.qbitspark.com"
}

auto_auth {
  method "approle" {
    config = {
      role_id_file_path                   = "/vault/role-id"
      secret_id_file_path                 = "/vault/secret-id"
      remove_secret_id_file_after_reading = false
    }
  }

  sink "file" {
    config = { path = "/run/secrets/token" }
  }
}

template {
  destination = "/run/secrets/application-secrets.properties"
  contents = <<EOT
{{- with secret "nexgate/data/ai/prod/postgres" }}
spring.datasource.username={{ .Data.data.username }}
spring.datasource.password={{ .Data.data.password }}
{{- end }}
{{- with secret "nexgate/data/ai/prod/openai" }}
openai.api-key={{ .Data.data.api_key }}
{{- end }}
EOT
}

Agent logs in, renders the file, renews the token, and re-authenticates when token_max_ttl is hit. The app just reads /run/secrets/application-secrets.properties.

Note {{ .Data.data.username }} — the double .data is KV v2's envelope. Another place the data/ structure surfaces.

Alternative — Spring Cloud Vault, if you'd rather the app authenticate directly:

spring:
  cloud:
    vault:
      uri: https://vault.qbitspark.com
      authentication: APPROLE
      app-role:
        role-id: ${VAULT_ROLE_ID}
        secret-id: ${VAULT_SECRET_ID}
      kv:
        enabled: true
        backend: nexgate
        default-context: ai/prod

Fewer containers, but the app handles the Secret ID. Prefer Agent — it keeps credentials out of application process memory and gives you one renewal implementation instead of one per language.

5.8 Two tiers, unchanged from v1

  • _infra tier — raw container credentials (Postgres/Redis/RabbitMQ/MinIO passwords) needed to boot the base containers
  • App tier — the service's own secrets (JWT, DB creds, third-party API keys)

5.9 Static vs dynamic — the migration target

Everything above stores static secrets in KV. That's the starting point, not the destination.

The limitation, concretely: if a static credential leaks, Vault cannot revoke it. Vault stores and guards it, but it can't kill something a third party issued. A leaked OpenAI key stays valid until someone revokes it at OpenAI.

Dynamic secrets invert this. Vault generates a real PostgreSQL user on demand with a TTL, then drops it:

vault read database/creds/ai-prod
# → v-approle-ai-prod-x7Kd92 / A1a-8sKdm3nQp   (lease 1h)
vault read database/creds/ai-prod
# → v-approle-ai-prod-mN4vB1 / Z9z-2pLwq7Rt    (lease 1h)

Different every time. vault lease revoke runs DROP ROLE against Postgres directly. A stolen credential dies on its own.

Migration order:

  1. PostgreSQL — all services, highest value
  2. RabbitMQ — notification-server, backend
  3. MinIO — via the AWS engine (S3-compatible STS)
  4. PKI — internal CA for mTLS between services
  5. Transit — encryption-as-a-service for sensitive payloads

What must stay static: third-party API keys (OpenAI, Cloudflare, payment providers). Vault didn't issue them, so it can't rotate them. These get quarterly manual rotation and versioned storage in KV.

Success measure: a leaked database credential is a non-event because it expired an hour ago.

6. The trust chain

v1 said "services authenticate via AppRole" and stopped there. That leaves the important question unanswered: who gives Komodo permission to mint Secret IDs, and who gives that to Komodo?

6.1 Komodo's Vault credentials

Komodo needs a Vault credential to mint per-service Secret IDs. That credential becomes the new secret zero, so it must be narrow.

# policies/komodo-deployer.hcl
path "auth/approle/role/+/secret-id" {
  capabilities = ["update"]
}
path "auth/approle/role/+/secret-id-accessor/destroy" {
  capabilities = ["update"]
}

Can mint and destroy Secret IDs. Cannot read a single secret. Cannot change a policy. Cannot create a role. If it leaks, an attacker can generate Secret IDs — but must guess role names, and every generation appears in the audit log.

vault write auth/approle/role/komodo-deployer \
    token_policies="komodo-deployer" \
    token_ttl=15m \
    token_max_ttl=30m \
    secret_id_ttl=0 \
    secret_id_bound_cidrs="<host-ip>/32"

15-minute tokens. Komodo logs in per deploy, uses the token, it dies.

6.2 Placing Komodo's credentials — the one manual step

ssh <host>
sudo mkdir -p /etc/vault

vault read -field=role_id auth/approle/role/komodo-deployer/role-id \
  | sudo tee /etc/vault/komodo-role-id > /dev/null
vault write -f -field=secret_id auth/approle/role/komodo-deployer/secret-id \
  | sudo tee /etc/vault/komodo-secret-id > /dev/null

sudo chmod 400 /etc/vault/komodo-*
sudo chown root:root /etc/vault/komodo-*

This is the only credential a human ever places by hand, and only once per host.

6.3 Komodo's deploy hook

#!/usr/bin/env bash
# scripts/deploy-service.sh — run by Komodo on tag change
set -euo pipefail

SERVICE="$1"        # e.g. ai
ENV="$2"            # staging | prod
ROLE="${SERVICE}-${ENV}"

# 1. Komodo authenticates (15-minute token)
TOKEN=$(vault write -field=token auth/approle/login \
  role_id="$(cat /etc/vault/komodo-role-id)" \
  secret_id="$(cat /etc/vault/komodo-secret-id)")

# 2. Record existing Secret ID accessors so we can revoke them after
OLD_ACCESSORS=$(VAULT_TOKEN=$TOKEN vault list -format=json \
  "auth/approle/role/${ROLE}/secret-id" 2>/dev/null || echo '[]')

# 3. Mint a fresh Secret ID for this deploy
VAULT_TOKEN=$TOKEN vault write -f -field=secret_id \
  "auth/approle/role/${ROLE}/secret-id" \
  > "/etc/vault/${ROLE}-secret-id"
chmod 400 "/etc/vault/${ROLE}-secret-id"
chown root:root "/etc/vault/${ROLE}-secret-id"

# 4. Deploy
docker compose pull "$SERVICE"
docker compose up -d "$SERVICE"

# 5. Wait for health, THEN revoke the old Secret IDs
if ./scripts/wait-for-health.sh "$SERVICE" 120; then
  echo "$OLD_ACCESSORS" | jq -r '.[]' | while read -r acc; do
    VAULT_TOKEN=$TOKEN vault write \
      "auth/approle/role/${ROLE}/secret-id-accessor/destroy" \
      secret_id_accessor="$acc"
  done
  echo "✓ ${ROLE} deployed, old Secret IDs revoked"
else
  echo "✗ ${ROLE} health check failed — old Secret IDs left intact for rollback"
  exit 1
fi

# Komodo's own token expires in 15 minutes regardless

Order matters. Revoke old Secret IDs after the new container is healthy, never before — otherwise a failed deploy leaves you unable to roll back.

6.4 The chain, complete

YOU                          ← SSH key. The actual root of trust.
 │
 └─ place Komodo's role-id + secret-id     (once per host, by hand)
     │
     └─ KOMODO mints service Secret IDs    (every deploy, 15-min tokens)
         │
         └─ VAULT AGENT logs in per service (1–4h tokens, auto-renewed)
             │
             └─ APP reads rendered secret files (never touches Vault)

Four levels, each narrower than the one above. Only the top requires a human, and only once per host.

Your SSH key is the bottom of the chain — the real secret zero for the entire system. It deserves a passphrase, a hardware key if possible, and an entry in the break-glass documentation.

6.5 Rotating Komodo's own Secret ID

Manual, deliberately.

ssh <host>
vault write -f -field=secret_id auth/approle/role/komodo-deployer/secret-id \
  | sudo tee /etc/vault/komodo-secret-id > /dev/null
docker compose restart komodo

It cannot be automated from Komodo — that would let a compromised Komodo renew its own access indefinitely. Keeping it manual bounds the chain. Quarterly, or on any suspicion.

6.6 Secret zero — the honest accounting

You cannot bootstrap trust from nothing. Every auth system has a first credential. The question is only what it is and how bad a leak would be.

Before (shared .env):

DB_PASSWORD=real
JWT_PRIVATE_KEY=real
OPENAI_KEY=real
MINIO_SECRET=real

One file. Everything. Never expires. No log of who read it.

After:

/etc/vault/komodo-secret-id:  d4e5f6a7-b8c9-...

One file. One UUID.

Old .env Stolen Komodo Secret ID
What they get Every credential immediately Nothing — needs the Role ID too
With both halves — Can mint Secret IDs, cannot read secrets
Duration Forever Until rotated
IP-bindable No Yes
Do you find out No Yes — audit log
To revoke Change everything everywhere One command

Vault doesn't protect secrets by hiding them. It protects them by encrypting at rest, authenticating every request, authorising per path, expiring everything, logging every access, and revoking centrally. A .env file does none of these six.

7. Setup from scratch

Bootstrap order matters.

Step 1 — Provision hosts

Two Ubuntu VPS: staging and prod (separate boxes — §11). Docker Engine + Compose plugin. UFW allowing 22, 80, 443 only.

Also: confirm the Docker daemon API is not exposed. curl http://<host>:2375/version from outside must fail. An exposed Docker socket makes every other control here irrelevant.

Step 2 — Private networking (do it early)

WireGuard mesh (or provider private network) between hosts, even with one host each. Makes future splitting a one-line change instead of a migration. Services address each other by name / internal DNS, never hardcoded localhost.

Step 3 — Docker networks

docker network create proxy
docker network create nexgate-staging     # on staging host
docker network create nexgate-prod        # on prod host

Step 4 — Deploy the base layer

Traefik (Let's Encrypt resolver le), PostgreSQL, Redis, RabbitMQ, MinIO. MinIO always-on from day one.

Vault Agent comes after Vault is configured — step 6.

Step 5 — Vault: audit logging first

Before anything else. Non-disruptive, one command, and it means everything from here forward is attributable.

vault audit enable file file_path=/vault/logs/audit.log
vault audit list -detailed

Vault refuses to serve requests if all audit devices fail to write — deliberate fail-closed design. Ship the log to Loki; sensitive values are HMAC'd while paths and field names stay plaintext, so it's safe to forward.

Also confirm, and write down offline:

  • Which storage backend is vault.qbitspark.com using? (determines backup strategy)
  • Where are the 5 unseal key shares, physically?
  • Are they split across locations, or all in one place?
  • Has a snapshot restore ever been tested?

Solo operator: Shamir's 3-of-5 assumes multiple people. Split across locations instead — bank box, trusted party, encrypted offsite, hidden at home. An untested recovery scheme is a story you tell yourself.

Step 6 — Vault: mount, auth, policies, roles

vault secrets enable -path=nexgate kv-v2
vault secrets tune -description="NexGate application secrets" nexgate/
vault auth enable approle

Then run the bootstrap script (§7 step 8) which generates policies and roles from the service folders.

Retention worth setting per path:

vault kv metadata put -max-versions=10 nexgate/ai/prod/postgres

Default is unlimited history — every rotation accumulates forever and old passwords stay readable.

Step 7 — Komodo

Install per host. Point at nexgate-infra. Staging = auto-sync, prod = manual-sync.

Place Komodo's Vault credentials by hand (§6.2). This is the one manual credential step.

Step 8 — Bootstrap script

#!/usr/bin/env bash
# scripts/vault-bootstrap.sh
# Generates policies and AppRoles from services/*/ folders.
# Idempotent — safe to re-run.
set -euo pipefail

STAGING_CIDR="${STAGING_CIDR:?set STAGING_CIDR}"
PROD_CIDR="${PROD_CIDR:?set PROD_CIDR}"

for dir in services/*/; do
  svc=$(basename "$dir")

  for env in staging prod; do
    name="${svc}-${env}"

    # policy — generated, so the data/ prefix is never hand-typed wrong
    cat > "policies/${name}.hcl" <<EOF
path "nexgate/data/${svc}/${env}/*" {
  capabilities = ["read"]
}
path "nexgate/metadata/${svc}/${env}/*" {
  capabilities = ["list"]
}
EOF
    vault policy write "$name" "policies/${name}.hcl"

    if [ "$env" = "prod" ]; then
      TTL=1h; MAXTTL=4h; CIDR="$PROD_CIDR"
    else
      TTL=4h; MAXTTL=12h; CIDR="$STAGING_CIDR"
    fi

    vault write "auth/approle/role/${name}" \
      token_policies="$name" \
      token_ttl="$TTL" \
      token_max_ttl="$MAXTTL" \
      secret_id_ttl=720h \
      secret_id_num_uses=0 \
      secret_id_bound_cidrs="$CIDR"

    # Role ID is not secret — emit for config.env
    echo "${name} role_id: $(vault read -field=role_id auth/approle/role/${name}/role-id)"
  done
done

Step 9 — Verify isolation

Do not skip this. A policy tested only for success is untested.

#!/usr/bin/env bash
# scripts/verify-isolation.sh
set -uo pipefail
PASS=0; FAIL=0

check() {
  local role="$1" path="$2" expect="$3" rid sid tok result
  rid=$(vault read -field=role_id "auth/approle/role/${role}/role-id")
  sid=$(vault write -f -field=secret_id "auth/approle/role/${role}/secret-id")
  tok=$(vault write -field=token auth/approle/login role_id="$rid" secret_id="$sid")

  if VAULT_TOKEN="$tok" vault kv get "$path" >/dev/null 2>&1
  then result="ALLOW"; else result="DENY"; fi

  if [ "$result" = "$expect" ]; then
    echo "  ✓ ${role} → ${path} = ${result}"; PASS=$((PASS+1))
  else
    echo "  ✗ ${role} → ${path} = ${result} (expected ${expect})"; FAIL=$((FAIL+1))
  fi
}

echo "Positive (must ALLOW):"
check ai-prod          nexgate/ai/prod/postgres            ALLOW
check backend-staging  nexgate/backend/staging/postgres    ALLOW

echo "Negative (must DENY):"
check ai-prod          nexgate/backend/prod/jwt-signing    DENY
check ai-prod          nexgate/ai/staging/postgres         DENY   # ← the v1 bug
check ai-staging       nexgate/ai/prod/postgres            DENY   # ← the v1 bug
check backend-staging  nexgate/backend/prod/postgres       DENY
check file-thunder-prod nexgate/ai/prod/openai             DENY

echo; echo "Passed: $PASS  Failed: $FAIL"
[ "$FAIL" -eq 0 ] || exit 1

The two marked lines are exactly what v1's policy would have failed. Run this after every policy change, forever. Wire it into CI on nexgate-infra.

Step 10 — service-template + reusable CI

Template repo with Dockerfile, workflow_call caller, /health + /metrics wired. One reusable GitHub Actions workflow in the org that all services call.

Step 11 — Onboard the first service

Run the §8 checklist end to end to validate the chain.

Step 12 — Notifications and observability

Telegram webhook in the reusable workflow. Komodo deploy alerts. Grafana stack on a dedicated ops VPS — §10.

8. Adding a new service

Assuming base, Komodo, template, and reusable workflow exist — onboarding ai:

  1. Generate the repo — nexgate-hq/ai-service from service-template
  2. Write the code — the only real work; everything below is config
  3. First push to staging → CI builds ai:0.1.0 → GHCR (created)
  4. Add one folder — services/ai/ with service.yml, config.env, vault-agent.hcl, tag.env
  5. Write the secrets to nexgate/ai/staging/* and nexgate/ai/prod/*
  6. Run vault-bootstrap.sh — generates both policies and both AppRoles
  7. Run verify-isolation.sh — add positive and negative cases for the new service
  8. Put the Role IDs in config.env (safe to commit)
  9. Commit → Komodo syncs staging → mints a Secret ID → pulls ai:0.1.0 → live at ai.staging.nexgate.co (pulled #1) — validate
  10. Promote to prod — PR bumping the prod tag. Merge → Komodo syncs prod → pulls the same image (pulled #2, identical)

The wiring (steps 4–8) is the entire cost — the old four-place scavenger hunt collapsed to one folder + one script run.

File Thunder note: additionally uncomment MinIO in base/ and declare its own Postgres (5433) and a ClamAV container in its service.yml. One-time, and entirely within File Thunder's folder + the base layer.

9. Deploying a change

  1. Edit code, push to staging
  2. CI builds <svc>:<version>, pushes to GHCR, bumps the staging tag (created)
  3. Komodo mints a fresh Secret ID, pulls, restarts only that service in staging (pulled #1). Automatic, no gate.
  4. On health-check pass, Komodo revokes the previous Secret IDs
  5. Validate in staging
  6. Open a PR bumping the prod tag. The PR is the gate.
  7. Merge → Komodo repeats for prod with the same image (pulled #2, identical)

Prod runs the byte-for-byte artifact you tested.

10. Operations

Rollback

Revert the tag-bump commit in nexgate-infra, or use Komodo's "redeploy previous." Images are immutable and promoted by reference, so rollback is re-pointing a tag at the last-good image.

Secret ID note: because old Secret IDs are only revoked after a health check passes, a failed deploy leaves the previous credential valid. Rollback works without re-minting.

Rough edges fixed by design

v1 problem v2 fix
Shared .env clobber Per-service tag.env
One VAULT_TOKEN for both envs AppRole per service and environment
Policy granting staging + prod Separate policy file per environment
Flat service_env path Hierarchical service/env/component
Secret ID placement undefined File on host, 0400, minted by Komodo
Komodo's Vault auth undefined Narrow AppRole, §6
No isolation testing verify-isolation.sh in CI
MinIO commented out Always-on in base/

Secret rotation

Credential Frequency Method
Service Secret IDs Every deploy Komodo, automatic
Service tokens 1–4h Vault Agent, automatic
Komodo's Secret ID Quarterly Manual, by SSH (§6.5)
Third-party API keys Quarterly + on incident Manual, versioned in KV
DB credentials 1h Dynamic engine (target state)
Vault encryption key Quarterly vault operator rotate
Unseal keys Annually or on suspicion vault operator rekey
Root token Per use Generate → use → revoke

Static secret rotation — order matters:

1. Create the new credential at the provider
2. vault kv put ...              ← new version; old still readable
3. Redeploy / let Agent re-render
4. Verify healthy
5. ONLY THEN revoke the old one at the provider
6. Optionally vault kv destroy -versions=<n>

Revoking before services pick up the new value causes an outage. KV v2 versioning is what makes step 5 safe — if step 4 fails, the previous version is one command away.

Backups

  • Postgres — scheduled pg_dump per database, shipped off-host
  • MinIO — bucket replication or scheduled sync off-site
  • Vault — vault operator raft snapshot save, encrypted, useless without unseal keys. Back up both, separately. Test the restore quarterly into a scratch instance.
  • nexgate-infra — it's git; already versioned

Monitoring — the Grafana stack

Runs on a dedicated ops VPS outside dev/staging/prod — never on a box it watches. One ops host watches every environment; targets labelled by env (env=dev|staging|prod), dashboards filter on that label. Joins the WireGuard mesh for internal IPs; hits public endpoints through Traefik.

Component Question it answers
Prometheus Metrics over time + alerting (scrapes /metrics)
Loki Log search (services log JSON to stdout)
Tempo Distributed traces
Grafana Dashboards, single view
Alertmanager Routes alerts → Telegram / email

Collectors: node_exporter (host CPU/RAM/disk), cAdvisor (per-container resources), blackbox_exporter (uptime probes against /health — no separate Uptime Kuma needed), Grafana Alloy or Promtail (stdout JSON → Loki).

Vault-specific alerts — Vault exposes Prometheus metrics at /v1/sys/metrics?format=prometheus:

Condition Severity
Vault sealed unexpectedly Critical
Root token used Critical
Audit device failure Critical
Permission-denial spike High — attack or misconfiguration
Token creation anomaly High
Lease count growth Medium — leaked unreleased leases
Secret/cert nearing expiry Medium

Root token use should page you. In steady state it should never happen.

Because every service exposes /metrics and logs JSON to stdout (§12), adding a service to monitoring is a scrape target + a health probe, both env-labelled.

Komodo vs Grafana — you need both

Komodo Grafana stack
Watches Deployments (control plane) Runtime health (observability)
Tells you Did it deploy, what tag runs where Is it up, slow, erroring; resource use; history
Lets you Redeploy, roll back Alert, investigate — not deploy

Komodo reports "deploy succeeded" the moment a container starts, even if it then crash-loops or serves 500s under load — Grafana catches that. Grafana can tell you a service is unhealthy but can't redeploy — that's Komodo. One tells you something's wrong; the other lets you do something about it.

Komodo shows basic per-host CPU/RAM/disk, so it doubles as light resource monitoring — but no metrics history, no log search, no tracing. Run both on the ops VPS; both alert to Telegram.

Incident response

Suspected Vault compromise:

vault operator seal      # everything slams shut, instantly

Costs an outage; the attacker gets ciphertext. Recovery needs 3 of 5 unseal shares — which is why you must know where they are before the emergency.

Suspected service credential compromise:

vault list auth/approle/role/<svc>-<env>/secret-id
vault write auth/approle/role/<svc>-<env>/secret-id-accessor/destroy \
    secret_id_accessor="<accessor>"
vault lease revoke -prefix auth/approle/

Redeploy to mint a fresh one.

Note the limit: revoking in Vault kills Vault access. It does not invalidate a third-party API key an attacker already read — that must be revoked at the provider. This is precisely why dynamic secrets matter (§5.9).

Upgrading base images

Postgres/Redis/etc. are pinned per service in service.yml (§11.4). Bump the pin, test in staging, promote.

Vault upgrades: read the changelog for every version between current and target. Snapshot first, always. Test on a scratch instance. Never skip major versions.

11. Scaling

11.1 Two kinds of scale

  • Many services (control plane) — scales freely. The 40th onboards like the 4th.
  • Heavy load (data plane) — single-host Compose has a ceiling. Climb the ladder without rebuilding the pattern.

11.2 The ladder

Rung Setup When
1 Single VPS, Compose + Komodo Now
2 Bigger VPS Vertical scale buys runway
3 Services split across hosts (Komodo multi-host) When a service gets heavy
4 Swarm or K3s Replicas + failover, much later

The repo and onboarding journey are identical at every rung. Moving up is a config change (which host a folder runs on), plus — at rung 3 — the private network from §7 step 2.

Secrets note: at rung 3, each host needs its own Komodo credentials placed by hand, and secret_id_bound_cidrs must list every host that runs a given service. At rung 4, switch to the Kubernetes auth method — pods authenticate via their service account and the Secret ID disappears entirely.

11.3 VPS topology

  • Split by environment (staging + prod VPS) — do this now. A safety baseline, not a scaling choice. It's also what makes secret_id_bound_cidrs meaningful: staging credentials physically cannot be used from the prod host and vice versa.
  • Split by service — a growth move. When File Thunder's transcoding starves the API, move it and its workers to their own host.
  • Every VPS runs everything — careful. Fine for stateless services behind Traefik. A trap for stateful ones: three MinIOs / Postgres / RabbitMQs are three diverging databases, not one system. Only works as true active-active with clustered datastores (Postgres replication, MinIO distributed mode, RabbitMQ quorum queues, Redis Sentinel) on Swarm/K3s — a distant milestone, never a casual switch.

Sequence: env-split today → service-split as load appears → all-have-all as a distant HA project.

11.4 Different versions per service

Postgres/Redis/MinIO are upstream public images pinned by tag — unrelated to "build once," which is only about your app image. Each service's datastore is a separate container in its own folder:

# services/file-thunder/service.yml
  ft-postgres:
    image: postgres:17
    volumes: [ft-pgdata:/var/lib/postgresql/data]

# services/backend/service.yml
  backend-postgres:
    image: postgres:15
    volumes: [backend-pgdata:/var/lib/postgresql/data]

Different names, volumes, versions — no collision. Shared vs dedicated is a per-service call: shared base Postgres saves RAM for light services; dedicated gives version/extension freedom and isolation (why File Thunder has its own). Never a base-layer constraint.

12. Conventions and invariants

Don't break these.

Deployment

  • Everything runs in a container — no exceptions. Every service and every base component. A static frontend ships as a container (nginx serving built assets), not a special case. This is what makes the model uniform.
  • One service = one repo = one folder in nexgate-infra
  • One deploy path for all service types. Backend, frontend, and AI share the same tail (push GHCR → bump tag → Komodo deploys). Only the build head differs per language — flavors of one reusable workflow, never separate deploy systems.
  • CI writes only services/<svc>/tag.env — never a shared file
  • Prod is promoted by reference, never rebuilt
  • Staging auto-deploys; prod is gated by a PR
  • The base layer is stable — deploying an app service never touches it

Secrets

  • No credential is ever committed to git. Role IDs are, because they're not credentials on their own.
  • Role ID and Secret ID never live in the same place
  • Secret IDs are files (0400 root), never environment variables — docker inspect must not reveal one
  • One policy per service and environment. Never one policy spanning both.
  • Policies are generated, not hand-written — the data/ prefix is too easy to get wrong
  • Policies are read-only for application roles. No create, update, or delete.
  • verify-isolation.sh runs on every policy change, and includes negative tests
  • Applications never handle Vault credentials — Vault Agent renders files
  • Audit logging is always on. No exceptions, no "temporarily off to debug."
  • Root token is generated, used, and revoked. Never stored, never used for routine work.

Runtime

  • Services address each other by name / internal DNS, never hardcoded localhost
  • Every service exposes /health + /metrics and logs JSON to stdout (the service contract)

13. Glossary

  • Reconciler — Komodo. Watches nexgate-infra and applies changes on hosts. Replaces Jenkins' deploy role. Not code you write.
  • Desired state — what nexgate-infra declares should be running
  • Base layer — always-on platform containers
  • Service layer — application services deployed against the base layer
  • AppRole — Vault auth method for machines. Two halves: Role ID (identity, semi-public) + Secret ID (credential, sensitive).
  • Role ID — identifies which AppRole. Committed to git. Useless alone.
  • Secret ID — proves the identity. Never in git. File on host, 0400, minted per deploy.
  • Vault Agent — sidecar that performs AppRole login, token renewal, and secret rendering so applications never handle Vault credentials
  • Secret zero — the first credential in any trust chain, which must be placed by a human. Here: Komodo's Secret ID, placed once per host over SSH.
  • Static secret — stored in Vault's KV engine. Vault guards it but cannot revoke it upstream.
  • Dynamic secret — generated by Vault on demand with a TTL. Vault issued it, so Vault can revoke it.
  • Promote by reference — moving the exact same image artifact from staging to prod, rather than rebuilding
  • Paved road — the standardized template + workflow + folder pattern that makes onboarding near-free

Migration checklist from v1

  • Split every combined policy into per-environment files — the v1 policy granted staging and prod together
  • Migrate paths from flat nexgate/<svc>_<env> to nexgate/<svc>/<env>/<component>
  • Enable audit logging — do this first, it's non-disruptive
  • Locate and verify the unseal key shares; confirm they're split across locations
  • Test a Vault snapshot restore into a scratch instance
  • Create the komodo-deployer policy and AppRole
  • Place Komodo's credentials on each host by hand
  • Move Secret IDs out of any environment variables and into 0400 files
  • Add secret_id_bound_cidrs to every staging and prod role
  • Write and run verify-isolation.sh; wire it into CI
  • Set max-versions on KV metadata paths
  • Add Vault alerts (root token use, denial spikes, seal events) to Alertmanager
  • Audit every running container: docker inspect <c> | grep -iE "secret|token|password|key"
  • Confirm the Docker daemon API is not exposed on any host
  • Plan the dynamic-secrets migration, starting with PostgreSQL

Versioned in nexgate-infra. Update whenever the architecture changes — it is the guide the whole organization relies on.

No comments to display

Back to top