NexGate Deployment Guide (New)

The paved road for shipping every NexGate service — architecture, flow, secrets, setup, and maintenance.operations.

Owner: Kibuti · Organization: NexGate / nexgate-hq · Status: living document · v2

How to read this

This is the singleSingle source of truth for how services are built, deployed, secured, and operatedoperated. across every NexGate service. It covers fourFive things:

1. Architecture — what the pieces are and how they fit.fit
2. Flow — what happens when you push code,code
and
3. Secrets — how credentials reach a changerunning reachescontainer production.without ever touching git
4. Setup — howstanding to stand the whole thingit up from zero.zero
5. MaintenanceOperations — how to runrunning it day to day, addadding services, rollrolling back, and scale.scaling

If you only remember one sentence, remember this:sentence:

You write code, drop one folder, and register one secret role. An image is built once when you push, and pulled once per environment by the reconciler. NothingNo elsecredential isever touched.enters git, and no human ever SSHes to deploy.

v2 changes: the secrets model is rebuilt from §5 onward. v1 described AppRole correctly at a high level but left the mechanics undefined — where Role IDs and Secret IDs physically live, how Komodo authenticates to Vault, how the trust chain terminates, and how any of it is verified. It also shipped a policy that granted one service both staging and prod. All fixed below.

Table of contents

1. Core principles
2. Architecture
3. The repositoryRepository layout
4. The deployDeploy flow
5. Secrets model
6. The trust chain
7. Setup from scratch
8. Adding a new service
9. Deploying a change
10. Maintenance and operationsOperations
11. Scaling
12. Conventions and invariants
13. Glossary

1. Core principles

The whole design turns on one shift away from the old Jenkins-drivenJenkins pipeline:

Stop editing shared state imperatively. Declare desired state, and let a reconciler make reality match it.

Everything else follows from that:follows:

• Every service has the same shape — a template so onboarding the 40th service is no harder than the 4th.4th
• Each service is self-contained — its own folder describes its containers, config, and secret permissions. One service's deploy can never touch another's.
• Build once, promote by reference — the image tested in staging is the exact artifact that runs in production. Prod never rebuilds.
• Git is the control plane — approvals are pull requests, history is the audit log, and rollback is a revert.revert
• TheNo patterncredential is substrate-ever in git — only pointers to credentials
• Every access is attributable — audit logs name the service and environment, never "root"
• Substrate-agnostic — the same repo and flow run on one VPS today and on many hosts later, without a rebuild.later

2. Architecture

2.1 The threefour moving parts

There is no Jenkins. The reconciler is an off-the-shelf tool (Komodo, or Portainer as an alternative) — not code you write. It does exactly what Jenkins did on the deploy side (docker compose pull + up -d), driven by git instead of by a webhook.

2.2 Runtime layers

Each host runs two layers:

• Base layer — the always-on platform: Traefik, Vault Agent, PostgreSQL, Redis, RabbitMQ, MinIO. Rarely redeployed. Deploying an app service never touches it.
• Service layer — the application services (backend, notifications, File Thunder, ai, …), each deployed independently against the base layer.

2.3 Component roles

• GHCR (ghcr.io/nexgate-hq/*) — image registry. CI pushes here;pushes; the reconciler pulls from here.pulls.
• Komodo — the reconciler. Watches nexgate-infra, applies stack changes, provides a dashboard for logs/redeploy/rollback, and emits deploy notifications. Also mints per-service Vault Secret IDs at deploy time (§6).
• Vault (vault.qbitspark.com) — secrets. Services authenticate via AppRole, scoped per service and per environment.
• Vault Agent — sidecar per host. Handles AppRole login, token renewal, and secret rendering so application containers never handle Vault credentials directly.
• Traefik — edge router. Auto-discovers services by container labels andlabels, serves *.nexgate.co over HTTPS (Let's Encrypt).

3. The repositoryRepository layout

nexgate-infra is the heart of the system.heart. Everything a service needs to run is described in its own folder — the same information that used to be crammed into two giant shared docker-compose.ymlcompose files, now cut apart so each service owns its slice.

nexgate-infra/
├── base/                          # the always-on platform layer
│   ├── traefik.yml
│   ├── vault-agent.yml
│   ├── postgres.yml               # optional shared Postgres for light services
│   ├── redis.yml
│   ├── rabbitmq.yml
│   └── minio.yml                  # always-on (File Thunder needs it live)
│
├── services/
│   ├── nexgate-backend/
│   │   ├── service.yml            # compose block: image, labels, depends_on
│   │   ├── config.env             # non-secret config + Vault pathsONLY
│   │   ├── vault-policy.hcl       # which Vault paths this service may read
│   │   ├── vault-agent.hcl        # secret rendering template
│   │   └── tag.env                # IMAGE_TAG=...  ← the one line CI edits
│   ├── notification-server/
│   ├── file-thunder/
│   └── ai/
│
├── policies/                      # generated from services/*/vault-policy.hcl
│   └── .gitkeep
│
├── scripts/
│   ├── vault-bootstrap.sh         # create policies + approles from service folders
│   └── verify-isolation.sh        # prove cross-service access is denied
│
├── staging/
│   └── stack.yml                  # include: base + services (staging tags)
└── prod/
    └── stack.yml                  # include: base + services (prod tags)

Nothing in a service folder is a program. service.yml is thea compose snippet you already write for a service.snippet. tag.env is one line. vault-policy.hcl is a few lines of permissions. The layout just guarantees that one service's deploy is isolated to its own files.

Example: aExample service folder

# services/ai/service.yml
services:
  ai:
    image: ghcr.io/nexgate-hq/ai-service:${AI_TAG:-latest}
    env_file: [config.env, tag.env]
    networks: [nexgate-prod, proxy]
    depends_on: [postgres, redis, rabbitmq]
    volumes:
      # Vault Agent renders secrets here. The app reads a file.
      # It never sees a role_id or secret_id.
      - vault-secrets-ai:/run/secrets:ro
    labels:
      - traefik.enable=true
      - traefik.http.routers.ai.rule=Host(`ai.nexgate.co`)
      - traefik.http.routers.ai.tls.certresolver=le
    restart: unless-stopped

# services/ai/config.env      ← non-secret ONLY. Committed to git.
SPRING_PROFILES_ACTIVE=prod
LOG_LEVEL=INFO
VAULT_ADDR=https://vault.qbitspark.com

# services/ai/tag.env         ← CI rewrites ONLY this file on each deploy
AI_TAG=0.1.0

Because tag.env is per-service, a deploy ofdeploying ai cannot reset the backend's tag. The shared-.env clobber problem is structurally impossible.

4. The deployDeploy flow

4.1 The one rule: built once, pulled per environment

• Created (built + pushed to GHCR): once, by CI, the momentwhen you push code.push. Tied to the code, not the environment.
• Pulled: by the reconciler, once per environment, when that environment's tag changes to point at the image.changes.

Staging pulls the image right after the change. Prod pulls the same image after approval. Prod never rebuilds.

4.2 End-to-end

git push (staging/master)
   │
   ▼
CI (GitHub Actions, reusable workflow)
   • build image via Buildx
   • push to GHCR (taggedversion versiontag + latest)
   • bump tag in nexgate-infra (services/<svc>/tag.env)env in nexgate-infra
   │
   ▼
nexgate-infra  ← declarative desired state
   │
   ▼  (prod only: approval gate — PR review / environment reviewer)review)
   │
   ▼
Komodo reconciler on the VPS
   • detects the tag change
   • mints a fresh Secret ID for <svc>-<env>      ← §6
   • writes it to /etc/vault/<svc>-<env>-secret-id (0400 root)
   • docker compose pull <svc> && up -d <svc>     (only that service)
   • on health-check pass: destroys prior Secret IDs
   │
   ▼
Vault Agent authenticates via AppRole → renders secrets to /run/secrets
   │
   ▼
Service container upstarts, →reads Vaultsecret Agent injects secrets at bootfiles
   │
   ▼
Traefik routes it over HTTPS (*.nexgate.co)

No SSH. No Jenkins job. AutomatedNo hopscredential requirein zero server interaction.git.

4.3 Approvals

The gate lives in git,git which is— stronger than a button because it is reviewable and permanent.

• Staging — no gate. Merge to staging, it auto-syncs. Fast feedback.
• Prod — promotion is a pull request that bumpsbumping the prod tag (or merges staging → master).tag. The PR is the approval:approval; you see exactly which service and tag are moving. Merge = deploy.

Optional hard gates on top of the PR:gates:

• A GitHub production environment with a required reviewer (may need a paid plan for private repos; if so, branch protection requiring a review before merge to master gives the same gate for free).
• Komodo'sKomodo manual-sync for the prod stack — auto-sync staging, hold prod for a manual "deploy" click. This is the closestClosest one-to-one replacement for the old Jenkins button.

Recommended: PR promotion + Komodo manual-sync for prod.

4.4 Notifications

Three taps, together richer than the old success/failure email:

• CI → a webhook step posts build + tag-bump status to Telegram (a bot is ideal — instant on mobile, free).
• Komodo → deploy success/failure to the same channel, reported after a health check rather than just "compose exited 0."0"
• GitHub → free email/mobile pings on workflow failures and pending PRapprovals
approvals.
• Vault audit → alert on root token use, permission-denial spikes, seal events (§10)

Post-deploy runtime health (the thing Jenkins never gave you) comes from Uptime Kuma or the Grafana stack — see §9.10.

5. Secrets model

This is the section that was thin in v1. It is now the longest, because it's where the real risk lives.

5.1 TwoThe tiers, per servicerule

No

• _infracredential tieris —ever rawin containergit. credentialsOnly (Postgres/Redis/RabbitMQ/MinIO passwords) neededpointers to bootcredentials.
  thebasecontainers.
  App
  tiertheservice'sownsecrets creds,APItheseitself

  Artifact	In —git?	Why
  config.env (JWT,non-secret DBconfig)	✅ third-partyYes	Log keys).levels, Theprofiles, serviceVault readsaddress
  vault-policy.hcl	✅ Yes	Permission rules, not credentials
  Role ID	✅ Yes	Identifies a role. Useless alone.
  Secret ID	❌ Never	Delivered at boot.deploy, file on host only
  Actual secrets	❌ Never	Live in Vault, fetched at runtime

  5.2 AppRole,Path scopedhierarchy

  v1 used a flat nexgate/data/ai_staging — one segment for service_env. That blocks globbing, so every new component needs a new policy line, and eventually someone "fixes" it with a wildcard that grants too much.

  v2 uses a proper hierarchy:

  nexgate/<service>/<environment>/<component>
  
  nexgate/ai/staging/postgres
  nexgate/ai/prod/postgres
  nexgate/ai/prod/openai
  nexgate/backend/prod/postgres
  nexgate/backend/prod/jwt-signing
  nexgate/file-thunder/prod/minio
  nexgate/file-thunder/prod/postgres
  nexgate/notification-server/prod/rabbitmq
  nexgate/shared/prod/cloudflare
  

  Why service/environment above component: that's the boundary policies cut along. One rule covers every component a service will ever have, including ones added next year.

  Put the thing you cut permissions along highest in the path.

  5.3 The KV v2 path gotcha

  The single most common Vault mistake, and it will bite whoever writes the next policy.

  What you type: vault kv get nexgate/ai/prod/postgres What goes over HTTP: GET /v1/nexgate/data/ai/prod/postgres

  KV v2 splits operations into sub-paths — data/, metadata/, delete/, undelete/, destroy/. The kv CLI inserts the right one for you. Policies have no wrapper.

  Write the obvious-looking thing and you get permission denied with no hint why:

  path "nexgate/ai/prod/*" { capabilities = ["read"] }   # ← WRONG. Silently fails.
  

  5.4 Policies — one per service and environment

  DoThe notv1 bug: use one long-lived VAULT_TOKEN shared across environments (the old blast-radius problem). Instead:

  • Each service has a Vault AppRole per environment (ai_staging, ai_prod, …) with a policy limited to its own paths.
  • A Vault Agent authenticates with the AppRole, receives a short-TTL token, renders secrets to the service, and auto-renews.
  • Staging credentials cannot reach prod paths. A leak in one environment is contained.

  # services/ai/vault-policy.hcl  ← BROKEN
  path "nexgate/data/ai_staging" { capabilities = ["read"] }
  path "nexgate/data/ai_prod"    { capabilities = ["read"] }
  

  One policy granting both environments. If both ai_staging and ai_prod roles attach it, staging can read production secrets — defeating the isolation claimed two paragraphs above it in v1.

  Correct — separate file per environment:

  # services/ai/vault-policy-staging.hcl
  path "nexgate/data/ai/staging/*" {
    capabilities = ["read"]
  }
  path "nexgate/metadata/ai/staging/*" {
    capabilities = ["list"]
  }
  

  # services/ai/vault-policy-prod.hcl
  path "nexgate/data/ai/prod/*" {
    capabilities = ["read"]
  }
  path "nexgate/metadata/ai/prod/*" {
    capabilities = ["list"]
  }
  

  Read only. No create, no update, no delete. An app that reads its config has no business rewriting it. Deny-by-default means anything not listed is denied — no explicit deny rule needed.

  Prefer generating these from a template so nobody hand-writes the data/ prefix wrong (§7 step 6).

  5.5 AppRole per service per environment

  # staging — looser, no CIDR binding if staging hosts vary
  vault write auth/approle/role/ai-staging \
      token_policies="ai-staging" \
      token_ttl=4h \
      token_max_ttl=12h \
      secret_id_ttl=720h \
      secret_id_num_uses=0 \
      secret_id_bound_cidrs="<staging-host-ip>/32"
  
  # prod — tightest
  vault write auth/approle/role/ai-prod \
      token_policies="ai-prod" \
      token_ttl=1h \
      token_max_ttl=4h \
      secret_id_ttl=720h \
      secret_id_num_uses=0 \
      secret_id_bound_cidrs="<prod-host-ip>/32"
  

  token_max_ttl is your actual worst-case exposure window. Renewal extends but never resets — it's measured from creation. A stolen prod token is dead in 4 hours regardless of how often the thief renews it.

  secret_id_bound_cidrs is free hardening. A stolen Role ID and Secret ID are useless from anywhere but that host. The attacker would have to run the attack from inside your infrastructure.

  5.6 Where the two halves physically live

  The rule: Role ID and Secret ID never in the same place. If one docker inspect yields both, the split has bought nothing.

  Role ID    → services/<svc>/config.env, committed to git
  Secret ID  → /etc/vault/<svc>-<env>-secret-id on the host
               chmod 400, chown root:root
               minted by Komodo at deploy, never in git
  

  Never do this:

  environment:
    VAULT_ROLE_ID: "..."
    VAULT_SECRET_ID: "..."   # ← docker inspect prints this in plaintext
  

  One command gives an attacker both halves. This is the single most common way AppRole deployments get compromised.

  Do this — file mount:

  services:
    vault-agent-ai:
      image: hashicorp/vault:1.20.3
      command: agent -config=/vault/agent.hcl
      environment:
        VAULT_ADDR: https://vault.qbitspark.com
      volumes:
        - ./services/ai/vault-agent.hcl:/vault/agent.hcl:ro
        - /etc/vault/ai-prod-role-id:/vault/role-id:ro
        - /etc/vault/ai-prod-secret-id:/vault/secret-id:ro
        - vault-secrets-ai:/run/secrets
  

  docker inspect now shows only mount paths. docker compose config shows nothing sensitive. Git holds only the Role ID.

  5.7 Vault Agent — the app never touches Vault

  This is why Agent is worth the extra container. The application never handles a Role ID, a Secret ID, or a Vault token. It reads a file.

  # services/ai/vault-agent.hcl
  vault {
    address = "https://vault.qbitspark.com"
  }
  
  auto_auth {
    method "approle" {
      config = {
        role_id_file_path                   = "/vault/role-id"
        secret_id_file_path                 = "/vault/secret-id"
        remove_secret_id_file_after_reading = false
      }
    }
  
    sink "file" {
      config = { path = "/run/secrets/token" }
    }
  }
  
  template {
    destination = "/run/secrets/application-secrets.properties"
    contents = <<EOT
  {{- with secret "nexgate/data/ai/prod/postgres" }}
  spring.datasource.username={{ .Data.data.username }}
  spring.datasource.password={{ .Data.data.password }}
  {{- end }}
  {{- with secret "nexgate/data/ai/prod/openai" }}
  openai.api-key={{ .Data.data.api_key }}
  {{- end }}
  EOT
  }
  

  Agent logs in, renders the file, renews the token, and re-authenticates when token_max_ttl is hit. The app just reads /run/secrets/application-secrets.properties.

  Note {{ .Data.data.username }} — the double .data is KV v2's envelope. Another place the data/ structure surfaces.

  Alternative — Spring Cloud Vault, if you'd rather the app authenticate directly:

  spring:
    cloud:
      vault:
        uri: https://vault.qbitspark.com
        authentication: APPROLE
        app-role:
          role-id: ${VAULT_ROLE_ID}
          secret-id: ${VAULT_SECRET_ID}
        kv:
          enabled: true
          backend: nexgate
          default-context: ai/prod
  

  Fewer containers, but the app handles the Secret ID. Prefer Agent — it keeps credentials out of application process memory and gives you one renewal implementation instead of one per language.

  5.8 Two tiers, unchanged from v1

  • _infra tier — raw container credentials (Postgres/Redis/RabbitMQ/MinIO passwords) needed to boot the base containers
  • App tier — the service's own secrets (JWT, DB creds, third-party API keys)

  5.9 Static vs dynamic — the migration target

  Everything above stores static secrets in KV. That's the starting point, not the destination.

  The limitation, concretely: if a static credential leaks, Vault cannot revoke it. Vault stores and guards it, but it can't kill something a third party issued. A leaked OpenAI key stays valid until someone revokes it at OpenAI.

  Dynamic secrets invert this. Vault generates a real PostgreSQL user on demand with a TTL, then drops it:

  vault read database/creds/ai-prod
  # → v-approle-ai-prod-x7Kd92 / A1a-8sKdm3nQp   (lease 1h)
  vault read database/creds/ai-prod
  # → v-approle-ai-prod-mN4vB1 / Z9z-2pLwq7Rt    (lease 1h)
  

  Different every time. vault lease revoke runs DROP ROLE against Postgres directly. A stolen credential dies on its own.

  Migration order:

  1. PostgreSQL — all services, highest value
  2. RabbitMQ — notification-server, backend
  3. MinIO — via the AWS engine (S3-compatible STS)
  4. PKI — internal CA for mTLS between services
  5. Transit — encryption-as-a-service for sensitive payloads

  What must stay static: third-party API keys (OpenAI, Cloudflare, payment providers). Vault didn't issue them, so it can't rotate them. These get quarterly manual rotation and versioned storage in KV.

  Success measure: a leaked database credential is a non-event because it expired an hour ago.

  ---

  6. The trust chain

  v1 said "services authenticate via AppRole" and stopped there. That leaves the important question unanswered: who gives Komodo permission to mint Secret IDs, and who gives that to Komodo?

  6.1 Komodo's Vault credentials

  Komodo needs a Vault credential to mint per-service Secret IDs. That credential becomes the new secret zero, so it must be narrow.

  # policies/komodo-deployer.hcl
  path "auth/approle/role/+/secret-id" {
    capabilities = ["update"]
  }
  path "auth/approle/role/+/secret-id-accessor/destroy" {
    capabilities = ["update"]
  }
  

  Can mint and destroy Secret IDs. Cannot read a single secret. Cannot change a policy. Cannot create a role. If it leaks, an attacker can generate Secret IDs — but must guess role names, and every generation appears in the audit log.

  vault write auth/approle/role/komodo-deployer \
      token_policies="komodo-deployer" \
      token_ttl=15m \
      token_max_ttl=30m \
      secret_id_ttl=0 \
      secret_id_bound_cidrs="<host-ip>/32"
  

  15-minute tokens. Komodo logs in per deploy, uses the token, it dies.

  6.2 Placing Komodo's credentials — the one manual step

  ssh <host>
  sudo mkdir -p /etc/vault
  
  vault read -field=role_id auth/approle/role/komodo-deployer/role-id \
    | sudo tee /etc/vault/komodo-role-id > /dev/null
  vault write -f -field=secret_id auth/approle/role/komodo-deployer/secret-id \
    | sudo tee /etc/vault/komodo-secret-id > /dev/null
  
  sudo chmod 400 /etc/vault/komodo-*
  sudo chown root:root /etc/vault/komodo-*
  

  This is the only credential a human ever places by hand, and only once per host.

  6.3 Komodo's deploy hook

  #!/usr/bin/env bash
  # scripts/deploy-service.sh — run by Komodo on tag change
  set -euo pipefail
  
  SERVICE="$1"        # e.g. ai
  ENV="$2"            # staging | prod
  ROLE="${SERVICE}-${ENV}"
  
  # 1. Komodo authenticates (15-minute token)
  TOKEN=$(vault write -field=token auth/approle/login \
    role_id="$(cat /etc/vault/komodo-role-id)" \
    secret_id="$(cat /etc/vault/komodo-secret-id)")
  
  # 2. Record existing Secret ID accessors so we can revoke them after
  OLD_ACCESSORS=$(VAULT_TOKEN=$TOKEN vault list -format=json \
    "auth/approle/role/${ROLE}/secret-id" 2>/dev/null || echo '[]')
  
  # 3. Mint a fresh Secret ID for this deploy
  VAULT_TOKEN=$TOKEN vault write -f -field=secret_id \
    "auth/approle/role/${ROLE}/secret-id" \
    > "/etc/vault/${ROLE}-secret-id"
  chmod 400 "/etc/vault/${ROLE}-secret-id"
  chown root:root "/etc/vault/${ROLE}-secret-id"
  
  # 4. Deploy
  docker compose pull "$SERVICE"
  docker compose up -d "$SERVICE"
  
  # 5. Wait for health, THEN revoke the old Secret IDs
  if ./scripts/wait-for-health.sh "$SERVICE" 120; then
    echo "$OLD_ACCESSORS" | jq -r '.[]' | while read -r acc; do
      VAULT_TOKEN=$TOKEN vault write \
        "auth/approle/role/${ROLE}/secret-id-accessor/destroy" \
        secret_id_accessor="$acc"
    done
    echo "✓ ${ROLE} deployed, old Secret IDs revoked"
  else
    echo "✗ ${ROLE} health check failed — old Secret IDs left intact for rollback"
    exit 1
  fi
  
  # Komodo's own token expires in 15 minutes regardless
  

  Order matters. Revoke old Secret IDs after the new container is healthy, never before — otherwise a failed deploy leaves you unable to roll back.

  6.4 The chain, complete

  YOU                          ← SSH key. The actual root of trust.
   │
   └─ place Komodo's role-id + secret-id     (once per host, by hand)
       │
       └─ KOMODO mints service Secret IDs    (every deploy, 15-min tokens)
           │
           └─ VAULT AGENT logs in per service (1–4h tokens, auto-renewed)
               │
               └─ APP reads rendered secret files (never touches Vault)
  

  Four levels, each narrower than the one above. Only the top requires a human, and only once per host.

  Your SSH key is the bottom of the chain — the real secret zero for the entire system. It deserves a passphrase, a hardware key if possible, and an entry in the break-glass documentation.

  6.5 Rotating Komodo's own Secret ID

  Manual, deliberately.

  ssh <host>
  vault write -f -field=secret_id auth/approle/role/komodo-deployer/secret-id \
    | sudo tee /etc/vault/komodo-secret-id > /dev/null
  docker compose restart komodo
  

  It cannot be automated from Komodo — that would let a compromised Komodo renew its own access indefinitely. Keeping it manual bounds the chain. Quarterly, or on any suspicion.

  6.6 Secret zero — the honest accounting

  You cannot bootstrap trust from nothing. Every auth system has a first credential. The question is only what it is and how bad a leak would be.

  Before (shared .env):

  DB_PASSWORD=real
  JWT_PRIVATE_KEY=real
  OPENAI_KEY=real
  MINIO_SECRET=real
  

  One file. Everything. Never expires. No log of who read it.

  After:

  /etc/vault/komodo-secret-id:  d4e5f6a7-b8c9-...
  

  One file. One UUID.

  	Old .env	Stolen Komodo Secret ID
  What they get	Every credential immediately	Nothing — needs the Role ID too
  With both halves	—	Can mint Secret IDs, cannot read secrets
  Duration	Forever	Until rotated
  IP-bindable	No	Yes
  Do you find out	No	Yes — audit log
  To revoke	Change everything everywhere	One command

  Vault doesn't protect secrets by hiding them. It protects them by encrypting at rest, authenticating every request, authorising per path, expiring everything, logging every access, and revoking centrally. A .env file does none of these six.

  ---

  7. Setup from scratch

  Bootstrap order matters. Do these in sequence the first time.

  Step 1 — Provision hosts

  • Two Ubuntu VPS: staging and prod (separate boxes — see §10)11).

  • Install Docker Engine + the Compose plugin.
  • Lock down with UFW (allowallowing 22, 80, 443;443 denyonly.

    Also: confirm the rest).

  Docker
  daemon API is not exposed. curl http://<host>:2375/version from outside must fail. An exposed Docker socket makes every other control here irrelevant.

  Step 2 — Private networking (do it early)

  • Set up a

    WireGuard mesh (or your provider'sprovider private network) between hosts, even if you start with one host each. This makesMakes future host-splitting a one-line change instead of a migration.

  • Services should address each other by name / internal DNS, never hardcoded localhost.

  Step 3 — Docker networks

  • One external proxy network for Traefik:

    docker network create proxy.

  • One internaldocker network percreate environment: nexgate-staging, # on staging host docker network create nexgate-prod # on prod host .

  Step 4 — Deploy the base layer

  On each host, bring up: Traefik (with the Let's Encrypt resolver le), Vault Agent, PostgreSQL, Redis, RabbitMQ, MinIO. MinIO is always-on from day one.

  Vault Agent comes after Vault is configured — step 6.

  Step 5 — ConfigureVault: audit logging first

  Before anything else. Non-disruptive, one command, and it means everything from here forward is attributable.

  vault audit enable file file_path=/vault/logs/audit.log
  vault audit list -detailed
  

  Vault refuses to serve requests if all audit devices fail to write — deliberate fail-closed design. Ship the log to Loki; sensitive values are HMAC'd while paths and field names stay plaintext, so it's safe to forward.

  Also confirm, and write down offline:

  • Enable Which storage backend is vault.qbitspark.com using? (determines backup strategy)
  •  Where are the AppRole5 authunseal backend.key shares, physically?
  • For eachAre servicethey +split environment,across createlocations, or all in one place?
  •  Has a policysnapshot andrestore ever been tested?

  Solo operator: Shamir's 3-of-5 assumes multiple people. Split across locations instead — bank box, trusted party, encrypted offsite, hidden at home. An untested recovery scheme is a role.

story
• Writeyou thetell serviceyourself.
  secrets to their app paths (nexgate/<svc>_<env>).

Step 6 — CreateVault: nexgate-inframount, auth, policies, roles

vault 
Scaffoldsecrets base/enable -path=nexgate kv-v2
vault secrets tune -description="NexGate application secrets" nexgate/
vault auth enable approle
,

services/,

Then run the bootstrap script (§7 step 8) which generates policies and staging/stack.ymlroles + prod/stack.yml.

• Each stack includes the base plusfrom the service foldersfolders.
  for

  Retention thatworth environment.

setting

vault kv metadata put -max-versions=10 nexgate/ai/prod/postgres

#!/usr/bin/env bash
# scripts/vault-bootstrap.sh
# Generates policies and AppRoles from services/*/ folders.
# Idempotent — safe to re-run.
set -euo pipefail

STAGING_CIDR="${STAGING_CIDR:?set STAGING_CIDR}"
PROD_CIDR="${PROD_CIDR:?set PROD_CIDR}"

for dir in services/*/; do
  svc=$(basename "$dir")

  for env in staging prod; do
    name="${svc}-${env}"

    # policy — generated, so the data/ prefix is never hand-typed wrong
    cat > "policies/${name}.hcl" <<EOF
path "nexgate/data/${svc}/${env}/*" {
  capabilities = ["read"]
}
path "nexgate/metadata/${svc}/${env}/*" {
  capabilities = ["list"]
}
EOF
    vault policy write "$name" "policies/${name}.hcl"

    if [ "$env" = "prod" ]; then
      TTL=1h; MAXTTL=4h; CIDR="$PROD_CIDR"
    else
      TTL=4h; MAXTTL=12h; CIDR="$STAGING_CIDR"
    fi

    vault write "auth/approle/role/${name}" \
      token_policies="$name" \
      token_ttl="$TTL" \
      token_max_ttl="$MAXTTL" \
      secret_id_ttl=720h \
      secret_id_num_uses=0 \
      secret_id_bound_cidrs="$CIDR"

    # Role ID is not secret — emit for config.env
    echo "${name} role_id: $(vault read -field=role_id auth/approle/role/${name}/role-id)"
  done
done

#!/usr/bin/env bash
# scripts/verify-isolation.sh
set -uo pipefail
PASS=0; FAIL=0

check() {
  local role="$1" path="$2" expect="$3" rid sid tok result
  rid=$(vault read -field=role_id "auth/approle/role/${role}/role-id")
  sid=$(vault write -f -field=secret_id "auth/approle/role/${role}/secret-id")
  tok=$(vault write -field=token auth/approle/login role_id="$rid" secret_id="$sid")

  if VAULT_TOKEN="$tok" vault kv get "$path" >/dev/null 2>&1
  then result="ALLOW"; else result="DENY"; fi

  if [ "$result" = "$expect" ]; then
    echo "  ✓ ${role} → ${path} = ${result}"; PASS=$((PASS+1))
  else
    echo "  ✗ ${role} → ${path} = ${result} (expected ${expect})"; FAIL=$((FAIL+1))
  fi
}

echo "Positive (must ALLOW):"
check ai-prod          nexgate/ai/prod/postgres            ALLOW
check backend-staging  nexgate/backend/staging/postgres    ALLOW

echo "Negative (must DENY):"
check ai-prod          nexgate/backend/prod/jwt-signing    DENY
check ai-prod          nexgate/ai/staging/postgres         DENY   # ← the v1 bug
check ai-staging       nexgate/ai/prod/postgres            DENY   # ← the v1 bug
check backend-staging  nexgate/backend/prod/postgres       DENY
check file-thunder-prod nexgate/ai/prod/openai             DENY

echo; echo "Passed: $PASS  Failed: $FAIL"
[ "$FAIL" -eq 0 ] || exit 1