NexGate Deployment

NexGate Deployment Guide

NexGate Deployment Guide

NexGate Deployment Guide (New)

The paved road for shipping every NexGate service — architecture, flow, setup, and maintenance.

Owner: Kibuti · Organization: NexGate / nexgate-hq · Status: living document

How to read this

This is the single source of truth for how services are built, deployed, and operated across every NexGate service. It covers four things:

  1. Architecture — what the pieces are and how they fit.
  2. Flow — what happens when you push code, and how a change reaches production.
  3. Setup — how to stand the whole thing up from zero.
  4. Maintenance — how to run it day to day, add services, roll back, and scale.

If you only remember one sentence, remember this:

You write code, drop one folder, and register one secret role. An image is built once when you push, and pulled once per environment by the reconciler. Nothing else is touched.

Table of contents

  1. Core principles
  2. Architecture
  3. The repository layout
  4. The deploy flow
  5. Secrets model
  6. Setup from scratch
  7. Adding a new service
  8. Deploying a change
  9. Maintenance and operations
  10. Scaling
  11. Conventions and invariants
  12. Glossary

1. Core principles

The whole design turns on one shift away from the old Jenkins-driven pipeline:

Stop editing shared state imperatively. Declare desired state, and let a reconciler make reality match it.

Everything else follows from that:

2. Architecture

2.1 The three moving parts

Part What it is Who owns it
Service repos Application code + a thin CI workflow (nexgate-hq/<service>) One repo per service
nexgate-infra Declarative desired state — what runs, where, at which version One shared repo
Reconciler (Komodo) Watches nexgate-infra and applies changes on the host(s) Installed once per host

There is no Jenkins. The reconciler is an off-the-shelf tool (Komodo, or Portainer as an alternative) — not code you write. It does exactly what Jenkins did on the deploy side (docker compose pull + up -d), driven by git instead of by a webhook.

2.2 Runtime layers

Each host runs two layers:

2.3 Component roles

3. The repository layout

nexgate-infra is the heart of the system. Everything a service needs to run is described in its own folder — the same information that used to be crammed into two giant shared docker-compose.yml files, now cut apart so each service owns its slice.

nexgate-infra/
├── base/                          # the always-on platform layer
│   ├── traefik.yml
│   ├── vault-agent.yml
│   ├── postgres.yml               # optional shared Postgres for light services
│   ├── redis.yml
│   ├── rabbitmq.yml
│   └── minio.yml                  # always-on (File Thunder needs it live)
│
├── services/
│   ├── nexgate-backend/
│   │   ├── service.yml            # compose block: image, labels, depends_on
│   │   ├── config.env             # non-secret config + Vault paths
│   │   ├── vault-policy.hcl       # which Vault paths this service may read
│   │   └── tag.env                # IMAGE_TAG=...  ← the one line CI edits
│   ├── notification-server/
│   ├── file-thunder/
│   └── ai/
│
├── staging/
│   └── stack.yml                  # include: base + services (staging tags)
└── prod/
    └── stack.yml                  # include: base + services (prod tags)

Nothing in a service folder is a program. service.yml is the compose snippet you already write for a service. tag.env is one line. vault-policy.hcl is a few lines of permissions. The layout just guarantees that one service's deploy is isolated to its own files.

Example: a service folder

# services/ai/service.yml
services:
  ai:
    image: ghcr.io/nexgate-hq/ai-service:${AI_TAG:-latest}
    env_file: [config.env, tag.env]
    networks: [nexgate-prod, proxy]
    depends_on: [postgres, redis, rabbitmq]
    labels:
      - traefik.enable=true
      - traefik.http.routers.ai.rule=Host(`ai.nexgate.co`)
      - traefik.http.routers.ai.tls.certresolver=le
    restart: unless-stopped

# services/ai/tag.env      ← CI rewrites ONLY this file on each deploy
AI_TAG=0.1.0

Because tag.env is per-service, a deploy of ai cannot reset the backend's tag. The shared-.env clobber problem is structurally impossible.

4. The deploy flow

4.1 The one rule: built once, pulled per environment

Staging pulls the image right after the change. Prod pulls the same image after approval. Prod never rebuilds.

4.2 End-to-end

git push (staging/master)
   │
   ▼
CI (GitHub Actions, reusable workflow)
   • build image via Buildx
   • push to GHCR (tagged version + latest)
   • bump tag in nexgate-infra (services/<svc>/tag.env)
   │
   ▼
nexgate-infra  ← declarative desired state
   │
   ▼  (prod only: approval gate — PR review / environment reviewer)
   │
   ▼
Komodo reconciler on the VPS
   • detects the tag change
   • docker compose pull <svc> && up -d <svc>   (only that service)
   │
   ▼
Service container up  →  Vault Agent injects secrets at boot
   │
   ▼
Traefik routes it over HTTPS (*.nexgate.co)

No SSH. No Jenkins job. Automated hops require zero server interaction.

4.3 Approvals

The gate lives in git, which is stronger than a button because it is reviewable and permanent.

Optional hard gates on top of the PR:

Recommended: PR promotion + Komodo manual-sync for prod.

4.4 Notifications

Three taps, together richer than the old success/failure email:

Post-deploy runtime health (the thing Jenkins never gave you) comes from Uptime Kuma or the Grafana stack — see §9.

5. Secrets model

5.1 Two tiers, per service

5.2 AppRole, scoped per service and environment

Do not use one long-lived VAULT_TOKEN shared across environments (the old blast-radius problem). Instead:

# services/ai/vault-policy.hcl
path "nexgate/data/ai_staging" { capabilities = ["read"] }
path "nexgate/data/ai_prod"    { capabilities = ["read"] }

6. Setup from scratch

Bootstrap order matters. Do these in sequence the first time.

Step 1 — Provision hosts

Step 2 — Private networking (do it early)

Step 3 — Docker networks

Step 4 — Deploy the base layer

On each host, bring up: Traefik (with the Let's Encrypt resolver le), Vault Agent, PostgreSQL, Redis, RabbitMQ, MinIO. MinIO is always-on from day one.

Step 5 — Configure Vault

Step 6 — Create nexgate-infra

Step 7 — Install and connect Komodo

Step 8 — Create service-template + reusable CI

Step 9 — Onboard the first service

Step 10 — Wire notifications and observability

7. Adding a new service

Assuming the base, Komodo, template, and reusable workflow already exist, onboarding a service (example: ai) is:

  1. Generate the repo — nexgate-hq/ai-service from service-template. Dockerfile, CI caller, health/metrics already wired.
  2. Write the code — your Spring Boot service. This is the only real work; everything below is config.
  3. First push to staging → CI builds ai:0.1.0 → GHCR. (image created, first time)
  4. Add one folder — services/ai/ with service.yml, config.env, vault-policy.hcl, tag.env.
  5. Register the Vault AppRole for staging + prod, and write the app secrets.
  6. Add it to the staging stack (or, if the stack globs services/*, Komodo picks it up automatically).
  7. Commit → Komodo syncs staging → pulls ai:0.1.0 → live at ai.staging.nexgate.co. (pulled #1) — validate.
  8. Promote to prod — PR bumping the prod tag. Merge → Komodo syncs prod → pulls the same image → live at ai.nexgate.co. (pulled #2, identical)

The wiring (steps 4–5) is the entire cost — the old four-place scavenger hunt collapsed to one folder + one AppRole. Every change after this is just §8.

File Thunder note: onboarding File Thunder additionally means uncommenting MinIO in base/ (it's the service that finally needs it live) and declaring its own Postgres (on 5433) and a ClamAV container in its service.yml. Those are one-time, and they live entirely in File Thunder's folder + the base layer.

8. Deploying a change

For a service already onboarded:

  1. Edit code, push to the staging branch.
  2. CI builds <svc>:<version>, pushes to GHCR, bumps the staging tag. (created)
  3. Komodo sees the staging tag change → pulls the image → restarts only that service in staging. (pulled #1) Automatic, no gate.
  4. Validate in staging.
  5. Open a PR bumping the prod tag (or merge staging → master). The PR is the gate.
  6. Approve/merge → Komodo pulls the same image into prod. (pulled #2, identical)

Prod runs the byte-for-byte artifact you tested.

9. Maintenance and operations

Rollback

Revert the tag-bump commit in nexgate-infra (or use Komodo's "redeploy previous"). Because images are immutable and promoted by reference, rolling back is re-pointing the tag at the last-good image — fast and safe.

Rough edges (now fixed by design)

Secret rotation

Rotate in Vault; the Vault Agent picks up the new value on renewal. No image rebuild, no redeploy needed for most secrets.

Backups

Monitoring — the Grafana stack

Monitoring runs on a dedicated ops VPS that sits outside dev, staging, and prod — never on a box it watches, so it survives that box going down. One ops host watches every environment; targets are labelled by env (env=dev|staging|prod) and dashboards filter on that label. It joins the WireGuard mesh to reach services over internal IPs, and hits public endpoints through Traefik.

The stack:

Component Question it answers
Prometheus Metrics over time + alerting rules (scrapes each service's /metrics)
Loki Log search (services log JSON to stdout)
Tempo Distributed traces across services
Grafana Dashboards and the single view across all of it
Alertmanager Routes alerts → Telegram / email

Collectors that feed it:

Because every service already exposes /metrics and logs JSON to stdout (the service contract in §11), adding a new service to monitoring is just a scrape target + a health probe, both env-labelled.

Komodo vs Grafana — you need both

They answer different questions and neither covers the other:

Komodo Grafana stack
Watches Deployments (control plane) Runtime health (observability)
Tells you Did it deploy, what tag runs where, deploy logs Is it up, is it slow, error rate, resource use, log/metric history
Lets you Redeploy, roll back Alert, investigate — but not deploy

The gap is real: Komodo reports "deploy succeeded" the moment a container starts, even if it then crash-loops or serves 500s under load — Grafana is what catches that. Conversely, Grafana can tell you a service is unhealthy but can't redeploy or roll it back — that's Komodo. One tells you something's wrong; the other lets you do something about it.

Komodo does show basic per-host/container CPU/RAM/disk, so it doubles as light resource monitoring — but it has no metrics history, no log search, no dashboards, no tracing. The moment you need "why was it slow last Tuesday" or "alert me when p95 latency climbs," that's Grafana. Run both on the ops VPS; both alert to Telegram.

Upgrading base images

Postgres/Redis/etc. versions are pinned per service in service.yml (see §10.4). Bump the pin, test in staging, promote.

The single pane of glass

Komodo's GUI replaces the old Jenkins dashboard: see every service, view logs, redeploy, roll back. Choosing Komodo (or Portainer) is what fills that slot — it isn't optional flavor.

10. Scaling

10.1 Two kinds of scale

10.2 The ladder

Rung Setup When
1 Single VPS, Compose + Komodo Now
2 Bigger VPS Vertical scale buys runway
3 Services split across hosts (Komodo, multi-host) When a service gets heavy
4 Swarm or K3s Replicas + failover, much later

The repo and onboarding journey are identical at every rung. Moving up is a config change (which host a folder runs on), plus — at rung 3 — the private network from [§6 step 2].

10.3 VPS topology options

Sequence to commit to: be on env-split today → grow into service-split as load appears → treat all-have-all as a distant HA project.

10.4 Different versions per service

Postgres/Redis/MinIO are upstream public images you pin by tag — unrelated to the "build once" rule (which is only about your app image). Different services can pin different versions freely, because each service's datastore is a separate container in its own folder:

# services/file-thunder/service.yml
  ft-postgres:
    image: postgres:17
    volumes: [ft-pgdata:/var/lib/postgresql/data]

# services/backend/service.yml
  backend-postgres:
    image: postgres:15
    volumes: [backend-pgdata:/var/lib/postgresql/data]

They don't collide — different container names, volumes, and versions. Shared vs dedicated is a per-service call: a shared base Postgres saves RAM for light services (each gets a database inside it); a dedicated Postgres gives version/extension freedom and isolation (why File Thunder has its own). The Postgres version is a per-service decision in service.yml, never a base-layer constraint.

11. Conventions and invariants

Rules that keep the system coherent. Don't break these.

12. Glossary

This document is versioned in nexgate-infra. Update it whenever the architecture changes — it is the guide the whole organization relies on.