Files-nexgate-service(2)

File Thunder Arch & Flow

va# File Thunder — Developer Documentation

File Thunder is the dedicated media-processing microservice for the NexGate / Veepii platform. It handles all file ingestion, processing, storage, and serving-readiness — the main backend never touches raw bytes.

Table of Contents

  1. Architecture Overview
  2. Storage: Buckets & Object Keys
  3. MediaDomain & MediaContext
  4. Upload Flow — End to End
  5. The Four Wheels
  6. Progress Tracking (Redis → SSE)
  7. Watermarking
  8. CDN & File Serving Per Environment
  9. Security
  10. API Reference & How to Consume

1. Architecture Overview

┌─────────────────────────────────────────────────────────┐
│                      Main Backend                       │
│  - Only public-facing API                               │
│  - Calls File Thunder over internal HTTP (HMAC signed)  │
│  - Subscribes to Redis for progress → pushes SSE        │
└────────────┬──────────────────────────┬─────────────────┘
             │ HTTP (sync)              │ Redis Pub/Sub (async)
             ▼                          ▼
┌─────────────────────────────────────────────────────────┐
│                     File Thunder                        │
│                                                         │
│   API Profile          Worker Profile                   │
│   ─────────────        ──────────────                   │
│   UploadController     UploadConfirmedWorker            │
│   MediaQueryController ThumbnailWorker                  │
│                        RawFilePurgeJob                  │
│                                                         │
│   Wheels (processing engines)                           │
│   ──────────────────────────                            │
│   ImageWheel  VideoWheel  ScanWheel  (MinIO ops)        │
└────────────┬──────────────────────────┬─────────────────┘
             │                          │
             ▼                          ▼
          MinIO                      RabbitMQ
    (4 private buckets)       (internal event routing)

Key constraints:

2. Storage: Buckets & Object Keys

Buckets

Bucket Purpose Access
nexgate-raw Temporary upload landing zone — 24h TTL Private
nexgate-public Processed media served to end users Private (CDN in front)
nexgate-private Internal system assets (outros, future forensic assets) Private
nexgate-digital Digital product files — ClamAV scanned, download only Private

All buckets are fully private. MinIO is never directly accessible from the internet. Public content is served exclusively through the CDN (Cloudflare), which pulls from MinIO origin.

Object Key Pattern

{domain}/{ownerId}/{fileId}/{variant}

Examples:

posts/550e8400-e29b-41d4-a716-446655440000/f7e8d9.../original
posts/550e8400-e29b-41d4-a716-446655440000/f7e8d9.../large.webp
posts/550e8400-e29b-41d4-a716-446655440000/f7e8d9.../thumb.webp
posts/550e8400-e29b-41d4-a716-446655440000/f7e8d9.../hls/master.m3u8
posts/550e8400-e29b-41d4-a716-446655440000/f7e8d9.../hls/360p/360p.m3u8

system/outro/{ownerId}/{height}p.mp4   ← in nexgate-private

Rule: The database stores object keys only, never full URLs. URLs are assembled at the resolver boundary (CDN base URL + key in prod, presigned GET in local).

3. MediaDomain & MediaContext

MediaDomain

Defines where in the storage hierarchy a file lives. It is the top-level folder in the object key. This is an enum — the main backend must send one of these exact values.

Value Used For
POSTS Social posts — images and videos
PROFILES Profile pictures, cover photos
MESSAGES Direct message attachments
PRODUCTS Product images, product videos, digital downloads
SHOPS Shop banners, shop logos
EVENTS Event covers, event gallery images

MediaContext

Defines how a file is processed. This drives the entire processing pipeline. No logic is attached to domain — all logic is driven by context.

Value Type Processing Pipeline
SOCIAL_IMAGE Image ImageWheel — orient, strip EXIF, WebP variants, blurhash, lqip
SOCIAL_VIDEO Video VideoWheel — transcode + thumbnail; social watermark + outro only if duration < 3 min (short clip)
PROFILE_PICTURE Image ImageWheel
COVER_PHOTO Image ImageWheel
DM_IMAGE Image ImageWheel
DM_VIDEO Video VideoWheel — transcode, no watermark
DM_DOCUMENT Any ScanWheel — ClamAV scan → private bucket
PRODUCT_IMAGE Image ImageWheel
PRODUCT_VIDEO Video VideoWheel — transcode + text watermark, no outro
DIGITAL_PRODUCT Any ScanWheel — ClamAV scan → digital bucket
PRODUCT_PREVIEW_IMAGE Image ImageWheel — identical pipeline to PRODUCT_IMAGE
PRODUCT_PREVIEW_VIDEO Video VideoWheel — identical pipeline to PRODUCT_VIDEO; watermarkLabel required
PRODUCT_PREVIEW_DOCUMENT Any ScanWheel — identical pipeline to DIGITAL_PRODUCT → digital bucket
SHOP_BANNER Image ImageWheel
SHOP_LOGO Image ImageWheel
EVENT_COVER Image ImageWheel
EVENT_GALLERY Image ImageWheel

Validation rules enforced at API layer:

4. Upload Flow — End to End

Standard Media Upload

Main Backend                    File Thunder                 MinIO           Client (Browser)
     │                               │                         │                   │
     │  POST /api/v1/upload/request  │                         │                   │
     │  (HMAC signed)                │                         │                   │
     │──────────────────────────────▶│                         │                   │
     │                               │ validate + create DB    │                   │
     │                               │ record (PENDING)        │                   │
     │                               │ publish PENDING         │                   │
     │                               │ to Redis                │                   │
     │                               │ generate presigned PUT  │                   │
     │                               │─────────────────────────▶                  │
     │  { fileId, presignedUrl }     │◀────────────────────────                   │
     │◀──────────────────────────────│                         │                   │
     │                               │                         │                   │
     │  return presignedUrl          │                         │                   │
     │  to client                    │                         │                   │
     │──────────────────────────────────────────────────────────────────────────▶ │
     │                               │                         │                   │
     │                               │                         │  PUT (file bytes) │
     │                               │                         │◀──────────────────│
     │                               │                         │  200 OK           │
     │                               │                         │──────────────────▶│
     │                               │                         │                   │
     │                               │ ◀── MinIO event ────────│ s3:ObjectCreated  │
     │                               │ MinioEventListener      │                   │
     │                               │ PENDING → UPLOADED      │                   │
     │                               │ publish UPLOADED→Redis  │                   │
     │                               │ publish UploadConfirmed │                   │
     │                               │ to RabbitMQ             │                   │
     │                               │                         │                   │
     │                               │ UploadConfirmedWorker   │                   │
     │                               │ routes by mimeType      │                   │
     │                               │ UPLOADED → PROCESSING   │                   │
     │                               │ publish PROCESSING→Redis│                   │
     │                               │                         │                   │
     │                               │ [ImageWheel or          │                   │
     │                               │  VideoWheel runs]       │                   │
     │                               │                         │                   │
     │                               │ variants uploaded       │                   │
     │                               │ to nexgate-public ──────▶                  │
     │                               │ raw deleted from        │                   │
     │                               │ nexgate-raw ────────────▶                  │
     │                               │ PROCESSING → READY      │                   │
     │                               │ publish READY → Redis   │                   │
     │                               │                         │                   │

User Custom Thumbnail Upload

Main Backend                    File Thunder                 MinIO
     │                               │                         │
     │  POST /api/v1/upload/         │                         │
     │  thumbnail/{fileId}           │                         │
     │──────────────────────────────▶│                         │
     │                               │ generate presigned PUT  │
     │                               │ key: .../custom-        │
     │                               │ thumbnail               │
     │  { presignedUrl }             │                         │
     │◀──────────────────────────────│                         │
     │  client uploads thumbnail     │                         │
     │─────────────────────────────────────────────────────────▶
     │                               │◀── MinIO event ─────────│
     │                               │ MinioEventListener      │
     │                               │ detects suffix=         │
     │                               │ "custom-thumbnail"      │
     │                               │ → ThumbnailWorker       │
     │                               │ download → verify magic │
     │                               │ bytes → ImageMagick     │
     │                               │ variants → save to      │
     │                               │ entity.userThumbnail    │
     │                               │ → delete raw            │

File Status Lifecycle

PENDING → UPLOADING → UPLOADED → SCANNING → PROCESSING → LIVE_PARTIAL → READY
                                                                ▲
                                              (all videos: after 360p HLS segments
                                               uploaded and partial master.m3u8 written)

5. The Four Wheels

Wheels are the processing engines. Each runs in the worker profile.

Wheel 1 — ImageWheel (ImageMagick)

Triggered by: image MIME type on any image context

Pipeline:

download raw from nexgate-raw
      │
      ▼
auto-orient + strip EXIF
      │
      ├──▶ large.webp    (1920px max width, shrink only, Q85)
      ├──▶ medium.webp   (800px max width, shrink only, Q82)
      ├──▶ thumb.webp    (300px max width, shrink only, Q80)
      ├──▶ og.webp       (1200×630 center crop, Q85)
      │    └─ falls back to medium if source width < 1200px
      ├──▶ blurhash      (encoded from 32×32 downsample)
      └──▶ lqip          (10×10 forced, base64 WebP data URI)

all WebP variants → nexgate-public
keys + blurhash + lqip → media_files.variants (JSONB)
raw deleted from nexgate-raw
status → READY

Skipping logic: variant is skipped if source is already within bounds (never upscale).

Wheel 2 — VideoWheel (FFmpeg / Jaffree)

Triggered by: video MIME type on any video context

Step 1 — Probe & Validate

download raw from nexgate-raw
magic byte verify (must be real video container)
FFprobe → codedWidth, codedHeight, rotation, duration, codec
displayWidth/displayHeight = rotation-aware (swap for 90°/270°)
size gate: > 2 GB → reject
duration gate: > 4h → reject
entity.shortClip = duration < 3min
route: shortClip=true  → processShort()
       shortClip=false → processLong()

Step 2a — Short Path (< 3 min) — HLS Portrait

transcodeHlsPortrait() runs 3 variants in sequence:
  360p  (CRF 28, 96k audio)  → HLS segments + playlist → upload → status LIVE_PARTIAL
  720p  (CRF 23, 128k audio) → HLS segments + playlist → upload
  1080p (CRF 21, 192k audio) → HLS segments + playlist → upload → final master.m3u8

filter_complex (portrait blur-pad, same as before):
  [0:v] split=2 [tmp_bg][tmp_fg]
  [tmp_bg] scale={W}:{H}:increase, crop={W}:{H}, boxblur=20:1 [bg]
  [tmp_fg] scale={W}:{H}:decrease, setsar=1 [fg]
  [bg][fg] overlay=(W-w)/2:(H-h)/2 [out]
  + rotation transpose prepended if needed

output format: HLS (-f hls), 2-second segments, VOD playlist type
variant keys:  {keyBase}/hls/{name}/{name}.m3u8
master key:    {keyBase}/hls/master.m3u8
segment pattern: {keyBase}/hls/{name}/{name}_%03d.ts

watermarked MP4 download (if profile.watermark()):
  transcode() — MP4 only, not HLS
  if SOCIAL_VIDEO: append outro via concat (no re-encode)
  key: {keyBase}/{name}_watermarked.mp4

skip logic: source displayWidth >= targetW OR displayHeight >= targetH

Step 2b — Long Path (≥ 3 min) — HLS Adaptive

transcodeHls() per variant:
  360p → segments + .m3u8 → upload → partial master.m3u8 → LIVE_PARTIAL
  720p → segments + .m3u8 → upload
  1080p → segments + .m3u8 → upload → full master.m3u8 → READY

segment length: 2s
playlist type: VOD
master.m3u8 key: {keyBase}/hls/master.m3u8
variant keys:   {keyBase}/hls/{name}/{name}.m3u8

No watermark generated — watermark is only produced for short clips (shortClip=true).

Step 3 — Thumbnail & Preview (both short and long)

selectBestFrame():
  5 candidates at 15/30/45/60/75% of duration
  scored by: brightness gate (30–230) + Laplacian variance (sharpness)
  FFmpeg extracts 720px JPEG per candidate → pick winner

buildThumbnailVariants():
  poster.webp      (1280px, Q85)
  thumb.webp       (480px, Q80)
  og.webp          (1200×630 center crop, Q85)
  blurhash         (from 32×32 downsample)
  lqip             (10×10 forced, base64 WebP data URI)
  dominant_color   (#RRGGBB from 1×1 squish)

extractPreviewClip():
  skip first 5% (max 2s) → read 6s → setpts=0.5*PTS → 3s at 2× speed
  blur-pad 360×640, muted
  watermarked (same moving watermark as main video)

all thumbnail variants → nexgate-public
raw deleted from nexgate-raw after all variants done

Wheel 3 — ScanWheel (ClamAV)

Triggered by: DIGITAL_PRODUCT, DM_DOCUMENT, and PRODUCT_PREVIEW_DOCUMENT contexts only Social content (images and videos) is never scanned by ClamAV — FFmpeg/ImageMagick re-encode is the sanitisation.

Pipeline:

download raw from nexgate-raw
      │
      ▼
SHA-256 hash → check file_hashes table
      │
      ├── hash known + clean → skip scan, copy to target bucket, READY (dedup fast lane)
      │
      └── hash unknown →
            ClamAV scan
            if clean → move to target bucket, save hash, status READY
            if infected → delete raw, status FAILED (VIRUS_DETECTED)

Target bucket by context:
  DIGITAL_PRODUCT        → nexgate-digital
  PRODUCT_PREVIEW_DOCUMENT → nexgate-digital
  DM_DOCUMENT            → nexgate-private

Wheel 4 — MinIO Operations (used by all wheels)

Not a standalone service — MinIO operations are helpers used across all wheels:

Operation Used By
Presigned PUT URL Upload request endpoint
Download object All wheels (download raw for processing)
Upload object All wheels (upload processed variants)
Remove object All wheels (delete raw after processing)
Stat object OutroService (cache check), ScanWheel (dedup check)

6. Progress Tracking (Redis → SSE)

File Thunder publishes status changes to Redis. The main backend subscribes and drives SSE to the client. SSE is the main backend's responsibility — File Thunder only publishes.

What File Thunder Does

On every status change:

// 1. Cache current status in Redis (24h TTL)
redisTemplate.opsForValue().set("ft:status:{fileId}", status.name(), 24h);

// 2. Publish to per-file channel
redisTemplate.convertAndSend("ft:progress:{fileId}", status.name());

// 3. Append to timeline in Postgres
entity.timeline.add({ status, at })

What the Main Backend Must Do

// Subscribe to the file's progress channel
redisTemplate.subscribe((message, pattern) -> {
    String status = message.toString();
    sseEmitter.send(SseEmitter.event().data(status));
}, "ft:progress:" + fileId);

Status Channel Key

ft:progress:{fileId}     ← subscribe here for live updates
ft:status:{fileId}       ← read here for current status (24h cached)

7. Watermarking

When Watermarks Are Applied

Context Condition Watermark type
SOCIAL_VIDEO shortClip = true (duration < 3 min) Social watermark (logo + watermarkLabel)
SOCIAL_VIDEO shortClip = false (duration ≥ 3 min) None
PRODUCT_VIDEO Always Text-only
PRODUCT_PREVIEW_VIDEO Always Text-only (same as PRODUCT_VIDEO)
DM_VIDEO Never —

Short-clip detection happens after FFprobe during VideoWheel processing. The shortClip flag is persisted on the entity and returned in every media response.

Moving Watermark

Watermark position cycles through corners on a timer to resist cropping.

Text-only (PRODUCT_VIDEO):

drawtext: fontsize=max(14,H/40), white@0.65 + black shadow
position: floor(t/3) mod 4 → top-left, top-right, bottom-right, bottom-left
text: from watermark.text property (default: "NexGate")

Social watermark (SOCIAL_VIDEO short clips only):

2-point diagonal: Pos A (22%, 28%) ↔ Pos B (58%, 65%) — switches every 5s
Logo: nexgate_logo_white.svg, 36×36, 65% opacity
@ symbol: orange #F06023@0.85 + shadow
watermarkLabel: white@0.85 + shadow, below logo

Outro (SOCIAL_VIDEO only)

A personalized outro clip is appended to the watermarked variant (not clean variants).

Generation:

template: outro_template_with_sfx.mp4 (classpath resource)
font: Sora SemiBold 600 (classpath resource)
drawtext: @ in orange #F06023, watermarkLabel in cream #F4EEE9
fade-in: invisible before t=0.75s, fully opaque by t=1.2s
scale to match variant resolution (360p / 720p / 1080p)
append via: -f concat -safe 0 -c copy (no re-encode, fast)

Caching:

Generated once per ownerId + resolution
Cached to: nexgate-private / system/outro/{ownerId}/{height}p.mp4
On next upload: cache hit → download and reuse, skip generation

8. CDN & File Serving Per Environment

All MinIO buckets are private. Access to public content is environment-dependent.

Environment Matrix

Environment Serving Method Config Value
Local dev Presigned GET URLs (MinIO direct, short-lived) ft.storage.mode=local
Staging CDN — cdn-staging.nexgate.com ft.storage.mode=cdn
Production CDN — cdn.nexgate.com ft.storage.mode=cdn

Local / Dev

Staging & Production (CDN)

Properties

# Local
ft.storage.mode=local
ft.cdn.base-url=

# Staging
ft.storage.mode=cdn
ft.cdn.base-url=https://cdn-staging.nexgate.com

# Production
ft.storage.mode=cdn
ft.cdn.base-url=https://cdn.nexgate.com

URL Assembly Logic (resolver boundary)

// local mode → presigned GET from MinIO
// cdn mode   → CDN base + object key
String url = storageMode.equals("cdn")
    ? cdnBaseUrl + "/" + objectKey
    : minioClient.getPresignedObjectUrl(GET, bucket, objectKey, 1h);

Private content (DIGITAL_PRODUCT, PRODUCT_PREVIEW_DOCUMENT, DM_DOCUMENT): always presigned GET URLs regardless of environment — these are never cached by CDN. Generated fresh per download request (15-minute expiry, audit logged).

9. Security

CORS

Configured via ft.cors.* properties. The CORS filter runs before all other filters so OPTIONS preflight requests are handled without requiring HMAC headers.

# Local
ft.cors.allowed-origins=http://localhost:3000

# Production
ft.cors.allowed-origins=https://nexgate.com,https://www.nexgate.com

ft.cors.allowed-methods=GET,POST,PUT,DELETE,OPTIONS
ft.cors.allowed-headers=*
ft.cors.allow-credentials=false
ft.cors.max-age-seconds=3600

Service-to-Service HMAC Authentication

Every HTTP request from the main backend to File Thunder must be HMAC-SHA256 signed. File Thunder rejects any unsigned or incorrectly signed request with 401 Unauthorized.

Filter order:

Request → CorsFilter (HIGHEST_PRECEDENCE)
        → HmacAuthFilter (HIGHEST_PRECEDENCE + 1)
        → Controllers

Required Headers

Header Description Example
X-Service-Id Identifier of the calling service nexgate-main
X-Timestamp Unix epoch seconds at time of request 1718123456
X-Nonce Random UUID — unique per request f47ac10b-58cc-...
X-Signature HMAC-SHA256 hex of the canonical string a3f5c2...

Canonical String

METHOD\n
REQUEST_URI\n
TIMESTAMP\n
NONCE\n
HEX(SHA-256(requestBody))

Example for POST /api/v1/upload/request:

POST
/api/v1/upload/request
1718123456
f47ac10b-58cc-4372-a567-0e02b2c3d479
e3b0c44298fc1c149afb...   ← SHA-256 of the JSON body

Signature Computation

// Main backend — how to sign a request
String bodyHash = HexFormat.of().formatHex(
    MessageDigest.getInstance("SHA-256").digest(requestBodyBytes)
);

String canonical = method + "\n" + uri + "\n" + timestamp + "\n" + nonce + "\n" + bodyHash;

Mac mac = Mac.getInstance("HmacSHA256");
mac.init(new SecretKeySpec(sharedSecret.getBytes(UTF_8), "HmacSHA256"));
String signature = HexFormat.of().formatHex(mac.doFinal(canonical.getBytes(UTF_8)));

// Set headers on outgoing request
request.setHeader("X-Service-Id", "nexgate-main");
request.setHeader("X-Timestamp",  String.valueOf(Instant.now().getEpochSecond()));
request.setHeader("X-Nonce",      UUID.randomUUID().toString());
request.setHeader("X-Signature",  signature);

What File Thunder Verifies (in order)

1. All 4 headers present                → 401 if any missing
2. X-Service-Id in allowed list         → 401 if unknown
3. X-Timestamp within ±5 minutes        → 401 if stale (replay protection)
4. X-Nonce not seen before              → 401 if duplicate
   (stored in Redis with 10-min TTL)       (replay attack blocked)
5. Recompute HMAC, timing-safe compare  → 401 if mismatch
   (MessageDigest.isEqual)                 (prevents timing oracle)

Properties

ft.security.hmac.secret=<long-random-secret-same-in-both-services>
ft.security.hmac.allowed-service-ids=nexgate-main
ft.security.hmac.timestamp-tolerance-seconds=300
ft.security.hmac.nonce-ttl-seconds=600

The secret must be identical in both File Thunder and the main backend config. Use a different value per environment (local / staging / prod).

10. API Reference & How to Consume

Base URL: http://file-thunder:8081 (internal network only) All requests must include HMAC headers — see Section 9.

POST /api/v1/upload/request

Request a presigned URL to upload a file directly to MinIO.

Request body:

{
  "ownerId":          "550e8400-e29b-41d4-a716-446655440000",
  "domain":           "POSTS",
  "context":          "SOCIAL_VIDEO",
  "originalFilename": "my-video.mp4",
  "mimeType":         "video/mp4",
  "fileSizeBytes":    104857600,
  "watermarkLabel":   "@josh_dev"
}

watermarkLabel is required when context is SOCIAL_VIDEO (pass @username) or PRODUCT_VIDEO (pass $shopName). All other fields are always required.

Response:

{
  "status": "success",
  "message": "Presigned upload URL generated",
  "data": {
    "fileId":           "f7e8d9fa-...",
    "presignedUrl":     "http://minio:9000/nexgate-raw/posts/.../original?X-Amz-...",
    "objectKey":        "posts/{ownerId}/{entityId}/{fileId}/original",
    "bucket":           "nexgate-raw",
    "expiresInSeconds": 1800
  }
}

What to do next: PUT the file bytes directly to presignedUrl from the client browser. No auth headers needed on the PUT — the presigned URL is self-authenticating.

POST /api/v1/upload/thumbnail/{fileId}

Request a presigned URL to replace the system-generated video thumbnail with a user-supplied one. Only valid after the video file has been processed (status READY).

Response:

{
  "status": "success",
  "message": "Thumbnail presigned upload URL generated",
  "data": {
    "fileId":           "f7e8d9fa-...",
    "presignedUrl":     "http://minio:9000/nexgate-raw/.../custom-thumbnail?X-Amz-...",
    "objectKey":        "posts/.../custom-thumbnail",
    "bucket":           "nexgate-raw",
    "expiresInSeconds": 1800
  }
}

Accepted formats: JPEG, PNG, WebP only (verified by magic bytes).

GET /api/v1/media/{fileId}

Get full metadata for a file, including all processed variants.

Response — SOCIAL_VIDEO short clip (shortClip: true, duration < 3 min):

{
  "status": "success",
  "data": {
    "fileId":        "f7e8d9fa-...",
    "ownerId":       "550e8400-...",
    "domain":        "POSTS",
    "context":       "SOCIAL_VIDEO",
    "status":        "READY",
    "mimeType":      "video/mp4",
    "shortClip":     true,
    "variants": {
      "360p_playlist":    "posts/.../hls/360p/360p.m3u8",
      "720p_playlist":    "posts/.../hls/720p/720p.m3u8",
      "1080p_playlist":   "posts/.../hls/1080p/1080p.m3u8",
      "master":           "posts/.../hls/master.m3u8",
      "720p_watermarked": "posts/.../720p_watermarked.mp4",
      "poster":           "posts/.../poster.webp",
      "thumb":            "posts/.../thumb.webp",
      "og":               "posts/.../og.webp",
      "preview":          "posts/.../preview_3s.mp4",
      "blurhash":         "LKO2?U%2Tw=w]~RBVZRi};RPxuwH",
      "lqip":             "data:image/webp;base64,...",
      "dominant_color":   "#1A2B3C"
    },
    "userThumbnail": null,
    "timeline": [
      { "status": "PENDING",     "at": "2026-06-15T10:00:00" },
      { "status": "UPLOADED",    "at": "2026-06-15T10:00:05" },
      { "status": "PROCESSING",  "at": "2026-06-15T10:00:06" },
      { "status": "LIVE_PARTIAL","at": "2026-06-15T10:00:45" },
      { "status": "READY",       "at": "2026-06-15T10:02:10" }
    ],
    "createdAt": "2026-06-15T10:00:00",
    "updatedAt": "2026-06-15T10:02:10"
  }
}

Response — SOCIAL_VIDEO long clip (shortClip: false, duration ≥ 3 min) — HLS:

{
  "status": "success",
  "data": {
    "fileId":        "f7e8d9fa-...",
    "ownerId":       "550e8400-...",
    "domain":        "POSTS",
    "context":       "SOCIAL_VIDEO",
    "status":        "READY",
    "mimeType":      "video/mp4",
    "shortClip":     false,
    "variants": {
      "360p_playlist":  "posts/.../hls/360p/360p.m3u8",
      "720p_playlist":  "posts/.../hls/720p/720p.m3u8",
      "1080p_playlist": "posts/.../hls/1080p/1080p.m3u8",
      "master":         "posts/.../hls/master.m3u8",
      "poster":         "posts/.../poster.webp",
      "thumb":          "posts/.../thumb.webp",
      "og":             "posts/.../og.webp",
      "preview":        "posts/.../preview_3s.mp4",
      "blurhash":       "LKO2?U%2Tw=w]~RBVZRi};RPxuwH",
      "lqip":           "data:image/webp;base64,...",
      "dominant_color": "#1A2B3C"
    },
    "userThumbnail": null,
    "timeline": [
      { "status": "PENDING",     "at": "2026-06-15T10:00:00" },
      { "status": "UPLOADED",    "at": "2026-06-15T10:00:05" },
      { "status": "PROCESSING",  "at": "2026-06-15T10:00:06" },
      { "status": "LIVE_PARTIAL","at": "2026-06-15T10:01:00" },
      { "status": "READY",       "at": "2026-06-15T10:05:30" }
    ],
    "createdAt": "2026-06-15T10:00:00",
    "updatedAt": "2026-06-15T10:05:30"
  }
}

Thumbnail resolution rule:

userThumbnail != null ? use userThumbnail : use variants

Both userThumbnail and variants use the same key names (poster, thumb, og, blurhash, lqip, dominant_color).

GET /api/v1/media/{fileId}/download?requesterId={uuid}

Generate a time-limited download URL for private files (DIGITAL_PRODUCT, PRODUCT_PREVIEW_DOCUMENT, DM_DOCUMENT only). Every call is audit-logged.

Response:

{
  "status": "success",
  "data": {
    "url":              "http://minio:9000/nexgate-digital/...?X-Amz-...",
    "expiresInSeconds": 900
  }
}

GET /api/v1/quota/{ownerId}

Get the total storage used by an owner across all their processed files. Updated atomically as each file is processed — never stale.

Response:

{
  "status": "success",
  "message": "Storage usage",
  "data": {
    "ownerId":   "550e8400-e29b-41d4-a716-446655440000",
    "usedBytes": 1073741824,
    "usedMb":    1024.0,
    "usedGb":    1.0
  }
}

How quota is tracked:

Use this endpoint to:

Error Responses

{ "error": "Missing required security headers" }          ← 401
{ "error": "Request timestamp out of acceptable window" } ← 401
{ "error": "Nonce already used — possible replay attack"} ← 401
{ "error": "Invalid signature" }                          ← 401

Validation errors return 400 with field-level messages via @ControllerAdvice.

This document covers the full File Thunder system as of 2026-06-15. CDN wiring (Cloudflare), Kafka integration, lazy transcode, and forensic watermarks are deferred.

Files Handling API (NEW)

Author: Josh S. Sakweli, Backend Lead Team
Last Updated: 2026-06-21
Version: v1.0

Base URL: https://your-api-domain.com/api/v1

Short Description: This document covers everything a frontend developer needs to handle files on the NexGate/Veepii platform — uploading, tracking processing progress, reading variant URLs from entity responses, and downloading private files. Files are managed by an internal service called FileThunder; the frontend never talks to FileThunder directly. All file-related operations go through the main backend endpoints documented here.

Hints:

How File Handling Works — Big Picture

Understanding this flow will save you from confusion. There are two separate progress concepts:

  1. Upload progress — bytes travelling from the client to object storage. You can show a progress bar for this using XMLHttpRequest.
  2. Processing progress — FileThunder processing those bytes into variants (WebP images, transcoded video resolutions, virus scans). You track this via SSE or polling.
┌─────────┐     Step 1: Request presigned URL      ┌──────────────┐
│  Client │ ──────────────────────────────────────► │ Main Backend │
│         │ ◄────────────────────────────────────── │              │
│         │     { fileId, presignedUrl, expiresIn } └──────────────┘
│         │
│         │     Step 2: PUT file bytes directly         ┌─────────┐
│         │ ────────────────────────────────────────►   │  MinIO  │
│         │ ◄────────────────────────────────────────   │         │
│         │     HTTP 200 (upload complete)              └────┬────┘
│         │                                                  │ MinIO fires event
│         │     Step 3: Open SSE stream                      ▼
│         │ ──────────────────────────────────────► ┌──────────────┐
│         │ ◄── status events (PROCESSING, READY) ── │ Main Backend │
└─────────┘                                          └──────────────┘
                                                          │ on READY
                                                          ▼
                                                  Variants saved to entity in DB
                                                  Entity response includes variant URLs

Standard Response Format

Success Response Structure

{
  "success": true,
  "httpStatus": "OK",
  "message": "Operation completed successfully",
  "action_time": "2025-09-23T10:30:45",
  "data": {}
}

Error Response Structure

{
  "success": false,
  "httpStatus": "BAD_REQUEST",
  "message": "Error description",
  "action_time": "2025-09-23T10:30:45",
  "data": "Error description"
}

Standard Response Fields

Field Type Description
success boolean true for success, false for errors
httpStatus string HTTP status name (OK, BAD_REQUEST, NOT_FOUND, etc.)
message string Human-readable result description
action_time string ISO 8601 timestamp of the response
data object/string Response payload or error details

HTTP Method Badge Standards

File Contexts

Every upload must declare a context. Context tells FileThunder what this file is for, which determines how it is processed, what variants are generated, and how it is stored.

Context Domain What it is Who uploads it
SOCIAL_IMAGE Posts Image attached to a post or story Authenticated user
SOCIAL_VIDEO Posts Video attached to a post or story Authenticated user
PROFILE_PICTURE Profiles User avatar / profile photo Authenticated user
COVER_PHOTO Profiles Profile cover / banner image Authenticated user
DM_IMAGE Messages Image sent in a direct message Authenticated user
DM_VIDEO Messages Video sent in a direct message Authenticated user
DM_DOCUMENT Messages Document sent in a direct message Authenticated user
PRODUCT_IMAGE Products Product listing photo Shop owner
PRODUCT_VIDEO Products Product demo/preview video Shop owner
DIGITAL_PRODUCT Products Purchasable digital file (PDF, ZIP, software, etc.) Shop owner
SHOP_BANNER Shops Shop header banner image Shop owner
SHOP_LOGO Shops Shop logo / avatar Shop owner
EVENT_COVER Events Event banner/hero image Event organiser
EVENT_GALLERY Events Additional event gallery image Event organiser

File Format & Size Constraints

These are the accepted formats and recommended limits per context. FileThunder enforces the actual limits server-side — passing incorrect mimeType or oversized files will result in a rejection at the presigned URL stage.

Images

Context Accepted MIME Types Max Size Min Dimensions Recommended Dimensions
SOCIAL_IMAGE image/jpeg, image/png, image/webp, image/gif 20 MB 200 × 200 px 1080 × 1080 px (square) or 1080 × 1920 px (portrait)
PROFILE_PICTURE image/jpeg, image/png, image/webp 10 MB 100 × 100 px 400 × 400 px square
COVER_PHOTO image/jpeg, image/png, image/webp 15 MB 600 × 200 px 1500 × 500 px
DM_IMAGE image/jpeg, image/png, image/webp, image/gif 20 MB — —
PRODUCT_IMAGE image/jpeg, image/png, image/webp 20 MB 400 × 400 px 1000 × 1000 px square
SHOP_BANNER image/jpeg, image/png, image/webp 15 MB 800 × 200 px 1200 × 400 px
SHOP_LOGO image/jpeg, image/png, image/webp 5 MB 100 × 100 px 400 × 400 px square
EVENT_COVER image/jpeg, image/png, image/webp 15 MB 800 × 400 px 1200 × 630 px
EVENT_GALLERY image/jpeg, image/png, image/webp 20 MB 200 × 200 px 1080 × 1080 px

Videos

Context Accepted MIME Types Max Size Max Duration Notes
SOCIAL_VIDEO video/mp4, video/quicktime, video/webm, video/x-msvideo, video/x-matroska 100 MB 60 min All durations use HLS adaptive streaming
DM_VIDEO video/mp4, video/quicktime, video/webm 100 MB 60 min HLS adaptive streaming, no watermark
PRODUCT_VIDEO video/mp4, video/quicktime, video/webm, video/x-msvideo, video/x-matroska 100 MB 60 min All durations use HLS adaptive streaming

Other

Context Accepted MIME Types Max Size Notes
DM_DOCUMENT application/pdf, application/msword, application/vnd.openxmlformats-officedocument.wordprocessingml.document, application/vnd.ms-excel, application/vnd.openxmlformats-officedocument.spreadsheetml.sheet, application/zip, text/plain 50 MB ClamAV scanned; no variants generated
DIGITAL_PRODUCT Any — PDF, ZIP, EXE, APK, MP3, etc. 5 GB ClamAV dual-scan + SHA-256 deduplication; never CDN served; always presigned download

Processing Status Lifecycle

After the file reaches object storage, FileThunder processes it through these states. You will receive these values from both the SSE stream and the status polling endpoint.

PENDING ──► UPLOADING ──► UPLOADED ──► SCANNING ──► PROCESSING ──► READY
                                                                  └──► FAILED
                                                         │
                                                    LIVE_PARTIAL
                                               (video 360p is done,
                                                 higher resolutions
                                                  still processing)
Status Meaning UI suggestion
PENDING Upload slot created, waiting for file bytes Show spinner
UPLOADING File bytes are being received by storage Show upload progress bar (XHR)
UPLOADED All bytes received, handing off to processing Show spinner
SCANNING ClamAV virus scan in progress (digital products / DM docs only) "Scanning for safety..."
PROCESSING Transcoding / image variant generation in progress "Processing..."
LIVE_PARTIAL Videos only — 360p variant is ready, higher resolutions still processing Show video with 360p, overlay "HD processing" badge
READY All variants generated and available Dismiss progress UI, display media
FAILED Processing failed Show error, offer re-upload option

Variant Keys

When a file reaches READY, its variants are stored in the entity's database record and returned in entity API responses. Here are all possible variant keys and what they contain.

Image Variants

Key Format Typical Use
large WebP URL Full-size display, lightbox, detail view
medium WebP URL Feed cards, grid thumbnails
thumb WebP URL Tiny previews, avatar chips, comment icons
og WebP URL <meta property="og:image"> Open Graph tag
blurhash String (e.g. LGF5?xYk^6...) CSS blur placeholder while image loads
lqip data:image/webp;base64,... inline data URI Inline <img src> placeholder, no extra request
dominant_color Hex string (e.g. #F06023) Background color while loading, skeleton screen tint

Not every key is present for every context. blurhash and lqip are always generated. large, medium, og, and dominant_color depend on the context — see the Context → Variants Cheat Sheet at the bottom of this document for the exact set per context.

Usage priority: Render lqip or apply blurhash immediately. Swap in medium or large once loaded. Use thumb for tiny contexts. Always include og in page meta where available.

Video Watermarking

Video processing produces two separate sets of variants: clean variants for streaming, and a watermarked variant for download. They are different keys in the variants map.

Context Streaming variants (HLS) Watermarked download Watermark style Cycle
SOCIAL_VIDEO master, 360p_playlist, 720p_playlist, 1080p_playlist 720p_watermarked.mp4 (or 360p_watermarked.mp4) Diagonal 2-point: NexGate logo + watermarkLabel overlay Every 5 seconds, alternates between upper-left and lower-right
PRODUCT_VIDEO master, 360p_playlist, 720p_playlist, 1080p_playlist 720p_watermarked.mp4 (or 360p_watermarked.mp4) Text-only (NexGate label) cycling all 4 corners Every 3 seconds
DM_VIDEO master, 360p_playlist, 720p_playlist None No watermark —

Which watermarked key do you get?

When to use which:

Social video outro: SOCIAL_VIDEO files have a personalized branded outro clip appended (watermarkLabel + orange accent, no re-encode). The outro is baked into the 720p_watermarked variant only, not the clean variants.

The watermarkLabel is passed to FileThunder by the backend on upload — you send nothing extra for this. For SOCIAL_VIDEO it is the user's @handle; for PRODUCT_VIDEO it is the $shopName.

Video Variants — Short Clips (< 3 minutes, HLS)

Short clips use HLS with the same portrait blur-pad filter (blurred background fills the frame). The shortClip: true flag in the response tells you it is a short clip.

Key Format Typical Use
master HLS master playlist URL Default — use this for all playback. Plug into HLS.js or native <video> on Safari; adaptive quality switching is automatic
360p_playlist HLS per-rendition playlist URL First available at LIVE_PARTIAL — only use when building a manual quality selector
720p_playlist HLS per-rendition playlist URL Only use when building a manual quality selector
1080p_playlist HLS per-rendition playlist URL Only use when building a manual quality selector (only present if source is 1080p-capable)
720p_watermarked MP4 URL Watermarked download variant — offer this for user downloads
360p_watermarked MP4 URL Watermarked download fallback — present instead of 720p_watermarked when source is too small
preview MP4 URL Auto-generated 3-second silent preview clip (speed-doubled from 6s of footage at ~5% into the video) — use for hover previews on cards
poster WebP URL Best auto-selected frame — show as video thumbnail before play
thumb WebP URL Small thumbnail for cards and grids
og WebP URL 1200×630 Open Graph image
blurhash String BlurHash string for placeholder
lqip data URI Inline WebP placeholder, no extra request
dominant_color Hex string Background tint for skeleton screens

Video Variants — Long Form (≥ 3 minutes, HLS)

Key Format Typical Use
master HLS master playlist URL Default — use this for all playback. Plug into HLS.js or native <video> on Safari; the player handles quality switching automatically based on bandwidth
360p_playlist HLS per-rendition playlist URL Only use when building a manual quality selector — swap the player src to this to force 360p
720p_playlist HLS per-rendition playlist URL Only use when building a manual quality selector — swap the player src to this to force 720p
1080p_playlist HLS per-rendition playlist URL Only use when building a manual quality selector — swap the player src to this to force 1080p (only present if source is 1080p-capable)
preview MP4 URL 3-second preview clip for card hover effects
poster WebP URL Best auto-selected frame
thumb WebP URL Card thumbnail
og WebP URL Open Graph image
blurhash String BlurHash placeholder
lqip data URI Inline placeholder
dominant_color Hex string Background tint

HLS Note: Always use the master key for adaptive streaming — it switches automatically between 360p/720p/1080p based on viewer bandwidth. Long-form videos do not include an MP4 fallback — if the environment does not support HLS, you will need to inform the user or handle it at the player level (e.g. HLS.js handles this for most browsers). Long-form videos do not produce a watermarked download variant — downloads of long videos are not supported.

The Upload Flow — Step by Step

Step 1 — Request a presigned upload URL (backend)

Call POST /api/v1/files/request-upload with the file metadata. You get back a fileId and a presignedUrl.

Step 2 — Upload the file directly to object storage (MinIO/CDN)

Make an HTTP PUT request to the presignedUrl with the file bytes as the body. This request does not go to the backend — it goes directly to object storage. Use XMLHttpRequest (not fetch) if you want to track upload progress, because XHR exposes an upload.onprogress event that fetch does not.

The request must include:

A successful upload returns HTTP 200 with an empty body.

Upload progress tracking via XHR: XHR fires upload.onprogress events with loaded (bytes sent) and total (total bytes). Divide loaded / total to get a 0–1 progress value for your progress bar. This tracks network transfer progress, not processing progress. The progress bar should complete at 100% when the PUT returns 200, then transition to the processing state UI.

Step 3 — Track processing progress (SSE or polling)

After the PUT succeeds, open an SSE stream to GET /api/v1/files/progress/{fileId}. You will receive server-sent events as FileThunder processes the file. Close the stream when you receive READY or FAILED.

If SSE is inconvenient in your environment, use GET /api/v1/files/status/{fileId} to poll instead. Recommended polling interval: every 2–3 seconds.

Step 4 — Entity response contains the variants

Once READY, the entity API (product, shop, post, etc.) will include the variant map in its response. You do not need to call any file endpoint to get URLs — they come embedded in the entity response.

Endpoints

1. Request Upload URL

Purpose: Generate a presigned PUT URL and a fileId for a new file upload. Always call this before any file upload.

Endpoint: POST {base_url}/files/request-upload

Access Level: 🔒 Protected (Requires authenticated user)

Authentication: Bearer Token — Authorization: Bearer <token>

Request Headers:

Header Type Required Description
Authorization string Yes Bearer <your_jwt_token>
Content-Type string Yes application/json

Request JSON Sample:

{
  "context": "PRODUCT_IMAGE",
  "originalFilename": "shoe-red-side.jpg",
  "mimeType": "image/jpeg",
  "fileSizeBytes": 2457600
}

Request Body Parameters:

Parameter Type Required Description Validation
context string Yes The purpose of this file — determines processing pipeline and storage bucket Must be one of the valid context values listed in the File Contexts section
originalFilename string Yes Original name of the file including extension Max 255 characters
mimeType string Yes MIME type of the file exactly as the browser reports it Must match the context's accepted formats; see File Format & Size Constraints
fileSizeBytes number Yes File size in bytes Must be a positive integer

Success Response JSON Sample:

{
  "success": true,
  "httpStatus": "OK",
  "message": "Upload URL generated",
  "action_time": "2026-06-21T14:22:10",
  "data": {
    "fileId": "a3f7c21b-09d4-4e8b-bf12-3c7d09e1f556",
    "presignedUrl": "https://storage.nexgate.com/nexgate-raw/products/.../shoe-red-side.jpg?X-Amz-Signature=...",
    "expiresInSeconds": 3600
  }
}

Success Response Fields:

Field Description
data.fileId UUID identifying this file — save this, you will need it for SSE tracking and to associate the file with an entity
data.presignedUrl The URL to PUT the file bytes to — send directly to this URL, do not proxy through your server
data.expiresInSeconds How many seconds before the presigned URL expires (typically 3600 = 1 hour) — start the upload immediately

Error Response JSON Sample:

{
  "success": false,
  "httpStatus": "BAD_REQUEST",
  "message": "context is required. Valid values: SOCIAL_IMAGE, SOCIAL_VIDEO, PROFILE_PICTURE, ...",
  "action_time": "2026-06-21T14:22:10",
  "data": "context is required."
}

Standard Error Types:

2. Upload File to Object Storage (Presigned PUT)

Purpose: Upload the raw file bytes directly to object storage using the presigned URL from Step 1. This is not a backend endpoint — it is a direct PUT to the object storage URL. It is documented here because understanding this step is essential to the flow.

Endpoint: PUT {presignedUrl} (the full URL returned in Step 1)

Access Level: 🌐 Self-authenticated (The presigned URL carries authentication in its query parameters — no Authorization header needed or allowed)

Request Headers:

Header Type Required Description
Content-Type string Yes Must exactly match the mimeType sent in Step 1 — e.g. image/jpeg, video/mp4

Request Body: Raw file bytes. No multipart encoding. No form fields. Just the file content.

Upload Progress:

Use XMLHttpRequest for this PUT request to gain access to upload progress events. The XMLHttpRequest.upload object fires progress events containing:

Property Type Description
loaded number Bytes sent so far
total number Total bytes to send (equals your fileSizeBytes)

Divide loaded / total to get a 0–1 ratio for a progress bar. This tracks network transfer only — once the PUT returns 200, transition the UI to the "Processing…" state and begin tracking via SSE (Step 3).

Success Response: HTTP 200 with an empty body. No JSON.

Failure Responses:

Status Cause
400 Content-Type header does not match what was declared in Step 1
403 Presigned URL has expired — go back to Step 1 and request a new one
413 File exceeds the size declared in fileSizeBytes in Step 1

3. Stream File Processing Progress (SSE)

Purpose: Receive real-time server-sent events as FileThunder processes the uploaded file. Open this stream immediately after the PUT in Step 2 returns 200. The stream closes automatically when READY or FAILED is reached.

Endpoint: GET {base_url}/files/progress/{fileId}

Access Level: 🔒 Protected (Only the file owner can stream progress for a given fileId)

Authentication: Bearer Token — Authorization: Bearer <token>

Request Headers:

Header Type Required Description
Authorization string Yes Bearer <your_jwt_token>
Accept string Yes text/event-stream

Path Parameters:

Parameter Type Required Description Validation
fileId UUID Yes The fileId returned in Step 1 Must be a valid UUID owned by the authenticated user

Response: This endpoint returns a stream of Server-Sent Events, not a JSON body. Each event has a name (the status) and data (JSON payload).

SSE Event Format:

event: PROCESSING
data: {"status":"PROCESSING","fileId":"a3f7c21b-09d4-4e8b-bf12-3c7d09e1f556"}

event: LIVE_PARTIAL
data: {"status":"LIVE_PARTIAL","fileId":"a3f7c21b-09d4-4e8b-bf12-3c7d09e1f556"}

event: READY
data: {"status":"READY","fileId":"a3f7c21b-09d4-4e8b-bf12-3c7d09e1f556"}

SSE Event Sequence:

Short video:

PENDING → UPLOADING → UPLOADED → PROCESSING → LIVE_PARTIAL → READY

Image:

PENDING → UPLOADING → UPLOADED → PROCESSING → READY

Long video (≥ 3 minutes):

PENDING → UPLOADING → UPLOADED → PROCESSING → LIVE_PARTIAL → READY

Digital product or DM document:

PENDING → UPLOADING → UPLOADED → SCANNING → READY

Event Handling:

Event name Action
PENDING Show spinner
UPLOADING Show spinner (XHR progress bar already running from Step 2)
UPLOADED Show "Processing…" state
SCANNING Show "Scanning for safety…" state
PROCESSING Show "Processing…" state
LIVE_PARTIAL Video 360p is available — can begin playback, show "HD loading" badge
READY Close the SSE connection. Show success. Refresh entity data.
FAILED Close the SSE connection. Show error. Offer re-upload.

SSE Timeout: The stream has a 5-minute server-side timeout. If your file is still processing after 5 minutes (large video), reconnect to this endpoint — it will immediately send the current status snapshot before continuing to stream.

Error Responses:

4. Poll File Processing Status

Purpose: An alternative to SSE for environments where persistent connections are difficult (e.g. React Native background states, certain browser extensions). Returns the current processing status as a single JSON response. Poll every 2–3 seconds; stop when ready is true or failed is true.

Endpoint: GET {base_url}/files/status/{fileId}

Access Level: 🔒 Protected (Only the file owner)

Authentication: Bearer Token — Authorization: Bearer <token>

Request Headers:

Header Type Required Description
Authorization string Yes Bearer <your_jwt_token>

Path Parameters:

Parameter Type Required Description Validation
fileId UUID Yes The fileId returned in Step 1 Must be a valid UUID owned by the authenticated user

Success Response JSON Sample:

{
  "success": true,
  "httpStatus": "OK",
  "message": "Status retrieved",
  "action_time": "2026-06-21T14:23:45",
  "data": {
    "fileId": "a3f7c21b-09d4-4e8b-bf12-3c7d09e1f556",
    "status": "PROCESSING",
    "ready": false,
    "failed": false,
    "processing": true
  }
}

Success Response Fields:

Field Type Description
data.fileId UUID The file identifier
data.status string Current status — one of PENDING, UPLOADING, UPLOADED, SCANNING, PROCESSING, LIVE_PARTIAL, READY, FAILED
data.ready boolean true when file is fully processed and variants are available
data.failed boolean true when processing failed — stop polling and show error
data.processing boolean true while processing is in progress — continue polling

Ready state response:

{
  "success": true,
  "httpStatus": "OK",
  "message": "Status retrieved",
  "action_time": "2026-06-21T14:24:10",
  "data": {
    "fileId": "a3f7c21b-09d4-4e8b-bf12-3c7d09e1f556",
    "status": "READY",
    "ready": true,
    "failed": false,
    "processing": false
  }
}

Error Response JSON Sample:

{
  "success": false,
  "httpStatus": "NOT_FOUND",
  "message": "File not found or access denied",
  "action_time": "2026-06-21T14:24:10",
  "data": "File not found or access denied"
}

Standard Error Types:

5. Replace Video Thumbnail

Purpose: Upload a custom thumbnail image to replace the auto-selected thumbnail for a video file. Call this after the video has reached READY status. Returns a presigned URL — upload the image bytes to it (same pattern as Step 2 above).

Endpoint: POST {base_url}/files/thumbnail/{fileId}

Access Level: 🔒 Protected (Only the video owner)

Authentication: Bearer Token — Authorization: Bearer <token>

Request Headers:

Header Type Required Description
Authorization string Yes Bearer <your_jwt_token>

Path Parameters:

Parameter Type Required Description Validation
fileId UUID Yes The fileId of the video whose thumbnail you want to replace Must be a valid video file owned by the authenticated user

Request Body: None — this is a POST with no body.

Success Response JSON Sample:

{
  "success": true,
  "httpStatus": "OK",
  "message": "Thumbnail upload URL generated",
  "action_time": "2026-06-21T14:25:00",
  "data": {
    "fileId": "a3f7c21b-09d4-4e8b-bf12-3c7d09e1f556",
    "presignedUrl": "https://storage.nexgate.com/nexgate-raw/products/.../thumb-a3f7c21b.jpg?X-Amz-Signature=...",
    "expiresInSeconds": 3600
  }
}

Success Response Fields:

Field Description
data.fileId The video file's ID
data.presignedUrl PUT the thumbnail image bytes here — same process as the main upload in Step 2
data.expiresInSeconds Seconds until the presigned URL expires

After uploading: PUT image/jpeg or image/png bytes to the presignedUrl. FileThunder will generate the thumbnail variants and update the video's effective thumbnail. Refresh the entity response to get the updated thumbnail URLs.

Error Response JSON Sample:

{
  "success": false,
  "httpStatus": "UNAUTHORIZED",
  "message": "Token has expired",
  "action_time": "2026-06-21T14:25:00",
  "data": "Token has expired"
}

Standard Error Types:

6. List Digital Files Available for Download (Order)

Purpose: For a purchased digital product order, list all the files the buyer is entitled to download, along with download availability and remaining download counts.

Endpoint: GET {base_url}/e-commerce/orders/{orderId}/downloads

Access Level: 🔒 Protected (Only the order buyer)

Authentication: Bearer Token — Authorization: Bearer <token>

Request Headers:

Header Type Required Description
Authorization string Yes Bearer <your_jwt_token>

Path Parameters:

Parameter Type Required Description Validation
orderId UUID Yes The order ID containing digital files Must be an order owned by the authenticated buyer

Success Response JSON Sample:

{
  "success": true,
  "httpStatus": "OK",
  "message": "3 file(s) available for download",
  "action_time": "2026-06-21T14:26:00",
  "data": [
    {
      "fileId": "b9c1d33e-22f4-4a0b-9e67-1d4f22c3a881",
      "fileName": "nexgate-design-kit-v2.zip",
      "contentType": "application/zip",
      "fileSize": 52428800,
      "downloadCount": 1,
      "downloadsRemaining": 4,
      "accessExpiresAt": "2026-12-21T00:00:00",
      "canDownload": true
    },
    {
      "fileId": "c2e5f44a-33g5-5b1c-af78-2e5g33d4b992",
      "fileName": "user-manual.pdf",
      "contentType": "application/pdf",
      "fileSize": 1048576,
      "downloadCount": 0,
      "downloadsRemaining": 5,
      "accessExpiresAt": "2026-12-21T00:00:00",
      "canDownload": true
    }
  ]
}

Success Response Fields:

Field Type Description
[].fileId UUID File identifier — use this in the next endpoint to get a download URL
[].fileName string Original filename including extension
[].contentType string MIME type of the file
[].fileSize number File size in bytes
[].downloadCount number How many times the buyer has already downloaded this file
[].downloadsRemaining number How many more downloads the buyer is allowed
[].accessExpiresAt string ISO 8601 datetime after which download access expires
[].canDownload boolean false if download limit reached or access has expired — show disabled button

Error Response JSON Sample:

{
  "success": false,
  "httpStatus": "NOT_FOUND",
  "message": "Order not found",
  "action_time": "2026-06-21T14:26:00",
  "data": "Order not found"
}

Standard Error Types:

7. Generate Digital File Download URL

Purpose: Generate a time-limited, single-use download URL for a specific digital file in an order. Call this endpoint when the user clicks the download button — do not cache this URL. Open it immediately in a new tab or trigger a browser download.

Endpoint: GET {base_url}/e-commerce/orders/{orderId}/downloads/{fileId}

Access Level: 🔒 Protected (Only the order buyer)

Authentication: Bearer Token — Authorization: Bearer <token>

Request Headers:

Header Type Required Description
Authorization string Yes Bearer <your_jwt_token>

Path Parameters:

Parameter Type Required Description Validation
orderId UUID Yes The order ID Must belong to the authenticated buyer
fileId UUID Yes The specific file to download (from endpoint 6) Must be a file within the specified order

Success Response JSON Sample:

{
  "success": true,
  "httpStatus": "OK",
  "message": "Download URL generated — link expires in 15 minutes",
  "action_time": "2026-06-21T14:27:00",
  "data": {
    "fileId": "b9c1d33e-22f4-4a0b-9e67-1d4f22c3a881",
    "fileName": "nexgate-design-kit-v2.zip",
    "downloadUrl": "https://storage.nexgate.com/nexgate-digital/products/.../nexgate-design-kit-v2.zip?X-Amz-Signature=...&X-Amz-Expires=300",
    "expiresAt": "2026-06-21T14:32:00",
    "downloadsRemaining": 3,
    "downloadCount": 2
  }
}

Success Response Fields:

Field Type Description
data.fileId UUID File identifier
data.fileName string Filename to use when saving locally
data.downloadUrl string Pre-signed download URL — valid for 15 minutes only. Open immediately. Do not cache.
data.expiresAt string ISO 8601 datetime when the download URL expires
data.downloadsRemaining number How many downloads remain after this one
data.downloadCount number Total downloads made so far including this one

Error Response JSON Sample:

{
  "success": false,
  "httpStatus": "BAD_REQUEST",
  "message": "Download limit reached for this file",
  "action_time": "2026-06-21T14:27:00",
  "data": "Download limit reached for this file"
}

Standard Error Types:

How Variants Appear in Entity Responses

You never call a dedicated endpoint to get variant URLs. When an entity (product, post, shop, profile, event) is fetched via its own API, the response already contains the assembled variant map. Here is what to expect per entity type:

Product Image Variants

{
  "productImageVariants": [
    {
      "large": "https://cdn.nexgate.com/products/owner123/fileabc/large.webp",
      "medium": "https://cdn.nexgate.com/products/owner123/fileabc/medium.webp",
      "thumb": "https://cdn.nexgate.com/products/owner123/fileabc/thumb.webp",
      "og": "https://cdn.nexgate.com/products/owner123/fileabc/og.webp",
      "blurhash": "LGF5?xYk^6#M@-5c,1J5@[or[Q6",
      "lqip": "data:image/webp;base64,/9j/4AAQ...",
      "dominant_color": "#C8A882"
    }
  ]
}

productImageVariants is an array — one entry per uploaded image. Index 0 is the primary image.

Video Variants (short clip, HLS)

{
  "previewVariants": {
    "master": "https://cdn.nexgate.com/products/owner123/filevid/hls/master.m3u8",
    "360p_playlist": "https://cdn.nexgate.com/products/owner123/filevid/hls/360p/360p.m3u8",
    "720p_playlist": "https://cdn.nexgate.com/products/owner123/filevid/hls/720p/720p.m3u8",
    "1080p_playlist": "https://cdn.nexgate.com/products/owner123/filevid/hls/1080p/1080p.m3u8",
    "720p_watermarked": "https://cdn.nexgate.com/products/owner123/filevid/720p_watermarked.mp4",
    "preview": "https://cdn.nexgate.com/products/owner123/filevid/preview_3s.mp4",
    "poster": "https://cdn.nexgate.com/products/owner123/filevid/poster.webp",
    "thumb": "https://cdn.nexgate.com/products/owner123/filevid/thumb.webp",
    "og": "https://cdn.nexgate.com/products/owner123/filevid/og.webp",
    "blurhash": "LGF5?xYk^6#M@-5c,1J5@[or[Q6",
    "lqip": "data:image/webp;base64,/9j/4AAQ...",
    "dominant_color": "#1A1A2E"
  }
}

If the source video was too small for 720p, 720p_watermarked will be absent and 360p_watermarked will be present instead. Always check which watermarked key exists before rendering a download button.

Video Variants (long form / HLS)

{
  "previewVariants": {
    "master": "https://cdn.nexgate.com/products/owner123/filevid/hls/master.m3u8",
    "360p_playlist": "https://cdn.nexgate.com/products/owner123/filevid/hls/360p/360p.m3u8",
    "720p_playlist": "https://cdn.nexgate.com/products/owner123/filevid/hls/720p/720p.m3u8",
    "1080p_playlist": "https://cdn.nexgate.com/products/owner123/filevid/hls/1080p/1080p.m3u8",
    "preview": "https://cdn.nexgate.com/products/owner123/filevid/preview_3s.mp4",
    "poster": "https://cdn.nexgate.com/products/owner123/filevid/poster.webp",
    "thumb": "https://cdn.nexgate.com/products/owner123/filevid/thumb.webp",
    "og": "https://cdn.nexgate.com/products/owner123/filevid/og.webp",
    "blurhash": "LGF5?xYk^6#M@-5c,1J5@[or[Q6",
    "lqip": "data:image/webp;base64,/9j/4AAQ...",
    "dominant_color": "#1A1A2E"
  }
}

Feed master into HLS.js or a native <video> src on Safari. Long-form videos do not include an MP4 fallback and do not have a watermarked download variant.

LIVE_PARTIAL state (video still processing)

When you get an entity response while the video is in LIVE_PARTIAL:

You can safely begin playback in both cases and show a "HD loading" badge.

Null variants

If a file is still in PROCESSING and you fetch the entity, variants fields may be null or an empty array. Always null-check before rendering. Display a placeholder (blurhash, dominant_color, or a skeleton) until variants are available.

Best Practices

Upload

Progress UI

SSE vs Polling

Use SSE when Use polling when
Web browser context React Native / mobile apps
Single in-progress upload screen Background upload (app minimised)
You can keep the tab open Network conditions drop connections often

SSE is preferred — it is lower overhead and gives instant events. Polling at 2-second intervals is a reliable fallback.

Variants

Digital Downloads

Quick Reference

Context → Variants Cheat Sheet

Context Variants Generated
SOCIAL_IMAGE large, medium, thumb, og, blurhash, lqip, dominant_color
SOCIAL_VIDEO (short, HLS) master, 360p_playlist, 720p_playlist, 1080p_playlist, 720p_watermarked (or 360p_watermarked), preview, poster, thumb, og, blurhash, lqip, dominant_color
SOCIAL_VIDEO (long, HLS) master, 360p_playlist, 720p_playlist, 1080p_playlist, preview, poster, thumb, og, blurhash, lqip, dominant_color
PROFILE_PICTURE medium, thumb, blurhash, lqip
COVER_PHOTO large, thumb, og, blurhash, lqip
DM_IMAGE medium, thumb, blurhash, lqip
DM_VIDEO master, 360p_playlist, 720p_playlist
DM_DOCUMENT (none — secure download only)
PRODUCT_IMAGE large, medium, thumb, og, blurhash, lqip, dominant_color
PRODUCT_VIDEO (short, HLS) master, 360p_playlist, 720p_playlist, 1080p_playlist, 720p_watermarked (or 360p_watermarked), preview, poster, thumb, og, blurhash, lqip, dominant_color
PRODUCT_VIDEO (long, HLS) master, 360p_playlist, 720p_playlist, 1080p_playlist, preview, poster, thumb, og, blurhash, lqip, dominant_color
DIGITAL_PRODUCT (none — secure download only)
SHOP_BANNER large, thumb, og, blurhash, lqip
SHOP_LOGO medium, thumb, blurhash, lqip
EVENT_COVER large, medium, thumb, og, blurhash, lqip
EVENT_GALLERY large, medium, thumb, blurhash, lqip

Status Codes

Code Meaning
200 OK Success
400 BAD_REQUEST Invalid request data, limit reached, or item already exists
401 UNAUTHORIZED Missing, expired, or invalid token
403 FORBIDDEN Authenticated but not permitted (not the file owner, not the buyer)
404 NOT_FOUND File, order, or resource not found
422 UNPROCESSABLE_ENTITY Validation error — response data will contain field-level error messages
500 INTERNAL_SERVER_ERROR Server error — report to backend team with action_time

Authentication

All endpoints require: Authorization: Bearer <your_jwt_token>

Tokens are obtained from the auth endpoints (see Auth API documentation). A 401 means the token is expired or invalid — redirect the user to login.

CDN & File Delivery — Full Walkthrough

This is the learning-session writeup for how Cloudflare fits into File Thunder's delivery story. It complements FILE_THUNDER.md §8 (which is the terse reference version) with full end-to-end walkthroughs, diagrams, and setup steps.

Nothing in this doc changes the processing pipeline. FFmpeg/ImageMagick/ClamAV/watermarking stay exactly as they are. This is only about the last mile — how a finished file reaches a viewer.

Table of Contents

  1. The Core Idea, In One Picture
  2. The Three Delivery Lanes
  3. Category → Lane Mapping
  4. How Cloudflare Physically Sits in the Middle
  5. Walkthrough A — Social Video (Lane 1, Public/Cached)
  6. Walkthrough B — Direct Message Image (Lane 2, Private/Bypassed)
  7. Group Chats — Same Lane, Different Scale
  8. Voice Notes & Live Streaming — Do the Same Rules Apply?
  9. Video/Audio Calls — Not a File Thunder Concern
  10. Walkthrough C — Digital Product Purchase (Lane 3, Protected/Audited)
  11. Cache Invalidation — The Only Sharp Edge
  12. Cloudflare Setup Checklist
  13. Video-at-Scale Caveat

1. The Core Idea, In One Picture

Cloudflare is not storage. It's a worldwide network of caches sitting in front of your one real origin (MinIO). Think of MinIO as a single warehouse, and Cloudflare as thousands of small local libraries that keep copies of the popular stuff close to whoever's asking.

                         ┌───────────────────────────────────────────┐
                         │              CLOUDFLARE EDGE               │
                         │   (hundreds of mini-caches worldwide)      │
                         │                                             │
   Bob in Nairobi ──────▶│   Nairobi edge node                        │
                         │      │                                     │
                         │      │  cache HIT  → serve instantly        │
                         │      │  cache MISS → ask origin, cache it   │
                         └──────┼─────────────────────────────────────┘
                                │
                                │  (only on a cache miss, via secure tunnel)
                                ▼
                     ┌─────────────────────┐
                     │   MinIO (your VPS)   │
                     │   nexgate-public     │◀── ONLY this bucket is tunnel-reachable
                     │   nexgate-private    │◀── never exposed to Cloudflare
                     │   nexgate-digital    │◀── never exposed to Cloudflare
                     │   nexgate-raw        │◀── never exposed to anything external
                     └─────────────────────┘

Every request after the first one, from anyone near that edge node, never touches your VPS at all.

2. The Three Delivery Lanes

Every file File Thunder ever produces falls into exactly one lane. The lane is decided by who is allowed to see the file, nothing else.

┌────────────────────┬───────────────────────┬────────────────────────┬───────────────────────┐
│                     │  LANE 1: PUBLIC       │  LANE 2: PRIVATE       │  LANE 3: PROTECTED     │
├────────────────────┼───────────────────────┼────────────────────────┼───────────────────────┤
│ Who can view it     │ Anyone                │ The 1-2 people in the  │ Only the verified      │
│                     │                       │ conversation           │ buyer                  │
├────────────────────┼───────────────────────┼────────────────────────┼───────────────────────┤
│ Cloudflare cache?   │ YES — cached forever  │ NO — always bypassed   │ NO — always bypassed   │
├────────────────────┼───────────────────────┼────────────────────────┼───────────────────────┤
│ URL type            │ Plain permanent URL   │ Signed, 15-min expiry  │ Signed, 15-min expiry  │
│                     │ cdn.nexgate.com/key   │ straight from MinIO    │ + audit log + re-scan  │
├────────────────────┼───────────────────────┼────────────────────────┼───────────────────────┤
│ Bucket              │ nexgate-public        │ nexgate-private        │ nexgate-digital        │
├────────────────────┼───────────────────────┼────────────────────────┼───────────────────────┤
│ Who assembles URL   │ Main backend, no call │ Main backend calls FT  │ Main backend calls FT  │
│                     │ to File Thunder needed│ generateDownloadUrl()  │ generateDownloadUrl()  │
└────────────────────┴───────────────────────┴────────────────────────┴───────────────────────┘

Why Lane 2/3 can never be cached: their URLs are unique per request (a signature + expiry baked into the query string). Caching is keyed on the URL — a signed URL is never requested twice, so there's nothing to cache, and worse, a shared cache is the last place private bytes should sit.

3. Category → Lane Mapping

Every MediaContext value maps to exactly one lane:

LANE 1 — PUBLIC / CDN-CACHED (nexgate-public)
├── SOCIAL_IMAGE, SOCIAL_VIDEO
├── PROFILE_PICTURE, COVER_PHOTO
├── PRODUCT_IMAGE, PRODUCT_VIDEO
├── PRODUCT_PREVIEW_IMAGE, PRODUCT_PREVIEW_VIDEO   ← teasers, meant to be public
├── SHOP_BANNER, SHOP_LOGO
└── EVENT_COVER, EVENT_GALLERY

LANE 2 — PRIVATE / SIGNED, NEVER CACHED (nexgate-private)
├── DM_IMAGE
├── DM_VIDEO
└── DM_DOCUMENT

LANE 3 — PROTECTED / SIGNED + AUDITED, NEVER CACHED (nexgate-digital)
├── DIGITAL_PRODUCT
└── PRODUCT_PREVIEW_DOCUMENT   ← a document preview is still gated like a purchase

This matches DOWNLOAD_CONTEXTS already defined in MediaQueryServiceImpl — Lane 2 + Lane 3 together are exactly the contexts that go through generateDownloadUrl(). Everything else is Lane 1.

4. How Cloudflare Physically Sits in the Middle

Three ingredients, none of which touch your Java code:

  1. A subdomain: cdn.nexgate.com (and cdn-staging.nexgate.com for staging).
  2. DNS pointed at Cloudflare for that subdomain, "proxied" (orange cloud ON) — this is what makes Cloudflare the middleman instead of a pass-through record.
  3. A Cloudflare Tunnel (cloudflared, a small background process on your VPS) that opens an outbound connection from your server to Cloudflare. This is the important safety property:
   WITHOUT a tunnel                       WITH a tunnel
   ─────────────────                       ─────────────
   Internet ──▶ your VPS:9000              Internet ──▶ Cloudflare ──▶ tunnel ──▶ your VPS
   (port must be open to the world,        (port 9000 is never opened to the world —
    firewall rules, real attack surface)    the VPS "calls out", nobody calls in)

The tunnel is configured to route only to nexgate-public. nexgate-private, nexgate-digital, and nexgate-raw are not part of the tunnel's routing table — Cloudflare has no path to them even if it wanted one.

5. Walkthrough A — Social Video (Lane 1, Public/Cached)

Alice posts a 40-second clip. Bob, her friend, watches it later.

 ALICE (upload)                                    BOB (view, later)
 ──────────────                                    ─────────────────
 1. Tap "post"                                     6. Opens app, scrolls to Alice's post
        │                                                  │
        ▼                                                  ▼
 2. Main backend → File Thunder:                   7. Main backend reads post from its DB,
    "need upload slot for SOCIAL_VIDEO"                sees context = SOCIAL_VIDEO (Lane 1)
        │                                                  │
        ▼                                                  ▼
 3. File Thunder → presigned PUT URL              8. No call to File Thunder needed —
    → nexgate-raw (private, temp)                     main backend just glues:
        │                                              cdn.nexgate.com + stored objectKey
        ▼                                                  │
 4. Phone uploads raw bytes directly to MinIO              ▼
    (File Thunder never touches the bytes)         9. Bob's phone requests that URL
        │                                                  │
        ▼                                                  ▼
 5. MinIO event → File Thunder:                   10. Nearest Cloudflare edge (e.g. Nairobi):
    PENDING → UPLOADED → VideoWheel                    - cache MISS (first viewer ever)
      - FFmpeg: short-path transcode                      → tunnel → MinIO → fetch once
        (360p/720p/1080p, watermark, outro)              - cached at that edge from now on
      - thumbnail + blurhash + preview clip                   │
      - LIVE_PARTIAL after 360p, READY after all           ▼
      - raw original deleted                       11. Every later viewer near Nairobi:
      - keys written to Postgres (not URLs)             cache HIT — instant, MinIO untouched

Key point: steps 1–5 (upload + processing) are entirely unrelated to CDN — that pipeline already works today. The CDN only enters at step 7 onward, and it's a pure "glue the domain onto the key" operation. No signing, no expiry, no per-viewer logic, because it's public by design.

If the clip had been ≥3 minutes (HLS path), step 10/11 repeats per segment: the player first fetches master.m3u8 from the CDN, then a stream of .ts segments, each cached the same way.

6. Walkthrough B — Direct Message Image (Lane 2, Private/Bypassed)

Alice sends Bob a private photo in DM.

 1. Alice sends photo → uploads to nexgate-raw (same presigned-upload pattern as Lane 1)
 2. ImageWheel processes it → variants written to nexgate-private (NOT nexgate-public)
 3. Bob opens the conversation
        │
        ▼
 4. Main backend checks: is Bob actually a participant in this conversation? (authorization)
        │
        ▼
 5. Main backend calls File Thunder → generateDownloadUrl(fileId, requesterId=Bob)
        │
        ▼
 6. File Thunder issues a PRESIGNED GET URL — 15 min expiry, straight from MinIO
        │
        ▼
 7. Bob's phone fetches directly from MinIO — the CDN is never involved in this request at all
        │
        ▼
 8. URL expires after 15 minutes — if Bob reopens the chat later, a fresh one is issued

If a stranger somehow obtained this URL, it would already be expired or tied to a bucket that Cloudflare's tunnel can't even route to — two independent layers of protection.

7. Group Chats — Same Lane, Different Scale

Group chat media (photos/videos/documents shared in a group conversation, not a 1:1 DM) still belongs in Lane 2 — same bucket (nexgate-private), same generateDownloadUrl() call, never cached. File Thunder doesn't actually know or care whether it's a 1:1 DM or a 300-person group; it only trusts whatever requesterId the main backend passes in.

What actually changes is who's allowed to ask:

1:1 DM:        "is requesterId one of the 2 people in this conversation?"
Group chat:    "is requesterId a CURRENT member of this group?"

That authorization check lives entirely in the main backend, before it ever calls File Thunder — nothing to design differently on the File Thunder side for correctness.

The real new problem: fan-out

In a 1:1 DM, at most 2 people will ever request a given file — cheap no matter how it's delivered. In a group chat, one photo sent to a 300-person group means up to 300 people opening it:

300 members → 300 separate calls to generateDownloadUrl()
            → 300 separate presigned URLs generated
            → 300 separate direct-to-MinIO fetches
            → none of it cached (Lane 2 is always cache-bypassed) — all of it hits the one VPS

That's a materially different load profile than a private DM, and it's worth designing around before group chat ships — but the fix is about reducing redundant origin calls, not about caching private files at a shared edge.

Option 1 — do nothing special (fine for small/medium groups)

A few dozen members fetching a photo once is trivial load for MinIO on a VPS. If groups stay small (family/friend-sized, not broadcast channels), Lane 2 as-is is genuinely fine — no need to build anything extra ahead of time.

Option 2 — share ONE signed URL instead of one per member (for large groups)

Instead of every member independently triggering generateDownloadUrl(), main backend generates the presigned URL once — when the message is sent, or on first view — caches that single URL (e.g. in Redis, keyed by message/file id), and hands the same URL to every member who opens the chat until it expires (15 min). This turns "300 origin hits" into "1 origin hit, 300 people reusing the same short-lived link":

Member 1 opens chat ──▶ main backend: URL cached? NO ──▶ call generateDownloadUrl() ──▶ cache it
Member 2 opens chat ──▶ main backend: URL cached? YES ─▶ reuse same URL, no MinIO call
Member 3-300 ...      ──▶ same — reuse until the 15-min expiry, then regenerate once

Still never touches Cloudflare's shared cache — this is purely about cutting redundant presigned-URL generation and redundant MinIO traffic, which is a much simpler problem than "how do we safely cache private files."

When a "group" stops being private

If a group grows into "public broadcast channel" territory (thousands of members, more like a public feed than a private conversation), that's usually a sign the content has drifted out of Lane 2 entirely — worth revisiting whether it should be public (Lane 1) or a members-only public tier, rather than forcing an increasingly public thing through the private/signed lane. Not a decision to make now — just a pattern to recognize if it comes up.

8. Voice Notes & Live Streaming — Do the Same Rules Apply?

Neither exists in File Thunder today (no MediaContext for either, no live-ingest infra). They split sharply: one fits the existing model perfectly, the other breaks a core assumption.

Voice notes — yes, exactly the same rules

A voice note is just a short, finished audio file. It doesn't break anything this guide relies on: upload once → process once → immutable key → serve. It drops straight into the existing lanes:

Voice note in a DM/group chat  → Lane 2  (same as DM_IMAGE/DM_VIDEO — private bucket, 15-min
                                           presigned URL, never cached)
Voice note on a public post     → Lane 1  (same as SOCIAL_IMAGE — public bucket, plain CDN URL,
                                           cached forever)

The only new work is a new MediaContext (e.g. DM_VOICE_NOTE) and a small audio wheel (FFmpeg re-encode to a consistent format/bitrate, maybe waveform-peaks generation for the UI — same spirit as ImageWheel's blurhash). Delivery-wise, there is nothing new to design.

Live streaming — no, this breaks a core assumption

Everything so far relies on one idea: a file is finished, then served. Upload → process → immutable key → cache forever. Live streaming violates that at its core — the "file" is still being created while people are watching it. That changes several things at once:

  1. No presigned-PUT-to-MinIO upload. A live stream needs a real-time ingest protocol (RTMP, SRT, or WebRTC) hitting an ingest server continuously — not a client uploading one finished file.
  2. The manifest never stabilizes. VOD HLS's master.m3u8 flips once (LIVE_PARTIAL → READY) and never changes again — that's why "short-cache the manifest, cache segments forever" (§10) works for VOD. A live HLS playlist is a sliding window — rewritten every few seconds to point at only the newest 3-5 segments, continuously, for the whole stream duration. That manifest needs a near-zero cache TTL (1-2s) the entire time, not a one-time flip.
  3. Individual segments are still cacheable — each 2-6 second .ts chunk is immutable once written, so Cloudflare still helps a lot for segment traffic, just not for the manifest.
  4. Private/restricted live streams (e.g. a paid live-shopping event — Lane 2/3 equivalent) can't use one 15-minute signed link the way a DM photo does, because the session usually runs longer than that. This needs rotating/refreshing tokens or signed cookies for the stream's duration — new machinery, not something the current design already provides.

Because of points 1 and 4, this genuinely isn't "reuse the existing lanes" — it needs its own ingest + packaging pipeline. Building that in-house (RTMP ingest server + real-time FFmpeg transcode + rolling manifest) is a serious lift, well beyond a delivery-lane decision. The pragmatic path most platforms take at this stage is to not build live ingest in-house and use a managed service (Cloudflare Stream Live, Mux, Agora, etc.) that hands back an HLS URL to embed — which then does slide back into the Lane 1 viewing model, just with someone else running the real-time part.

Bottom line: voice notes are a non-event — add the context, reuse the lanes. Live streaming is a separate architectural track that shouldn't be forced into this document's model; buying that piece is worth strongly considering before building it in-house.

9. Video/Audio Calls — Not a File Thunder Concern

This is the sharpest split of all. Everything else in this guide — even live streaming — is about media that gets stored (however briefly) and then fetched by someone else, later or from elsewhere. A 1:1 or group call is neither.

  1. Nothing is ever "delivered" through the storage layer. A call is devices talking to each other in as close to real time as possible — 100-300ms round trip matters, not "eventually arrives." Routing it through MinIO → Cloudflare → viewer would add multiple seconds of latency. Wrong tool entirely.
  2. The transport is peer-to-peer or SFU, not HTTP/CDN. Calls use WebRTC: media streams connect device-to-device directly when possible, and go through a lightweight SFU (Selective Forwarding Unit) relay when a direct connection isn't possible (NAT/firewall issues) — the SFU forwards packets, it doesn't store them. A completely different network model from "upload a file, cache it, serve it."
  3. Normally nothing to process, scan, or watermark. No FFmpeg wheel, no ClamAV, no thumbnails — there's no file, just a live signal that's gone the instant it's forwarded.
  4. Cloudflare's role, if any, is different too: not caching — it would be TURN/STUN relay (helping two devices behind NATs find each other, or relaying when they can't connect directly). Cloudflare Calls / Cloudflare Realtime is their product here, but it conceptually replaces the SFU, not the CDN edge cache used everywhere else in this guide.

Calls need their own separate service, not a feature of File Thunder — worth calling out explicitly so this doesn't get scoped into "file delivery" work by mistake later.

The one place calls do touch File Thunder: if a call gets recorded and saved afterward (e.g. a missed-call "voicemail," or a saved video-call recording), it's no longer a live call at that point — it's a finished file, and it goes right back into the existing model: probably DM_VIDEO or a new CALL_RECORDING context, Lane 2, same rules as any other private media.

10. Walkthrough C — Digital Product Purchase (Lane 3, Protected/Audited)

Alice sells a paid PDF course. Bob buys it.

 1. Bob completes payment → main backend confirms order in its own DB
        │
        ▼
 2. Main backend calls File Thunder → generateDownloadUrl(fileId, requesterId=Bob)
        │
        ▼
 3. File Thunder checks:
      - context is DIGITAL_PRODUCT? ✓ (only DIGITAL_PRODUCT / DM_DOCUMENT / PRODUCT_PREVIEW_DOCUMENT
        are allowed through this endpoint at all)
      - file status == READY? ✓
        │
        ▼
 4. File Thunder writes a DownloadAuditEntity row:
      fileId, ownerId, requesterId=Bob, objectKey, requestedAt, expiresAt
      (a permanent record of exactly who downloaded what, and when)
        │
        ▼
 5. Presigned GET URL issued — 15 min expiry, bucket = nexgate-digital
        │
        ▼
 6. Bob downloads directly from MinIO — never cached, never at a shared edge

The audit row is what lets you answer "who downloaded this file, and when" months later — important for a paid-goods platform if there's ever a dispute or a leak investigation.

11. Cache Invalidation — The Only Sharp Edge

Because object keys are unique per fileId/variant and are never overwritten, a cached copy at the edge can basically never go stale — new content always gets a new key. That removes 99% of the usual "how do I bust the cache" headache. Three known exceptions:

Case Problem Fix
HLS master.m3u8 Same key, first written when only 360p is ready (LIVE_PARTIAL), then rewritten when all qualities finish (READY) Short cache lifetime on this one file only (10–30s). The .ts segments never change — cache those forever
User thumbnail overwrite user-poster.webp reuses the same key on re-upload Prefer versioning the key (user-poster-v2.webp) over a cache purge call
Profile picture / cover / shop logo Same "identity slot gets overwritten" pattern Same fix — version the key rather than relying on purge

Versioning the key is preferred over calling Cloudflare's purge API because it needs no extra network call and can never race (old URL still valid until the new one is swapped in by the app).

12. Cloudflare Setup Checklist

Concrete steps, in order, when it's time to actually turn this on:

  1. Add the domain to Cloudflare (or subdomain, if the root domain is hosted elsewhere).
  2. Create cdn.nexgate.com DNS record, proxied (orange cloud ON). Create cdn-staging.nexgate.com the same way for staging.
  3. Install cloudflared on the VPS, authenticate it to your Cloudflare account, create a tunnel, and route it to http://localhost:9000 (MinIO) — but only for requests that resolve to the nexgate-public bucket path.
  4. Re-enable anonymous read, but only on nexgate-public (the other three buckets stay fully private — do not touch their bucket policy).
  5. Firewall the MinIO port so it's not reachable from the raw internet at all — the tunnel is the only path in. (If a tunnel isn't used yet, at minimum restrict to Cloudflare's published IP ranges.)
  6. Set a Cache Rule in Cloudflare for cdn.nexgate.com/*: cache everything, respect Cache-Control: public, max-age=31536000, immutable from origin — except */master.m3u8, which gets its own short-TTL rule (10–30s).
  7. Flip the app config (already scaffolded per FILE_THUNDER.md §8): 
    ft.storage.mode=cdn
ft.cdn.base-url=https://cdn.nexgate.com
  8. Smoke test: upload a test image, confirm the CDN URL 200s, re-request it and confirm the cf-cache-status response header flips from MISS to HIT on the second request.

13. Video-at-Scale Caveat

Cloudflare's free/pro plans restrict using the CDN as a bulk video host at large scale (their terms are written around "no non-HTML heavy media" abuse). Images, thumbnails, blurhash/lqip, and HLS manifests are completely fine on the free tier — this whole design is ideal for those. But if social video volume grows large, the two Cloudflare-native upgrade paths to know about later are:

Not a decision to make now — just something to keep in mind as a later migration, not a redesign.