File Thunder Arch & Flow
File Thunder is the dedicated media-processing microservice for the NexGate / Veepii platform. It handles all file ingestion, processing, storage, and serving-readiness — the main backend never touches raw bytes.
Table of Contents
- Architecture Overview
- Storage: Buckets & Object Keys
- MediaDomain & MediaContext
- Upload Flow — End to End
- The Four Wheels
- Progress Tracking (Redis → SSE)
- Watermarking
- CDN & File Serving Per Environment
- Security
- API Reference & How to Consume
1. Architecture Overview
┌─────────────────────────────────────────────────────────┐
│ Main Backend │
│ - Only public-facing API │
│ - Calls File Thunder over internal HTTP (HMAC signed) │
│ - Subscribes to Redis for progress → pushes SSE │
└────────────┬──────────────────────────┬─────────────────┘
│ HTTP (sync) │ Redis Pub/Sub (async)
▼ ▼
┌─────────────────────────────────────────────────────────┐
│ File Thunder │
│ │
│ API Profile Worker Profile │
│ ───────────── ────────────── │
│ UploadController UploadConfirmedWorker │
│ MediaQueryController ThumbnailWorker │
│ RawFilePurgeJob │
│ │
│ Wheels (processing engines) │
│ ────────────────────────── │
│ ImageWheel VideoWheel ScanWheel (MinIO ops) │
└────────────┬──────────────────────────┬─────────────────┘
│ │
▼ ▼
MinIO RabbitMQ
(4 private buckets) (internal event routing)
Key constraints:
- File Thunder is internal only — never exposed to the internet directly
- Main backend is the only caller of File Thunder HTTP APIs
- File Thunder never touches file bytes on upload — client uploads directly to MinIO via presigned URL
- Redis is FT-internal — used for progress pub/sub and nonce replay protection
- RabbitMQ is FT-internal — used to route events between listeners and workers
2. Storage: Buckets & Object Keys
Buckets
| Bucket | Purpose | Access |
|---|---|---|
nexgate-raw |
Temporary upload landing zone — 24h TTL | Private |
nexgate-public |
Processed media served to end users | Private (CDN in front) |
nexgate-private |
Internal system assets (outros, future forensic assets) | Private |
nexgate-digital |
Digital product files — ClamAV scanned, download only | Private |
All buckets are fully private. MinIO is never directly accessible from the internet. Public content is served exclusively through the CDN (Cloudflare), which pulls from MinIO origin.
Object Key Pattern
{domain}/{ownerId}/{fileId}/{variant}
Examples:
posts/550e8400-e29b-41d4-a716-446655440000/f7e8d9.../original
posts/550e8400-e29b-41d4-a716-446655440000/f7e8d9.../large.webp
posts/550e8400-e29b-41d4-a716-446655440000/f7e8d9.../thumb.webp
posts/550e8400-e29b-41d4-a716-446655440000/f7e8d9.../hls/master.m3u8
posts/550e8400-e29b-41d4-a716-446655440000/f7e8d9.../hls/360p/360p.m3u8
system/outro/{ownerId}/{height}p.mp4 ← in nexgate-private
Rule: The database stores object keys only, never full URLs. URLs are assembled at the resolver boundary (CDN base URL + key in prod, presigned GET in local).
3. MediaDomain & MediaContext
MediaDomain
Defines where in the storage hierarchy a file lives. It is the top-level folder in the object key. This is an enum — the main backend must send one of these exact values.
| Value | Used For |
|---|---|
POSTS |
Social posts — images and videos |
PROFILES |
Profile pictures, cover photos |
MESSAGES |
Direct message attachments |
PRODUCTS |
Product images, product videos, digital downloads |
SHOPS |
Shop banners, shop logos |
EVENTS |
Event covers, event gallery images |
MediaContext
Defines how a file is processed. This drives the entire processing pipeline. No logic is attached to domain — all logic is driven by context.
| Value | Type | Processing Pipeline |
|---|---|---|
SOCIAL_IMAGE |
Image | ImageWheel — orient, strip EXIF, WebP variants, blurhash, lqip |
SOCIAL_VIDEO |
Video | VideoWheel — transcode + social watermark + outro + thumbnail |
PROFILE_PICTURE |
Image | ImageWheel |
COVER_PHOTO |
Image | ImageWheel |
DM_IMAGE |
Image | ImageWheel |
DM_VIDEO |
Video | VideoWheel — transcode, no watermark |
DM_DOCUMENT |
Any | ScanWheel — ClamAV scan |
PRODUCT_IMAGE |
Image | ImageWheel |
PRODUCT_VIDEO |
Video | VideoWheel — transcode + text watermark, no outro |
DIGITAL_PRODUCT |
Any | ScanWheel — ClamAV dual scan |
SHOP_BANNER |
Image | ImageWheel |
SHOP_LOGO |
Image | ImageWheel |
EVENT_COVER |
Image | ImageWheel |
EVENT_GALLERY |
Image | ImageWheel |
Validation rules enforced at API layer:
- Image contexts only accept image MIME types; video contexts only accept video MIME types
SOCIAL_VIDEOrequiresusernamefield (used for personalized watermark + outro)- Max file sizes: images 20 MB, videos 2 GB, digital products 500 MB
- HEIC/HEIF are rejected — parser attack surface too high
4. Upload Flow — End to End
Standard Media Upload
Main Backend File Thunder MinIO Client (Browser)
│ │ │ │
│ POST /api/v1/upload/request │ │ │
│ (HMAC signed) │ │ │
│──────────────────────────────▶│ │ │
│ │ validate + create DB │ │
│ │ record (PENDING) │ │
│ │ publish PENDING │ │
│ │ to Redis │ │
│ │ generate presigned PUT │ │
│ │─────────────────────────▶ │
│ { fileId, presignedUrl } │◀──────────────────────── │
│◀──────────────────────────────│ │ │
│ │ │ │
│ return presignedUrl │ │ │
│ to client │ │ │
│──────────────────────────────────────────────────────────────────────────▶ │
│ │ │ │
│ │ │ PUT (file bytes) │
│ │ │◀──────────────────│
│ │ │ 200 OK │
│ │ │──────────────────▶│
│ │ │ │
│ │ ◀── MinIO event ────────│ s3:ObjectCreated │
│ │ MinioEventListener │ │
│ │ PENDING → UPLOADED │ │
│ │ publish UPLOADED→Redis │ │
│ │ publish UploadConfirmed │ │
│ │ to RabbitMQ │ │
│ │ │ │
│ │ UploadConfirmedWorker │ │
│ │ routes by mimeType │ │
│ │ UPLOADED → PROCESSING │ │
│ │ publish PROCESSING→Redis│ │
│ │ │ │
│ │ [ImageWheel or │ │
│ │ VideoWheel runs] │ │
│ │ │ │
│ │ variants uploaded │ │
│ │ to nexgate-public ──────▶ │
│ │ raw deleted from │ │
│ │ nexgate-raw ────────────▶ │
│ │ PROCESSING → READY │ │
│ │ publish READY → Redis │ │
│ │ │ │
User Custom Thumbnail Upload
Main Backend File Thunder MinIO
│ │ │
│ POST /api/v1/upload/ │ │
│ thumbnail/{fileId} │ │
│──────────────────────────────▶│ │
│ │ generate presigned PUT │
│ │ key: .../custom- │
│ │ thumbnail │
│ { presignedUrl } │ │
│◀──────────────────────────────│ │
│ client uploads thumbnail │ │
│─────────────────────────────────────────────────────────▶
│ │◀── MinIO event ─────────│
│ │ MinioEventListener │
│ │ detects suffix= │
│ │ "custom-thumbnail" │
│ │ → ThumbnailWorker │
│ │ download → verify magic │
│ │ bytes → ImageMagick │
│ │ variants → save to │
│ │ entity.userThumbnail │
│ │ → delete raw │
File Status Lifecycle
PENDING → UPLOADING → UPLOADED → SCANNING → PROCESSING → LIVE_PARTIAL → READY
▲
(short video: after 360p uploaded)
(long video: after 360p + partial HLS)
5. The Four Wheels
Wheels are the processing engines. Each runs in the worker profile.
Wheel 1 — ImageWheel (ImageMagick)
Triggered by: image MIME type on any image context
Pipeline:
download raw from nexgate-raw
│
▼
auto-orient + strip EXIF
│
├──▶ large.webp (1920px max width, shrink only, Q85)
├──▶ medium.webp (800px max width, shrink only, Q82)
├──▶ thumb.webp (300px max width, shrink only, Q80)
├──▶ og.webp (1200×630 center crop, Q85)
│ └─ falls back to medium if source width < 1200px
├──▶ blurhash (encoded from 32×32 downsample)
└──▶ lqip (10×10 forced, base64 WebP data URI)
all WebP variants → nexgate-public
keys + blurhash + lqip → media_files.variants (JSONB)
raw deleted from nexgate-raw
status → READY
Skipping logic: variant is skipped if source is already within bounds (never upscale).
Wheel 2 — VideoWheel (FFmpeg / Jaffree)
Triggered by: video MIME type on any video context
Step 1 — Probe & Validate
download raw from nexgate-raw
magic byte verify (must be real video container)
FFprobe → codedWidth, codedHeight, rotation, duration, codec
displayWidth/displayHeight = rotation-aware (swap for 90°/270°)
size gate: > 2 GB → reject
duration gate: > 4h → reject
route: duration < 3min → processShort()
duration ≥ 3min → processLong()
Step 2a — Short Path (< 3 min)
transcode() runs 3 variants in sequence:
360p (CRF 28, 96k audio) → upload → status LIVE_PARTIAL
720p (CRF 23, 128k audio) → upload
1080p (CRF 21, 192k audio) → upload
filter_complex:
[0:v] split=2 [main][blur]
[blur] scale={W}:{H}, boxblur=20:5 [bg]
[main] scale={w}:{h} (fit within target, keep AR) [fg]
[bg][fg] overlay=x=(W-w)/2:y=(H-h)/2 [out]
+ rotation transpose prepended if needed
+ watermark drawtext chain appended
if SOCIAL_VIDEO:
append outro via concat (no re-encode)
skip logic: source displayWidth >= targetW OR displayHeight >= targetH
Step 2b — Long Path (≥ 3 min) — HLS Adaptive
transcodeHls() per variant:
360p → segments + .m3u8 → upload → partial master.m3u8 → LIVE_PARTIAL
720p → segments + .m3u8 → upload
1080p → segments + .m3u8 → upload → full master.m3u8 → READY
segment length: 2s
playlist type: VOD
master.m3u8 key: {keyBase}/hls/master.m3u8
variant keys: {keyBase}/hls/{name}/{name}.m3u8
Step 3 — Thumbnail & Preview (both short and long)
selectBestFrame():
5 candidates at 15/30/45/60/75% of duration
scored by: brightness gate (30–230) + Laplacian variance (sharpness)
FFmpeg extracts 720px JPEG per candidate → pick winner
buildThumbnailVariants():
poster.webp (1280px, Q85)
thumb.webp (480px, Q80)
og.webp (1200×630 center crop, Q85)
blurhash (from 32×32 downsample)
lqip (10×10 forced, base64 WebP data URI)
dominant_color (#RRGGBB from 1×1 squish)
extractPreviewClip():
skip first 5% (max 2s) → read 6s → setpts=0.5*PTS → 3s at 2× speed
blur-pad 360×640, muted
watermarked (same moving watermark as main video)
all thumbnail variants → nexgate-public
raw deleted from nexgate-raw after all variants done
Wheel 3 — ScanWheel (ClamAV)
Triggered by: DIGITAL_PRODUCT and DM_DOCUMENT contexts only
Social content (images and videos) is never scanned by ClamAV — FFmpeg/ImageMagick re-encode is the sanitisation.
Pipeline:
download raw from nexgate-raw
│
▼
SHA-256 hash → check file_hashes table
│
├── hash known + clean → skip scan, copy to nexgate-digital, READY (dedup fast lane)
│
└── hash unknown →
ClamAV scan #1 (upload scan)
if clean → move to nexgate-digital
ClamAV scan #2 (pre-download scan, confirms in-transit integrity)
if clean → save hash, status READY
if infected → status FAILED, quarantine
Wheel 4 — MinIO Operations (used by all wheels)
Not a standalone service — MinIO operations are helpers used across all wheels:
| Operation | Used By |
|---|---|
| Presigned PUT URL | Upload request endpoint |
| Download object | All wheels (download raw for processing) |
| Upload object | All wheels (upload processed variants) |
| Remove object | All wheels (delete raw after processing) |
| Stat object | OutroService (cache check), ScanWheel (dedup check) |
6. Progress Tracking (Redis → SSE)
File Thunder publishes status changes to Redis. The main backend subscribes and drives SSE to the client. SSE is the main backend's responsibility — File Thunder only publishes.
What File Thunder Does
On every status change:
// 1. Cache current status in Redis (24h TTL)
redisTemplate.opsForValue().set("ft:status:{fileId}", status.name(), 24h);
// 2. Publish to per-file channel
redisTemplate.convertAndSend("ft:progress:{fileId}", status.name());
// 3. Append to timeline in Postgres
entity.timeline.add({ status, at })
What the Main Backend Must Do
// Subscribe to the file's progress channel
redisTemplate.subscribe((message, pattern) -> {
String status = message.toString();
sseEmitter.send(SseEmitter.event().data(status));
}, "ft:progress:" + fileId);
Status Channel Key
ft:progress:{fileId} ← subscribe here for live updates
ft:status:{fileId} ← read here for current status (24h cached)
7. Watermarking
Moving Watermark
Applied to all video variants and preview clips. Watermark position cycles through 4 corners every 3 seconds (5 seconds for social).
Text-only (PRODUCT_VIDEO):
drawtext: fontsize=max(14,H/40), white@0.65 + black shadow
position: floor(t/3) mod 4 → top-left, top-right, bottom-right, bottom-left
text: from watermark.text property (default: "NexGate")
2-point diagonal: Pos A (22%, 28%) ↔ Pos B (58%, 65%) — switches every 5s
Logo: nexgate_logo_white.svg, 36×36, 65% opacity
@ symbol: orange #F06023@0.85 + shadow
username: white@0.85 + shadow, below logo
Outro (SOCIAL_VIDEO only)
A personalized outro clip is appended to the watermarked variant (not clean variants).
Generation:
template: outro_template_with_sfx.mp4 (classpath resource)
font: Sora SemiBold 600 (classpath resource)
drawtext: @ in orange #F06023, username in cream #F4EEE9
fade-in: invisible before t=0.75s, fully opaque by t=1.2s
scale to match variant resolution (360p / 720p / 1080p)
append via: -f concat -safe 0 -c copy (no re-encode, fast)
Caching:
Generated once per ownerId + resolution
Cached to: nexgate-private / system/outro/{ownerId}/{height}p.mp4
On next upload: cache hit → download and reuse, skip generation
8. CDN & File Serving Per Environment
All MinIO buckets are private. Access to public content is environment-dependent.
Environment Matrix
| Environment | Serving Method | Config Value |
|---|---|---|
| Local dev | Presigned GET URLs (MinIO direct, short-lived) | ft.storage.mode=local |
| Staging | CDN — cdn-staging.nexgate.com |
ft.storage.mode=cdn |
| Production | CDN — cdn.nexgate.com |
ft.storage.mode=cdn |
Local / Dev
- App generates a presigned GET URL from MinIO (e.g. 1-hour expiry)
- URL is returned directly to the client
- MinIO must be reachable by the client
- Good enough for development and manual testing
Staging & Production (CDN)
- App assembles:
{ft.cdn.base-url} + "/" + objectKey - Cloudflare sits in front of MinIO origin
- Cloudflare pulls from MinIO on cache miss using a service credential
- Every subsequent request served from Cloudflare edge — MinIO never hit again
- Cost: Cloudflare bandwidth is free; MinIO only pays VPS bandwidth once per cache miss
Properties
# Local
ft.storage.mode=local
ft.cdn.base-url=
# Staging
ft.storage.mode=cdn
ft.cdn.base-url=https://cdn-staging.nexgate.com
# Production
ft.storage.mode=cdn
ft.cdn.base-url=https://cdn.nexgate.com
URL Assembly Logic (resolver boundary)
// local mode → presigned GET from MinIO
// cdn mode → CDN base + object key
String url = storageMode.equals("cdn")
? cdnBaseUrl + "/" + objectKey
: minioClient.getPresignedObjectUrl(GET, bucket, objectKey, 1h);
Private content (DIGITAL_PRODUCT, DM_DOCUMENT): always presigned GET URLs regardless of environment — these are never cached by CDN. Generated fresh per download request (15-minute expiry, audit logged).
9. Security
CORS
Configured via ft.cors.* properties. The CORS filter runs before all other filters so OPTIONS preflight requests are handled without requiring HMAC headers.
# Local
ft.cors.allowed-origins=http://localhost:3000
# Production
ft.cors.allowed-origins=https://nexgate.com,https://www.nexgate.com
ft.cors.allowed-methods=GET,POST,PUT,DELETE,OPTIONS
ft.cors.allowed-headers=*
ft.cors.allow-credentials=false
ft.cors.max-age-seconds=3600
Service-to-Service HMAC Authentication
Every HTTP request from the main backend to File Thunder must be HMAC-SHA256 signed.
File Thunder rejects any unsigned or incorrectly signed request with 401 Unauthorized.
Filter order:
Request → CorsFilter (HIGHEST_PRECEDENCE)
→ HmacAuthFilter (HIGHEST_PRECEDENCE + 1)
→ Controllers
Required Headers
| Header | Description | Example |
|---|---|---|
X-Service-Id |
Identifier of the calling service | nexgate-main |
X-Timestamp |
Unix epoch seconds at time of request | 1718123456 |
X-Nonce |
Random UUID — unique per request | f47ac10b-58cc-... |
X-Signature |
HMAC-SHA256 hex of the canonical string | a3f5c2... |
Canonical String
METHOD\n
REQUEST_URI\n
TIMESTAMP\n
NONCE\n
HEX(SHA-256(requestBody))
Example for POST /api/v1/upload/request:
POST
/api/v1/upload/request
1718123456
f47ac10b-58cc-4372-a567-0e02b2c3d479
e3b0c44298fc1c149afb... ← SHA-256 of the JSON body
Signature Computation
// Main backend — how to sign a request
String bodyHash = HexFormat.of().formatHex(
MessageDigest.getInstance("SHA-256").digest(requestBodyBytes)
);
String canonical = method + "\n" + uri + "\n" + timestamp + "\n" + nonce + "\n" + bodyHash;
Mac mac = Mac.getInstance("HmacSHA256");
mac.init(new SecretKeySpec(sharedSecret.getBytes(UTF_8), "HmacSHA256"));
String signature = HexFormat.of().formatHex(mac.doFinal(canonical.getBytes(UTF_8)));
// Set headers on outgoing request
request.setHeader("X-Service-Id", "nexgate-main");
request.setHeader("X-Timestamp", String.valueOf(Instant.now().getEpochSecond()));
request.setHeader("X-Nonce", UUID.randomUUID().toString());
request.setHeader("X-Signature", signature);
What File Thunder Verifies (in order)
1. All 4 headers present → 401 if any missing
2. X-Service-Id in allowed list → 401 if unknown
3. X-Timestamp within ±5 minutes → 401 if stale (replay protection)
4. X-Nonce not seen before → 401 if duplicate
(stored in Redis with 10-min TTL) (replay attack blocked)
5. Recompute HMAC, timing-safe compare → 401 if mismatch
(MessageDigest.isEqual) (prevents timing oracle)
Properties
ft.security.hmac.secret=<long-random-secret-same-in-both-services>
ft.security.hmac.allowed-service-ids=nexgate-main
ft.security.hmac.timestamp-tolerance-seconds=300
ft.security.hmac.nonce-ttl-seconds=600
The secret must be identical in both File Thunder and the main backend config.
Use a different value per environment (local / staging / prod).
10. API Reference & How to Consume
Base URL: http://file-thunder:8081 (internal network only)
All requests must include HMAC headers — see Section 9.
POST /api/v1/upload/request
Request a presigned URL to upload a file directly to MinIO.
Request body:
{
"ownerId": "550e8400-e29b-41d4-a716-446655440000",
"domain": "POSTS",
"context": "SOCIAL_VIDEO",
"originalFilename": "my-video.mp4",
"mimeType": "video/mp4",
"fileSizeBytes": 104857600,
"username": "josh_dev"
}
username is required when context is SOCIAL_VIDEO. All other fields are always required.
Response:
{
"status": "success",
"message": "Presigned upload URL generated",
"data": {
"fileId": "f7e8d9fa-...",
"presignedUrl": "http://minio:9000/nexgate-raw/posts/.../original?X-Amz-...",
"objectKey": "posts/{ownerId}/{entityId}/{fileId}/original",
"bucket": "nexgate-raw",
"expiresInSeconds": 1800
}
}
What to do next: PUT the file bytes directly to presignedUrl from the client browser. No auth headers needed on the PUT — the presigned URL is self-authenticating.
POST /api/v1/upload/thumbnail/{fileId}
Request a presigned URL to replace the system-generated video thumbnail with a user-supplied one. Only valid after the video file has been processed (status READY).
Response:
{
"status": "success",
"message": "Thumbnail presigned upload URL generated",
"data": {
"fileId": "f7e8d9fa-...",
"presignedUrl": "http://minio:9000/nexgate-raw/.../custom-thumbnail?X-Amz-...",
"objectKey": "posts/.../custom-thumbnail",
"bucket": "nexgate-raw",
"expiresInSeconds": 1800
}
}
Accepted formats: JPEG, PNG, WebP only (verified by magic bytes).
GET /api/v1/media/{fileId}
Get full metadata for a file, including all processed variants.
Response:
{
"status": "success",
"data": {
"fileId": "f7e8d9fa-...",
"ownerId": "550e8400-...",
"domain": "POSTS",
"context": "SOCIAL_VIDEO",
"status": "READY",
"mimeType": "video/mp4",
"variants": {
"360p": "posts/.../360p.mp4",
"720p": "posts/.../720p.mp4",
"1080p": "posts/.../1080p.mp4",
"poster": "posts/.../poster.webp",
"thumb": "posts/.../thumb.webp",
"og": "posts/.../og.webp",
"preview_3s": "posts/.../preview_3s.mp4",
"blurhash": "LKO2?U%2Tw=w]~RBVZRi};RPxuwH",
"lqip": "data:image/webp;base64,...",
"dominant_color":"#1A2B3C"
},
"userThumbnail": null,
"timeline": [
{ "status": "PENDING", "at": "2026-06-15T10:00:00" },
{ "status": "UPLOADED", "at": "2026-06-15T10:00:05" },
{ "status": "PROCESSING", "at": "2026-06-15T10:00:06" },
{ "status": "LIVE_PARTIAL","at": "2026-06-15T10:00:45" },
{ "status": "READY", "at": "2026-06-15T10:02:10" }
],
"createdAt": "2026-06-15T10:00:00",
"updatedAt": "2026-06-15T10:02:10"
}
}
Thumbnail resolution rule:
userThumbnail != null ? use userThumbnail : use variants
Both userThumbnail and variants use the same key names (poster, thumb, og, blurhash, lqip, dominant_color).
GET /api/v1/media/{fileId}/download?requesterId={uuid}
Generate a time-limited download URL for private files (DIGITAL_PRODUCT, DM_DOCUMENT only).
Every call is audit-logged.
Response:
{
"status": "success",
"data": {
"url": "http://minio:9000/nexgate-digital/...?X-Amz-...",
"expiresInSeconds": 900
}
}
GET /api/v1/quota/{ownerId}
Get the total storage used by an owner across all their processed files. Updated atomically as each file is processed — never stale.
Response:
{
"status": "success",
"message": "Storage usage",
"data": {
"ownerId": "550e8400-e29b-41d4-a716-446655440000",
"usedBytes": 1073741824,
"usedMb": 1024.0,
"usedGb": 1.0
}
}
How quota is tracked:
- Every wheel (
ImageWheel,VideoWheel,ScanWheel) callsStorageUsageService.trackUsage(ownerId, bytes)after uploading processed variants - Tracked bytes = size of the processed variants, not the raw upload
- Stored in
user_storage_quotatable, upserted atomically per owner releaseUsage()is called on hard delete (soft deletes do not release quota)- Returns
0if the owner has no tracked usage yet (never uploaded)
Use this endpoint to:
- Display storage usage in the main backend dashboard
- Enforce storage limits before allowing a new upload request
Error Responses
{ "error": "Missing required security headers" } ← 401
{ "error": "Request timestamp out of acceptable window" } ← 401
{ "error": "Nonce already used — possible replay attack"} ← 401
{ "error": "Invalid signature" } ← 401
Validation errors return 400 with field-level messages via @ControllerAdvice.
This document covers the full File Thunder system as of 2026-06-15. CDN wiring (Cloudflare), Kafka integration, lazy transcode, and forensic watermarks are deferred.
No comments to display
No comments to display