Files-nexgate-service(2)

File Management Service
File Thunder Arch & Flow

File Management Service

Author: Josh S. Sakweli, Backend Lead Team
Last Updated: 2025-09-23
Version: v1.0

Short Description: The File Management Service handles file uploads, downloads, and management for the NextGate social commerce platform. This service provides secure file storage with support for multiple file types, automatic metadata extraction, and organized directory structures for profiles, categories, and shops.

Hints:

Maximum file size: 25MB per file
Supported file types: Images (JPEG, PNG, GIF, WebP, BMP, SVG), Videos (MP4, AVI, MOV, WebM, MKV), Documents (PDF, DOC, DOCX, TXT, XLS), Audio (MP3, WAV, OGG, AAC)
All uploaded files are publicly accessible via direct URLs
Files are automatically organized by directory type (profile, categories, shops)
Image dimensions and metadata are automatically extracted
Unique filenames are generated to prevent conflicts
MD5 checksums are calculated for file integrity verification

Endpoints

1. Upload Multiple Files

Purpose: Upload multiple files to a specified directory with comprehensive metadata extraction

Endpoint: POST https://apinexgate.glueauth.com/api/v1/files/upload

Access Level: 🔒 Protected (Requires JWT Bearer token)

Authentication: Bearer Token (JWT access token required)

Request Headers:

Header	Type	Required	Description
Authorization	string	Yes	Bearer {access_token}
Content-Type	string	Yes	Must be "multipart/form-data"

Request Form Data Parameters:

Parameter	Type	Required	Description	Validation
files	file[]	Yes	Array of files to upload	Max 25MB per file, supported file types only
directory	string	Yes	Target directory for files	enum: PROFILE, CATEGORIES, SHOPS

Response JSON Sample:

{
  "success": true,
  "httpStatus": "OK",
  "message": "Files uploaded successfully",
  "action_time": "2025-09-23T10:30:00",
  "data": {
    "uploadedFiles": [
      {
        "fileName": "123e4567-e89b-12d3-a456-426614174000.jpg",
        "originalFileName": "product-image.jpg",
        "objectKey": "profile/123e4567-e89b-12d3-a456-426614174000.jpg",
        "directory": "PROFILE",
        "contentType": "image/jpeg",
        "fileSize": 2048576,
        "fileSizeFormatted": "2.0 MB",
        "permanentUrl": "https://files.nextgate.com/bucket-uuid/profile/123e4567-e89b-12d3-a456-426614174000.jpg",
        "thumbnailUrl": "https://files.nextgate.com/bucket-uuid/profile/123e4567-e89b-12d3-a456-426614174000.jpg",
        "fileExtension": ".jpg",
        "fileType": "IMAGE",
        "isImage": true,
        "isVideo": false,
        "isDocument": false,
        "isAudio": false,
        "width": 1920,
        "height": 1080,
        "dimensions": "1920x1080",
        "checksum": "d41d8cd98f00b204e9800998ecf8427e",
        "uploadedAt": "2025-09-23T10:30:00",
        "uploadedBy": "123e4567-e89b-12d3-a456-426614174000",
        "isPublic": true
      }
    ],
    "totalFiles": 3,
    "successfulUploads": 2,
    "failedUploads": 1,
    "totalSize": 5242880,
    "totalSizeFormatted": "5.0 MB",
    "uploadedAt": "2025-09-23T10:30:00",
    "message": "2 files uploaded successfully, 1 failed",
    "errors": [
      "Failed to upload large-file.mp4: File size exceeds maximum limit of 25MB"
    ]
  }
}

Response Fields:

Field	Description
success	Boolean indicating operation success
httpStatus	HTTP status code as string
message	Overall operation success message
action_time	ISO 8601 timestamp of response generation
data.uploadedFiles	Array of successfully uploaded file details
data.uploadedFiles[].fileName	System-generated unique filename with UUID
data.uploadedFiles[].originalFileName	Original filename provided by user
data.uploadedFiles[].objectKey	Full object path in storage system
data.uploadedFiles[].directory	Directory where file was stored
data.uploadedFiles[].contentType	MIME type of the uploaded file
data.uploadedFiles[].fileSize	File size in bytes
data.uploadedFiles[].fileSizeFormatted	Human-readable file size (e.g., "2.0 MB")
data.uploadedFiles[].permanentUrl	Direct public URL to access the file
data.uploadedFiles[].thumbnailUrl	Thumbnail URL (same as permanentUrl for images)
data.uploadedFiles[].fileExtension	File extension including dot (e.g., ".jpg")
data.uploadedFiles[].fileType	File category (IMAGE, VIDEO, DOCUMENT, AUDIO, OTHER)
data.uploadedFiles[].isImage	Boolean indicating if file is an image
data.uploadedFiles[].isVideo	Boolean indicating if file is a video
data.uploadedFiles[].isDocument	Boolean indicating if file is a document
data.uploadedFiles[].isAudio	Boolean indicating if file is audio
data.uploadedFiles[].width	Image width in pixels (images only)
data.uploadedFiles[].height	Image height in pixels (images only)
data.uploadedFiles[].dimensions	Image dimensions as "WxH" format
data.uploadedFiles[].checksum	MD5 hash of file content for integrity verification
data.uploadedFiles[].uploadedAt	Upload timestamp
data.uploadedFiles[].uploadedBy	Account ID of uploader
data.uploadedFiles[].isPublic	Boolean indicating public accessibility (always true)
data.totalFiles	Total number of files in upload request
data.successfulUploads	Number of successfully uploaded files
data.failedUploads	Number of failed uploads
data.totalSize	Total size of all uploaded files in bytes
data.totalSizeFormatted	Human-readable total size
data.uploadedAt	Overall upload completion timestamp
data.message	Summary message of upload results
data.errors	Array of error messages for failed uploads (null if no errors)

Error Responses:

400 Bad Request: Invalid file format, missing parameters, or validation errors
401 Unauthorized: Missing or invalid JWT token
413 Payload Too Large: File size exceeds 25MB limit
422 Unprocessable Entity: Unsupported file type or directory
500 Internal Server Error: Storage system errors

2. Upload Single File

Purpose: Upload a single file to a specified directory with detailed metadata extraction

Endpoint: POST https://apinexgate.glueauth.com/api/v1/files/upload-single

Access Level: 🔒 Protected (Requires JWT Bearer token)

Authentication: Bearer Token (JWT access token required)

Request Headers:

Header	Type	Required	Description
Authorization	string	Yes	Bearer {access_token}
Content-Type	string	Yes	Must be "multipart/form-data"

Request Form Data Parameters:

Parameter	Type	Required	Description	Validation
file	file	Yes	Single file to upload	Max 25MB, supported file types only
directory	string	Yes	Target directory for the file	enum: PROFILE, CATEGORIES, SHOPS

Response JSON Sample:

{
  "success": true,
  "httpStatus": "OK",
  "message": "File uploaded successfully",
  "action_time": "2025-09-23T10:30:00",
  "data": {
    "fileName": "123e4567-e89b-12d3-a456-426614174000.pdf",
    "originalFileName": "product-catalog.pdf",
    "objectKey": "categories/123e4567-e89b-12d3-a456-426614174000.pdf",
    "directory": "CATEGORIES",
    "contentType": "application/pdf",
    "fileSize": 1048576,
    "fileSizeFormatted": "1.0 MB",
    "permanentUrl": "https://files.nextgate.com/bucket-uuid/categories/123e4567-e89b-12d3-a456-426614174000.pdf",
    "thumbnailUrl": null,
    "fileExtension": ".pdf",
    "fileType": "DOCUMENT",
    "isImage": false,
    "isVideo": false,
    "isDocument": true,
    "isAudio": false,
    "width": null,
    "height": null,
    "dimensions": null,
    "checksum": "5d41402abc4b2a76b9719d911017c592",
    "uploadedAt": "2025-09-23T10:30:00",
    "uploadedBy": "123e4567-e89b-12d3-a456-426614174000",
    "isPublic": true
  }
}

Response Fields:

Field	Description
success	Boolean indicating operation success
httpStatus	HTTP status code as string
message	Success message
action_time	ISO 8601 timestamp of response generation
data	Single FileResponse object with complete file metadata (same structure as uploadedFiles[] items above)

Error Responses:

400 Bad Request: Invalid file, missing parameters, or validation errors
401 Unauthorized: Missing or invalid JWT token
413 Payload Too Large: File size exceeds 25MB limit
422 Unprocessable Entity: Unsupported file type or directory
500 Internal Server Error: Storage system errors

3. Get User Files by Directory

Purpose: Retrieve all files uploaded by the authenticated user in a specific directory

Endpoint: GET https://apinexgate.glueauth.com/api/v1/files/directory/{directory}

Access Level: 🔒 Protected (Requires JWT Bearer token)

Authentication: Bearer Token (JWT access token required)

Request Headers:

Header	Type	Required	Description
Authorization	string	Yes	Bearer {access_token}
Content-Type	string	Yes	Must be "application/json"

Path Parameters:

Parameter	Type	Required	Description	Validation
directory	string	Yes	Directory to list files from	enum: PROFILE, CATEGORIES, SHOPS

Response JSON Sample:

{
  "success": true,
  "httpStatus": "OK",
  "message": "Files retrieved successfully",
  "action_time": "2025-09-23T10:30:00",
  "data": [
    {
      "fileName": "123e4567-e89b-12d3-a456-426614174000.jpg",
      "originalFileName": "profile-picture.jpg",
      "objectKey": "profile/123e4567-e89b-12d3-a456-426614174000.jpg",
      "directory": "PROFILE",
      "contentType": "image/jpeg",
      "fileSize": 2048576,
      "fileSizeFormatted": "2.0 MB",
      "permanentUrl": "https://files.nextgate.com/bucket-uuid/profile/123e4567-e89b-12d3-a456-426614174000.jpg",
      "thumbnailUrl": "https://files.nextgate.com/bucket-uuid/profile/123e4567-e89b-12d3-a456-426614174000.jpg",
      "fileExtension": ".jpg",
      "fileType": "IMAGE",
      "isImage": true,
      "isVideo": false,
      "isDocument": false,
      "isAudio": false,
      "width": 1920,
      "height": 1080,
      "dimensions": "1920x1080",
      "uploadedBy": "123e4567-e89b-12d3-a456-426614174000",
      "isPublic": true
    },
    {
      "fileName": "456e7890-e89b-12d3-a456-426614174111.png",
      "originalFileName": "banner-image.png",
      "objectKey": "profile/456e7890-e89b-12d3-a456-426614174111.png",
      "directory": "PROFILE",
      "contentType": "image/png",
      "fileSize": 1524288,
      "fileSizeFormatted": "1.5 MB",
      "permanentUrl": "https://files.nextgate.com/bucket-uuid/profile/456e7890-e89b-12d3-a456-426614174111.png",
      "thumbnailUrl": "https://files.nextgate.com/bucket-uuid/profile/456e7890-e89b-12d3-a456-426614174111.png",
      "fileExtension": ".png",
      "fileType": "IMAGE",
      "isImage": true,
      "isVideo": false,
      "isDocument": false,
      "isAudio": false,
      "width": 1600,
      "height": 900,
      "dimensions": "1600x900",
      "uploadedBy": "123e4567-e89b-12d3-a456-426614174000",
      "isPublic": true
    }
  ]
}

Response Fields:

Field	Description
success	Boolean indicating operation success
httpStatus	HTTP status code as string
message	Success message
action_time	ISO 8601 timestamp of response generation
data	Array of FileResponse objects for all files in the specified directory

Error Responses:

400 Bad Request: Invalid directory parameter
401 Unauthorized: Missing or invalid JWT token
404 Not Found: Directory not found or user not found
500 Internal Server Error: Storage system errors

4. Delete File

Purpose: Delete a specific file uploaded by the authenticated user

Endpoint: DELETE https://apinexgate.glueauth.com/api/v1/files/{objectKey}

Access Level: 🔒 Protected (Requires JWT Bearer token)

Authentication: Bearer Token (JWT access token required)

Request Headers:

Header	Type	Required	Description
Authorization	string	Yes	Bearer {access_token}
Content-Type	string	Yes	Must be "application/json"

Path Parameters:

Parameter	Type	Required	Description	Validation
objectKey	string	Yes	Full object key/path of the file to delete	Must be a valid object key from user's files

Response JSON Sample:

{
  "success": true,
  "httpStatus": "OK",
  "message": "File deleted successfully",
  "action_time": "2025-09-23T10:30:00",
  "data": null
}

Response Fields:

Field	Description
success	Boolean indicating operation success
httpStatus	HTTP status code as string
message	Success message confirming deletion
action_time	ISO 8601 timestamp of response generation
data	Always null for delete operations

Error Responses:

400 Bad Request: Invalid object key format
401 Unauthorized: Missing or invalid JWT token
404 Not Found: File not found or user not found
403 Forbidden: Attempting to delete file not owned by user
500 Internal Server Error: Storage system errors

Quick Reference Guide

File Directory Types

PROFILE: User profile images, avatars, and personal media
CATEGORIES: Category images, banners, and related media
SHOPS: Shop-related images, logos, and promotional materials

Supported File Types

Images: JPEG, JPG, PNG, GIF, WebP, BMP, SVG
Videos: MP4, AVI, MOV, WMV, FLV, WebM, MKV
Documents: PDF, DOC, DOCX, TXT, XLS
Audio: MP3, WAV, OGG, AAC

File Size and Limits

Maximum file size: 25MB per file
File naming: Original names preserved, unique UUIDs generated for storage
Public access: All uploaded files are publicly accessible via direct URLs
Checksums: MD5 hashes calculated for file integrity verification

Common HTTP Status Codes

200 OK: Successful GET/DELETE request
201 Created: Successful POST request (file upload)
400 Bad Request: Invalid request data or file validation errors
401 Unauthorized: Authentication required/failed
403 Forbidden: Insufficient permissions
404 Not Found: Resource not found
413 Payload Too Large: File size exceeds limit
422 Unprocessable Entity: Validation errors
500 Internal Server Error: Server error

Authentication Types

Bearer Token: Include Authorization: Bearer your_access_token in headers
Form Data: Use multipart/form-data for file uploads
JSON: Use application/json for other requests

Data Format Standards

Dates: Use ISO 8601 format (2025-09-23T10:30:00Z)
File sizes: Provided in bytes and human-readable format
UUIDs: Standard UUID format for all identifiers
URLs: Full HTTPS URLs for file access

File Thunder Arch & Flow

File Thunder is the dedicated media-processing microservice for the NexGate / Veepii platform. It handles all file ingestion, processing, storage, and serving-readiness — the main backend never touches raw bytes.

Architecture Overview
Storage: Buckets & Object Keys
MediaDomain & MediaContext
Upload Flow — End to End
The Four Wheels
Progress Tracking (Redis → SSE)
Watermarking
CDN & File Serving Per Environment
Security
API Reference & How to Consume

1. Architecture Overview

┌─────────────────────────────────────────────────────────┐
│                      Main Backend                       │
│  - Only public-facing API                               │
│  - Calls File Thunder over internal HTTP (HMAC signed)  │
│  - Subscribes to Redis for progress → pushes SSE        │
└────────────┬──────────────────────────┬─────────────────┘
             │ HTTP (sync)              │ Redis Pub/Sub (async)
             ▼                          ▼
┌─────────────────────────────────────────────────────────┐
│                     File Thunder                        │
│                                                         │
│   API Profile          Worker Profile                   │
│   ─────────────        ──────────────                   │
│   UploadController     UploadConfirmedWorker            │
│   MediaQueryController ThumbnailWorker                  │
│                        RawFilePurgeJob                  │
│                                                         │
│   Wheels (processing engines)                           │
│   ──────────────────────────                            │
│   ImageWheel  VideoWheel  ScanWheel  (MinIO ops)        │
└────────────┬──────────────────────────┬─────────────────┘
             │                          │
             ▼                          ▼
          MinIO                      RabbitMQ
    (4 private buckets)       (internal event routing)

Key constraints:

File Thunder is internal only — never exposed to the internet directly
Main backend is the only caller of File Thunder HTTP APIs
File Thunder never touches file bytes on upload — client uploads directly to MinIO via presigned URL
Redis is FT-internal — used for progress pub/sub and nonce replay protection
RabbitMQ is FT-internal — used to route events between listeners and workers

2. Storage: Buckets & Object Keys

Buckets

Bucket	Purpose	Access
`nexgate-raw`	Temporary upload landing zone — 24h TTL	Private
`nexgate-public`	Processed media served to end users	Private (CDN in front)
`nexgate-private`	Internal system assets (outros, future forensic assets)	Private
`nexgate-digital`	Digital product files — ClamAV scanned, download only	Private

All buckets are fully private. MinIO is never directly accessible from the internet. Public content is served exclusively through the CDN (Cloudflare), which pulls from MinIO origin.

Object Key Pattern

{domain}/{ownerId}/{fileId}/{variant}

Examples:

posts/550e8400-e29b-41d4-a716-446655440000/f7e8d9.../original
posts/550e8400-e29b-41d4-a716-446655440000/f7e8d9.../large.webp
posts/550e8400-e29b-41d4-a716-446655440000/f7e8d9.../thumb.webp
posts/550e8400-e29b-41d4-a716-446655440000/f7e8d9.../hls/master.m3u8
posts/550e8400-e29b-41d4-a716-446655440000/f7e8d9.../hls/360p/360p.m3u8

system/outro/{ownerId}/{height}p.mp4   ← in nexgate-private

Rule: The database stores object keys only, never full URLs. URLs are assembled at the resolver boundary (CDN base URL + key in prod, presigned GET in local).

3. MediaDomain & MediaContext

MediaDomain

Defines where in the storage hierarchy a file lives. It is the top-level folder in the object key. This is an enum — the main backend must send one of these exact values.

Value	Used For
`POSTS`	Social posts — images and videos
`PROFILES`	Profile pictures, cover photos
`MESSAGES`	Direct message attachments
`PRODUCTS`	Product images, product videos, digital downloads
`SHOPS`	Shop banners, shop logos
`EVENTS`	Event covers, event gallery images

MediaContext

Defines how a file is processed. This drives the entire processing pipeline. No logic is attached to domain — all logic is driven by context.

Value	Type	Processing Pipeline
`SOCIAL_IMAGE`	Image	ImageWheel — orient, strip EXIF, WebP variants, blurhash, lqip
`SOCIAL_VIDEO`	Video	VideoWheel — transcode + social watermark + outro + thumbnail
`PROFILE_PICTURE`	Image	ImageWheel
`COVER_PHOTO`	Image	ImageWheel
`DM_IMAGE`	Image	ImageWheel
`DM_VIDEO`	Video	VideoWheel — transcode, no watermark
`DM_DOCUMENT`	Any	ScanWheel — ClamAV scan
`PRODUCT_IMAGE`	Image	ImageWheel
`PRODUCT_VIDEO`	Video	VideoWheel — transcode + text watermark, no outro
`DIGITAL_PRODUCT`	Any	ScanWheel — ClamAV dual scan
`SHOP_BANNER`	Image	ImageWheel
`SHOP_LOGO`	Image	ImageWheel
`EVENT_COVER`	Image	ImageWheel
`EVENT_GALLERY`	Image	ImageWheel

Validation rules enforced at API layer:

Image contexts only accept image MIME types; video contexts only accept video MIME types
SOCIAL_VIDEO requires username field (used for personalized watermark + outro)
Max file sizes: images 20 MB, videos 2 GB, digital products 500 MB
HEIC/HEIF are rejected — parser attack surface too high

4. Upload Flow — End to End

Standard Media Upload

Main Backend                    File Thunder                 MinIO           Client (Browser)
     │                               │                         │                   │
     │  POST /api/v1/upload/request  │                         │                   │
     │  (HMAC signed)                │                         │                   │
     │──────────────────────────────▶│                         │                   │
     │                               │ validate + create DB    │                   │
     │                               │ record (PENDING)        │                   │
     │                               │ publish PENDING         │                   │
     │                               │ to Redis                │                   │
     │                               │ generate presigned PUT  │                   │
     │                               │─────────────────────────▶                  │
     │  { fileId, presignedUrl }     │◀────────────────────────                   │
     │◀──────────────────────────────│                         │                   │
     │                               │                         │                   │
     │  return presignedUrl          │                         │                   │
     │  to client                    │                         │                   │
     │──────────────────────────────────────────────────────────────────────────▶ │
     │                               │                         │                   │
     │                               │                         │  PUT (file bytes) │
     │                               │                         │◀──────────────────│
     │                               │                         │  200 OK           │
     │                               │                         │──────────────────▶│
     │                               │                         │                   │
     │                               │ ◀── MinIO event ────────│ s3:ObjectCreated  │
     │                               │ MinioEventListener      │                   │
     │                               │ PENDING → UPLOADED      │                   │
     │                               │ publish UPLOADED→Redis  │                   │
     │                               │ publish UploadConfirmed │                   │
     │                               │ to RabbitMQ             │                   │
     │                               │                         │                   │
     │                               │ UploadConfirmedWorker   │                   │
     │                               │ routes by mimeType      │                   │
     │                               │ UPLOADED → PROCESSING   │                   │
     │                               │ publish PROCESSING→Redis│                   │
     │                               │                         │                   │
     │                               │ [ImageWheel or          │                   │
     │                               │  VideoWheel runs]       │                   │
     │                               │                         │                   │
     │                               │ variants uploaded       │                   │
     │                               │ to nexgate-public ──────▶                  │
     │                               │ raw deleted from        │                   │
     │                               │ nexgate-raw ────────────▶                  │
     │                               │ PROCESSING → READY      │                   │
     │                               │ publish READY → Redis   │                   │
     │                               │                         │                   │

User Custom Thumbnail Upload

Main Backend                    File Thunder                 MinIO
     │                               │                         │
     │  POST /api/v1/upload/         │                         │
     │  thumbnail/{fileId}           │                         │
     │──────────────────────────────▶│                         │
     │                               │ generate presigned PUT  │
     │                               │ key: .../custom-        │
     │                               │ thumbnail               │
     │  { presignedUrl }             │                         │
     │◀──────────────────────────────│                         │
     │  client uploads thumbnail     │                         │
     │─────────────────────────────────────────────────────────▶
     │                               │◀── MinIO event ─────────│
     │                               │ MinioEventListener      │
     │                               │ detects suffix=         │
     │                               │ "custom-thumbnail"      │
     │                               │ → ThumbnailWorker       │
     │                               │ download → verify magic │
     │                               │ bytes → ImageMagick     │
     │                               │ variants → save to      │
     │                               │ entity.userThumbnail    │
     │                               │ → delete raw            │

File Status Lifecycle

PENDING → UPLOADING → UPLOADED → SCANNING → PROCESSING → LIVE_PARTIAL → READY
                                                                ▲
                                              (short video: after 360p uploaded)
                                              (long video: after 360p + partial HLS)

5. The Four Wheels

Wheels are the processing engines. Each runs in the worker profile.

Wheel 1 — ImageWheel (ImageMagick)

Triggered by: image MIME type on any image context

Pipeline:

download raw from nexgate-raw
      │
      ▼
auto-orient + strip EXIF
      │
      ├──▶ large.webp    (1920px max width, shrink only, Q85)
      ├──▶ medium.webp   (800px max width, shrink only, Q82)
      ├──▶ thumb.webp    (300px max width, shrink only, Q80)
      ├──▶ og.webp       (1200×630 center crop, Q85)
      │    └─ falls back to medium if source width < 1200px
      ├──▶ blurhash      (encoded from 32×32 downsample)
      └──▶ lqip          (10×10 forced, base64 WebP data URI)

all WebP variants → nexgate-public
keys + blurhash + lqip → media_files.variants (JSONB)
raw deleted from nexgate-raw
status → READY

Skipping logic: variant is skipped if source is already within bounds (never upscale).

Wheel 2 — VideoWheel (FFmpeg / Jaffree)

Triggered by: video MIME type on any video context

Step 1 — Probe & Validate

download raw from nexgate-raw
magic byte verify (must be real video container)
FFprobe → codedWidth, codedHeight, rotation, duration, codec
displayWidth/displayHeight = rotation-aware (swap for 90°/270°)
size gate: > 2 GB → reject
duration gate: > 4h → reject
route: duration < 3min → processShort()
       duration ≥ 3min → processLong()

Step 2a — Short Path (< 3 min)

transcode() runs 3 variants in sequence:
  360p  (CRF 28, 96k audio)  → upload → status LIVE_PARTIAL
  720p  (CRF 23, 128k audio) → upload
  1080p (CRF 21, 192k audio) → upload

filter_complex:
  [0:v] split=2 [main][blur]
  [blur] scale={W}:{H}, boxblur=20:5 [bg]
  [main] scale={w}:{h} (fit within target, keep AR) [fg]
  [bg][fg] overlay=x=(W-w)/2:y=(H-h)/2 [out]
  + rotation transpose prepended if needed
  + watermark drawtext chain appended

if SOCIAL_VIDEO:
  append outro via concat (no re-encode)

skip logic: source displayWidth >= targetW OR displayHeight >= targetH

Step 2b — Long Path (≥ 3 min) — HLS Adaptive

transcodeHls() per variant:
  360p → segments + .m3u8 → upload → partial master.m3u8 → LIVE_PARTIAL
  720p → segments + .m3u8 → upload
  1080p → segments + .m3u8 → upload → full master.m3u8 → READY

segment length: 2s
playlist type: VOD
master.m3u8 key: {keyBase}/hls/master.m3u8
variant keys:   {keyBase}/hls/{name}/{name}.m3u8

Step 3 — Thumbnail & Preview (both short and long)

selectBestFrame():
  5 candidates at 15/30/45/60/75% of duration
  scored by: brightness gate (30–230) + Laplacian variance (sharpness)
  FFmpeg extracts 720px JPEG per candidate → pick winner

buildThumbnailVariants():
  poster.webp      (1280px, Q85)
  thumb.webp       (480px, Q80)
  og.webp          (1200×630 center crop, Q85)
  blurhash         (from 32×32 downsample)
  lqip             (10×10 forced, base64 WebP data URI)
  dominant_color   (#RRGGBB from 1×1 squish)

extractPreviewClip():
  skip first 5% (max 2s) → read 6s → setpts=0.5*PTS → 3s at 2× speed
  blur-pad 360×640, muted
  watermarked (same moving watermark as main video)

all thumbnail variants → nexgate-public
raw deleted from nexgate-raw after all variants done

Wheel 3 — ScanWheel (ClamAV)

Triggered by: DIGITAL_PRODUCT and DM_DOCUMENT contexts only Social content (images and videos) is never scanned by ClamAV — FFmpeg/ImageMagick re-encode is the sanitisation.

Pipeline:

download raw from nexgate-raw
      │
      ▼
SHA-256 hash → check file_hashes table
      │
      ├── hash known + clean → skip scan, copy to nexgate-digital, READY (dedup fast lane)
      │
      └── hash unknown →
            ClamAV scan #1 (upload scan)
            if clean → move to nexgate-digital
            ClamAV scan #2 (pre-download scan, confirms in-transit integrity)
            if clean → save hash, status READY
            if infected → status FAILED, quarantine

Wheel 4 — MinIO Operations (used by all wheels)

Not a standalone service — MinIO operations are helpers used across all wheels:

Operation	Used By
Presigned PUT URL	Upload request endpoint
Download object	All wheels (download raw for processing)
Upload object	All wheels (upload processed variants)
Remove object	All wheels (delete raw after processing)
Stat object	OutroService (cache check), ScanWheel (dedup check)

6. Progress Tracking (Redis → SSE)

File Thunder publishes status changes to Redis. The main backend subscribes and drives SSE to the client. SSE is the main backend's responsibility — File Thunder only publishes.

What File Thunder Does

On every status change:

// 1. Cache current status in Redis (24h TTL)
redisTemplate.opsForValue().set("ft:status:{fileId}", status.name(), 24h);

// 2. Publish to per-file channel
redisTemplate.convertAndSend("ft:progress:{fileId}", status.name());

// 3. Append to timeline in Postgres
entity.timeline.add({ status, at })

What the Main Backend Must Do

// Subscribe to the file's progress channel
redisTemplate.subscribe((message, pattern) -> {
    String status = message.toString();
    sseEmitter.send(SseEmitter.event().data(status));
}, "ft:progress:" + fileId);

Status Channel Key

ft:progress:{fileId}     ← subscribe here for live updates
ft:status:{fileId}       ← read here for current status (24h cached)

7. Watermarking

Moving Watermark

Applied to all video variants and preview clips. Watermark position cycles through 4 corners every 3 seconds (5 seconds for social).

Text-only (PRODUCT_VIDEO):

drawtext: fontsize=max(14,H/40), white@0.65 + black shadow
position: floor(t/3) mod 4 → top-left, top-right, bottom-right, bottom-left
text: from watermark.text property (default: "NexGate")

2-point diagonal: Pos A (22%, 28%) ↔ Pos B (58%, 65%) — switches every 5s
Logo: nexgate_logo_white.svg, 36×36, 65% opacity
@ symbol: orange #F06023@0.85 + shadow
username: white@0.85 + shadow, below logo

Outro (SOCIAL_VIDEO only)

A personalized outro clip is appended to the watermarked variant (not clean variants).

Generation:

template: outro_template_with_sfx.mp4 (classpath resource)
font: Sora SemiBold 600 (classpath resource)
drawtext: @ in orange #F06023, username in cream #F4EEE9
fade-in: invisible before t=0.75s, fully opaque by t=1.2s
scale to match variant resolution (360p / 720p / 1080p)
append via: -f concat -safe 0 -c copy (no re-encode, fast)

Caching:

Generated once per ownerId + resolution
Cached to: nexgate-private / system/outro/{ownerId}/{height}p.mp4
On next upload: cache hit → download and reuse, skip generation

8. CDN & File Serving Per Environment

All MinIO buckets are private. Access to public content is environment-dependent.

Environment Matrix

Environment	Serving Method	Config Value
Local dev	Presigned GET URLs (MinIO direct, short-lived)	`ft.storage.mode=local`
Staging	CDN — `cdn-staging.nexgate.com`	`ft.storage.mode=cdn`
Production	CDN — `cdn.nexgate.com`	`ft.storage.mode=cdn`

Local / Dev

App generates a presigned GET URL from MinIO (e.g. 1-hour expiry)
URL is returned directly to the client
MinIO must be reachable by the client
Good enough for development and manual testing

Staging & Production (CDN)

App assembles: {ft.cdn.base-url} + "/" + objectKey
Cloudflare sits in front of MinIO origin
Cloudflare pulls from MinIO on cache miss using a service credential
Every subsequent request served from Cloudflare edge — MinIO never hit again
Cost: Cloudflare bandwidth is free; MinIO only pays VPS bandwidth once per cache miss

Properties

# Local
ft.storage.mode=local
ft.cdn.base-url=

# Staging
ft.storage.mode=cdn
ft.cdn.base-url=https://cdn-staging.nexgate.com

# Production
ft.storage.mode=cdn
ft.cdn.base-url=https://cdn.nexgate.com

URL Assembly Logic (resolver boundary)

// local mode → presigned GET from MinIO
// cdn mode   → CDN base + object key
String url = storageMode.equals("cdn")
    ? cdnBaseUrl + "/" + objectKey
    : minioClient.getPresignedObjectUrl(GET, bucket, objectKey, 1h);

Private content (DIGITAL_PRODUCT, DM_DOCUMENT): always presigned GET URLs regardless of environment — these are never cached by CDN. Generated fresh per download request (15-minute expiry, audit logged).

9. Security

CORS

Configured via ft.cors.* properties. The CORS filter runs before all other filters so OPTIONS preflight requests are handled without requiring HMAC headers.

# Local
ft.cors.allowed-origins=http://localhost:3000

# Production
ft.cors.allowed-origins=https://nexgate.com,https://www.nexgate.com

ft.cors.allowed-methods=GET,POST,PUT,DELETE,OPTIONS
ft.cors.allowed-headers=*
ft.cors.allow-credentials=false
ft.cors.max-age-seconds=3600

Service-to-Service HMAC Authentication

Every HTTP request from the main backend to File Thunder must be HMAC-SHA256 signed. File Thunder rejects any unsigned or incorrectly signed request with 401 Unauthorized.

Filter order:

Request → CorsFilter (HIGHEST_PRECEDENCE)
        → HmacAuthFilter (HIGHEST_PRECEDENCE + 1)
        → Controllers

Required Headers

Header	Description	Example
`X-Service-Id`	Identifier of the calling service	`nexgate-main`
`X-Timestamp`	Unix epoch seconds at time of request	`1718123456`
`X-Nonce`	Random UUID — unique per request	`f47ac10b-58cc-...`
`X-Signature`	HMAC-SHA256 hex of the canonical string	`a3f5c2...`

Canonical String

METHOD\n
REQUEST_URI\n
TIMESTAMP\n
NONCE\n
HEX(SHA-256(requestBody))

Example for POST /api/v1/upload/request:

POST
/api/v1/upload/request
1718123456
f47ac10b-58cc-4372-a567-0e02b2c3d479
e3b0c44298fc1c149afb...   ← SHA-256 of the JSON body

Signature Computation

// Main backend — how to sign a request
String bodyHash = HexFormat.of().formatHex(
    MessageDigest.getInstance("SHA-256").digest(requestBodyBytes)
);

String canonical = method + "\n" + uri + "\n" + timestamp + "\n" + nonce + "\n" + bodyHash;

Mac mac = Mac.getInstance("HmacSHA256");
mac.init(new SecretKeySpec(sharedSecret.getBytes(UTF_8), "HmacSHA256"));
String signature = HexFormat.of().formatHex(mac.doFinal(canonical.getBytes(UTF_8)));

// Set headers on outgoing request
request.setHeader("X-Service-Id", "nexgate-main");
request.setHeader("X-Timestamp",  String.valueOf(Instant.now().getEpochSecond()));
request.setHeader("X-Nonce",      UUID.randomUUID().toString());
request.setHeader("X-Signature",  signature);

What File Thunder Verifies (in order)

1. All 4 headers present                → 401 if any missing
2. X-Service-Id in allowed list         → 401 if unknown
3. X-Timestamp within ±5 minutes        → 401 if stale (replay protection)
4. X-Nonce not seen before              → 401 if duplicate
   (stored in Redis with 10-min TTL)       (replay attack blocked)
5. Recompute HMAC, timing-safe compare  → 401 if mismatch
   (MessageDigest.isEqual)                 (prevents timing oracle)

Properties

ft.security.hmac.secret=<long-random-secret-same-in-both-services>
ft.security.hmac.allowed-service-ids=nexgate-main
ft.security.hmac.timestamp-tolerance-seconds=300
ft.security.hmac.nonce-ttl-seconds=600

The secret must be identical in both File Thunder and the main backend config. Use a different value per environment (local / staging / prod).

10. API Reference & How to Consume

Base URL: http://file-thunder:8081 (internal network only) All requests must include HMAC headers — see Section 9.

POST /api/v1/upload/request

Request a presigned URL to upload a file directly to MinIO.

Request body:

{
  "ownerId":          "550e8400-e29b-41d4-a716-446655440000",
  "domain":           "POSTS",
  "context":          "SOCIAL_VIDEO",
  "originalFilename": "my-video.mp4",
  "mimeType":         "video/mp4",
  "fileSizeBytes":    104857600,
  "username":         "josh_dev"
}

username is required when context is SOCIAL_VIDEO. All other fields are always required.

Response:

{
  "status": "success",
  "message": "Presigned upload URL generated",
  "data": {
    "fileId":           "f7e8d9fa-...",
    "presignedUrl":     "http://minio:9000/nexgate-raw/posts/.../original?X-Amz-...",
    "objectKey":        "posts/{ownerId}/{entityId}/{fileId}/original",
    "bucket":           "nexgate-raw",
    "expiresInSeconds": 1800
  }
}

What to do next: PUT the file bytes directly to presignedUrl from the client browser. No auth headers needed on the PUT — the presigned URL is self-authenticating.

POST /api/v1/upload/thumbnail/{fileId}

Request a presigned URL to replace the system-generated video thumbnail with a user-supplied one. Only valid after the video file has been processed (status READY).

Response:

{
  "status": "success",
  "message": "Thumbnail presigned upload URL generated",
  "data": {
    "fileId":           "f7e8d9fa-...",
    "presignedUrl":     "http://minio:9000/nexgate-raw/.../custom-thumbnail?X-Amz-...",
    "objectKey":        "posts/.../custom-thumbnail",
    "bucket":           "nexgate-raw",
    "expiresInSeconds": 1800
  }
}

Accepted formats: JPEG, PNG, WebP only (verified by magic bytes).

GET /api/v1/media/{fileId}

Get full metadata for a file, including all processed variants.

Response:

{
  "status": "success",
  "data": {
    "fileId":    "f7e8d9fa-...",
    "ownerId":   "550e8400-...",
    "domain":    "POSTS",
    "context":   "SOCIAL_VIDEO",
    "status":    "READY",
    "mimeType":  "video/mp4",
    "variants": {
      "360p":          "posts/.../360p.mp4",
      "720p":          "posts/.../720p.mp4",
      "1080p":         "posts/.../1080p.mp4",
      "poster":        "posts/.../poster.webp",
      "thumb":         "posts/.../thumb.webp",
      "og":            "posts/.../og.webp",
      "preview_3s":    "posts/.../preview_3s.mp4",
      "blurhash":      "LKO2?U%2Tw=w]~RBVZRi};RPxuwH",
      "lqip":          "data:image/webp;base64,...",
      "dominant_color":"#1A2B3C"
    },
    "userThumbnail": null,
    "timeline": [
      { "status": "PENDING",    "at": "2026-06-15T10:00:00" },
      { "status": "UPLOADED",   "at": "2026-06-15T10:00:05" },
      { "status": "PROCESSING", "at": "2026-06-15T10:00:06" },
      { "status": "LIVE_PARTIAL","at": "2026-06-15T10:00:45" },
      { "status": "READY",      "at": "2026-06-15T10:02:10" }
    ],
    "createdAt": "2026-06-15T10:00:00",
    "updatedAt": "2026-06-15T10:02:10"
  }
}

Thumbnail resolution rule:

userThumbnail != null ? use userThumbnail : use variants

Both userThumbnail and variants use the same key names (poster, thumb, og, blurhash, lqip, dominant_color).

GET /api/v1/media/{fileId}/download?requesterId={uuid}

Generate a time-limited download URL for private files (DIGITAL_PRODUCT, DM_DOCUMENT only). Every call is audit-logged.

Response:

{
  "status": "success",
  "data": {
    "url":              "http://minio:9000/nexgate-digital/...?X-Amz-...",
    "expiresInSeconds": 900
  }
}

GET /api/v1/quota/{ownerId}

Get the total storage used by an owner across all their processed files. Updated atomically as each file is processed — never stale.

Response:

{
  "status": "success",
  "message": "Storage usage",
  "data": {
    "ownerId":   "550e8400-e29b-41d4-a716-446655440000",
    "usedBytes": 1073741824,
    "usedMb":    1024.0,
    "usedGb":    1.0
  }
}

How quota is tracked:

Every wheel (ImageWheel, VideoWheel, ScanWheel) calls StorageUsageService.trackUsage(ownerId, bytes) after uploading processed variants
Tracked bytes = size of the processed variants, not the raw upload
Stored in user_storage_quota table, upserted atomically per owner
releaseUsage() is called on hard delete (soft deletes do not release quota)
Returns 0 if the owner has no tracked usage yet (never uploaded)

Use this endpoint to:

Display storage usage in the main backend dashboard
Enforce storage limits before allowing a new upload request

Error Responses

{ "error": "Missing required security headers" }          ← 401
{ "error": "Request timestamp out of acceptable window" } ← 401
{ "error": "Nonce already used — possible replay attack"} ← 401
{ "error": "Invalid signature" }                          ← 401

Validation errors return 400 with field-level messages via @ControllerAdvice.

This document covers the full File Thunder system as of 2026-06-15. CDN wiring (Cloudflare), Kafka integration, lazy transcode, and forensic watermarks are deferred.

Files-nexgate-service(2)

File Management Service

Endpoints

1. Upload Multiple Files

2. Upload Single File

3. Get User Files by Directory

4. Delete File

Quick Reference Guide

File Directory Types

Supported File Types

File Size and Limits

Common HTTP Status Codes

Authentication Types

Data Format Standards

File Thunder Arch & Flow

Table of Contents

1. Architecture Overview

2. Storage: Buckets & Object Keys

Buckets

Object Key Pattern

3. MediaDomain & MediaContext

MediaDomain

MediaContext

4. Upload Flow — End to End

Standard Media Upload

User Custom Thumbnail Upload

File Status Lifecycle

5. The Four Wheels

Wheel 1 — ImageWheel (ImageMagick)

Wheel 2 — VideoWheel (FFmpeg / Jaffree)

Wheel 3 — ScanWheel (ClamAV)

Wheel 4 — MinIO Operations (used by all wheels)

6. Progress Tracking (Redis → SSE)

What File Thunder Does

What the Main Backend Must Do

Status Channel Key

7. Watermarking

Moving Watermark

Outro (SOCIAL_VIDEO only)

8. CDN & File Serving Per Environment

Environment Matrix

Local / Dev

Staging & Production (CDN)

Properties

URL Assembly Logic (resolver boundary)

9. Security

CORS

Service-to-Service HMAC Authentication

Required Headers

Canonical String

Signature Computation

What File Thunder Verifies (in order)

Properties

10. API Reference & How to Consume

POST /api/v1/upload/request

POST /api/v1/upload/thumbnail/{fileId}

GET /api/v1/media/{fileId}

GET /api/v1/media/{fileId}/download?requesterId={uuid}

GET /api/v1/quota/{ownerId}

Error Responses