# Files-nexgate-service(2)

# File Management Service

**Author**: Josh S. Sakweli, Backend Lead Team  
**Last Updated**: 2025-09-23  
**Version**: v1.0

**Short Description**: The File Management Service handles file uploads, downloads, and management for the NextGate social commerce platform. This service provides secure file storage with support for multiple file types, automatic metadata extraction, and organized directory structures for profiles, categories, and shops.

**Hints**:

- Maximum file size: 25MB per file
- Supported file types: Images (JPEG, PNG, GIF, WebP, BMP, SVG), Videos (MP4, AVI, MOV, WebM, MKV), Documents (PDF, DOC, DOCX, TXT, XLS), Audio (MP3, WAV, OGG, AAC)
- All uploaded files are publicly accessible via direct URLs
- Files are automatically organized by directory type (profile, categories, shops)
- Image dimensions and metadata are automatically extracted
- Unique filenames are generated to prevent conflicts
- MD5 checksums are calculated for file integrity verification

---

## Endpoints

## 1. Upload Multiple Files

**Purpose**: Upload multiple files to a specified directory with comprehensive metadata extraction

**Endpoint**: <span style="background-color: #fd7e14; color: white; padding: 4px 8px; border-radius: 4px; font-family: monospace; font-size: 12px; font-weight: bold;">POST</span> `https://apinexgate.glueauth.com/api/v1/files/upload`

**Access Level**: 🔒 Protected (Requires JWT Bearer token)

**Authentication**: Bearer Token (JWT access token required)

**Request Headers**:

<table id="bkmrk-header-type-required"><thead><tr><th>Header</th><th>Type</th><th>Required</th><th>Description</th></tr></thead><tbody><tr><td>Authorization</td><td>string</td><td>Yes</td><td>Bearer {access\_token}</td></tr><tr><td>Content-Type</td><td>string</td><td>Yes</td><td>Must be "multipart/form-data"</td></tr></tbody></table>

**Request Form Data Parameters**:

<table id="bkmrk-parameter-type-requi"><thead><tr><th>Parameter</th><th>Type</th><th>Required</th><th>Description</th><th>Validation</th></tr></thead><tbody><tr><td>files</td><td>file\[\]</td><td>Yes</td><td>Array of files to upload</td><td>Max 25MB per file, supported file types only</td></tr><tr><td>directory</td><td>string</td><td>Yes</td><td>Target directory for files</td><td>enum: PROFILE, CATEGORIES, SHOPS</td></tr></tbody></table>

**Response JSON Sample**:

```json
{
  "success": true,
  "httpStatus": "OK",
  "message": "Files uploaded successfully",
  "action_time": "2025-09-23T10:30:00",
  "data": {
    "uploadedFiles": [
      {
        "fileName": "123e4567-e89b-12d3-a456-426614174000.jpg",
        "originalFileName": "product-image.jpg",
        "objectKey": "profile/123e4567-e89b-12d3-a456-426614174000.jpg",
        "directory": "PROFILE",
        "contentType": "image/jpeg",
        "fileSize": 2048576,
        "fileSizeFormatted": "2.0 MB",
        "permanentUrl": "https://files.nextgate.com/bucket-uuid/profile/123e4567-e89b-12d3-a456-426614174000.jpg",
        "thumbnailUrl": "https://files.nextgate.com/bucket-uuid/profile/123e4567-e89b-12d3-a456-426614174000.jpg",
        "fileExtension": ".jpg",
        "fileType": "IMAGE",
        "isImage": true,
        "isVideo": false,
        "isDocument": false,
        "isAudio": false,
        "width": 1920,
        "height": 1080,
        "dimensions": "1920x1080",
        "checksum": "d41d8cd98f00b204e9800998ecf8427e",
        "uploadedAt": "2025-09-23T10:30:00",
        "uploadedBy": "123e4567-e89b-12d3-a456-426614174000",
        "isPublic": true
      }
    ],
    "totalFiles": 3,
    "successfulUploads": 2,
    "failedUploads": 1,
    "totalSize": 5242880,
    "totalSizeFormatted": "5.0 MB",
    "uploadedAt": "2025-09-23T10:30:00",
    "message": "2 files uploaded successfully, 1 failed",
    "errors": [
      "Failed to upload large-file.mp4: File size exceeds maximum limit of 25MB"
    ]
  }
}

```

**Response Fields**:

<table id="bkmrk-field-description-su"><thead><tr><th>Field</th><th>Description</th></tr></thead><tbody><tr><td>success</td><td>Boolean indicating operation success</td></tr><tr><td>httpStatus</td><td>HTTP status code as string</td></tr><tr><td>message</td><td>Overall operation success message</td></tr><tr><td>action\_time</td><td>ISO 8601 timestamp of response generation</td></tr><tr><td>data.uploadedFiles</td><td>Array of successfully uploaded file details</td></tr><tr><td>data.uploadedFiles\[\].fileName</td><td>System-generated unique filename with UUID</td></tr><tr><td>data.uploadedFiles\[\].originalFileName</td><td>Original filename provided by user</td></tr><tr><td>data.uploadedFiles\[\].objectKey</td><td>Full object path in storage system</td></tr><tr><td>data.uploadedFiles\[\].directory</td><td>Directory where file was stored</td></tr><tr><td>data.uploadedFiles\[\].contentType</td><td>MIME type of the uploaded file</td></tr><tr><td>data.uploadedFiles\[\].fileSize</td><td>File size in bytes</td></tr><tr><td>data.uploadedFiles\[\].fileSizeFormatted</td><td>Human-readable file size (e.g., "2.0 MB")</td></tr><tr><td>data.uploadedFiles\[\].permanentUrl</td><td>Direct public URL to access the file</td></tr><tr><td>data.uploadedFiles\[\].thumbnailUrl</td><td>Thumbnail URL (same as permanentUrl for images)</td></tr><tr><td>data.uploadedFiles\[\].fileExtension</td><td>File extension including dot (e.g., ".jpg")</td></tr><tr><td>data.uploadedFiles\[\].fileType</td><td>File category (IMAGE, VIDEO, DOCUMENT, AUDIO, OTHER)</td></tr><tr><td>data.uploadedFiles\[\].isImage</td><td>Boolean indicating if file is an image</td></tr><tr><td>data.uploadedFiles\[\].isVideo</td><td>Boolean indicating if file is a video</td></tr><tr><td>data.uploadedFiles\[\].isDocument</td><td>Boolean indicating if file is a document</td></tr><tr><td>data.uploadedFiles\[\].isAudio</td><td>Boolean indicating if file is audio</td></tr><tr><td>data.uploadedFiles\[\].width</td><td>Image width in pixels (images only)</td></tr><tr><td>data.uploadedFiles\[\].height</td><td>Image height in pixels (images only)</td></tr><tr><td>data.uploadedFiles\[\].dimensions</td><td>Image dimensions as "WxH" format</td></tr><tr><td>data.uploadedFiles\[\].checksum</td><td>MD5 hash of file content for integrity verification</td></tr><tr><td>data.uploadedFiles\[\].uploadedAt</td><td>Upload timestamp</td></tr><tr><td>data.uploadedFiles\[\].uploadedBy</td><td>Account ID of uploader</td></tr><tr><td>data.uploadedFiles\[\].isPublic</td><td>Boolean indicating public accessibility (always true)</td></tr><tr><td>data.totalFiles</td><td>Total number of files in upload request</td></tr><tr><td>data.successfulUploads</td><td>Number of successfully uploaded files</td></tr><tr><td>data.failedUploads</td><td>Number of failed uploads</td></tr><tr><td>data.totalSize</td><td>Total size of all uploaded files in bytes</td></tr><tr><td>data.totalSizeFormatted</td><td>Human-readable total size</td></tr><tr><td>data.uploadedAt</td><td>Overall upload completion timestamp</td></tr><tr><td>data.message</td><td>Summary message of upload results</td></tr><tr><td>data.errors</td><td>Array of error messages for failed uploads (null if no errors)</td></tr></tbody></table>

**Error Responses**:

- `400 Bad Request`: Invalid file format, missing parameters, or validation errors
- `401 Unauthorized`: Missing or invalid JWT token
- `413 Payload Too Large`: File size exceeds 25MB limit
- `422 Unprocessable Entity`: Unsupported file type or directory
- `500 Internal Server Error`: Storage system errors

---

## 2. Upload Single File

**Purpose**: Upload a single file to a specified directory with detailed metadata extraction

**Endpoint**: <span style="background-color: #fd7e14; color: white; padding: 4px 8px; border-radius: 4px; font-family: monospace; font-size: 12px; font-weight: bold;">POST</span> `https://apinexgate.glueauth.com/api/v1/files/upload-single`

**Access Level**: 🔒 Protected (Requires JWT Bearer token)

**Authentication**: Bearer Token (JWT access token required)

**Request Headers**:

<table id="bkmrk-header-type-required-1"><thead><tr><th>Header</th><th>Type</th><th>Required</th><th>Description</th></tr></thead><tbody><tr><td>Authorization</td><td>string</td><td>Yes</td><td>Bearer {access\_token}</td></tr><tr><td>Content-Type</td><td>string</td><td>Yes</td><td>Must be "multipart/form-data"</td></tr></tbody></table>

**Request Form Data Parameters**:

<table id="bkmrk-parameter-type-requi-1"><thead><tr><th>Parameter</th><th>Type</th><th>Required</th><th>Description</th><th>Validation</th></tr></thead><tbody><tr><td>file</td><td>file</td><td>Yes</td><td>Single file to upload</td><td>Max 25MB, supported file types only</td></tr><tr><td>directory</td><td>string</td><td>Yes</td><td>Target directory for the file</td><td>enum: PROFILE, CATEGORIES, SHOPS</td></tr></tbody></table>

**Response JSON Sample**:

```json
{
  "success": true,
  "httpStatus": "OK",
  "message": "File uploaded successfully",
  "action_time": "2025-09-23T10:30:00",
  "data": {
    "fileName": "123e4567-e89b-12d3-a456-426614174000.pdf",
    "originalFileName": "product-catalog.pdf",
    "objectKey": "categories/123e4567-e89b-12d3-a456-426614174000.pdf",
    "directory": "CATEGORIES",
    "contentType": "application/pdf",
    "fileSize": 1048576,
    "fileSizeFormatted": "1.0 MB",
    "permanentUrl": "https://files.nextgate.com/bucket-uuid/categories/123e4567-e89b-12d3-a456-426614174000.pdf",
    "thumbnailUrl": null,
    "fileExtension": ".pdf",
    "fileType": "DOCUMENT",
    "isImage": false,
    "isVideo": false,
    "isDocument": true,
    "isAudio": false,
    "width": null,
    "height": null,
    "dimensions": null,
    "checksum": "5d41402abc4b2a76b9719d911017c592",
    "uploadedAt": "2025-09-23T10:30:00",
    "uploadedBy": "123e4567-e89b-12d3-a456-426614174000",
    "isPublic": true
  }
}

```

**Response Fields**:

<table id="bkmrk-field-description-su-1"><thead><tr><th>Field</th><th>Description</th></tr></thead><tbody><tr><td>success</td><td>Boolean indicating operation success</td></tr><tr><td>httpStatus</td><td>HTTP status code as string</td></tr><tr><td>message</td><td>Success message</td></tr><tr><td>action\_time</td><td>ISO 8601 timestamp of response generation</td></tr><tr><td>data</td><td>Single FileResponse object with complete file metadata (same structure as uploadedFiles\[\] items above)</td></tr></tbody></table>

**Error Responses**:

- `400 Bad Request`: Invalid file, missing parameters, or validation errors
- `401 Unauthorized`: Missing or invalid JWT token
- `413 Payload Too Large`: File size exceeds 25MB limit
- `422 Unprocessable Entity`: Unsupported file type or directory
- `500 Internal Server Error`: Storage system errors

---

## 3. Get User Files by Directory

**Purpose**: Retrieve all files uploaded by the authenticated user in a specific directory

**Endpoint**: <span style="background-color: #28a745; color: white; padding: 4px 8px; border-radius: 4px; font-family: monospace; font-size: 12px; font-weight: bold;">GET</span> `https://apinexgate.glueauth.com/api/v1/files/directory/{directory}`

**Access Level**: 🔒 Protected (Requires JWT Bearer token)

**Authentication**: Bearer Token (JWT access token required)

**Request Headers**:

<table id="bkmrk-header-type-required-2"><thead><tr><th>Header</th><th>Type</th><th>Required</th><th>Description</th></tr></thead><tbody><tr><td>Authorization</td><td>string</td><td>Yes</td><td>Bearer {access\_token}</td></tr><tr><td>Content-Type</td><td>string</td><td>Yes</td><td>Must be "application/json"</td></tr></tbody></table>

**Path Parameters**:

<table id="bkmrk-parameter-type-requi-2"><thead><tr><th>Parameter</th><th>Type</th><th>Required</th><th>Description</th><th>Validation</th></tr></thead><tbody><tr><td>directory</td><td>string</td><td>Yes</td><td>Directory to list files from</td><td>enum: PROFILE, CATEGORIES, SHOPS</td></tr></tbody></table>

**Response JSON Sample**:

```json
{
  "success": true,
  "httpStatus": "OK",
  "message": "Files retrieved successfully",
  "action_time": "2025-09-23T10:30:00",
  "data": [
    {
      "fileName": "123e4567-e89b-12d3-a456-426614174000.jpg",
      "originalFileName": "profile-picture.jpg",
      "objectKey": "profile/123e4567-e89b-12d3-a456-426614174000.jpg",
      "directory": "PROFILE",
      "contentType": "image/jpeg",
      "fileSize": 2048576,
      "fileSizeFormatted": "2.0 MB",
      "permanentUrl": "https://files.nextgate.com/bucket-uuid/profile/123e4567-e89b-12d3-a456-426614174000.jpg",
      "thumbnailUrl": "https://files.nextgate.com/bucket-uuid/profile/123e4567-e89b-12d3-a456-426614174000.jpg",
      "fileExtension": ".jpg",
      "fileType": "IMAGE",
      "isImage": true,
      "isVideo": false,
      "isDocument": false,
      "isAudio": false,
      "width": 1920,
      "height": 1080,
      "dimensions": "1920x1080",
      "uploadedBy": "123e4567-e89b-12d3-a456-426614174000",
      "isPublic": true
    },
    {
      "fileName": "456e7890-e89b-12d3-a456-426614174111.png",
      "originalFileName": "banner-image.png",
      "objectKey": "profile/456e7890-e89b-12d3-a456-426614174111.png",
      "directory": "PROFILE",
      "contentType": "image/png",
      "fileSize": 1524288,
      "fileSizeFormatted": "1.5 MB",
      "permanentUrl": "https://files.nextgate.com/bucket-uuid/profile/456e7890-e89b-12d3-a456-426614174111.png",
      "thumbnailUrl": "https://files.nextgate.com/bucket-uuid/profile/456e7890-e89b-12d3-a456-426614174111.png",
      "fileExtension": ".png",
      "fileType": "IMAGE",
      "isImage": true,
      "isVideo": false,
      "isDocument": false,
      "isAudio": false,
      "width": 1600,
      "height": 900,
      "dimensions": "1600x900",
      "uploadedBy": "123e4567-e89b-12d3-a456-426614174000",
      "isPublic": true
    }
  ]
}

```

**Response Fields**:

<table id="bkmrk-field-description-su-2"><thead><tr><th>Field</th><th>Description</th></tr></thead><tbody><tr><td>success</td><td>Boolean indicating operation success</td></tr><tr><td>httpStatus</td><td>HTTP status code as string</td></tr><tr><td>message</td><td>Success message</td></tr><tr><td>action\_time</td><td>ISO 8601 timestamp of response generation</td></tr><tr><td>data</td><td>Array of FileResponse objects for all files in the specified directory</td></tr></tbody></table>

**Error Responses**:

- `400 Bad Request`: Invalid directory parameter
- `401 Unauthorized`: Missing or invalid JWT token
- `404 Not Found`: Directory not found or user not found
- `500 Internal Server Error`: Storage system errors

---

## 4. Delete File

**Purpose**: Delete a specific file uploaded by the authenticated user

**Endpoint**: <span style="background-color: #dc3545; color: white; padding: 4px 8px; border-radius: 4px; font-family: monospace; font-size: 12px; font-weight: bold;">DELETE</span> `https://apinexgate.glueauth.com/api/v1/files/{objectKey}`

**Access Level**: 🔒 Protected (Requires JWT Bearer token)

**Authentication**: Bearer Token (JWT access token required)

**Request Headers**:

<table id="bkmrk-header-type-required-3"><thead><tr><th>Header</th><th>Type</th><th>Required</th><th>Description</th></tr></thead><tbody><tr><td>Authorization</td><td>string</td><td>Yes</td><td>Bearer {access\_token}</td></tr><tr><td>Content-Type</td><td>string</td><td>Yes</td><td>Must be "application/json"</td></tr></tbody></table>

**Path Parameters**:

<table id="bkmrk-parameter-type-requi-3"><thead><tr><th>Parameter</th><th>Type</th><th>Required</th><th>Description</th><th>Validation</th></tr></thead><tbody><tr><td>objectKey</td><td>string</td><td>Yes</td><td>Full object key/path of the file to delete</td><td>Must be a valid object key from user's files</td></tr></tbody></table>

**Response JSON Sample**:

```json
{
  "success": true,
  "httpStatus": "OK",
  "message": "File deleted successfully",
  "action_time": "2025-09-23T10:30:00",
  "data": null
}

```

**Response Fields**:

<table id="bkmrk-field-description-su-3"><thead><tr><th>Field</th><th>Description</th></tr></thead><tbody><tr><td>success</td><td>Boolean indicating operation success</td></tr><tr><td>httpStatus</td><td>HTTP status code as string</td></tr><tr><td>message</td><td>Success message confirming deletion</td></tr><tr><td>action\_time</td><td>ISO 8601 timestamp of response generation</td></tr><tr><td>data</td><td>Always null for delete operations</td></tr></tbody></table>

**Error Responses**:

- `400 Bad Request`: Invalid object key format
- `401 Unauthorized`: Missing or invalid JWT token
- `404 Not Found`: File not found or user not found
- `403 Forbidden`: Attempting to delete file not owned by user
- `500 Internal Server Error`: Storage system errors

---

## Quick Reference Guide

### File Directory Types

- **PROFILE**: User profile images, avatars, and personal media
- **CATEGORIES**: Category images, banners, and related media
- **SHOPS**: Shop-related images, logos, and promotional materials

### Supported File Types

- **Images**: JPEG, JPG, PNG, GIF, WebP, BMP, SVG
- **Videos**: MP4, AVI, MOV, WMV, FLV, WebM, MKV
- **Documents**: PDF, DOC, DOCX, TXT, XLS
- **Audio**: MP3, WAV, OGG, AAC

### File Size and Limits

- **Maximum file size**: 25MB per file
- **File naming**: Original names preserved, unique UUIDs generated for storage
- **Public access**: All uploaded files are publicly accessible via direct URLs
- **Checksums**: MD5 hashes calculated for file integrity verification

### Common HTTP Status Codes

- `200 OK`: Successful GET/DELETE request
- `201 Created`: Successful POST request (file upload)
- `400 Bad Request`: Invalid request data or file validation errors
- `401 Unauthorized`: Authentication required/failed
- `403 Forbidden`: Insufficient permissions
- `404 Not Found`: Resource not found
- `413 Payload Too Large`: File size exceeds limit
- `422 Unprocessable Entity`: Validation errors
- `500 Internal Server Error`: Server error

### Authentication Types

- **Bearer Token**: Include `Authorization: Bearer your_access_token` in headers
- **Form Data**: Use `multipart/form-data` for file uploads
- **JSON**: Use `application/json` for other requests

### Data Format Standards

- **Dates**: Use ISO 8601 format (2025-09-23T10:30:00Z)
- **File sizes**: Provided in bytes and human-readable format
- **UUIDs**: Standard UUID format for all identifiers
- **URLs**: Full HTTPS URLs for file access

# File Thunder Arch & Flow

File Thunder is the dedicated media-processing microservice for the NexGate / Veepii platform. It handles all file ingestion, processing, storage, and serving-readiness — the main backend never touches raw bytes.

---

## Table of Contents

1. [Architecture Overview](#1-architecture-overview)
2. [Storage: Buckets &amp; Object Keys](#2-storage-buckets--object-keys)
3. [MediaDomain &amp; MediaContext](#3-mediadomain--mediacontext)
4. [Upload Flow — End to End](#4-upload-flow--end-to-end)
5. [The Four Wheels](#5-the-four-wheels)
6. [Progress Tracking (Redis → SSE)](#6-progress-tracking-redis--sse)
7. [Watermarking](#7-watermarking)
8. [CDN &amp; File Serving Per Environment](#8-cdn--file-serving-per-environment)
9. [Security](#9-security)
10. [API Reference &amp; How to Consume](#10-api-reference--how-to-consume)

---

## 1. Architecture Overview

```
┌─────────────────────────────────────────────────────────┐
│                      Main Backend                       │
│  - Only public-facing API                               │
│  - Calls File Thunder over internal HTTP (HMAC signed)  │
│  - Subscribes to Redis for progress → pushes SSE        │
└────────────┬──────────────────────────┬─────────────────┘
             │ HTTP (sync)              │ Redis Pub/Sub (async)
             ▼                          ▼
┌─────────────────────────────────────────────────────────┐
│                     File Thunder                        │
│                                                         │
│   API Profile          Worker Profile                   │
│   ─────────────        ──────────────                   │
│   UploadController     UploadConfirmedWorker            │
│   MediaQueryController ThumbnailWorker                  │
│                        RawFilePurgeJob                  │
│                                                         │
│   Wheels (processing engines)                           │
│   ──────────────────────────                            │
│   ImageWheel  VideoWheel  ScanWheel  (MinIO ops)        │
└────────────┬──────────────────────────┬─────────────────┘
             │                          │
             ▼                          ▼
          MinIO                      RabbitMQ
    (4 private buckets)       (internal event routing)

```

**Key constraints:**

- File Thunder is **internal only** — never exposed to the internet directly
- Main backend is the **only caller** of File Thunder HTTP APIs
- File Thunder **never touches file bytes on upload** — client uploads directly to MinIO via presigned URL
- Redis is **FT-internal** — used for progress pub/sub and nonce replay protection
- RabbitMQ is **FT-internal** — used to route events between listeners and workers

---

## 2. Storage: Buckets &amp; Object Keys

### Buckets

<table id="bkmrk-bucket-purpose-acces"><thead><tr><th>Bucket</th><th>Purpose</th><th>Access</th></tr></thead><tbody><tr><td>`nexgate-raw`</td><td>Temporary upload landing zone — 24h TTL</td><td>Private</td></tr><tr><td>`nexgate-public`</td><td>Processed media served to end users</td><td>Private (CDN in front)</td></tr><tr><td>`nexgate-private`</td><td>Internal system assets (outros, future forensic assets)</td><td>Private</td></tr><tr><td>`nexgate-digital`</td><td>Digital product files — ClamAV scanned, download only</td><td>Private</td></tr></tbody></table>

All buckets are **fully private**. MinIO is never directly accessible from the internet. Public content is served exclusively through the CDN (Cloudflare), which pulls from MinIO origin.

### Object Key Pattern

```
{domain}/{ownerId}/{fileId}/{variant}

```

**Examples:**

```
posts/550e8400-e29b-41d4-a716-446655440000/f7e8d9.../original
posts/550e8400-e29b-41d4-a716-446655440000/f7e8d9.../large.webp
posts/550e8400-e29b-41d4-a716-446655440000/f7e8d9.../thumb.webp
posts/550e8400-e29b-41d4-a716-446655440000/f7e8d9.../hls/master.m3u8
posts/550e8400-e29b-41d4-a716-446655440000/f7e8d9.../hls/360p/360p.m3u8

system/outro/{ownerId}/{height}p.mp4   ← in nexgate-private

```

**Rule:** The database stores **object keys only**, never full URLs. URLs are assembled at the resolver boundary (CDN base URL + key in prod, presigned GET in local).

---

## 3. MediaDomain &amp; MediaContext

### MediaDomain

Defines **where** in the storage hierarchy a file lives. It is the top-level folder in the object key. This is an enum — the main backend must send one of these exact values.

<table id="bkmrk-value-used-for-posts"><thead><tr><th>Value</th><th>Used For</th></tr></thead><tbody><tr><td>`POSTS`</td><td>Social posts — images and videos</td></tr><tr><td>`PROFILES`</td><td>Profile pictures, cover photos</td></tr><tr><td>`MESSAGES`</td><td>Direct message attachments</td></tr><tr><td>`PRODUCTS`</td><td>Product images, product videos, digital downloads</td></tr><tr><td>`SHOPS`</td><td>Shop banners, shop logos</td></tr><tr><td>`EVENTS`</td><td>Event covers, event gallery images</td></tr></tbody></table>

### MediaContext

Defines **how** a file is processed. This drives the entire processing pipeline. No logic is attached to domain — all logic is driven by context.

<table id="bkmrk-value-type-processin"><thead><tr><th>Value</th><th>Type</th><th>Processing Pipeline</th></tr></thead><tbody><tr><td>`SOCIAL_IMAGE`</td><td>Image</td><td>ImageWheel — orient, strip EXIF, WebP variants, blurhash, lqip</td></tr><tr><td>`SOCIAL_VIDEO`</td><td>Video</td><td>VideoWheel — transcode + **social watermark + outro** + thumbnail</td></tr><tr><td>`PROFILE_PICTURE`</td><td>Image</td><td>ImageWheel</td></tr><tr><td>`COVER_PHOTO`</td><td>Image</td><td>ImageWheel</td></tr><tr><td>`DM_IMAGE`</td><td>Image</td><td>ImageWheel</td></tr><tr><td>`DM_VIDEO`</td><td>Video</td><td>VideoWheel — transcode, no watermark</td></tr><tr><td>`DM_DOCUMENT`</td><td>Any</td><td>ScanWheel — ClamAV scan</td></tr><tr><td>`PRODUCT_IMAGE`</td><td>Image</td><td>ImageWheel</td></tr><tr><td>`PRODUCT_VIDEO`</td><td>Video</td><td>VideoWheel — transcode + text watermark, no outro</td></tr><tr><td>`DIGITAL_PRODUCT`</td><td>Any</td><td>ScanWheel — ClamAV dual scan</td></tr><tr><td>`SHOP_BANNER`</td><td>Image</td><td>ImageWheel</td></tr><tr><td>`SHOP_LOGO`</td><td>Image</td><td>ImageWheel</td></tr><tr><td>`EVENT_COVER`</td><td>Image</td><td>ImageWheel</td></tr><tr><td>`EVENT_GALLERY`</td><td>Image</td><td>ImageWheel</td></tr></tbody></table>

**Validation rules enforced at API layer:**

- Image contexts only accept image MIME types; video contexts only accept video MIME types
- `SOCIAL_VIDEO` requires `username` field (used for personalized watermark + outro)
- Max file sizes: images 20 MB, videos 2 GB, digital products 500 MB
- HEIC/HEIF are rejected — parser attack surface too high

---

## 4. Upload Flow — End to End

### Standard Media Upload

```
Main Backend                    File Thunder                 MinIO           Client (Browser)
     │                               │                         │                   │
     │  POST /api/v1/upload/request  │                         │                   │
     │  (HMAC signed)                │                         │                   │
     │──────────────────────────────▶│                         │                   │
     │                               │ validate + create DB    │                   │
     │                               │ record (PENDING)        │                   │
     │                               │ publish PENDING         │                   │
     │                               │ to Redis                │                   │
     │                               │ generate presigned PUT  │                   │
     │                               │─────────────────────────▶                  │
     │  { fileId, presignedUrl }     │◀────────────────────────                   │
     │◀──────────────────────────────│                         │                   │
     │                               │                         │                   │
     │  return presignedUrl          │                         │                   │
     │  to client                    │                         │                   │
     │──────────────────────────────────────────────────────────────────────────▶ │
     │                               │                         │                   │
     │                               │                         │  PUT (file bytes) │
     │                               │                         │◀──────────────────│
     │                               │                         │  200 OK           │
     │                               │                         │──────────────────▶│
     │                               │                         │                   │
     │                               │ ◀── MinIO event ────────│ s3:ObjectCreated  │
     │                               │ MinioEventListener      │                   │
     │                               │ PENDING → UPLOADED      │                   │
     │                               │ publish UPLOADED→Redis  │                   │
     │                               │ publish UploadConfirmed │                   │
     │                               │ to RabbitMQ             │                   │
     │                               │                         │                   │
     │                               │ UploadConfirmedWorker   │                   │
     │                               │ routes by mimeType      │                   │
     │                               │ UPLOADED → PROCESSING   │                   │
     │                               │ publish PROCESSING→Redis│                   │
     │                               │                         │                   │
     │                               │ [ImageWheel or          │                   │
     │                               │  VideoWheel runs]       │                   │
     │                               │                         │                   │
     │                               │ variants uploaded       │                   │
     │                               │ to nexgate-public ──────▶                  │
     │                               │ raw deleted from        │                   │
     │                               │ nexgate-raw ────────────▶                  │
     │                               │ PROCESSING → READY      │                   │
     │                               │ publish READY → Redis   │                   │
     │                               │                         │                   │

```

### User Custom Thumbnail Upload

```
Main Backend                    File Thunder                 MinIO
     │                               │                         │
     │  POST /api/v1/upload/         │                         │
     │  thumbnail/{fileId}           │                         │
     │──────────────────────────────▶│                         │
     │                               │ generate presigned PUT  │
     │                               │ key: .../custom-        │
     │                               │ thumbnail               │
     │  { presignedUrl }             │                         │
     │◀──────────────────────────────│                         │
     │  client uploads thumbnail     │                         │
     │─────────────────────────────────────────────────────────▶
     │                               │◀── MinIO event ─────────│
     │                               │ MinioEventListener      │
     │                               │ detects suffix=         │
     │                               │ "custom-thumbnail"      │
     │                               │ → ThumbnailWorker       │
     │                               │ download → verify magic │
     │                               │ bytes → ImageMagick     │
     │                               │ variants → save to      │
     │                               │ entity.userThumbnail    │
     │                               │ → delete raw            │

```

### File Status Lifecycle

```
PENDING → UPLOADING → UPLOADED → SCANNING → PROCESSING → LIVE_PARTIAL → READY
                                                                ▲
                                              (short video: after 360p uploaded)
                                              (long video: after 360p + partial HLS)

```

---

## 5. The Four Wheels

Wheels are the processing engines. Each runs in the worker profile.

---

### Wheel 1 — ImageWheel (ImageMagick)

**Triggered by:** image MIME type on any image context

**Pipeline:**

```
download raw from nexgate-raw
      │
      ▼
auto-orient + strip EXIF
      │
      ├──▶ large.webp    (1920px max width, shrink only, Q85)
      ├──▶ medium.webp   (800px max width, shrink only, Q82)
      ├──▶ thumb.webp    (300px max width, shrink only, Q80)
      ├──▶ og.webp       (1200×630 center crop, Q85)
      │    └─ falls back to medium if source width < 1200px
      ├──▶ blurhash      (encoded from 32×32 downsample)
      └──▶ lqip          (10×10 forced, base64 WebP data URI)

all WebP variants → nexgate-public
keys + blurhash + lqip → media_files.variants (JSONB)
raw deleted from nexgate-raw
status → READY

```

**Skipping logic:** variant is skipped if source is already within bounds (never upscale).

---

### Wheel 2 — VideoWheel (FFmpeg / Jaffree)

**Triggered by:** video MIME type on any video context

**Step 1 — Probe &amp; Validate**

```
download raw from nexgate-raw
magic byte verify (must be real video container)
FFprobe → codedWidth, codedHeight, rotation, duration, codec
displayWidth/displayHeight = rotation-aware (swap for 90°/270°)
size gate: > 2 GB → reject
duration gate: > 4h → reject
route: duration < 3min → processShort()
       duration ≥ 3min → processLong()

```

**Step 2a — Short Path (&lt; 3 min)**

```
transcode() runs 3 variants in sequence:
  360p  (CRF 28, 96k audio)  → upload → status LIVE_PARTIAL
  720p  (CRF 23, 128k audio) → upload
  1080p (CRF 21, 192k audio) → upload

filter_complex:
  [0:v] split=2 [main][blur]
  [blur] scale={W}:{H}, boxblur=20:5 [bg]
  [main] scale={w}:{h} (fit within target, keep AR) [fg]
  [bg][fg] overlay=x=(W-w)/2:y=(H-h)/2 [out]
  + rotation transpose prepended if needed
  + watermark drawtext chain appended

if SOCIAL_VIDEO:
  append outro via concat (no re-encode)

skip logic: source displayWidth >= targetW OR displayHeight >= targetH

```

**Step 2b — Long Path (≥ 3 min) — HLS Adaptive**

```
transcodeHls() per variant:
  360p → segments + .m3u8 → upload → partial master.m3u8 → LIVE_PARTIAL
  720p → segments + .m3u8 → upload
  1080p → segments + .m3u8 → upload → full master.m3u8 → READY

segment length: 2s
playlist type: VOD
master.m3u8 key: {keyBase}/hls/master.m3u8
variant keys:   {keyBase}/hls/{name}/{name}.m3u8

```

**Step 3 — Thumbnail &amp; Preview (both short and long)**

```
selectBestFrame():
  5 candidates at 15/30/45/60/75% of duration
  scored by: brightness gate (30–230) + Laplacian variance (sharpness)
  FFmpeg extracts 720px JPEG per candidate → pick winner

buildThumbnailVariants():
  poster.webp      (1280px, Q85)
  thumb.webp       (480px, Q80)
  og.webp          (1200×630 center crop, Q85)
  blurhash         (from 32×32 downsample)
  lqip             (10×10 forced, base64 WebP data URI)
  dominant_color   (#RRGGBB from 1×1 squish)

extractPreviewClip():
  skip first 5% (max 2s) → read 6s → setpts=0.5*PTS → 3s at 2× speed
  blur-pad 360×640, muted
  watermarked (same moving watermark as main video)

all thumbnail variants → nexgate-public
raw deleted from nexgate-raw after all variants done

```

---

### Wheel 3 — ScanWheel (ClamAV)

**Triggered by:** `DIGITAL_PRODUCT` and `DM_DOCUMENT` contexts only Social content (images and videos) is **never** scanned by ClamAV — FFmpeg/ImageMagick re-encode is the sanitisation.

**Pipeline:**

```
download raw from nexgate-raw
      │
      ▼
SHA-256 hash → check file_hashes table
      │
      ├── hash known + clean → skip scan, copy to nexgate-digital, READY (dedup fast lane)
      │
      └── hash unknown →
            ClamAV scan #1 (upload scan)
            if clean → move to nexgate-digital
            ClamAV scan #2 (pre-download scan, confirms in-transit integrity)
            if clean → save hash, status READY
            if infected → status FAILED, quarantine

```

---

### Wheel 4 — MinIO Operations (used by all wheels)

Not a standalone service — MinIO operations are helpers used across all wheels:

<table id="bkmrk-operation-used-by-pr"><thead><tr><th>Operation</th><th>Used By</th></tr></thead><tbody><tr><td>Presigned PUT URL</td><td>Upload request endpoint</td></tr><tr><td>Download object</td><td>All wheels (download raw for processing)</td></tr><tr><td>Upload object</td><td>All wheels (upload processed variants)</td></tr><tr><td>Remove object</td><td>All wheels (delete raw after processing)</td></tr><tr><td>Stat object</td><td>OutroService (cache check), ScanWheel (dedup check)</td></tr></tbody></table>

---

## 6. Progress Tracking (Redis → SSE)

File Thunder publishes status changes to Redis. The main backend subscribes and drives SSE to the client. **SSE is the main backend's responsibility — File Thunder only publishes.**

### What File Thunder Does

On every status change:

```java
// 1. Cache current status in Redis (24h TTL)
redisTemplate.opsForValue().set("ft:status:{fileId}", status.name(), 24h);

// 2. Publish to per-file channel
redisTemplate.convertAndSend("ft:progress:{fileId}", status.name());

// 3. Append to timeline in Postgres
entity.timeline.add({ status, at })

```

### What the Main Backend Must Do

```java
// Subscribe to the file's progress channel
redisTemplate.subscribe((message, pattern) -> {
    String status = message.toString();
    sseEmitter.send(SseEmitter.event().data(status));
}, "ft:progress:" + fileId);

```

### Status Channel Key

```
ft:progress:{fileId}     ← subscribe here for live updates
ft:status:{fileId}       ← read here for current status (24h cached)

```

---

## 7. Watermarking

### Moving Watermark

Applied to all video variants and preview clips. Watermark position cycles through 4 corners every 3 seconds (5 seconds for social).

**Text-only (PRODUCT\_VIDEO):**

```
drawtext: fontsize=max(14,H/40), white@0.65 + black shadow
position: floor(t/3) mod 4 → top-left, top-right, bottom-right, bottom-left
text: from watermark.text property (default: "NexGate")

```

**Social watermark (SOCIAL\_VIDEO):**

```
2-point diagonal: Pos A (22%, 28%) ↔ Pos B (58%, 65%) — switches every 5s
Logo: nexgate_logo_white.svg, 36×36, 65% opacity
@ symbol: orange #F06023@0.85 + shadow
username: white@0.85 + shadow, below logo

```

### Outro (SOCIAL\_VIDEO only)

A personalized outro clip is appended to the watermarked variant (not clean variants).

**Generation:**

```
template: outro_template_with_sfx.mp4 (classpath resource)
font: Sora SemiBold 600 (classpath resource)
drawtext: @ in orange #F06023, username in cream #F4EEE9
fade-in: invisible before t=0.75s, fully opaque by t=1.2s
scale to match variant resolution (360p / 720p / 1080p)
append via: -f concat -safe 0 -c copy (no re-encode, fast)

```

**Caching:**

```
Generated once per ownerId + resolution
Cached to: nexgate-private / system/outro/{ownerId}/{height}p.mp4
On next upload: cache hit → download and reuse, skip generation

```

---

## 8. CDN &amp; File Serving Per Environment

All MinIO buckets are private. Access to public content is environment-dependent.

### Environment Matrix

<table id="bkmrk-environment-serving-"><thead><tr><th>Environment</th><th>Serving Method</th><th>Config Value</th></tr></thead><tbody><tr><td>Local dev</td><td>Presigned GET URLs (MinIO direct, short-lived)</td><td>`ft.storage.mode=local`</td></tr><tr><td>Staging</td><td>CDN — `cdn-staging.nexgate.com`</td><td>`ft.storage.mode=cdn`</td></tr><tr><td>Production</td><td>CDN — `cdn.nexgate.com`</td><td>`ft.storage.mode=cdn`</td></tr></tbody></table>

### Local / Dev

- App generates a presigned GET URL from MinIO (e.g. 1-hour expiry)
- URL is returned directly to the client
- MinIO must be reachable by the client
- Good enough for development and manual testing

### Staging &amp; Production (CDN)

- App assembles: `{ft.cdn.base-url} + "/" + objectKey`
- Cloudflare sits in front of MinIO origin
- Cloudflare pulls from MinIO on cache miss using a service credential
- Every subsequent request served from Cloudflare edge — MinIO never hit again
- Cost: Cloudflare bandwidth is free; MinIO only pays VPS bandwidth once per cache miss

### Properties

```properties
# Local
ft.storage.mode=local
ft.cdn.base-url=

# Staging
ft.storage.mode=cdn
ft.cdn.base-url=https://cdn-staging.nexgate.com

# Production
ft.storage.mode=cdn
ft.cdn.base-url=https://cdn.nexgate.com

```

### URL Assembly Logic (resolver boundary)

```java
// local mode → presigned GET from MinIO
// cdn mode   → CDN base + object key
String url = storageMode.equals("cdn")
    ? cdnBaseUrl + "/" + objectKey
    : minioClient.getPresignedObjectUrl(GET, bucket, objectKey, 1h);

```

**Private content (DIGITAL\_PRODUCT, DM\_DOCUMENT):** always presigned GET URLs regardless of environment — these are never cached by CDN. Generated fresh per download request (15-minute expiry, audit logged).

---

## 9. Security

### CORS

Configured via `ft.cors.*` properties. The CORS filter runs **before all other filters** so OPTIONS preflight requests are handled without requiring HMAC headers.

```properties
# Local
ft.cors.allowed-origins=http://localhost:3000

# Production
ft.cors.allowed-origins=https://nexgate.com,https://www.nexgate.com

ft.cors.allowed-methods=GET,POST,PUT,DELETE,OPTIONS
ft.cors.allowed-headers=*
ft.cors.allow-credentials=false
ft.cors.max-age-seconds=3600

```

---

### Service-to-Service HMAC Authentication

Every HTTP request from the main backend to File Thunder must be HMAC-SHA256 signed. File Thunder rejects any unsigned or incorrectly signed request with `401 Unauthorized`.

**Filter order:**

```
Request → CorsFilter (HIGHEST_PRECEDENCE)
        → HmacAuthFilter (HIGHEST_PRECEDENCE + 1)
        → Controllers

```

#### Required Headers

<table id="bkmrk-header-description-e"><thead><tr><th>Header</th><th>Description</th><th>Example</th></tr></thead><tbody><tr><td>`X-Service-Id`</td><td>Identifier of the calling service</td><td>`nexgate-main`</td></tr><tr><td>`X-Timestamp`</td><td>Unix epoch seconds at time of request</td><td>`1718123456`</td></tr><tr><td>`X-Nonce`</td><td>Random UUID — unique per request</td><td>`f47ac10b-58cc-...`</td></tr><tr><td>`X-Signature`</td><td>HMAC-SHA256 hex of the canonical string</td><td>`a3f5c2...`</td></tr></tbody></table>

#### Canonical String

```
METHOD\n
REQUEST_URI\n
TIMESTAMP\n
NONCE\n
HEX(SHA-256(requestBody))

```

**Example for `POST /api/v1/upload/request`:**

```
POST
/api/v1/upload/request
1718123456
f47ac10b-58cc-4372-a567-0e02b2c3d479
e3b0c44298fc1c149afb...   ← SHA-256 of the JSON body

```

#### Signature Computation

```java
// Main backend — how to sign a request
String bodyHash = HexFormat.of().formatHex(
    MessageDigest.getInstance("SHA-256").digest(requestBodyBytes)
);

String canonical = method + "\n" + uri + "\n" + timestamp + "\n" + nonce + "\n" + bodyHash;

Mac mac = Mac.getInstance("HmacSHA256");
mac.init(new SecretKeySpec(sharedSecret.getBytes(UTF_8), "HmacSHA256"));
String signature = HexFormat.of().formatHex(mac.doFinal(canonical.getBytes(UTF_8)));

// Set headers on outgoing request
request.setHeader("X-Service-Id", "nexgate-main");
request.setHeader("X-Timestamp",  String.valueOf(Instant.now().getEpochSecond()));
request.setHeader("X-Nonce",      UUID.randomUUID().toString());
request.setHeader("X-Signature",  signature);

```

#### What File Thunder Verifies (in order)

```
1. All 4 headers present                → 401 if any missing
2. X-Service-Id in allowed list         → 401 if unknown
3. X-Timestamp within ±5 minutes        → 401 if stale (replay protection)
4. X-Nonce not seen before              → 401 if duplicate
   (stored in Redis with 10-min TTL)       (replay attack blocked)
5. Recompute HMAC, timing-safe compare  → 401 if mismatch
   (MessageDigest.isEqual)                 (prevents timing oracle)

```

#### Properties

```properties
ft.security.hmac.secret=<long-random-secret-same-in-both-services>
ft.security.hmac.allowed-service-ids=nexgate-main
ft.security.hmac.timestamp-tolerance-seconds=300
ft.security.hmac.nonce-ttl-seconds=600

```

The `secret` must be identical in both File Thunder and the main backend config. Use a different value per environment (local / staging / prod).

---

## 10. API Reference &amp; How to Consume

Base URL: `http://file-thunder:8081` (internal network only) All requests must include HMAC headers — see Section 9.

---

### POST /api/v1/upload/request

Request a presigned URL to upload a file directly to MinIO.

**Request body:**

```json
{
  "ownerId":          "550e8400-e29b-41d4-a716-446655440000",
  "domain":           "POSTS",
  "context":          "SOCIAL_VIDEO",
  "originalFilename": "my-video.mp4",
  "mimeType":         "video/mp4",
  "fileSizeBytes":    104857600,
  "username":         "josh_dev"
}

```

`username` is required when `context` is `SOCIAL_VIDEO`. All other fields are always required.

**Response:**

```json
{
  "status": "success",
  "message": "Presigned upload URL generated",
  "data": {
    "fileId":           "f7e8d9fa-...",
    "presignedUrl":     "http://minio:9000/nexgate-raw/posts/.../original?X-Amz-...",
    "objectKey":        "posts/{ownerId}/{entityId}/{fileId}/original",
    "bucket":           "nexgate-raw",
    "expiresInSeconds": 1800
  }
}

```

**What to do next:** PUT the file bytes directly to `presignedUrl` from the client browser. No auth headers needed on the PUT — the presigned URL is self-authenticating.

---

### POST /api/v1/upload/thumbnail/{fileId}

Request a presigned URL to replace the system-generated video thumbnail with a user-supplied one. Only valid after the video file has been processed (status READY).

**Response:**

```json
{
  "status": "success",
  "message": "Thumbnail presigned upload URL generated",
  "data": {
    "fileId":           "f7e8d9fa-...",
    "presignedUrl":     "http://minio:9000/nexgate-raw/.../custom-thumbnail?X-Amz-...",
    "objectKey":        "posts/.../custom-thumbnail",
    "bucket":           "nexgate-raw",
    "expiresInSeconds": 1800
  }
}

```

Accepted formats: JPEG, PNG, WebP only (verified by magic bytes).

---

### GET /api/v1/media/{fileId}

Get full metadata for a file, including all processed variants.

**Response:**

```json
{
  "status": "success",
  "data": {
    "fileId":    "f7e8d9fa-...",
    "ownerId":   "550e8400-...",
    "domain":    "POSTS",
    "context":   "SOCIAL_VIDEO",
    "status":    "READY",
    "mimeType":  "video/mp4",
    "variants": {
      "360p":          "posts/.../360p.mp4",
      "720p":          "posts/.../720p.mp4",
      "1080p":         "posts/.../1080p.mp4",
      "poster":        "posts/.../poster.webp",
      "thumb":         "posts/.../thumb.webp",
      "og":            "posts/.../og.webp",
      "preview_3s":    "posts/.../preview_3s.mp4",
      "blurhash":      "LKO2?U%2Tw=w]~RBVZRi};RPxuwH",
      "lqip":          "data:image/webp;base64,...",
      "dominant_color":"#1A2B3C"
    },
    "userThumbnail": null,
    "timeline": [
      { "status": "PENDING",    "at": "2026-06-15T10:00:00" },
      { "status": "UPLOADED",   "at": "2026-06-15T10:00:05" },
      { "status": "PROCESSING", "at": "2026-06-15T10:00:06" },
      { "status": "LIVE_PARTIAL","at": "2026-06-15T10:00:45" },
      { "status": "READY",      "at": "2026-06-15T10:02:10" }
    ],
    "createdAt": "2026-06-15T10:00:00",
    "updatedAt": "2026-06-15T10:02:10"
  }
}

```

**Thumbnail resolution rule:**

```
userThumbnail != null ? use userThumbnail : use variants

```

Both `userThumbnail` and `variants` use the same key names (`poster`, `thumb`, `og`, `blurhash`, `lqip`, `dominant_color`).

---

### GET /api/v1/media/{fileId}/download?requesterId={uuid}

Generate a time-limited download URL for private files (`DIGITAL_PRODUCT`, `DM_DOCUMENT` only). Every call is audit-logged.

**Response:**

```json
{
  "status": "success",
  "data": {
    "url":              "http://minio:9000/nexgate-digital/...?X-Amz-...",
    "expiresInSeconds": 900
  }
}

```

---

### GET /api/v1/quota/{ownerId}

Get the total storage used by an owner across all their processed files. Updated atomically as each file is processed — never stale.

**Response:**

```json
{
  "status": "success",
  "message": "Storage usage",
  "data": {
    "ownerId":   "550e8400-e29b-41d4-a716-446655440000",
    "usedBytes": 1073741824,
    "usedMb":    1024.0,
    "usedGb":    1.0
  }
}

```

**How quota is tracked:**

- Every wheel (`ImageWheel`, `VideoWheel`, `ScanWheel`) calls `StorageUsageService.trackUsage(ownerId, bytes)` after uploading processed variants
- Tracked bytes = size of the **processed variants**, not the raw upload
- Stored in `user_storage_quota` table, upserted atomically per owner
- `releaseUsage()` is called on hard delete (soft deletes do not release quota)
- Returns `0` if the owner has no tracked usage yet (never uploaded)

**Use this endpoint to:**

- Display storage usage in the main backend dashboard
- Enforce storage limits before allowing a new upload request

---

### Error Responses

```json
{ "error": "Missing required security headers" }          ← 401
{ "error": "Request timestamp out of acceptable window" } ← 401
{ "error": "Nonce already used — possible replay attack"} ← 401
{ "error": "Invalid signature" }                          ← 401

```

Validation errors return 400 with field-level messages via `@ControllerAdvice`.

---

*This document covers the full File Thunder system as of 2026-06-15.**CDN wiring (Cloudflare), Kafka integration, lazy transcode, and forensic watermarks are deferred.*