File & Media Thunder
File Thunder — Architecture & Design Guide
File Thunder — Architecture & Design Guide
NexGate Media Engine | Version 1.0
Table of Contents
What is File Thunder?
The Four Wheels
Storage Architecture (MinIO Buckets)
Accepted File Formats
Upload Flow
Processing Pipelines
Image Pipeline
Short Video Pipeline
Long Video Pipeline
Digital Products Pipeline
DM Attachments Pipeline
Watermarking Strategy
File Protection Layers
URL Strategy
OG (Open Graph) Strategy
Compression & Transcoding
BlurHash, LQIP & Dominant Color
Deduplication
Quota Management
Storage Lifecycle
Abandoned Upload Handling
Progress Tracking (SSE)
Communication Architecture
CDN Strategy
Shareable Links
Technology Stack
Database Schema
Docker & Infrastructure
What File Thunder Does NOT Do
1. What is File Thunder?
File Thunder is NexGate's dedicated media processing engine. It is a standalone Spring Boot microservice responsible for all file and media operations across the entire NexGate platform.
Core principle: The Main Backend never touches raw files. It delegates all media operations to File Thunder and acts as a thin client.
Responsibilities:
Receive file upload requests
Validate files (type, size, quota)
Virus scanning
Transcoding and processing
Generating variants (thumbnails, WebP, HLS, MP4)
Watermarking
Storing files in MinIO
Serving files via CDN
Publishing media-ready events
What it is NOT:
Not an AI/ML service
Not a recommendation engine
Not a social graph service
Not a payment service
2. The Four Wheels
File Thunder is powered by four core engines:
FILE THUNDER ENGINE
│
┌──────┼──────┬──────┐
│ │ │ │
FFmpeg IM ClamAV MinIO
Video Image Sec Storage
Wheel Wheel Wheel Wheel
Wheel 1 — FFmpeg (Video)
Video transcoding (any format → H.264 MP4 or HLS)
Audio extraction (for Rec Engine)
HLS segment generation
Thumbnail extraction (best frame detection)
3-second preview clip generation
Watermark overlay (logo + username)
Aspect ratio handling and blur padding
Fragmented MP4 (fMP4) generation
GIF → MP4 conversion
Format conversion (MOV, MKV, AVI, WEBM → MP4)
Audio normalization
Java wrapper: Jaffree (calls /usr/bin/ffmpeg via ProcessBuilder)
Wheel 2 — ImageMagick (Images)
Resize to multiple variants
Format conversion (JPEG, PNG, HEIC → WebP)
EXIF stripping (privacy — removes GPS data)
Auto-orientation (fix rotated phone photos)
Quality compression
Dominant color extraction
LQIP generation (10×10 base64)
OG image generation (1200×630)
GIF → WebP animated conversion
PDF → preview image conversion
Java wrapper: IM4Java (calls /usr/bin/convert via ProcessBuilder)
Additional library: blurhash-java for BlurHash generation
Wheel 4 — MinIO (Storage)
Stores ALL file variants across 4 buckets
S3-compatible object storage (self-hosted)
Receives direct uploads via presigned URLs (client never touches File Thunder bandwidth)
Serves as origin for Cloudflare CDN
Supports multipart uploads (files > 5MB)
Supports byte-range requests (progressive streaming, resumable downloads)
Lifecycle policies (auto-delete raw uploads after 24h)
Storage class tiering (hot → warm → cold)
Bucket event notifications (upload complete events)
Never exposed publicly (only Cloudflare and File Thunder can reach it)
Java SDK: MinIO Java SDK ( io.minio:minio )
Wheel 3 — ClamAV (Security)
Virus and malware scanning for every uploaded file
Hash-based scan cache (skip re-scanning known clean files)
Runs as separate Docker container (daemon-based)
File Thunder connects via TCP socket ( clamav:3310 )
Digital products scanned twice (on upload + before download)
Java client: clamd4j or custom TCP socket client
3. Storage Architecture (MinIO Buckets)
Rule: 4 buckets total. Never one bucket per user.
nexgate-raw/ ← temporary upload landing zone
nexgate-public/ ← CDN cached, social content
nexgate-private/ ← signed URLs, DMs + docs
nexgate-digital/ ← purchase-gated, digital products
nexgate-raw (Temporary)
nexgate-raw/
uploads/{userId}/{fileId}/
original.ext ← raw upload lands here
Auto-delete lifecycle policy: 24 hours
Incomplete multipart uploads aborted: 24 hours
Never served to any client
Deleted by File Thunder after processing completes
nexgate-public
nexgate-public/
profiles/{userId}/{fileId}/
avatar_400.webp
avatar_150.webp
avatar_50.webp
cover.webp
posts/{userId}/{fileId}/
360p_clean.mp4
720p_clean.mp4
1080p_clean.mp4
360p_watermarked.mp4
720p_watermarked.mp4
1080p_watermarked.mp4
preview_3s.mp4
hls/master.m3u8
hls/360p/segment_000.ts ...
hls/720p/segment_000.ts ...
thumbnail.webp
og_clean.webp
og_play.webp
og_preview.mp4
stories/{userId}/{fileId}/
720p_clean.mp4
thumbnail.webp
events/{accountId}/{eventId}/{fileId}/
banner.webp
banner_mobile.webp
banner_thumb.webp
shops/{shopId}/{productId}/{fileId}/
large.webp
medium.webp
thumb.webp
categories/{categoryId}/{fileId}.webp
nexgate-private
nexgate-private/
messages/{conversationId}/{fileId}/
original.jpg
thumb.webp
audio/{fileId}.wav ← temp for Rec Engine (deleted after transcription)
documents/{userId}/{fileId}/
original.pdf
preview.webp
kyc/{userId}/{fileId}/
id_front.jpg
id_back.jpg
nexgate-digital
nexgate-digital/
products/{shopId}/{productId}/{fileId}/
original/
file.pdf ← actual purchased product
checksum.txt ← SHA-256 hash
preview/
preview.webp ← low-res watermarked preview
cover.webp ← product cover image
Object Key Pattern (Consistent Rule)
{bucket}/{domain}/{ownerId}/{entityId}/{fileId}/{variant}
Examples:
nexgate-public/posts/usr_123/post_456/file_789/thumb.webp
nexgate-private/messages/conv_abc/file_789/original.jpg
nexgate-digital/products/shop_xyz/prod_123/file_789/original.pdf
4. Accepted File Formats
Images
Accept
Reject
JPEG/JPG
BMP
PNG
TIFF
HEIC/HEIF
RAW
WebP
SVG (security risk)
GIF
Output: Always WebP
Videos
Accept
Reject
MP4
WMV
MOV
FLV
MKV
VOB
WEBM
AVI
3GP
Output: H.264 MP4 (short) or HLS H.264 (long)
Digital Products
PDF, DOCX, XLSX, PPTX, GLB, OBJ, FBX, STL, MP3, WAV, FLAC, ZIP, RAR, PSD, AI, PNG (stock art)
5. Upload Flow
Step-by-Step
CLIENT
↓
[1] Intelligent client-side compression
- Detect: resolution, bitrate, codec, network type, device tier
- Compress if: over-bitrated, slow network, large file
- Never compress if: already optimized
- Target: max 1080p, CRF 18 (light)
- HEIC → JPEG conversion (iOS)
↓
[2] POST /media/upload-request
{
fileName, fileSize, mimeType,
directory, clientMeta: {
clientApp, networkType,
deviceTier, wasCompressed
}
}
↓
[3] File Thunder validates:
- MIME type in allowed list?
- File size within per-upload limit?
- User quota not exceeded? (atomic SQL check)
- Creates DB record: status PENDING
- Generates presigned MinIO URL → nexgate-raw
- Returns { fileId, uploadUrl, expiresIn: 1800 }
↓
[4] Client uploads directly to MinIO
- TUS resumable protocol for large files
- Multipart upload for files > 5MB
- Progress tracked client-side
- Upload starts during caption writing (parallel)
↓
[5] Client confirms: POST /media/confirm { fileId }
(or cleanup job recovers if confirm missed)
↓
[6] Processing pipeline starts
Quota + Duration Enforcement
Size check → at presigned URL generation (before upload):
Atomic SQL:
UPDATE user_storage_quota
SET used_bytes = used_bytes + fileSize
WHERE user_id = ?
AND (used_bytes + fileSize) <= quota_bytes
RETURNING used_bytes
0 rows affected → quota exceeded → reject 403
1 row affected → quota reserved → proceed
Duration check → after upload (FFprobe analysis):
Client sends estimated duration in clientMeta (hint only)
FFprobe confirms actual duration after upload
If actual duration > plan limit:
→ delete file from nexgate-raw
→ reject with 403: { error: "DURATION_EXCEEDED", plan: "FREE", maxDuration: 300 }
→ release quota reservation
→ notify user: "Upgrade plan to upload longer videos"
Why duration checked after upload (not before):
Client-reported duration = not trusted
= could be manipulated
FFprobe = ground truth
= server side, reliable
= only way to confirm actual duration
Per-Upload Size + Duration Limits
Plan
Image
Short Video (social)
Long Video
Digital Product Video
Duration
FREE
20MB
200MB
❌ Not allowed
❌ Not allowed
Max 5 min
PRO
20MB
500MB
2GB
5GB per file
Max 60 min
BUSINESS
20MB
2GB
5GB
20GB per file
Unlimited
Why duration matters as much as size:
Same 200MB could be:
3 minute 1080p reel → acceptable ✅
45 minute long video → not acceptable for FREE ❌
Size check alone = not enough
Duration check = required alongside size
Enforcement:
Size → checked at upload request (before presigned URL)
Duration → checked AFTER upload (FFprobe analysis)
client also sends estimated duration in clientMeta
If duration exceeds plan limit after FFprobe:
→ delete from nexgate-raw
→ reject processing
→ notify user: "Upgrade to upload longer videos"
→ refund quota reservation
Duration limits by plan:
FREE:
Reels/short video → max 5 minutes
Long video → not allowed
PRO:
Reels/short video → max 5 minutes (treated as short)
Long video → max 60 minutes
BUSINESS:
All video → unlimited duration
6. Processing Pipelines
6.1 Image Pipeline
Upload confirmed
↓
RabbitMQ: SCAN job
↓
ClamAV scan → VIRUS: quarantine + notify | CLEAN: continue
↓
SHA-256 hash check (deduplication)
Exists + clean → skip processing, reuse variants
New → continue
↓
RabbitMQ: PROCESS job → Image Worker
↓
[1] Auto-orient (apply EXIF rotation before stripping)
[2] Extract useful EXIF (orientation only)
[3] Strip ALL EXIF (privacy — remove GPS, device info)
[4] AI moderation check (NSFW detection)
UNSAFE → quarantine | SAFE → continue
[5] Extract dominant color (resize to 1×1 pixel)
[6] Generate LQIP (resize to 10×10, base64 encode)
[7] Generate BlurHash (blurhash-java library)
[8] Generate variants by content type:
Profile → 400px, 150px, 50px (WebP)
Post → 1600px, 800px, 300px (WebP)
Product → 1000px, 500px, 200px (WebP, 1:1 square)
Event → 1200×630px, 800×420px (WebP)
Story → 1080×1920px (WebP)
[9] Convert all variants → WebP
[10] Generate OG image (1200×630, WebP):
- og_clean.webp (no play button)
- For non-16:9 → blur pad background to fill 16:9
[11] Store all variants → nexgate-public
[12] Delete original from nexgate-raw
[13] Update DB: status READY, populate variants JSONB
[14] Publish to Kafka: FILE_READY event
[15] Publish to RabbitMQ: MEDIA_READY event → Main Backend
Variant sizes per content type:
Content
Variants
Profile picture
400px, 150px, 50px
Post image
1600px (large), 800px (medium), 300px (thumb)
Product image
1000px, 500px, 200px (always 1:1 square)
Event banner
1200×630, 800×420, 400×210
Category
400px wide
6.2 Short Video Pipeline (MP4)
Threshold: Duration < 3 minutes → MP4
Upload confirmed
↓
RabbitMQ: SCAN job
↓
ClamAV scan → VIRUS: quarantine | CLEAN: continue
↓
SHA-256 hash check (deduplication)
↓
FFprobe analysis:
- resolution, bitrate, fps
- codec, duration, aspectRatio
- hasAudio, qualityScore
↓
Adaptive transcode decision:
- Never upscale (never exceed input resolution)
- Never inflate (if output > input size, use original)
- Only generate eligible variants
↓
RabbitMQ: PROCESS job → Video Worker
↓
[FAST LANE — runs first for UX]:
Quick 360p transcode → post goes LIVE_PARTIAL
User notified: "Almost ready!"
↓
[FULL PROCESSING — continues async]:
[1] Detect aspect ratio (FFprobe):
9:16 (vertical) → native reel format → no padding needed
16:9 (landscape) → needs blur pad for reel pool
1:1 (square) → needs blur pad for reel pool
4:5 (portrait) → needs blur pad for reel pool
Other → preserve original, blur pad if reel eligible
[2] Transcode CLEAN variants (H.264, faststart):
For NATIVE 9:16 videos:
Transcode directly to 9:16 variants
360p_clean.mp4 (360×640, CRF 28)
720p_clean.mp4 (720×1280, CRF 23)
1080p_clean.mp4 (1080×1920, CRF 21, if eligible)
For NON-9:16 videos (isReelEligible = true):
Apply BLUR PAD to fill 9:16 frame (TikTok/Instagram style)
What blur pad looks like:
┌──────────────┐
│▓▓▓▓▓▓▓▓▓▓▓▓▓│ ← blurred background (scaled up + blurred)
│▓▓▓▓▓▓▓▓▓▓▓▓▓│
│┌────────────┐│
││ ││
││ original ││ ← actual video centered (no cropping)
││ video ││
││ ││
│└────────────┘│
│▓▓▓▓▓▓▓▓▓▓▓▓▓│
│▓▓▓▓▓▓▓▓▓▓▓▓▓│
└──────────────┘
Why blur pad (not black bars, not crop):
Black bars → looks amateur ❌
Crop → destroys content ❌
Blur pad → professional, intentional ✅
TikTok + Instagram do this ✅
No content lost ✅
FFmpeg blur pad command:
ffmpeg -i input.mp4 \
-filter_complex \
"[0]scale=720:1280:force_original_aspect_ratio=increase,
crop=720:1280,
boxblur=20:5[bg];
[0]scale=720:1280:force_original_aspect_ratio=decrease,
pad=720:1280:(ow-iw)/2:(oh-ih)/2[fg];
[bg][fg]overlay=(W-w)/2:(H-h)/2" \
output_9_16_blurpad.mp4
Steps:
bg layer → scale to FILL 9:16, crop excess, apply blur (20px radius)
fg layer → scale to FIT in 9:16, letterbox, center
overlay → sharp video (fg) on top of blurred background (bg)
For NON-9:16 videos (isReelEligible = false):
Keep original aspect ratio
No blur pad
Feed player handles letterboxing client-side
[3] Generate WATERMARKED variants:
FFmpeg overlay — MOVING watermark (TikTok style):
Watermark jumps between corners every 3 seconds:
0-3s → top left (10px, 10px)
3-6s → top right (W-w-10, 10px)
6-9s → bottom right (W-w-10, H-h-10)
9-12s → bottom left (10px, H-h-10)
repeat...
Logo PNG: 80×80px, 60% opacity
Username text: same position as logo (below it)
Based on clientApp field:
NEXGATE_ANDROID / NEXGATE_IOS / NEXGATE_WEB → "NexGate"
NEXGATE_LITE → "NexGate Lite"
Why moving watermark:
Static corner = easy to crop out ❌
Moving = appears in all corners = impossible to crop ✅
TikTok confirmed uses this strategy
FFmpeg command (moving watermark):
overlay=
x='if(lt(mod(t,12),3), 10,
if(lt(mod(t,12),6), W-w-10,
if(lt(mod(t,12),9), W-w-10, 10)))':
y='if(lt(mod(t,12),3), 10,
if(lt(mod(t,12),6), 10,
if(lt(mod(t,12),9), H-h-10, H-h-10)))'
Variants generated:
360p_watermarked.mp4
720p_watermarked.mp4
1080p_watermarked.mp4
[4] Generate extras:
thumbnail.webp → best frame (brightness + sharpness scored)
thumbnail LQIP → 10×10 base64
thumbnail BlurHash → blurhash-java
thumbnail dominantColor → hex
preview_3s.mp4 → first 3 seconds, 360p, muted, watermarked
og_clean.webp → 1200×630, clean thumbnail
og_play.webp → 1200×630, NexGate play button burned in
og_preview.mp4 → 360p, watermarked, permanent CDN URL
[5] Generate Fragmented MP4 (fMP4):
ffmpeg flags: frag_keyframe+empty_moov+faststart
1-second fragment intervals (keyframe every 30 frames at 30fps)
Enables byte-range requests for progressive streaming
[6] Extract audio for Rec Engine:
ffmpeg -vn -acodec pcm_s16le -ar 16000 -ac 1 audio.wav
Store → nexgate-private/audio/{fileId}.wav
(Rec Engine fetches, transcribes, deletes)
[7] Store all variants → nexgate-public
[8] Delete original from nexgate-raw
[9] Update DB:
status: READY
isReelEligible: true (duration < 3min)
streamingFormat: MP4
variants: JSONB with all paths
aspectRatio, qualityScore, duration
[10] Publish to Kafka: FILE_READY (with audioRef)
[11] Publish to RabbitMQ: MEDIA_READY → Main Backend
Main Backend:
→ update post status LIVE
→ index into reel pool
→ trigger fan-out to followers
Reel pool eligibility:
duration < 3 minutes → isReelEligible: true → indexed in reel pool
duration ≥ 3 minutes → isReelEligible: false → feed only
MP4 variant resolutions by aspect ratio:
Aspect
360p
540p
720p
1080p
9:16 (vertical)
360×640
540×960
720×1280
1080×1920
16:9 (landscape)
640×360
—
1280×720
1920×1080
1:1 (square)
360×360
—
720×720
1080×1080
Why 1080p is max (not 4K):
All major platforms compress 4K down to 1080p on delivery anyway:
TikTok → accepts 4K, serves 1080p max
Instagram → accepts 4K, serves 1080p max
YouTube Shorts → 1080×1920 optimal
Accepting 4K but processing/serving max 1080p = correct strategy.
Above 1080p = no visible benefit for social content, just wasted storage.
Platform max resolution reference (2026):
Platform
Max Resolution
Aspect
Max File Size
TikTok
1080×1920
9:16
287MB mobile / 4GB web
Instagram Reels
1080×1920
9:16
4GB
YouTube Shorts
1080×1920
9:16
—
Facebook Reels
1080×1920
9:16
1GB
Twitter/X
1280×1024
any
512MB
Safe zone for 9:16 reels (1080×1920):
┌──────────────────┐
│░░░░ top 120px ░░░│ ← UI bar — avoid placing content here
│ │
│ SAFE ZONE │ ← faces, text, important content here
│ 860×1550px │
│ │
│░ right 180px ░░░░│ ← action buttons (like, comment, share)
│░░ bottom 250px ░░│ ← caption area
└──────────────────┘
Watermark placement must respect safe zones.
6.3 Long Video Pipeline (HLS)
Threshold: Duration ≥ 3 minutes → HLS
Same as short video EXCEPT:
Format → HLS adaptive streaming (not MP4)
No fast lane (too long, post stays PROCESSING)
Post goes live only when fully processed
Transcode to HLS:
hls/360p/ → segments + 360p.m3u8
hls/720p/ → segments + 720p.m3u8
hls/1080p/ → segments + 1080p.m3u8
hls/master.m3u8 → points to all quality manifests
Segment duration: 2 seconds
Codec: H.264, AAC audio
Container: MPEG-TS (.ts chunks)
Still generated:
preview_3s.mp4 ← for feed inline autoplay
thumbnail.webp
LQIP, BlurHash, dominantColor
og_clean.webp, og_play.webp, og_preview.mp4
isReelEligible: false
streamingFormat: HLS
HLS Cache-Control:
.ts chunks: Cache-Control: public, max-age=31536000
master.m3u8: Cache-Control: public, max-age=31536000
6.4 Digital Products Pipeline
Upload Processing
Seller uploads product file
↓
Validate:
- File type allowed?
- Seller quota OK?
- Product exists + owned by seller?
↓
ClamAV scan (DOUBLE scan — extra safety)
↓
SHA-256 checksum generated + stored
↓
Light processing by file type:
PDF:
Extract page 1 → preview image (72 DPI)
Watermark "PREVIEW" across image
Store: preview/preview.webp
Video course (MP4):
Extract first 2 minutes
Transcode to 480p
Watermark burned in
Store: preview/preview.mp4
Audio (MP3/WAV/FLAC):
Extract first 30 seconds
Lower bitrate (96kbps)
Store: preview/preview.mp3
3D Models (GLB/OBJ/FBX):
Render thumbnail from multiple angles
Store: preview/thumb_*.webp
Images (stock art, PNG):
Resize to low resolution (600px max)
"PREVIEW" watermark burned in
Store: preview/preview.webp
Software/ZIP:
No preview file
Cover image only
↓
Store → nexgate-digital/products/{shopId}/{productId}/{fileId}/
original/ ← actual product (never exposed publicly)
preview/ ← shown to everyone
cover/ ← product listing thumbnail
↓
Update DB: READY
Publish to RabbitMQ: DIGITAL_PRODUCT_READY → Main Backend
Download Flow (Buyer)
Buyer clicks Download
↓
POST /digital/download { productId, orderId, fileId }
↓
File Thunder validates:
[1] orderId exists in DB?
[2] orderId belongs to requesting user?
[3] Order status = PAID?
[4] download_count < download_limit?
[5] Current time < expires_at?
[6] User account in good standing?
↓
All pass:
Generate single-use signed URL:
10-minute TTL
Order bound
Device bound
Encrypted token
Stored in Redis (single-use enforcement)
↓
Log download event:
orderId, buyerId, timestamp
IP address, deviceId, country
↓
Increment download_count
↓
Return signed URL to client
Client downloads (chunked, resumable via byte-range)
↓
URL marked as used in Redis → 403 on reuse
No forensic watermarking for now. Planned for future phase when piracy becomes a real problem.
6.5 DM Attachments Pipeline
Same core pipeline with different rules:
Social Posts
DM Attachments
Bucket
nexgate-public
nexgate-private
CDN
✅ Cloudflare
❌ No CDN
Access check
Public/followers
Conversation membership
Variants
Full (all sizes)
Thumb + original only
Watermark
✅ On download
❌ Never
Reel pool
✅ If eligible
❌ Never
Virus scan
✅ Yes
✅ Yes
URL type
Permanent CDN
Signed 5min TTL
Processing
Full
Minimal
Access control for DM files:
Client requests DM file URL
↓
File Thunder checks:
Is requesting user a member of this conversation?
YES → generate signed URL (5min TTL)
NO → 403 Forbidden
7. Watermarking Strategy
When Applied
Processing time (server-side) — NOT at serve time, NOT client-side
Done ONCE when video is processed
Stored as separate _watermarked variant
When Served
Scenario
Variant Served
Watching on NexGate (in-app/web)
Clean variant
Downloading via NexGate download button
Watermarked variant
Accessing via CDN URL directly
Clean variant (unavoidable)
Sharing via savefromnet-style tools
Clean variant (acceptable)
Watermark Position — Moving (TikTok Style)
Jumps between 4 corners every 3 seconds:
0-3s → top left
3-6s → top right
6-9s → bottom right
9-12s → bottom left
repeat for full video duration
Logo: 80×80px, 60% opacity
Username: directly below logo, same position
Why moving:
Static corner → easy to crop out ❌
Moving → appears everywhere → impossible to crop ✅
TikTok confirmed uses this strategy
App-based Watermark
clientApp = NEXGATE_ANDROID → "NexGate" logo
clientApp = NEXGATE_IOS → "NexGate" logo
clientApp = NEXGATE_WEB → "NexGate" logo
clientApp = NEXGATE_LITE → "NexGate Lite" logo
Determined at upload time. Burned at processing time. Never changes.
Pre-Watermarked Content (Uploaded from Other Platforms)
The Problem:
User downloads TikTok video (has TikTok watermark)
Re-uploads to NexGate
NexGate adds NexGate watermark on top
= two watermarks visible = bad UX 😂
What major platforms do:
Platform
Approach
Instagram
Algorithm suppresses watermarked content from discovery
TikTok
No detection, just adds own watermark on top
YouTube Shorts
Suppresses competitor watermarked content
Instagram head Adam Mosseri confirmed: videos with competitor watermarks receive significantly reduced algorithmic reach.
NexGate approach by phase:
Phase 1 (launch — current):
No watermark detection at all
NexGate watermark burned on top of everything
Pre-existing watermarks = overwritten/overlaid
= zero complexity ✅
= same pipeline for all videos ✅
= double watermark acceptable at this stage ✅
Phase 2 (growth):
Client-side warning before upload:
"This video appears to have a watermark
from another platform. Remove it for
better reach on NexGate."
User can proceed anyway
= user educated, NexGate protected ✅
Phase 3 (scale):
Server-side AI detection:
Scan first frame for known platform logos
Flag: hasExternalWatermark: true
Suppress in reel pool (isReelEligible: false)
Show post-level label: "Cross-posted content"
= same approach as Instagram ✅
Note: NexGate never BLOCKS pre-watermarked uploads. Only suppresses reach in discovery/recommendations. Content still visible to uploader's followers.
Username Change Policy
Old videos keep old username watermark (same as TikTok). Re-watermarking = future premium feature.
8. File Protection Layers
Layer
Method
Stops
1
Content-Disposition: inline header
Casual right-click downloaders
2
Dynamic URL via JS (never in HTML)
HTML source inspection
3
Signed expiring URLs (private content)
URL sharing between users
4
Session + device binding
Cross-device URL reuse
5
Disable right-click on player (frontend)
Non-technical users
6
Watermark burned on download
Casual sharing without credit
7
HLS chunking for video
Download and reassembly
Philosophy: Goal is not impossible to download. Goal is not worth the effort for 99% of users + traceable if they do.
9. URL Strategy
Public Content → Permanent CDN URLs
Thumbnail:
https://media.nexgate.com/posts/usr_123/vid_789/thumb.webp
Video (public, clean):
https://media.nexgate.com/posts/usr_123/vid_789/720p_clean.mp4
OG preview (permanent):
https://media.nexgate.com/og/vid_789/og_preview.mp4
Profile picture:
https://media.nexgate.com/profiles/usr_123/avatar_400.webp
No signing, no expiry
Cache-Control: public, max-age=31536000
Cloudflare CDN caches globally
Private Content → Signed Expiring URLs
DM attachment:
https://media.nexgate.com/private/...?
x-expires=1716305400
&x-sig=hmac_signature
&x-uid=usr_xyz
&x-sid=sess_abc
Digital product download:
https://media.nexgate.com/digital/...?
x-expires=1716305400
&x-sig=hmac_signature
&x-order=enc_orderId
&x-device=device_abc
Signed URL Params Explained
Param
Purpose
Without It
x-expires
URL dies after TTL
Permanent link = unrevokable
x-sig
HMAC tamper protection
Anyone can forge expiry
x-uid
Bound to user session
URL shareable between users
x-sid
Bound to session
Works after logout
Who Generates vs Validates
Main Backend / File Thunder → generates signed URL (knows secret key)
Cloudflare CDN → validates signature at edge (configured with same secret)
MinIO → never exposed to clients
TTL by Content Type
Content
TTL
Stream URL (video)
10 min
DM attachment
5 min
Digital product download
10 min (single-use)
Public CDN content
Permanent
OG preview video
Permanent
10. OG (Open Graph) Strategy
For Video Posts
For Image Posts
Play Button Behavior
Platform
Play Button Source
WhatsApp
Added by WhatsApp (reads og:type)
Telegram
Added by Telegram
Twitter/X
Added by Twitter
SMS/Email
Must burn into og:image (og_play.webp)
Unknown apps
Must burn into og:image (og_play.webp)
NexGate serves og_clean.webp to known platforms, og_play.webp to unknown platforms (detected via crawler User-Agent).
OG Video URL — Why Permanent
WhatsApp/Telegram crawl on share, user opens hours later
Signed URL expired = "Error loading video"
Solution: permanent og_preview.mp4 in CDN (360p, watermarked, small)
Access control = server-side (private posts return 403)
11. Compression & Transcoding
Client-Side Intelligent Compression
Before upload, client analyzes:
Inputs:
resolution, bitrate, codec
network type (WiFi/4G/3G/2G)
device tier (high/mid/low)
Decision:
Already optimized + good network → upload direct
Over-bitrated → compress
Slow network → compress regardless
Target (if compressing):
Max 1080p
CRF 18 (light — preserve quality for server)
H.264
Never upscale on client either.
Server-Side Adaptive Transcoding
After upload, FFprobe analyzes:
Extract: width, height, bitrate, fps, codec, duration
Decision tree:
height ≥ 1080 → generate 1080p, 720p, 360p
height ≥ 720 → generate 720p, 360p
height ≥ 480 → generate 480p, 360p
height < 480 → keep original resolution
Rules:
NEVER upscale (never exceed input resolution)
NEVER inflate (if output > input size → use original as-is: -c:v copy)
CRF adapts to input quality
CRF Values (H.264)
Variant
CRF
Use
1080p
21
High quality
720p
23
Standard
480p
25
Medium
360p
28
Low / slow networks
OG preview
30
Very small file
Video Codec Strategy
Now
Future
H.264 (universal support)
AV1 (50% smaller, open source)
AAC audio 128kbps
Opus audio
MP4 container
CMAF container
Image Format Strategy
Input
Output
JPEG
WebP (quality 80%)
PNG
WebP (quality 85%)
HEIC
WebP (quality 80%)
GIF
WebP animated + MP4 loop
Critical FFmpeg Flags
-movflags +faststart → metadata at start of file
= video starts playing before fully downloaded
= essential for web streaming
-movflags frag_keyframe+empty_moov+faststart → fragmented MP4
= byte-range requests work
= reel prefetch works
Generation Loss (Double Compression)
Client compresses lightly (CRF 18) → preserves quality
Server compresses per variant (CRF 23-28) → optimizes delivery
Two passes but first is near-lossless → acceptable tradeoff
Benefit: fast uploads on Tanzania mobile networks
12. BlurHash, LQIP & Dominant Color
All three generated for every image and video thumbnail:
Dominant Color
ImageMagick resize to 1×1 pixel
→ average color of entire image
→ stored as hex: "#2D7BC8"
→ shown instantly as CSS background (0ms, 0 bytes extra)
BlurHash
Algorithm: DCT-based image encoding
Output: ~30 character string "LGF5]+Yk^6#M@-5c,1J5@[or[Q6."
Library: blurhash-java
Components: 4x3 (width x height)
→ decoded client-side to blurry placeholder
→ no network request
→ shows shape/composition of image
LQIP (Low Quality Image Placeholder)
ImageMagick resize to 10×10px
Quality: 20%
Convert to base64 string
→ embedded in feed JSON
→ client scales up (naturally blurry)
→ more accurate than BlurHash
→ no network request
Progressive Loading Stages
Stage 0 (instant, 0ms): dominantColor CSS background
Stage 1 (instant, 0ms): BlurHash decoded → blurry shape
Stage 2 (instant, 0ms): LQIP scaled up → accurate blur
Stage 3 (CDN, 50-200ms): thumb.webp loads → sharp thumbnail
Stage 4 (on demand): large.webp loads → full quality (on tap)
13. Deduplication
Hash-Based (Exact Match)
File uploaded
↓
Compute SHA-256 of raw file
↓
Check file_hashes table:
HASH EXISTS + was CLEAN → skip processing entirely
create reference to existing variants
increment reference_count
HASH NOT FOUND → process normally
save hash + object_key to file_hashes
Benefits
Same viral video uploaded 1000x = stored once
Known clean files skip ClamAV scan (hash cache)
Known virus files rejected instantly (hash cache)
Storage savings up to 99% for viral content
Reference Counting (Deletion)
User deletes file
↓
Decrement reference_count
reference_count > 0 → keep file (others reference it)
reference_count = 0 → delete from MinIO
Near-Duplicate Detection
Exact SHA-256 only (now)
Perceptual hashing (pHash) = future feature for copyright detection
14. Quota Management
Enforcement Point
At presigned URL generation — never after upload.
Request upload
↓
Atomic check:
UPDATE user_storage_quota
SET used_bytes = used_bytes + fileSize
WHERE user_id = ?
AND (used_bytes + fileSize) <= quota_bytes
RETURNING used_bytes
0 rows affected → quota exceeded → reject with 403
1 row affected → quota reserved → proceed
What Counts Against Quota
Content
Charged
Social posts
Original upload size only
Digital products
Actual stored bytes (all variants)
Plan Limits
Plan
Storage
Max Image
Max Video
FREE
1GB
20MB
200MB, 5min
PRO
20GB
20MB
1GB, 60min
BUSINESS
100GB
20MB
2GB, unlimited
Warning System
80% used → email + in-app notification
95% used → in-app banner
100% used → uploads blocked, existing content untouched
Quota Release
On ABANDONED: immediate release (file never stored)
On soft delete: NOT released (file still exists)
On hard delete (30 days after soft): quota freed
Quota Cache (Redis)
GET quota:usr_123 → used_bytes (cached)
INCRBY quota:usr_123 fileSize → atomic increment
Sync to PostgreSQL every 5 minutes
15. Storage Lifecycle
nexgate-raw
Auto-delete: 24 hours (MinIO lifecycle policy)
Incomplete multiparts: aborted after 24 hours
nexgate-public (Social Content)
All variants: keep FOREVER (never delete)
Move to cold storage tier based on access frequency:
Hot (< 30 days or high views) → NVMe SSD
Warm (30-365 days, moderate) → HDD
Cold (1yr+, rare access) → Cold object tier
CDN serves from cache after first request
MinIO barely hit after content warms up
Smart lifecycle (access-based, not just age):
< 100 views/day for 30 days → move to warm
< 10 views/day for 90 days → move to cold
Viral spike on old content → auto-promote back to hot
nexgate-private
DM attachments: 1 year
Audio temp files (for Rec Engine): 24 hours OR deleted by Rec Engine after transcription
KYC documents: per legal requirement
nexgate-digital
Original product files: FOREVER (seller's product)
Preview files: FOREVER
Cover images: FOREVER
Deleted Content (Soft Delete)
User deletes post
↓
DB: soft delete (mark deleted, keep record)
↓
30-day grace period:
User can appeal/restore
Legal holds possible
CDN cache still valid (purge takes time)
↓
30 days → hard delete from MinIO
Cloudflare cache purge
DB record updated
Quota freed
16. Abandoned Upload Handling
The Problem
File uploaded to nexgate-raw ✅
Client never sends POST /media/confirm ❌
Reasons: app crash, network drop, user closed app
→ file stuck in raw bucket = wasted storage
Three-Layer Solution
Layer 1 — Client Confirm (Primary fast path)
Client sends POST /media/confirm after upload
→ processing triggered immediately
→ best UX (instant)
Layer 2 — Cleanup Job (Safety net, every 30 min)
// Runs every 30 minutes
// Finds PENDING records > 1 hour old
// Checks if file exists in MinIO
File exists → RECOVER (upload done, confirm missed)
→ update status UPLOADED
→ trigger processing
→ Tanzania network drop scenario handled ✅
File not exists → ABANDONED (user closed app)
→ update status ABANDONED
→ release quota reservation
Layer 3 — Raw Bucket Lifecycle (Final safety)
nexgate-raw auto-delete: 24 hours
Incomplete multiparts aborted: 24 hours
= catches anything missed by layers 1 + 2
Cleanup Job Scaling
Add DB index:
CREATE INDEX idx_media_cleanup
ON media_files (created_at)
WHERE status = 'PENDING'
Process in batches of 100
Parallel MinIO checks (parallelStream)
Redis distributed lock (prevent overlap):
SET lock:cleanup_job "running" NX EX 2100
fileId in Object Key (Enables Recovery)
objectKey = uploads/{userId}/{fileId}/original.mp4
↑
fileId encoded in path
→ cleanup job extracts fileId from objectKey
→ can recover without client confirm
17. Progress Tracking (SSE)
Why SSE (not polling, not WebSocket)
Polling
SSE
WebSocket
Direction
Both
Server→Client only
Both
Connection
New each time
Persistent
Persistent
Auto-reconnect
Manual
Built-in ✅
Manual
Complexity
Medium
Simple
Complex
Tanzania networks
Wasteful
Resilient ✅
Complex
Upload progress = one direction only → SSE is perfect fit.
Status Flow
Redis key: upload_status:{fileId}
PENDING → "Upload requested"
UPLOADING → "Uploading..."
UPLOADED → "File received"
SCANNING → "Checking file..."
PROCESSING → "Processing video..."
LIVE_PARTIAL → "Almost ready!" + { available: ["360p"] }
READY → "Your post is live!" + { liveUrl: "..." }
FAILED → "Something went wrong"
QUARANTINED → "File failed security check"
ABANDONED → "Upload timed out"
Spring Boot SSE
@GetMapping(
value = "/media/{fileId}/progress",
produces = MediaType.TEXT_EVENT_STREAM_VALUE
)
SseEmitter trackProgress(@PathVariable String fileId) {
SseEmitter emitter = new SseEmitter(7_200_000L); // 2hr
emitterRegistry.register(fileId, emitter);
// Send current status immediately
emitter.send(redis.get("upload_status:" + fileId));
return emitter;
}
Multi-Instance Scaling
File Thunder (Server A) updates Redis
File Thunder (Server B) has SSE connection
→ Redis Pub/Sub bridges them:
PUBLISH media_status:{fileId} { status: "READY" }
Server B receives → finds local emitter → pushes to client
18. Communication Architecture
Rule: No Direct HTTP Between Services
❌ No REST API calls between File Thunder and Main Backend
❌ No webhooks
✅ RabbitMQ for internal operations
✅ Kafka for event streaming to external services
RabbitMQ (File Thunder ↔ Main Backend)
Exchange: nexgate.media (topic exchange)
Routing Key
Direction
Consumer
media.upload.request
MB → FT
File Thunder
media.upload.url.ready
FT → MB
Main Backend
media.ready
FT → MB
Main Backend
media.live.partial
FT → MB
Main Backend
media.failed
FT → MB
Main Backend
media.quarantined
FT → MB
Main Backend
digital.product.ready
FT → MB
Main Backend
Example MEDIA_READY event:
{
"event": "MEDIA_READY",
"fileId": "vid_789",
"ownerId": "usr_123",
"type": "SHORT_VIDEO",
"status": "READY",
"variants": {
"360p_clean": "posts/usr_123/vid_789/360p_clean.mp4",
"720p_clean": "posts/usr_123/vid_789/720p_clean.mp4",
"thumbnail": "posts/usr_123/vid_789/thumb.webp",
"blurhash": "LGF5]+Yk^6#M@-5c,1J5@[or[Q6.",
"lqip": "data:image/webp;base64,...",
"dominantColor": "#1a1a2e",
"preview_3s": "posts/usr_123/vid_789/preview_3s.mp4",
"og_clean": "posts/usr_123/vid_789/og_clean.webp",
"og_preview_video": "og/vid_789/og_preview.mp4"
},
"isReelEligible": true,
"streamingFormat": "MP4",
"duration": 28,
"aspectRatio": "9:16"
}
Kafka (File Thunder → Rec Engine + Analytics)
Topic: content.new
{
"event": "FILE_READY",
"fileId": "vid_789",
"type": "SHORT_VIDEO",
"ownerId": "usr_123",
"media": {
"duration": 28,
"aspectRatio": "9:16",
"qualityScore": 75,
"isReelEligible": true,
"hasAudio": true,
"streamingFormat": "MP4"
},
"userProvided": {
"caption": "Check this Spring Boot setup",
"hashtags": ["#tech", "#java", "#springboot"],
"location": null
},
"audioRef": {
"bucket": "nexgate-private",
"objectKey": "audio/vid_789.wav",
"expiresAt": "2026-05-23T10:00:00Z"
},
"thumbnailUrl": "https://media.nexgate.com/...",
"processedAt": "2026-05-22T10:00:00Z"
}
Rec Engine consumes this, fetches audio.wav, runs Whisper transcription, classifies content, deletes audio.wav.
19. CDN Strategy
How It Works
First request (cache miss):
Client → Cloudflare edge → MinIO → Cloudflare caches → serves
Every subsequent request (cache hit):
Client → Cloudflare edge → serves from cache
MinIO never touched
Cache-Control Headers
Content
Cache-Control
TTL
Post thumbnails
public, max-age=31536000
1 year
Video MP4 variants
public, max-age=31536000
1 year
HLS .ts chunks
public, max-age=31536000
1 year
HLS .m3u8 manifests
public, max-age=31536000
1 year
OG images
public, max-age=31536000
1 year
OG preview video
public, max-age=31536000
1 year
Profile pictures
public, max-age=86400
1 day
API responses
no-store
Never cached
Private content
private, no-store
Never cached
Cloudflare Phases
Phase
Setup
Cost
Now (launch)
Cloudflare Free
$0
Growth
Cloudflare Pro
$20/month
Scale
Migrate to Cloudflare R2 (zero egress)
Pay storage only
MinIO — Never Publicly Exposed
Public access: BLOCKED
Only Cloudflare IPs can reach MinIO
Client → media.nexgate.com (Cloudflare) → MinIO (internal)
MinIO URL never seen by clients
20. Shareable Links
Share URL Format
Generated client-side (no server call):
nexgate.com/reels/reel_abc123?ref=whatsapp
ref values:
whatsapp, telegram, twitter, sms, copy, email
Follow Prompt (On Link Click)
Sarah taps share link
→ NexGate loads post
→ Server looks up post owner from DB (postId → authorId)
→ Shows follow prompt for correct creator
→ No uid in URL (no manipulation possible)
→ No token needed (public content)
Analytics from ref Parameter
Track per share:
Which platform shared from
Click-through rate per platform
Follow conversion rate per platform
New signups from shares
Watch completion rate from shares
21. Technology Stack
Component
Technology
Language
Java 21 (Spring Boot 3.x)
Video processing
FFmpeg (global install) via Jaffree
Image processing
ImageMagick (global install) via IM4Java
BlurHash
blurhash-java library
Virus scanning
ClamAV (Docker container, TCP :3310)
Object storage
MinIO (S3 compatible)
CDN
Cloudflare
Message queue (ops)
RabbitMQ
Event streaming
Kafka
Cache / progress
Redis
Database
PostgreSQL
Resumable uploads
TUS protocol
Secrets
HashiCorp Vault (vault.qbitspark.com)
Container
Docker + Docker Compose
Progress delivery
SSE (Server-Sent Events)
Why FFmpeg + ImageMagick Global Install
Both are CLI tools (not daemons)
Called on demand via ProcessBuilder
Jaffree/IM4Java call /usr/bin/ffmpeg and /usr/bin/convert
Same filesystem = works perfectly
No network overhead
Simpler than separate containers
ClamAV = separate Docker container because:
Runs as daemon (long-running process)
Official Docker image exists
Connects via TCP = natural for Docker
Auto-updates virus definitions
22. Database Schema
-- Core media file record
media_files
file_id UUID PK
owner_id UUID -- accountId
directory ENUM -- PROFILE, POSTS, STORIES, EVENTS, SHOPS, MESSAGES, PRODUCTS
original_name TEXT
object_key TEXT -- raw upload path in nexgate-raw
mime_type TEXT
file_size BIGINT
status ENUM -- PENDING, UPLOADING, UPLOADED, SCANNING, PROCESSING,
-- LIVE_PARTIAL, READY, FAILED, QUARANTINED, ABANDONED, EXPIRED
variants JSONB -- all processed variant paths
metadata JSONB -- dimensions, duration, fps, codec
scan_result TEXT -- "clean" or virus name
hash TEXT -- SHA-256 for deduplication
is_reel_eligible BOOLEAN
streaming_format ENUM -- MP4, HLS
created_at TIMESTAMPTZ
ready_at TIMESTAMPTZ
-- Content signals for Rec Engine
media_content_signals
file_id UUID FK
has_audio BOOLEAN
audio_ref TEXT -- nexgate-private/audio/{fileId}.wav
dominant_color TEXT -- hex color
blurhash TEXT
lqip TEXT -- base64
aspect_ratio TEXT -- "9:16", "16:9", "1:1", "4:5"
quality_score INT -- 0-100
width INT
height INT
duration_seconds DECIMAL
-- User storage quota
user_storage_quota
user_id UUID PK
plan ENUM -- FREE, PRO, BUSINESS
quota_bytes BIGINT
used_bytes BIGINT
file_count INT
updated_at TIMESTAMPTZ
-- Per bucket breakdown
user_storage_breakdown
user_id UUID
bucket TEXT
used_bytes BIGINT
file_count INT
updated_at TIMESTAMPTZ
-- Deduplication hash index
file_hashes
hash TEXT PK -- SHA-256
object_key TEXT -- canonical storage path
file_size BIGINT
mime_type TEXT
reference_count INT
created_at TIMESTAMPTZ
-- Variant storage class tracking
media_variants
file_id UUID FK
quality TEXT -- "360p", "720p", "1080p", "thumb", etc
status ENUM -- HOT, WARM, COLD, FROZEN
storage_class TEXT
object_key TEXT
file_size BIGINT
last_accessed TIMESTAMPTZ
access_count INT
-- Digital products
digital_product_files
file_id UUID PK
product_id UUID FK
file_type ENUM -- MAIN, PREVIEW, COVER
original_name TEXT
mime_type TEXT
file_size BIGINT
checksum TEXT -- SHA-256
object_key TEXT
status ENUM -- PROCESSING, READY, FAILED
created_at TIMESTAMPTZ
-- Digital orders
digital_orders
order_id UUID PK
product_id UUID FK
buyer_id UUID FK
seller_id UUID FK
amount_paid DECIMAL
currency TEXT
status ENUM -- PAID, REFUNDED, DISPUTED
download_count INT DEFAULT 0
download_limit INT
expires_at TIMESTAMPTZ
purchased_at TIMESTAMPTZ
-- Download audit log
digital_download_logs
log_id UUID PK
order_id UUID FK
buyer_id UUID FK
file_id UUID FK
downloaded_at TIMESTAMPTZ
ip_address TEXT
device_id TEXT
user_agent TEXT
country TEXT
download_number INT -- which download (1st, 2nd, etc)
Critical indexes:
-- Cleanup job performance
CREATE INDEX idx_media_cleanup
ON media_files (created_at)
WHERE status = 'PENDING';
-- Hash lookup for deduplication
CREATE UNIQUE INDEX idx_file_hashes_hash
ON file_hashes (hash);
-- Owner lookup
CREATE INDEX idx_media_files_owner
ON media_files (owner_id, status, created_at DESC);
23. Docker & Infrastructure
File Thunder Dockerfile
FROM eclipse-temurin:21-jdk-alpine
# Install FFmpeg (global, called by Jaffree)
RUN apk add --no-cache ffmpeg
# Install ImageMagick with WebP support (called by IM4Java)
RUN apk add --no-cache \
imagemagick \
imagemagick-webp \
imagemagick-heic
# Copy Spring Boot jar
COPY target/file-thunder.jar app.jar
ENTRYPOINT ["java", "-jar", "/app.jar"]
Docker Compose (Relevant Services)
file-thunder:
build: ./file-thunder
ports: ["8081:8081"]
environment:
SPRING_DATASOURCE_URL: jdbc:postgresql://postgres:5432/nexgate
SPRING_REDIS_HOST: redis
RABBITMQ_HOST: rabbitmq
KAFKA_BOOTSTRAP_SERVERS: kafka:9092
MINIO_ENDPOINT: http://minio:9000
CLAMAV_HOST: clamav
CLAMAV_PORT: 3310
VAULT_ADDR: https://vault.qbitspark.com
depends_on: [postgres, redis, rabbitmq, kafka, minio, clamav]
clamav:
image: clamav/clamav:stable
ports: ["3310:3310"]
volumes:
- clamav_data:/var/lib/clamav
# Auto-updates virus definitions via freshclam
VPS Resource Allocation
File Thunder: 4 cores, 16GB RAM (FFmpeg is CPU intensive)
ClamAV: 1 core, 2GB RAM
24. What File Thunder Does NOT Do
❌ ML / AI content classification
❌ Speech-to-text (Whisper) — that's Rec Engine
❌ Image visual analysis (CLIP) — that's Rec Engine
❌ Feed computation or ranking
❌ Social graph operations (follow/unfollow)
❌ Payment processing
❌ User authentication / authorization
❌ Push notifications
❌ Search indexing
❌ Business logic (post creation, comments, likes)
❌ Fan-out to followers
❌ Recommendation logic
File Thunder extracts audio.wav and stores temporarily. The Rec Engine fetches it, runs Whisper, classifies content, and deletes the audio file. File Thunder never knows what's in the audio.
Summary — File Thunder in One Sentence
File Thunder receives raw files, validates and secures them via ClamAV, processes them into optimized delivery variants using FFmpeg and ImageMagick, stores everything in MinIO across four isolated buckets, serves public content efficiently via Cloudflare CDN, tracks progress via SSE and Redis, communicates with the Main Backend via RabbitMQ, and publishes content signals to Kafka for downstream services — without ever performing any business logic, ML, or social operations.
File Thunder Architecture Guide v1.0 — NexGate / QBIT SPARK
Document compiled: May 2026
NextGate Media Delivery — Redesign
Target: a safe, scalable media layer for the feed, modelled on patterns observed in
TikTok's delivery (signed expiring URLs, CDN edge, variant matrix, batch minting) and
mapped onto NextGate's existing media engine (MinIO + RabbitMQ + workers + Redis).
1. What's wrong today
Observed in the current GET /posts (feed) response.
#
Problem
Why it matters
1
originalUrl is a raw, permanent, unsigned link to s3.nexgate.co
Post visibility is enforced at the API only. A PRIVATE post's bytes stay reachable at a permanent public URL → media bypasses access control entirely.
2
Storage origin exposed, no CDN
Every view hits MinIO directly. High latency from East Africa, full egress cost, zero edge cache.
3
thumbnailUrl: null , placeholderBase64: null
Full-res images (e.g. 1536×2048) served into feed thumbnail slots. Heavy on expensive mobile data. Blurhash field exists but unused.
4
Video = raw .mp4 , width/height/duration = null
No transcode, no poster frame, no ladder, no HLS. No adaptive playback.
5
Per-user path prefix doesn't match owner
e.g. joshdoe's avatar stored under eddiesmith19's stg-acc-{uuid} prefix. Leaks UUIDs and misattributes ownership.
6
dev.s3.nexgate.co leaking into staging data
Environment bleed in a live feed response.
What to KEEP (already good): batch-hydrated feed (one call, all posts + nested quotes +
engagement), permanent shareCode short IDs, graceful deleted-post handling
( unavailable: true ), the rich post data model.
The fix is scoped to the media delivery layer , not the data model.
2. Core principle
Two URLs, two lifespans (same split TikTok uses):
Permanent identity — the post id / shareCode and the storage objectKey . Stored in DB. Shareable forever.
Ephemeral delivery ticket — a signed, expiring CDN URL. Minted on every feed build. Never stored.
Rule: store the objectKey , never store the signed URL.
3. Three delivery tiers
Tier
Content
Auth gate?
URL signing
Expiry
CDN cache
1 — public asset
avatars, thumbnails
no
none (or very long)
n/a / long
aggressive, long TTL
2 — public content, protected delivery
public post images/video
no
yes (short-ish)
hours
careful (see §6)
3 — private content
private/followers posts, paid digital
yes (ownership/visibility check)
yes
minutes
none / private
The only code difference between tier 2 and tier 3 is whether the permission check runs
before minting . Same minting call underneath.
4. Storage layer changes
4.1 Buckets
public — tier 1 assets (avatars, generated thumbnails)
private — tier 2 + tier 3 post media
digital — paid products (tier 3, strictest)
Shared buckets, owner-prefixed object keys , prefix must match the real owner.
Opaque object keys — no user UUID needed in the public URL.
private/{ownerId}/{mediaId}/original.jpg
private/{ownerId}/{mediaId}/thumb_320.webp
private/{ownerId}/{mediaId}/thumb_720.webp
4.2 DB: store keys, not URLs
Persist the storage coordinates and processing state, not a full URL.
// ── MediaEntity (delivery-relevant fields) ──
private String bucket; // "private"
private String objectKey; // "ownerId/mediaId/original.jpg" (opaque, owner-correct)
private MediaType mediaType; // IMAGE | VIDEO
private Integer width;
private Integer height;
private Double duration; // video only
private String placeholder; // blurhash / LQIP — fills the empty placeholderBase64
private MediaStatus status; // PROCESSING | READY | FAILED
// variants: thumb sizes, video rungs, formats — JSONB
private String variantsJson; // {"thumb":["320","720"],"formats":["webp"],"rungs":[...]}
5. Delivery layer changes
5.1 CDN in front of MinIO
Serve everything through media.nexgate.co (Cloudflare) → MinIO origin.
Never return s3.nexgate.co (or dev.s3… ) in an API response again.
5.2 Minting helper (SDK does the crypto)
// ── Presigned URL minting (MinIO SigV4) ──
public String mintUrl(String bucket, String objectKey, int seconds) {
return minioClient.getPresignedObjectUrl(
GetPresignedObjectUrlArgs.builder()
.method(Method.GET)
.bucket(bucket)
.object(objectKey)
.expiry(seconds, TimeUnit.SECONDS) // → X-Amz-Expires + X-Amz-Signature
.build());
// returned host is rewritten to media.nexgate.co at the edge / via MinIO public endpoint
}
5.3 Feed response shape (mint at build time, batch)
Mirror TikTok's batch mint: build the page, mint every media URL inline, ship once.
No per-item round trip from the client.
"media": [{
"id": "014ab4eb-...",
"mediaType": "IMAGE",
"url": "https://media.nexgate.co/.../original.jpg?X-Amz-Expires=3600&X-Amz-Signature=...",
"thumbnailUrl": "https://media.nexgate.co/.../thumb_720.webp?...sig...",
"placeholderBase64": "LEHV6nWB2yk8...", // blurhash, no longer null
"width": 945, "height": 2048,
"expiresIn": 3600
}]
// ── In the feed mapper ──
String url = mintUrl(m.getBucket(), m.getObjectKey(), tier2Seconds); // hours
String thumb = mintUrl(m.getBucket(), thumbKey(m, 720), tier2Seconds);
// tier 3 (private/paid): run accessService.canAccess(user, m) first, use short expiry
6. CDN + signed-URL gotcha
Do not let the CDN cache a per-user signed URL under a key that includes the signature,
or one user's link can be served to another. Options:
cache key = path only (strip query/signature), short edge TTL for tier 2; or
tier 1 (avatars/thumbs) = truly public objects, cache hard; tier 2/3 originals = bypass
shared cache or use very short edge TTL.
Tier 1 assets are the ones you cache aggressively. Tier 3 never shares cache.
7. Variant + transcode pipeline (reuse existing media engine)
On upload (presigned PUT direct to MinIO → enqueue job → workers → webhook back):
ClamAV scan first; reject on hit.
Image worker : generate thumb_320 + thumb_720 (WebP), compute blurhash → placeholder ,
read width/height. Write variants to JSONB, set status READY.
Video worker : extract poster frame (tier-1 thumbnail), read width/height/duration,
transcode to 1–2 MP4 rungs now (e.g. 540p + 720p), HLS later when analytics/scale justify it.
Redis tracks job state; webhook flips MediaEntity.status PROCESSING → READY.
Until a media row is READY, the feed returns the blurhash placeholder + a processing flag
instead of a broken/raw link.
8. Migration sequence (non-breaking)
CDN — stand up media.nexgate.co → MinIO, dual-serve. Stop emitting s3.nexgate.co .
Schema — add bucket / objectKey / status / placeholder / variantsJson . Backfill
objectKey by parsing existing originalUrl ; fix owner prefix + drop dev. rows here.
Mint — change feed to mint from objectKey instead of returning raw originalUrl .
Roll out tier 1 + tier 2 first (public) so nothing visibly breaks.
Backfill variants — run image/video workers over existing media; populate thumbs + blurhash.
Lock the bucket — remove public-read from MinIO; force all access through signed CDN URLs.
At this point the visibility-bypass hole (problem #1) is closed.
Tier 3 — gate private/followers/paid media behind canAccess + short expiry.
Closing problem #1 happens at step 5; everything before it is safe groundwork.
9. Security checklist ("safe" part)
No storage origin host ( s3.nexgate.co , dev.s3… ) in any API response.
No user UUID in public URLs (opaque object keys).
Private/followers post media served tier 3 (auth gate + short signed URL).
MinIO buckets are not public-read once migration completes.
Owner prefix on every key matches the real owner.
Dev/staging/prod storage hosts strictly separated; no cross-env URLs.
File Thunder — Architecture & Design Guide V2
NexGate Media Engine | Version 2.0
v2 integrates the media-security refinements worked out in design review.
Every change from v1 is tagged inline as [v2 FIX] so it can be reviewed individually.
Section 0 lists all of them in one place — read that first.
0. What changed in v2 (read this first)
These are the careful corrections. Each is explained in full in its section.
Signing is three tiers, not two. "Public = unsigned, private = signed" was wrong. Correct model: trivial public assets unsigned; public content (full images/videos) signed even though public ; private/paid signed + auth. → §8
Public content gets signed too. A signed URL is anti-scrape / anti-hotlink, not only access control. TikTok signs even public videos. → §8, §14
The visibility-bypass hole must be closed. In v1, media URLs were permanent and unsigned, so a PRIVATE post's bytes were reachable by anyone with the link — the post was hidden by the API but the file was not. Fixed by locking the bucket (no public-read) and routing everything through signed CDN URLs. → §14
Backend signs, CDN validates, they never talk at sign time. Signing is pure math on (path + expiry + secret) . No file, no CDN contact. CDN fetches from MinIO lazily on first view. → §9
Domain is media.nexgate.co , not .com . v1 mixed both. One domain everywhere. → §10
Object keys must be owner-correct. v1 had one user's avatar stored under another user's prefix. The owner segment in the key must match the real owner. → §3
Store the objectKey , never the signed URL. Signed URLs are minted per request and thrown away. Never persisted in the DB or feed cache. → §13, §22
MP4 and HLS use different token strategies. Short MP4 = one signed URL. Long HLS = dual-token (short manifest + long segments) or prefix-signing, so playback never dies mid-video. → §11
Single-use is for downloads only, never streams. A stream is many requests; single-use would break playback. Streams use short expiry + user-binding instead. → §11, §12
Watermarked file is NEVER in the feed response. Feed = clean playback URLs only. Watermarked download comes from a separate endpoint, minted on tap. → §12, §13
Cache gotcha: never cache a signed object keyed by its signature. A cached object can outlive its stamp; private/paid content must bypass shared cache. → §10, §11
Bot gate is the missing half of anti-scraping. Signed URLs stop permanent theft; the bot gate stops the act of mass-pulling. v1 had no bot gate section. → §14
Open decision parked: NextGate's moat (content vs commerce) drives how hard to stamp public media and how heavy the bot gate needs to be. → §30
1. What is File Thunder?
File Thunder is NexGate's dedicated media processing engine — a standalone Spring Boot
microservice responsible for all file and media operations across the platform.
Core principle: The Main Backend never touches raw files. It delegates all media
operations to File Thunder and acts as a thin client.
Responsibilities: receive uploads, validate (type/size/quota), virus scan, transcode,
generate variants (thumbnails, WebP, HLS, MP4), watermark, store in MinIO, serve via CDN,
mint signed delivery URLs, publish media-ready events.
Not: AI/ML, recommendation, social graph, payments, auth, business logic.
2. The Four Wheels
FILE THUNDER
┌───────────┬───────────┬───────────┬───────────┐
│ FFmpeg │ ImageMagic│ ClamAV │ MinIO │
│ (video) │ (images) │ (security)│ (storage) │
└───────────┴───────────┴───────────┴───────────┘
transcode resize + virus object
HLS, w/mark WebP, blur scan store + CDN
FFmpeg (Jaffree) — transcode, HLS segmenting, thumbnail/best-frame, 3s preview, watermark overlay, blur-pad, fMP4, format conversion, audio extract/normalize.
ImageMagick (IM4Java) — resize variants, format→WebP, EXIF strip (privacy), auto-orient, dominant color, LQIP, OG image, GIF→WebP, PDF→preview. Plus blurhash-java .
ClamAV — virus scan every upload (TCP clamav:3310 ), hash-cache, digital products scanned twice.
MinIO — S3-compatible storage, presigned direct uploads, origin for Cloudflare, multipart + byte-range, lifecycle policies. Never publicly exposed (only Cloudflare + File Thunder reach it).
3. Storage Architecture (MinIO Buckets)
Rule: 4 buckets total. Never one bucket per user.
nexgate-raw/ ← temporary upload landing (auto-delete 24h)
nexgate-public/ ← social content (avatars, posts, stories, events, shops)
nexgate-private/ ← DMs, documents, KYC, temp audio
nexgate-digital/ ← purchase-gated products
Object Key Pattern
{bucket}/{domain}/{ownerId}/{entityId}/{fileId}/{variant}
nexgate-public/posts/usr_123/post_456/file_789/720p_clean.mp4
nexgate-private/messages/conv_abc/file_789/original.jpg
nexgate-digital/products/shop_xyz/prod_123/file_789/original/file.pdf
[v2 FIX] Owner-correct keys. The {ownerId} segment MUST be the real owner of the
file. v1 had cases where one user's avatar lived under a different user's prefix — that
leaks identity and misattributes ownership. The owner in the key = the owner in the DB.
Keys are otherwise opaque (no usernames, no PII).
nexgate-public layout (unchanged from v1)
profiles/{userId}/{fileId}/ avatar_400.webp, avatar_150.webp, avatar_50.webp, cover.webp
posts/{userId}/{fileId}/ {360,720,1080}p_clean.mp4, {360,720,1080}p_watermarked.mp4,
preview_3s.mp4, hls/master.m3u8, hls/{360,720,1080}p/seg_*.ts,
thumbnail.webp, og_clean.webp, og_play.webp, og_preview.mp4
stories/{userId}/{fileId}/ 720p_clean.mp4, thumbnail.webp
events/{accountId}/{eventId}/{fileId}/ banner.webp, banner_mobile.webp, banner_thumb.webp
shops/{shopId}/{productId}/{fileId}/ large.webp, medium.webp, thumb.webp
(See §12 for the watermarked-variant storage decision — store one rung, not all three.)
nexgate-private / nexgate-digital
As v1: messages, documents, kyc, temp audio (private); products with original/ (never
exposed) + preview/ + cover/ (digital).
4. Accepted File Formats
Images accept JPEG/PNG/HEIC/WebP/GIF; reject BMP/TIFF/RAW/SVG (SVG = security risk). Output always WebP .
Videos accept MP4/MOV/MKV/WEBM/AVI/3GP; reject WMV/FLV/VOB. Output H.264 MP4 (short) or HLS H.264 (long).
Digital PDF, DOCX/XLSX/PPTX, GLB/OBJ/FBX/STL, MP3/WAV/FLAC, ZIP/RAR, PSD/AI, PNG.
5. Upload Flow
[1] Client-side intelligent compression (max 1080p, CRF 18 light, HEIC→JPEG)
[2] POST /media/upload-request { fileName, fileSize, mimeType, directory, clientMeta }
[3] File Thunder: validate MIME + size + quota (atomic SQL) → DB record PENDING
→ presigned MinIO URL (nexgate-raw) → { fileId, uploadUrl, expiresIn: 1800 }
[4] Client uploads directly to MinIO (TUS resumable / multipart >5MB)
[5] Client confirms POST /media/confirm { fileId } (cleanup job recovers if missed)
[6] Processing pipeline starts
client compress
│
▼
POST upload-request ──▶ validate + quota ──▶ presigned URL
│
▼
upload direct to MinIO (raw)
│
▼
confirm ──▶ ClamAV ──▶ dedup ──▶ process ──▶ READY
│
▼
RabbitMQ media.ready ──▶ Main Backend
Quota checked at presigned-URL time (atomic UPDATE ... WHERE used+size <= quota ).
Duration checked AFTER upload via FFprobe (client value is a hint, not trusted) — over
limit → delete from raw, release quota, 403 DURATION_EXCEEDED .
Per-Upload Limits
Plan
Image
Short Video
Long Video
Digital Video
Duration
FREE
20MB
200MB
❌
❌
5 min
PRO
20MB
500MB
2GB
5GB
60 min
BUSINESS
20MB
2GB
5GB
20GB
Unlimited
6. Processing Pipelines (summary)
All pipelines: ClamAV scan → SHA-256 dedup check → process → store → delete raw →
DB READY → publish events. Watermarking and all placeholders are generated here, at
processing time , never at serve time.
6.1 Image
Auto-orient → strip EXIF (privacy) → NSFW check → dominant color + LQIP + BlurHash →
size variants per content type (Post: 1600/800/300; Profile: 400/150/50; Product: 1000/500/200
square; Event: 1200×630/800×420) → WebP → OG image → store public → READY.
6.2 Short Video (MP4, duration < 3 min)
FFprobe → adaptive transcode decision (never upscale, never inflate) → fast lane (quick
360p → LIVE_PARTIAL ) → full processing:
Clean variants {360,720,1080}p_clean.mp4 (faststart). Non-9:16 reel-eligible → blur-pad to 9:16 (scale-fill+blur background, scale-fit sharp foreground, overlay).
Watermarked variant(s) — moving watermark (jumps corners every 3s; 80×80 logo 60% opacity + username; app-based label from clientApp ). [v2 FIX] store one download rung (720p), not all three (§12).
Extras: thumbnail.webp (best-frame scored) + its LQIP/BlurHash/dominantColor, preview_3s.mp4 (360p muted watermarked), og_clean.webp , og_play.webp , og_preview.mp4 .
fMP4 ( frag_keyframe+empty_moov+faststart ) for byte-range.
Extract audio.wav → nexgate-private (Rec Engine).
READY: isReelEligible (<3min), streamingFormat: MP4 , variants JSONB, aspectRatio, qualityScore, duration.
6.3 Long Video (HLS, duration ≥ 3 min)
Same as short EXCEPT: HLS adaptive (no fast lane — stays PROCESSING until done).
hls/{360,720,1080}p/ segments + master.m3u8 . Segment = 2s, H.264 + AAC, MPEG-TS.
Still makes preview_3s, thumbnail, placeholders, OG. isReelEligible: false , streamingFormat: HLS .
6.4 Digital Products
Double ClamAV scan → SHA-256 checksum → light preview by type (PDF p1 watermarked,
video first 2min 480p watermarked, audio 30s 96kbps, 3D multi-angle thumbs, image 600px
watermarked) → store original/ (never exposed) + preview/ + cover/ → READY.
6.5 DM Attachments
nexgate-private, no CDN , conversation-membership check, thumb+original only, no
watermark, no reel pool, signed 5-min URLs. Virus scanned like everything.
7. Watermarking Strategy
Applied at processing time, once. Stored as separate _watermarked variant. Never at serve time, never client-side.
Moving watermark (corners every 3s) — static is croppable, moving is not.
Served: in-app playback → clean variant; download button → watermarked variant (§12). Direct CDN URL → clean (unavoidable; accepted, same tradeoff every platform has).
Pre-watermarked uploads (from other platforms): Phase 1 = no detection, just overlay ours. Phase 3 = AI logo detection → suppress reach (Instagram model), never block.
Username change: old videos keep old username (like TikTok).
[v2 FIX] Honest scope. A watermark does NOT prevent download — it survives download.
Its job is attribution + traceability, not prevention. The clean file is reachable by a
determined ripper via the playback URL; that's accepted. Goal = "not worth the effort for
99% + traceable if they do."
8. The Three Delivery Tiers [v2 FIX — core correction]
v1 assumed two levels (public unsigned / private signed). Correct model is three :
Tier
Content
Login to view?
URL signed?
Expiry
CDN cache
1 — trivial public
avatars, thumbnails, posters, blur previews
no
no
n/a
aggressive, 1y
2 — public content
full post images, video rungs
no
YES
hours
careful (§10)
3 — private / paid
DM media, private posts, digital products, KYC
yes
YES
minutes
none / private
┌──────────────────────────────────────────────┐
│ TIER 1 trivial public │
│ avatars, thumbs → unsigned, cache 1y │
├──────────────────────────────────────────────┤
│ TIER 2 public content │
│ full images/video → SIGNED, hrs TTL │
├──────────────────────────────────────────────┤
│ TIER 3 private / paid │
│ DM, private, digital → SIGNED + auth + x-uid │
└──────────────────────────────────────────────┘
only diff T2 vs T3 = auth check before minting
The key insight: "public to view" and "signed URL" are independent. Tier 2 content is
open to watch by anyone (no login), but the file is still signed — because signing is
anti-scrape / anti-hotlink , not access control. This is exactly TikTok's model (public
video, yet the raw file link expires and 403s).
The only difference between tier 2 and tier 3 in code is whether an auth/visibility check
runs before minting — the signing itself is identical.
Tier 1 is the only unsigned tier. Anything a scraper would actually want (full media)
is tier 2+, i.e. signed.
9. Signed URLs — how they actually work [v2 — new, the mechanism]
A signed URL = a normal link with a stamp made from three things:
stamp (x-sig) = HMAC(secret, path + expiry)
path — the object key being requested
expiry ( x-expires ) — unix time the link dies
secret — a key known ONLY to File Thunder/backend AND the CDN. Never in the URL, browser, or JSON. Lives in Vault.
Who does what, and when — three separate moments:
Upload time: File Thunder makes the variants, stores them in MinIO. CDN not involved.
Feed-build time (signing): backend computes x-sig from (path + expiry + secret) . Pure string math — no file opened, no CDN contacted, no storage touched. It's just writing a stamped link. (You can sign a path whose bytes don't even exist yet — the math only sees the string.)
Playback time (serving): the phone requests the stamped URL → CDN re-computes the stamp with its copy of the secret. Match + not expired → serve. First viewer: CDN lazily fetches the file from MinIO and caches it. Later viewers: served from cache.
upload feed build playback
┌────────┐ store ┌──────────┐ stamped ┌──────────┐
│ File │ ───────▶ │ Backend │ ─ link ──▶ │ CDN │
│ Thunder│ to MinIO │ signs │ │ validates│
└────────┘ └──────────┘ └────┬─────┘
secret ▲ │ first view
Vault └──────── same ────────┘ fetch+cache
secret from MinIO
Tamper resistance (why a stripped/edited link fails):
No sig → CDN rejects (paths require a valid sig) → 403.
Fake sig → recompute doesn't match → 403.
Edited expiry → sig was computed from the old expiry, no longer matches → 403.
Swapped path → sig bound to original path → 403.
Untouched + in time → serve. After x-expires → even the real link 403s.
Forging anything requires the secret , which never leaves backend + CDN. Protect the
secret (Vault) and the whole scheme holds.
You don't write the HMAC. The MinIO SDK (or CDN signer) does it in one call:
// ── Mint a signed delivery URL ──
String url = minioClient.getPresignedObjectUrl(
GetPresignedObjectUrlArgs.builder()
.method(Method.GET)
.bucket(bucket)
.object(objectKey)
.expiry(seconds, TimeUnit.SECONDS) // → x-expires + x-sig
.build());
10. URL Strategy
Tier 1 — trivial public → permanent, unsigned, cache hard
https://media.nexgate.co/profiles/usr_123/avatar_400.webp
https://media.nexgate.co/posts/usr_123/vid_789/thumbnail.webp
Cache-Control: public, max-age=31536000 . Scraping these gains nothing.
Tier 2 — public content → signed, short-ish expiry
https://media.nexgate.co/posts/usr_123/vid_789/720p_clean.mp4?x-expires=1780297200&x-sig=8f3a9c
Tier 3 — private / paid → signed + user-bound + auth check first
https://media.nexgate.co/private/messages/conv_abc/file_789/original.jpg
?x-expires=...&x-sig=...&x-uid=usr_xyz
[v2 FIX] Domain is .co . All URLs use media.nexgate.co . v1 mixed .com / .co —
that produces broken links. One domain everywhere.
[v2 FIX] Cache-key gotcha. Never let the CDN cache a signed object using a cache key
that includes x-sig / x-uid — you'd fragment cache per user, and a cached object can
outlive its stamp (origin checks expiry, but a cache hit never reaches origin). Rules:
tier 1 cache hard; tier 2 cache by path only (strip query from cache key), short edge
TTL; tier 3 never on shared cache.
Signed URL params
Param
Purpose
x-expires
link dies after TTL
x-sig
HMAC tamper protection
x-uid
bound to user (tier 3) — shared link won't work for someone else
Who generates / validates
Backend (has secret) generates . Cloudflare edge (same secret) validates . MinIO never
seen by clients.
11. Streaming Token Strategy: MP4 vs HLS [v2 — new]
Expiry is checked when each request opens , not continuously. An in-flight transfer
completes even if the clock passes mid-download. The two formats differ because of how many
requests a single playback makes.
SHORT MP4 LONG HLS
one request hundreds of requests
┌──────────┐ ┌──────────┐ ┌──┬──┬──┬──┐
│ 720p.mp4 │ ← 1 signed URL │master.m3u│─▶│ts│ts│ts│..│
└──────────┘ few-hr TTL └──────────┘ └──┴──┴──┴──┘
opens once, short token long token
rides to end (~1 min) (> video len)
per-session one, shared
Short video (MP4) — one signed URL
One file = one request (or a few byte-ranges, all opened up front). A few-hour expiry is
plenty; the in-flight request completes, so mid-play expiry basically never happens.
→ mint one signed URL per rung, few-hour TTL. No dual-token, no single-use.
Long video (HLS) — dual-token (or prefix-signed)
HLS = hundreds of segment requests (one every ~2s), so expiry is re-checked constantly. A
single short token would die mid-video. Industry-standard fix (Google Media CDN / CloudFront):
master.m3u8 → short token (~1 min). Per-session entry gate. Fetched once at start.
child manifests + .ts segments → long token (> full video length, up to ~1 day), carried by signed cookie or path-prefix signing (sign the folder, not each segment).
Segments stay cacheable (same bytes for everyone); only the manifest is per-session. Preserves cache hit rate.
At our stage on Cloudflare Free/Pro, do the simpler version: backend mints a long
prefix-signed token for the whole hls/ folder when the post loads. Graduate to true
edge dual-token (short→long exchange) if/when we move to a heavier CDN.
Single-use — downloads ONLY, never streams
A stream is many requests; making the token single-use would 403 the viewer one segment in.
Streams use short expiry + x-uid binding instead. Single-use (Redis) is correct only
for one-file-one-request downloads (§12, digital products).
Edge case (both formats)
User pauses past expiry, then seeks → player fires a fresh request with a dead URL → 403.
Client safety net: catch the 403, silently re-fetch a fresh URL from backend, resume.
Short MP4
Long HLS
Requests / play
one (few byte-ranges)
hundreds
Token
single signed URL
dual-token / prefix-signed
TTL
few hours
> watch duration
Single-use
no
no
12. Watermark & Download Path [v2 FIX]
Decision: Option B — downloads are watermarked. (Full build, not the launch shortcut.)
The feed NEVER contains a watermarked URL. Feed = clean playback URLs only. Watermarked
download is a separate endpoint , minted on tap.
User taps Download
→ GET /media/{fileId}/download?quality=720p (quality optional, default 720p)
→ backend: check canDownload (post setting) + tier (private/paid → auth check)
→ mint signed URL → {quality}_watermarked.mp4
short TTL (~10 min), x-uid bound, single-use (Redis x-once)
→ log download event (orderId/buyerId/ip/device/country if digital)
→ return { downloadUrl, expiresIn, singleUse: true }
Client downloads. URL marked used in Redis → 403 on reuse.
Default quality, no picker (short video). Serve one default rung ( 720p ) — matches
TikTok/IG, sane on TZ data. Endpoint takes optional quality so a picker is an additive
change later (long-form only, where the size gap matters).
Storage decision: store one watermarked rung (720p), not all three. If a picker is
added later, lazy-generate other rungs on first request and cache — only spend
CPU/storage on a quality someone actually downloads.
Timing: watermarked 720p is generated at processing time (pre-ready), so download is
instant — no first-user FFmpeg penalty.
// ── Download endpoint (picker-ready, single-use, watermarked) ──
@GetMapping("/media/{fileId}/download")
ResponseEntity download(@PathVariable String fileId,
@RequestParam(defaultValue = "720p") String quality,
@AuthenticationPrincipal AccountEntity user) {
MediaEntity m = mediaService.findReady(fileId);
// ── access gate (tier 2 = open; tier 3 = ownership/visibility) ──
if (!m.isCanDownload()) return ResponseEntity.status(403).build();
if (m.isRestricted() && !accessService.canAccess(user, m))
return ResponseEntity.status(403).build();
// ── mint watermarked, user-bound, single-use URL ──
String key = m.watermarkedKey(quality); // lazy-generate if absent
String url = mediaService.mintDownloadUrl(key, user.getId(), Duration.ofMinutes(10));
downloadLog.record(fileId, user, request); // audit
return ResponseEntity.ok(Map.of("downloadUrl", url, "expiresIn", 600, "singleUse", true));
}
13. Feed Response Shape [v2 — new]
Rules baked in:
Batch-mint inline. While building the page (already looping posts), mint every signed URL right there — no per-item round trip from the client.
Store objectKey , mint URL per request. Never persist a signed URL (it expires).
Tier 1 unsigned (avatar, thumb, poster, preview3s); tier 2 signed (full image, video rungs).
Placeholders filled (blurhash, lqip, dominantColor) + width / height / duration present.
No watermarked URL here (§12). canDownload flag tells the client whether to show the button.
{
"data": [{
"id": "0e67aef4-...",
"author": {
"id": "ad03431a-...", "userName": "juliusdev_", "verified": true,
"profilePictureUrl": "https://media.nexgate.co/profiles/ad03431a-.../avatar_400.webp"
},
"content": "Testing video media post 🎬",
"shareCode": "waBbP82h",
"media": [{
"id": "436bb62b-...",
"mediaType": "VIDEO",
"status": "READY",
"width": 1080, "height": 1920, "aspectRatio": "9:16", "duration": 28,
"isReelEligible": true,
"streamingFormat": "MP4",
"canDownload": true,
"dominantColor": "#1a1a2e",
"blurhash": "L6Pj0^i_.AyE_3t7t7R**0o#DgR4",
"lqip": "data:image/webp;base64,UklGRjoAAAB...",
"poster": "https://media.nexgate.co/posts/ad03431a-.../436bb62b-.../thumbnail.webp",
"preview3s": "https://media.nexgate.co/posts/ad03431a-.../436bb62b-.../preview_3s.mp4",
"variants": {
"360p": "https://media.nexgate.co/posts/ad03431a-.../436bb62b-.../360p_clean.mp4?x-expires=1780297200&x-sig=a1f2b3",
"720p": "https://media.nexgate.co/posts/ad03431a-.../436bb62b-.../720p_clean.mp4?x-expires=1780297200&x-sig=b2e3c4",
"1080p": "https://media.nexgate.co/posts/ad03431a-.../436bb62b-.../1080p_clean.mp4?x-expires=1780297200&x-sig=c3f4d5"
},
"order": 1
}],
"privacySettings": { "visibility": "PUBLIC" }
}]
}
(HLS video: streamingFormat: "HLS" , replace variants with a single signed/prefix-signed master.m3u8 .)
14. Security & Anti-Scraping [v2 — expanded]
14.1 Close the visibility-bypass hole
v1's permanent unsigned URLs meant a PRIVATE post's bytes were reachable by anyone with the
link — the API hid the post, the file did not. Fix:
Remove public-read from MinIO buckets (already a stated rule — enforce it).
Route all access through signed CDN URLs.
Private/followers/DM/paid → tier 3 (auth check before minting + x-uid ).
After this, a hidden post's media returns 403 to anyone but the authorized viewer.
14.2 Two layers, both required
Signed URLs stop permanent theft: scraped links die in hours → no permanent mirror, no hotlinking. They do NOT stop a one-time download — a valid link is a valid link.
Bot gate stops the act of mass-pulling: it guards the feed API door so a bot can't cheaply pull fresh links over and over.
Neither alone is enough. Together = scraping isn't worth it. (You cannot make it impossible — the goal is "not worth the effort for 99%, traceable if they do.")
14.3 Bot gate (the missing half) — sized to moat (§30)
App attestation — the real app computes a hard-to-fake token (TikTok's X-Bogus / msToken equivalent). A plain script can't produce it without running app code.
Rate limiting — one "user" pulling thousands of posts/min = bot → throttle/block.
Auth + behaviour — require login; flag inhuman scroll/pull patterns.
Escalating friction — suspicious traffic → CAPTCHA / slowdown / ban.
Build depth depends on the moat decision (§30). Commerce-moat → lighter bot gate is
fine for launch. Content-moat → invest earlier.
14.4 Protection layers (reference)
Content-Disposition: inline · dynamic URL via JS · signed expiring URLs · session/device
binding · disable right-click (frontend) · watermark on download · HLS chunking.
15. OG (Open Graph) Strategy
OG images + og_preview.mp4 are permanent, unsigned, tier 1 — because crawlers
(WhatsApp/Telegram) fetch on share and the user may open hours later; a signed URL would
expire → "Error loading." Access control for private posts is enforced server-side (the
post page returns 403), not via the OG asset. Serve og_clean.webp to known platforms,
og_play.webp (burned play button) to unknown crawlers (User-Agent detection).
16. Compression & Transcoding (reference)
Client: light pre-compress (max 1080p, CRF 18, never upscale). Server: FFprobe → adaptive
ladder (never upscale, never inflate → -c:v copy if output > input). CRF 21/23/25/28 for
1080/720/480/360, 30 for OG preview. 1080p max (all platforms serve ≤1080p anyway).
-movflags +faststart always; frag_keyframe+empty_moov+faststart for fMP4.
Now: H.264 + AAC. Future: AV1 + Opus + CMAF.
17. BlurHash, LQIP & Dominant Color
Generated for every image + video thumbnail, at processing time. Progressive load:
dominantColor (0ms) → BlurHash (0ms) → LQIP (0ms) → thumb.webp (CDN) → full on tap.
All three stored in media_content_signals and shipped inline in the feed JSON.
18. Deduplication
SHA-256 on raw file → file_hashes . Exists+clean → skip processing, reference existing
variants, reference_count++ . Delete → decrement; 0 → remove from MinIO. Near-dup (pHash) =
future (copyright).
19. Quota Management
Enforced at presigned-URL generation (atomic SQL). Social = original size charged;
digital = all stored bytes. Plans: FREE 1GB / PRO 20GB / BUSINESS 100GB. Warn 80%/95%/100%.
Release on abandoned (immediate) and hard-delete (30d after soft). Redis cache, sync to PG
every 5 min.
20. Storage Lifecycle
raw: auto-delete 24h. public: keep forever, access-based tiering (hot NVMe → warm HDD → cold;
viral old content auto-promotes). private: DM 1y, audio 24h, KYC per law. digital: forever.
Soft delete → 30d grace → hard delete + Cloudflare purge + quota free.
21. Abandoned Upload Handling
Layer 1: client confirm (fast path). Layer 2: cleanup job every 30 min (PENDING >1h → file
exists = recover/process; not = abandoned, release quota). Layer 3: raw lifecycle 24h.
fileId in object key enables recovery without confirm. Redis distributed lock on the job.
22. Progress Tracking (SSE)
SSE (one-way, auto-reconnect, resilient on TZ networks). Redis key upload_status:{fileId} :
PENDING→UPLOADING→UPLOADED→SCANNING→PROCESSING→LIVE_PARTIAL→READY (or FAILED/QUARANTINED/
ABANDONED). Multi-instance via Redis Pub/Sub bridging emitters.
23. Communication Architecture
No direct HTTP between services. RabbitMQ (FT ↔ Main Backend, topic nexgate.media ):
media.upload.request , media.upload.url.ready , media.ready , media.live.partial ,
media.failed , media.quarantined , digital.product.ready . Kafka (FT → Rec Engine /
Analytics, topic content.new ).
[v2 FIX] Events carry objectKey s + variants, not signed URLs. MEDIA_READY ships
storage keys (e.g. posts/usr_123/vid_789/720p_clean.mp4 ) + placeholders. The Main
Backend mints signed URLs per feed request from those keys — it never stores a signed
URL.
24. CDN Strategy
Cloudflare in front of MinIO. First request → edge → MinIO → cache → serve; thereafter from
cache. Cache-Control: tier 1 = 1y; tier 2 = path-keyed short edge TTL; tier 3 / API =
private, no-store . MinIO never publicly exposed (only Cloudflare IPs). Phases: Free →
Pro ($20) → R2 (zero egress). See §10 cache-key gotcha.
25. Shareable Links
Client-side nexgate.co/reels/{shareCode}?ref=whatsapp . No uid in URL (server resolves
owner from shareCode → follow prompt). ref drives share analytics. Public content = no
token needed for the page; the media on the page still follows tier rules (§8).
26. Technology Stack
Java 21 / Spring Boot 3.x · FFmpeg (Jaffree) · ImageMagick (IM4Java) · blurhash-java ·
ClamAV (Docker, TCP 3310) · MinIO · Cloudflare · RabbitMQ · Kafka · Redis · PostgreSQL ·
TUS · Vault ( vault.qbitspark.com ) — holds the signing secret · Docker Compose · SSE.
27. Database Schema (key tables + v2 notes)
media_files (
file_id UUID PK, owner_id UUID, directory ENUM, original_name TEXT,
object_key TEXT, -- canonical storage key [v2: keys, never signed URLs]
mime_type TEXT, file_size BIGINT,
status ENUM, -- PENDING..READY..QUARANTINED..ABANDONED
variants JSONB, -- variant object_keys (NOT URLs)
metadata JSONB, scan_result TEXT, hash TEXT,
is_reel_eligible BOOLEAN, streaming_format ENUM,
can_download BOOLEAN DEFAULT true, -- [v2] drives feed canDownload + download gate
created_at TIMESTAMPTZ, ready_at TIMESTAMPTZ
);
media_content_signals (
file_id UUID FK, has_audio BOOLEAN, audio_ref TEXT,
dominant_color TEXT, blurhash TEXT, lqip TEXT,
aspect_ratio TEXT, quality_score INT, width INT, height INT, duration_seconds DECIMAL
);
-- user_storage_quota, file_hashes, media_variants (HOT/WARM/COLD),
-- digital_product_files, digital_orders, digital_download_logs (as v1)
Indexes: idx_media_cleanup (created_at) WHERE status='PENDING' ; unique file_hashes(hash) ;
media_files(owner_id, status, created_at DESC) .
[v2 FIX] No signed_url column anywhere. Signed URLs are minted per request and
discarded. The DB stores only object_key + variants (keys).
28. Docker & Infrastructure (reference)
eclipse-temurin:21 + ffmpeg + imagemagick(+webp/heic). ClamAV separate container
(daemon, auto-updates). FT: 4 cores / 16GB (FFmpeg CPU-heavy). ClamAV: 1 core / 2GB.
Env: PG, Redis, RabbitMQ, Kafka, MinIO, ClamAV, VAULT_ADDR .
29. What File Thunder Does NOT Do
❌ ML/AI classification · Whisper (Rec Engine) · CLIP · feed ranking · social graph ·
payments · auth · push · search · business logic · fan-out · recommendations.
Extracts audio.wav temporarily; Rec Engine fetches/transcribes/deletes. FT never knows
audio content.
30. Open Decisions (parked)
Moat: content vs commerce? This drives §14.3 (bot-gate depth) and §8 (how hard to
stamp tier-2 public media). If commerce is the moat (shops/payments/events), public
media scraping hurts less → lighter bot gate, tier-2 stamping is softer/optional. If
content is the moat → invest in bot gate + stamp tier-2 firmly. Decide before finalizing
anti-scraping work.
Bot gate depth for launch — minimum: login + rate limit. Full app-attestation later,
sized to (1).
HLS dual-token vs prefix-sign for launch — prefix-sign on Cloudflare now; edge
dual-token if/when heavier CDN (§11).
Download quality picker — default 720p now; picker + lazy-gen later (§12).
Summary — File Thunder v2 in one line
Receives raw files → ClamAV → processes into clean + watermarked variants (FFmpeg/IM) with
blurhash/lqip/dominant-color → stores object keys across 4 buckets in MinIO (never
public-read) → the Main Backend mints signed delivery URLs per request from those keys
using a three-tier policy (trivial-public unsigned, public-content signed, private/paid
signed + auth) → Cloudflare serves and caches by tier → streams use MP4-single-URL or
HLS-dual-token, downloads use single-use watermarked URLs → signed URLs + a bot gate
together make scraping not worth the effort — all without business logic, ML, or social ops.
File Thunder Architecture Guide v2.0 — NexGate / QBIT SPARK
v2 integrates media-security design review. [v2 FIX] tags mark every change from v1.