# File & Media Thunder # File Thunder — Architecture & Design Guide # File Thunder — Architecture & Design Guide ### NexGate Media Engine | Version 1.0 --- ## Table of Contents 1. [What is File Thunder?](#1-what-is-file-thunder) 2. [The Four Wheels](#2-the-four-wheels) 3. [Storage Architecture (MinIO Buckets)](#3-storage-architecture-minio-buckets) 4. [Accepted File Formats](#4-accepted-file-formats) 5. [Upload Flow](#5-upload-flow) 6. [Processing Pipelines](#6-processing-pipelines) - [Image Pipeline](#61-image-pipeline) - [Short Video Pipeline](#62-short-video-pipeline-mp4) - [Long Video Pipeline](#63-long-video-pipeline-hls) - [Digital Products Pipeline](#64-digital-products-pipeline) - [DM Attachments Pipeline](#65-dm-attachments-pipeline) 7. [Watermarking Strategy](#7-watermarking-strategy) 8. [File Protection Layers](#8-file-protection-layers) 9. [URL Strategy](#9-url-strategy) 10. [OG (Open Graph) Strategy](#10-og-open-graph-strategy) 11. [Compression & Transcoding](#11-compression--transcoding) 12. [BlurHash, LQIP & Dominant Color](#12-blurhash-lqip--dominant-color) 13. [Deduplication](#13-deduplication) 14. [Quota Management](#14-quota-management) 15. [Storage Lifecycle](#15-storage-lifecycle) 16. [Abandoned Upload Handling](#16-abandoned-upload-handling) 17. [Progress Tracking (SSE)](#17-progress-tracking-sse) 18. [Communication Architecture](#18-communication-architecture) 19. [CDN Strategy](#19-cdn-strategy) 20. [Shareable Links](#20-shareable-links) 21. [Technology Stack](#21-technology-stack) 22. [Database Schema](#22-database-schema) 23. [Docker & Infrastructure](#23-docker--infrastructure) 24. [What File Thunder Does NOT Do](#24-what-file-thunder-does-not-do) --- ## 1. What is File Thunder? File Thunder is NexGate's dedicated media processing engine. It is a standalone Spring Boot microservice responsible for all file and media operations across the entire NexGate platform. **Core principle:** The Main Backend never touches raw files. It delegates all media operations to File Thunder and acts as a thin client. **Responsibilities:** - Receive file upload requests - Validate files (type, size, quota) - Virus scanning - Transcoding and processing - Generating variants (thumbnails, WebP, HLS, MP4) - Watermarking - Storing files in MinIO - Serving files via CDN - Publishing media-ready events **What it is NOT:** - Not an AI/ML service - Not a recommendation engine - Not a social graph service - Not a payment service --- ## 2. The Four Wheels File Thunder is powered by four core engines: ``` FILE THUNDER ENGINE │ ┌──────┼──────┬──────┐ │ │ │ │ FFmpeg IM ClamAV MinIO Video Image Sec Storage Wheel Wheel Wheel Wheel ``` ### Wheel 1 — FFmpeg (Video) - Video transcoding (any format → H.264 MP4 or HLS) - Audio extraction (for Rec Engine) - HLS segment generation - Thumbnail extraction (best frame detection) - 3-second preview clip generation - Watermark overlay (logo + username) - Aspect ratio handling and blur padding - Fragmented MP4 (fMP4) generation - GIF → MP4 conversion - Format conversion (MOV, MKV, AVI, WEBM → MP4) - Audio normalization **Java wrapper:** Jaffree (calls `/usr/bin/ffmpeg` via ProcessBuilder) ### Wheel 2 — ImageMagick (Images) - Resize to multiple variants - Format conversion (JPEG, PNG, HEIC → WebP) - EXIF stripping (privacy — removes GPS data) - Auto-orientation (fix rotated phone photos) - Quality compression - Dominant color extraction - LQIP generation (10×10 base64) - OG image generation (1200×630) - GIF → WebP animated conversion - PDF → preview image conversion **Java wrapper:** IM4Java (calls `/usr/bin/convert` via ProcessBuilder) **Additional library:** `blurhash-java` for BlurHash generation ### Wheel 4 — MinIO (Storage) - Stores ALL file variants across 4 buckets - S3-compatible object storage (self-hosted) - Receives direct uploads via presigned URLs (client never touches File Thunder bandwidth) - Serves as origin for Cloudflare CDN - Supports multipart uploads (files > 5MB) - Supports byte-range requests (progressive streaming, resumable downloads) - Lifecycle policies (auto-delete raw uploads after 24h) - Storage class tiering (hot → warm → cold) - Bucket event notifications (upload complete events) - Never exposed publicly (only Cloudflare and File Thunder can reach it) **Java SDK:** MinIO Java SDK (`io.minio:minio`) --- ### Wheel 3 — ClamAV (Security) - Virus and malware scanning for every uploaded file - Hash-based scan cache (skip re-scanning known clean files) - Runs as separate Docker container (daemon-based) - File Thunder connects via TCP socket (`clamav:3310`) - Digital products scanned twice (on upload + before download) **Java client:** clamd4j or custom TCP socket client --- ## 3. Storage Architecture (MinIO Buckets) **Rule: 4 buckets total. Never one bucket per user.** ``` nexgate-raw/ ← temporary upload landing zone nexgate-public/ ← CDN cached, social content nexgate-private/ ← signed URLs, DMs + docs nexgate-digital/ ← purchase-gated, digital products ``` ### nexgate-raw (Temporary) ``` nexgate-raw/ uploads/{userId}/{fileId}/ original.ext ← raw upload lands here ``` - Auto-delete lifecycle policy: 24 hours - Incomplete multipart uploads aborted: 24 hours - Never served to any client - Deleted by File Thunder after processing completes ### nexgate-public ``` nexgate-public/ profiles/{userId}/{fileId}/ avatar_400.webp avatar_150.webp avatar_50.webp cover.webp posts/{userId}/{fileId}/ 360p_clean.mp4 720p_clean.mp4 1080p_clean.mp4 360p_watermarked.mp4 720p_watermarked.mp4 1080p_watermarked.mp4 preview_3s.mp4 hls/master.m3u8 hls/360p/segment_000.ts ... hls/720p/segment_000.ts ... thumbnail.webp og_clean.webp og_play.webp og_preview.mp4 stories/{userId}/{fileId}/ 720p_clean.mp4 thumbnail.webp events/{accountId}/{eventId}/{fileId}/ banner.webp banner_mobile.webp banner_thumb.webp shops/{shopId}/{productId}/{fileId}/ large.webp medium.webp thumb.webp categories/{categoryId}/{fileId}.webp ``` ### nexgate-private ``` nexgate-private/ messages/{conversationId}/{fileId}/ original.jpg thumb.webp audio/{fileId}.wav ← temp for Rec Engine (deleted after transcription) documents/{userId}/{fileId}/ original.pdf preview.webp kyc/{userId}/{fileId}/ id_front.jpg id_back.jpg ``` ### nexgate-digital ``` nexgate-digital/ products/{shopId}/{productId}/{fileId}/ original/ file.pdf ← actual purchased product checksum.txt ← SHA-256 hash preview/ preview.webp ← low-res watermarked preview cover.webp ← product cover image ``` ### Object Key Pattern (Consistent Rule) ``` {bucket}/{domain}/{ownerId}/{entityId}/{fileId}/{variant} Examples: nexgate-public/posts/usr_123/post_456/file_789/thumb.webp nexgate-private/messages/conv_abc/file_789/original.jpg nexgate-digital/products/shop_xyz/prod_123/file_789/original.pdf ``` --- ## 4. Accepted File Formats ### Images
AcceptReject
JPEG/JPGBMP
PNGTIFF
HEIC/HEIFRAW
WebPSVG (security risk)
GIF
**Output:** Always WebP ### Videos
AcceptReject
MP4WMV
MOVFLV
MKVVOB
WEBM
AVI
3GP
**Output:** H.264 MP4 (short) or HLS H.264 (long) ### Digital Products PDF, DOCX, XLSX, PPTX, GLB, OBJ, FBX, STL, MP3, WAV, FLAC, ZIP, RAR, PSD, AI, PNG (stock art) --- ## 5. Upload Flow ### Step-by-Step ``` CLIENT ↓ [1] Intelligent client-side compression - Detect: resolution, bitrate, codec, network type, device tier - Compress if: over-bitrated, slow network, large file - Never compress if: already optimized - Target: max 1080p, CRF 18 (light) - HEIC → JPEG conversion (iOS) ↓ [2] POST /media/upload-request { fileName, fileSize, mimeType, directory, clientMeta: { clientApp, networkType, deviceTier, wasCompressed } } ↓ [3] File Thunder validates: - MIME type in allowed list? - File size within per-upload limit? - User quota not exceeded? (atomic SQL check) - Creates DB record: status PENDING - Generates presigned MinIO URL → nexgate-raw - Returns { fileId, uploadUrl, expiresIn: 1800 } ↓ [4] Client uploads directly to MinIO - TUS resumable protocol for large files - Multipart upload for files > 5MB - Progress tracked client-side - Upload starts during caption writing (parallel) ↓ [5] Client confirms: POST /media/confirm { fileId } (or cleanup job recovers if confirm missed) ↓ [6] Processing pipeline starts ``` ### Quota + Duration Enforcement **Size check** → at presigned URL generation (before upload): ``` Atomic SQL: UPDATE user_storage_quota SET used_bytes = used_bytes + fileSize WHERE user_id = ? AND (used_bytes + fileSize) <= quota_bytes RETURNING used_bytes 0 rows affected → quota exceeded → reject 403 1 row affected → quota reserved → proceed ``` **Duration check** → after upload (FFprobe analysis): ``` Client sends estimated duration in clientMeta (hint only) FFprobe confirms actual duration after upload If actual duration > plan limit: → delete file from nexgate-raw → reject with 403: { error: "DURATION_EXCEEDED", plan: "FREE", maxDuration: 300 } → release quota reservation → notify user: "Upgrade plan to upload longer videos" ``` Why duration checked after upload (not before): ``` Client-reported duration = not trusted = could be manipulated FFprobe = ground truth = server side, reliable = only way to confirm actual duration ``` ### Per-Upload Size + Duration Limits
PlanImageShort Video (social)Long VideoDigital Product VideoDuration
FREE20MB200MB❌ Not allowed❌ Not allowedMax 5 min
PRO20MB500MB2GB5GB per fileMax 60 min
BUSINESS20MB2GB5GB20GB per fileUnlimited
**Why duration matters as much as size:** ``` Same 200MB could be: 3 minute 1080p reel → acceptable ✅ 45 minute long video → not acceptable for FREE ❌ Size check alone = not enough Duration check = required alongside size Enforcement: Size → checked at upload request (before presigned URL) Duration → checked AFTER upload (FFprobe analysis) client also sends estimated duration in clientMeta If duration exceeds plan limit after FFprobe: → delete from nexgate-raw → reject processing → notify user: "Upgrade to upload longer videos" → refund quota reservation ``` **Duration limits by plan:** ``` FREE: Reels/short video → max 5 minutes Long video → not allowed PRO: Reels/short video → max 5 minutes (treated as short) Long video → max 60 minutes BUSINESS: All video → unlimited duration ``` --- ## 6. Processing Pipelines ### 6.1 Image Pipeline ``` Upload confirmed ↓ RabbitMQ: SCAN job ↓ ClamAV scan → VIRUS: quarantine + notify | CLEAN: continue ↓ SHA-256 hash check (deduplication) Exists + clean → skip processing, reuse variants New → continue ↓ RabbitMQ: PROCESS job → Image Worker ↓ [1] Auto-orient (apply EXIF rotation before stripping) [2] Extract useful EXIF (orientation only) [3] Strip ALL EXIF (privacy — remove GPS, device info) [4] AI moderation check (NSFW detection) UNSAFE → quarantine | SAFE → continue [5] Extract dominant color (resize to 1×1 pixel) [6] Generate LQIP (resize to 10×10, base64 encode) [7] Generate BlurHash (blurhash-java library) [8] Generate variants by content type: Profile → 400px, 150px, 50px (WebP) Post → 1600px, 800px, 300px (WebP) Product → 1000px, 500px, 200px (WebP, 1:1 square) Event → 1200×630px, 800×420px (WebP) Story → 1080×1920px (WebP) [9] Convert all variants → WebP [10] Generate OG image (1200×630, WebP): - og_clean.webp (no play button) - For non-16:9 → blur pad background to fill 16:9 [11] Store all variants → nexgate-public [12] Delete original from nexgate-raw [13] Update DB: status READY, populate variants JSONB [14] Publish to Kafka: FILE_READY event [15] Publish to RabbitMQ: MEDIA_READY event → Main Backend ``` **Variant sizes per content type:**
ContentVariants
Profile picture400px, 150px, 50px
Post image1600px (large), 800px (medium), 300px (thumb)
Product image1000px, 500px, 200px (always 1:1 square)
Event banner1200×630, 800×420, 400×210
Category400px wide
--- ### 6.2 Short Video Pipeline (MP4) **Threshold:** Duration < 3 minutes → MP4 ``` Upload confirmed ↓ RabbitMQ: SCAN job ↓ ClamAV scan → VIRUS: quarantine | CLEAN: continue ↓ SHA-256 hash check (deduplication) ↓ FFprobe analysis: - resolution, bitrate, fps - codec, duration, aspectRatio - hasAudio, qualityScore ↓ Adaptive transcode decision: - Never upscale (never exceed input resolution) - Never inflate (if output > input size, use original) - Only generate eligible variants ↓ RabbitMQ: PROCESS job → Video Worker ↓ [FAST LANE — runs first for UX]: Quick 360p transcode → post goes LIVE_PARTIAL User notified: "Almost ready!" ↓ [FULL PROCESSING — continues async]: [1] Detect aspect ratio (FFprobe): 9:16 (vertical) → native reel format → no padding needed 16:9 (landscape) → needs blur pad for reel pool 1:1 (square) → needs blur pad for reel pool 4:5 (portrait) → needs blur pad for reel pool Other → preserve original, blur pad if reel eligible [2] Transcode CLEAN variants (H.264, faststart): For NATIVE 9:16 videos: Transcode directly to 9:16 variants 360p_clean.mp4 (360×640, CRF 28) 720p_clean.mp4 (720×1280, CRF 23) 1080p_clean.mp4 (1080×1920, CRF 21, if eligible) For NON-9:16 videos (isReelEligible = true): Apply BLUR PAD to fill 9:16 frame (TikTok/Instagram style) What blur pad looks like: ┌──────────────┐ │▓▓▓▓▓▓▓▓▓▓▓▓▓│ ← blurred background (scaled up + blurred) │▓▓▓▓▓▓▓▓▓▓▓▓▓│ │┌────────────┐│ ││ ││ ││ original ││ ← actual video centered (no cropping) ││ video ││ ││ ││ │└────────────┘│ │▓▓▓▓▓▓▓▓▓▓▓▓▓│ │▓▓▓▓▓▓▓▓▓▓▓▓▓│ └──────────────┘ Why blur pad (not black bars, not crop): Black bars → looks amateur ❌ Crop → destroys content ❌ Blur pad → professional, intentional ✅ TikTok + Instagram do this ✅ No content lost ✅ FFmpeg blur pad command: ffmpeg -i input.mp4 \ -filter_complex \ "[0]scale=720:1280:force_original_aspect_ratio=increase, crop=720:1280, boxblur=20:5[bg]; [0]scale=720:1280:force_original_aspect_ratio=decrease, pad=720:1280:(ow-iw)/2:(oh-ih)/2[fg]; [bg][fg]overlay=(W-w)/2:(H-h)/2" \ output_9_16_blurpad.mp4 Steps: bg layer → scale to FILL 9:16, crop excess, apply blur (20px radius) fg layer → scale to FIT in 9:16, letterbox, center overlay → sharp video (fg) on top of blurred background (bg) For NON-9:16 videos (isReelEligible = false): Keep original aspect ratio No blur pad Feed player handles letterboxing client-side [3] Generate WATERMARKED variants: FFmpeg overlay — MOVING watermark (TikTok style): Watermark jumps between corners every 3 seconds: 0-3s → top left (10px, 10px) 3-6s → top right (W-w-10, 10px) 6-9s → bottom right (W-w-10, H-h-10) 9-12s → bottom left (10px, H-h-10) repeat... Logo PNG: 80×80px, 60% opacity Username text: same position as logo (below it) Based on clientApp field: NEXGATE_ANDROID / NEXGATE_IOS / NEXGATE_WEB → "NexGate" NEXGATE_LITE → "NexGate Lite" Why moving watermark: Static corner = easy to crop out ❌ Moving = appears in all corners = impossible to crop ✅ TikTok confirmed uses this strategy FFmpeg command (moving watermark): overlay= x='if(lt(mod(t,12),3), 10, if(lt(mod(t,12),6), W-w-10, if(lt(mod(t,12),9), W-w-10, 10)))': y='if(lt(mod(t,12),3), 10, if(lt(mod(t,12),6), 10, if(lt(mod(t,12),9), H-h-10, H-h-10)))' Variants generated: 360p_watermarked.mp4 720p_watermarked.mp4 1080p_watermarked.mp4 [4] Generate extras: thumbnail.webp → best frame (brightness + sharpness scored) thumbnail LQIP → 10×10 base64 thumbnail BlurHash → blurhash-java thumbnail dominantColor → hex preview_3s.mp4 → first 3 seconds, 360p, muted, watermarked og_clean.webp → 1200×630, clean thumbnail og_play.webp → 1200×630, NexGate play button burned in og_preview.mp4 → 360p, watermarked, permanent CDN URL [5] Generate Fragmented MP4 (fMP4): ffmpeg flags: frag_keyframe+empty_moov+faststart 1-second fragment intervals (keyframe every 30 frames at 30fps) Enables byte-range requests for progressive streaming [6] Extract audio for Rec Engine: ffmpeg -vn -acodec pcm_s16le -ar 16000 -ac 1 audio.wav Store → nexgate-private/audio/{fileId}.wav (Rec Engine fetches, transcribes, deletes) [7] Store all variants → nexgate-public [8] Delete original from nexgate-raw [9] Update DB: status: READY isReelEligible: true (duration < 3min) streamingFormat: MP4 variants: JSONB with all paths aspectRatio, qualityScore, duration [10] Publish to Kafka: FILE_READY (with audioRef) [11] Publish to RabbitMQ: MEDIA_READY → Main Backend Main Backend: → update post status LIVE → index into reel pool → trigger fan-out to followers ``` **Reel pool eligibility:** ``` duration < 3 minutes → isReelEligible: true → indexed in reel pool duration ≥ 3 minutes → isReelEligible: false → feed only ``` **MP4 variant resolutions by aspect ratio:**
Aspect360p540p720p1080p
9:16 (vertical)360×640540×960720×12801080×1920
16:9 (landscape)640×360—1280×7201920×1080
1:1 (square)360×360—720×7201080×1080
**Why 1080p is max (not 4K):** All major platforms compress 4K down to 1080p on delivery anyway: - TikTok → accepts 4K, serves 1080p max - Instagram → accepts 4K, serves 1080p max - YouTube Shorts → 1080×1920 optimal Accepting 4K but processing/serving max 1080p = correct strategy. Above 1080p = no visible benefit for social content, just wasted storage. **Platform max resolution reference (2026):**
PlatformMax ResolutionAspectMax File Size
TikTok1080×19209:16287MB mobile / 4GB web
Instagram Reels1080×19209:164GB
YouTube Shorts1080×19209:16—
Facebook Reels1080×19209:161GB
Twitter/X1280×1024any512MB
**Safe zone for 9:16 reels (1080×1920):** ``` ┌──────────────────┐ │░░░░ top 120px ░░░│ ← UI bar — avoid placing content here │ │ │ SAFE ZONE │ ← faces, text, important content here │ 860×1550px │ │ │ │░ right 180px ░░░░│ ← action buttons (like, comment, share) │░░ bottom 250px ░░│ ← caption area └──────────────────┘ ``` Watermark placement must respect safe zones. --- ### 6.3 Long Video Pipeline (HLS) **Threshold:** Duration ≥ 3 minutes → HLS Same as short video EXCEPT: ``` Format → HLS adaptive streaming (not MP4) No fast lane (too long, post stays PROCESSING) Post goes live only when fully processed Transcode to HLS: hls/360p/ → segments + 360p.m3u8 hls/720p/ → segments + 720p.m3u8 hls/1080p/ → segments + 1080p.m3u8 hls/master.m3u8 → points to all quality manifests Segment duration: 2 seconds Codec: H.264, AAC audio Container: MPEG-TS (.ts chunks) Still generated: preview_3s.mp4 ← for feed inline autoplay thumbnail.webp LQIP, BlurHash, dominantColor og_clean.webp, og_play.webp, og_preview.mp4 isReelEligible: false streamingFormat: HLS ``` **HLS Cache-Control:** ``` .ts chunks: Cache-Control: public, max-age=31536000 master.m3u8: Cache-Control: public, max-age=31536000 ``` --- ### 6.4 Digital Products Pipeline #### Upload Processing ``` Seller uploads product file ↓ Validate: - File type allowed? - Seller quota OK? - Product exists + owned by seller? ↓ ClamAV scan (DOUBLE scan — extra safety) ↓ SHA-256 checksum generated + stored ↓ Light processing by file type: PDF: Extract page 1 → preview image (72 DPI) Watermark "PREVIEW" across image Store: preview/preview.webp Video course (MP4): Extract first 2 minutes Transcode to 480p Watermark burned in Store: preview/preview.mp4 Audio (MP3/WAV/FLAC): Extract first 30 seconds Lower bitrate (96kbps) Store: preview/preview.mp3 3D Models (GLB/OBJ/FBX): Render thumbnail from multiple angles Store: preview/thumb_*.webp Images (stock art, PNG): Resize to low resolution (600px max) "PREVIEW" watermark burned in Store: preview/preview.webp Software/ZIP: No preview file Cover image only ↓ Store → nexgate-digital/products/{shopId}/{productId}/{fileId}/ original/ ← actual product (never exposed publicly) preview/ ← shown to everyone cover/ ← product listing thumbnail ↓ Update DB: READY Publish to RabbitMQ: DIGITAL_PRODUCT_READY → Main Backend ``` #### Download Flow (Buyer) ``` Buyer clicks Download ↓ POST /digital/download { productId, orderId, fileId } ↓ File Thunder validates: [1] orderId exists in DB? [2] orderId belongs to requesting user? [3] Order status = PAID? [4] download_count < download_limit? [5] Current time < expires_at? [6] User account in good standing? ↓ All pass: Generate single-use signed URL: 10-minute TTL Order bound Device bound Encrypted token Stored in Redis (single-use enforcement) ↓ Log download event: orderId, buyerId, timestamp IP address, deviceId, country ↓ Increment download_count ↓ Return signed URL to client Client downloads (chunked, resumable via byte-range) ↓ URL marked as used in Redis → 403 on reuse ``` **No forensic watermarking for now.** Planned for future phase when piracy becomes a real problem. --- ### 6.5 DM Attachments Pipeline Same core pipeline with different rules:
Social PostsDM Attachments
Bucketnexgate-publicnexgate-private
CDN✅ Cloudflare❌ No CDN
Access checkPublic/followersConversation membership
VariantsFull (all sizes)Thumb + original only
Watermark✅ On download❌ Never
Reel pool✅ If eligible❌ Never
Virus scan✅ Yes✅ Yes
URL typePermanent CDNSigned 5min TTL
ProcessingFullMinimal
**Access control for DM files:** ``` Client requests DM file URL ↓ File Thunder checks: Is requesting user a member of this conversation? YES → generate signed URL (5min TTL) NO → 403 Forbidden ``` --- ## 7. Watermarking Strategy ### When Applied - **Processing time** (server-side) — NOT at serve time, NOT client-side - Done ONCE when video is processed - Stored as separate `_watermarked` variant ### When Served
ScenarioVariant Served
Watching on NexGate (in-app/web)Clean variant
Downloading via NexGate download buttonWatermarked variant
Accessing via CDN URL directlyClean variant (unavoidable)
Sharing via savefromnet-style toolsClean variant (acceptable)
### Watermark Position — Moving (TikTok Style) ``` Jumps between 4 corners every 3 seconds: 0-3s → top left 3-6s → top right 6-9s → bottom right 9-12s → bottom left repeat for full video duration Logo: 80×80px, 60% opacity Username: directly below logo, same position Why moving: Static corner → easy to crop out ❌ Moving → appears everywhere → impossible to crop ✅ TikTok confirmed uses this strategy ``` ### App-based Watermark ``` clientApp = NEXGATE_ANDROID → "NexGate" logo clientApp = NEXGATE_IOS → "NexGate" logo clientApp = NEXGATE_WEB → "NexGate" logo clientApp = NEXGATE_LITE → "NexGate Lite" logo ``` Determined at upload time. Burned at processing time. Never changes. ### Pre-Watermarked Content (Uploaded from Other Platforms) **The Problem:** ``` User downloads TikTok video (has TikTok watermark) Re-uploads to NexGate NexGate adds NexGate watermark on top = two watermarks visible = bad UX 😂 ``` **What major platforms do:**
PlatformApproach
InstagramAlgorithm suppresses watermarked content from discovery
TikTokNo detection, just adds own watermark on top
YouTube ShortsSuppresses competitor watermarked content
Instagram head Adam Mosseri confirmed: videos with competitor watermarks receive significantly reduced algorithmic reach. **NexGate approach by phase:** ``` Phase 1 (launch — current): No watermark detection at all NexGate watermark burned on top of everything Pre-existing watermarks = overwritten/overlaid = zero complexity ✅ = same pipeline for all videos ✅ = double watermark acceptable at this stage ✅ Phase 2 (growth): Client-side warning before upload: "This video appears to have a watermark from another platform. Remove it for better reach on NexGate." User can proceed anyway = user educated, NexGate protected ✅ Phase 3 (scale): Server-side AI detection: Scan first frame for known platform logos Flag: hasExternalWatermark: true Suppress in reel pool (isReelEligible: false) Show post-level label: "Cross-posted content" = same approach as Instagram ✅ ``` **Note:** NexGate never BLOCKS pre-watermarked uploads. Only suppresses reach in discovery/recommendations. Content still visible to uploader's followers. ### Username Change Policy Old videos keep old username watermark (same as TikTok). Re-watermarking = future premium feature. --- ## 8. File Protection Layers
LayerMethodStops
1`Content-Disposition: inline` headerCasual right-click downloaders
2Dynamic URL via JS (never in HTML)HTML source inspection
3Signed expiring URLs (private content)URL sharing between users
4Session + device bindingCross-device URL reuse
5Disable right-click on player (frontend)Non-technical users
6Watermark burned on downloadCasual sharing without credit
7HLS chunking for videoDownload and reassembly
**Philosophy:** Goal is not impossible to download. Goal is not worth the effort for 99% of users + traceable if they do. --- ## 9. URL Strategy ### Public Content → Permanent CDN URLs ``` Thumbnail: https://media.nexgate.com/posts/usr_123/vid_789/thumb.webp Video (public, clean): https://media.nexgate.com/posts/usr_123/vid_789/720p_clean.mp4 OG preview (permanent): https://media.nexgate.com/og/vid_789/og_preview.mp4 Profile picture: https://media.nexgate.com/profiles/usr_123/avatar_400.webp ``` - No signing, no expiry - Cache-Control: public, max-age=31536000 - Cloudflare CDN caches globally ### Private Content → Signed Expiring URLs ``` DM attachment: https://media.nexgate.com/private/...? x-expires=1716305400 &x-sig=hmac_signature &x-uid=usr_xyz &x-sid=sess_abc Digital product download: https://media.nexgate.com/digital/...? x-expires=1716305400 &x-sig=hmac_signature &x-order=enc_orderId &x-device=device_abc ``` ### Signed URL Params Explained
ParamPurposeWithout It
`x-expires`URL dies after TTLPermanent link = unrevokable
`x-sig`HMAC tamper protectionAnyone can forge expiry
`x-uid`Bound to user sessionURL shareable between users
`x-sid`Bound to sessionWorks after logout
### Who Generates vs Validates - **Main Backend / File Thunder** → generates signed URL (knows secret key) - **Cloudflare CDN** → validates signature at edge (configured with same secret) - **MinIO** → never exposed to clients ### TTL by Content Type
ContentTTL
Stream URL (video)10 min
DM attachment5 min
Digital product download10 min (single-use)
Public CDN contentPermanent
OG preview videoPermanent
--- ## 10. OG (Open Graph) Strategy ### For Video Posts ```html ``` ### For Image Posts ```html ``` ### Play Button Behavior
PlatformPlay Button Source
WhatsAppAdded by WhatsApp (reads og:type)
TelegramAdded by Telegram
Twitter/XAdded by Twitter
SMS/EmailMust burn into og:image (og\_play.webp)
Unknown appsMust burn into og:image (og\_play.webp)
**NexGate serves og\_clean.webp to known platforms, og\_play.webp to unknown platforms** (detected via crawler User-Agent). ### OG Video URL — Why Permanent - WhatsApp/Telegram crawl on share, user opens hours later - Signed URL expired = "Error loading video" - Solution: permanent `og_preview.mp4` in CDN (360p, watermarked, small) - Access control = server-side (private posts return 403) --- ## 11. Compression & Transcoding ### Client-Side Intelligent Compression Before upload, client analyzes: ``` Inputs: resolution, bitrate, codec network type (WiFi/4G/3G/2G) device tier (high/mid/low) Decision: Already optimized + good network → upload direct Over-bitrated → compress Slow network → compress regardless Target (if compressing): Max 1080p CRF 18 (light — preserve quality for server) H.264 Never upscale on client either. ``` ### Server-Side Adaptive Transcoding After upload, FFprobe analyzes: ``` Extract: width, height, bitrate, fps, codec, duration Decision tree: height ≥ 1080 → generate 1080p, 720p, 360p height ≥ 720 → generate 720p, 360p height ≥ 480 → generate 480p, 360p height < 480 → keep original resolution Rules: NEVER upscale (never exceed input resolution) NEVER inflate (if output > input size → use original as-is: -c:v copy) CRF adapts to input quality ``` ### CRF Values (H.264)
VariantCRFUse
1080p21High quality
720p23Standard
480p25Medium
360p28Low / slow networks
OG preview30Very small file
### Video Codec Strategy
NowFuture
H.264 (universal support)AV1 (50% smaller, open source)
AAC audio 128kbpsOpus audio
MP4 containerCMAF container
### Image Format Strategy
InputOutput
JPEGWebP (quality 80%)
PNGWebP (quality 85%)
HEICWebP (quality 80%)
GIFWebP animated + MP4 loop
### Critical FFmpeg Flags ``` -movflags +faststart → metadata at start of file = video starts playing before fully downloaded = essential for web streaming -movflags frag_keyframe+empty_moov+faststart → fragmented MP4 = byte-range requests work = reel prefetch works ``` ### Generation Loss (Double Compression) - Client compresses lightly (CRF 18) → preserves quality - Server compresses per variant (CRF 23-28) → optimizes delivery - Two passes but first is near-lossless → acceptable tradeoff - Benefit: fast uploads on Tanzania mobile networks --- ## 12. BlurHash, LQIP & Dominant Color All three generated for every image and video thumbnail: ### Dominant Color ``` ImageMagick resize to 1×1 pixel → average color of entire image → stored as hex: "#2D7BC8" → shown instantly as CSS background (0ms, 0 bytes extra) ``` ### BlurHash ``` Algorithm: DCT-based image encoding Output: ~30 character string "LGF5]+Yk^6#M@-5c,1J5@[or[Q6." Library: blurhash-java Components: 4x3 (width x height) → decoded client-side to blurry placeholder → no network request → shows shape/composition of image ``` ### LQIP (Low Quality Image Placeholder) ``` ImageMagick resize to 10×10px Quality: 20% Convert to base64 string → embedded in feed JSON → client scales up (naturally blurry) → more accurate than BlurHash → no network request ``` ### Progressive Loading Stages ``` Stage 0 (instant, 0ms): dominantColor CSS background Stage 1 (instant, 0ms): BlurHash decoded → blurry shape Stage 2 (instant, 0ms): LQIP scaled up → accurate blur Stage 3 (CDN, 50-200ms): thumb.webp loads → sharp thumbnail Stage 4 (on demand): large.webp loads → full quality (on tap) ``` --- ## 13. Deduplication ### Hash-Based (Exact Match) ``` File uploaded ↓ Compute SHA-256 of raw file ↓ Check file_hashes table: HASH EXISTS + was CLEAN → skip processing entirely create reference to existing variants increment reference_count HASH NOT FOUND → process normally save hash + object_key to file_hashes ``` ### Benefits - Same viral video uploaded 1000x = stored once - Known clean files skip ClamAV scan (hash cache) - Known virus files rejected instantly (hash cache) - Storage savings up to 99% for viral content ### Reference Counting (Deletion) ``` User deletes file ↓ Decrement reference_count reference_count > 0 → keep file (others reference it) reference_count = 0 → delete from MinIO ``` ### Near-Duplicate Detection - Exact SHA-256 only (now) - Perceptual hashing (pHash) = future feature for copyright detection --- ## 14. Quota Management ### Enforcement Point **At presigned URL generation** — never after upload. ``` Request upload ↓ Atomic check: UPDATE user_storage_quota SET used_bytes = used_bytes + fileSize WHERE user_id = ? AND (used_bytes + fileSize) <= quota_bytes RETURNING used_bytes 0 rows affected → quota exceeded → reject with 403 1 row affected → quota reserved → proceed ``` ### What Counts Against Quota
ContentCharged
Social postsOriginal upload size only
Digital productsActual stored bytes (all variants)
### Plan Limits
PlanStorageMax ImageMax Video
FREE1GB20MB200MB, 5min
PRO20GB20MB1GB, 60min
BUSINESS100GB20MB2GB, unlimited
### Warning System ``` 80% used → email + in-app notification 95% used → in-app banner 100% used → uploads blocked, existing content untouched ``` ### Quota Release - On ABANDONED: immediate release (file never stored) - On soft delete: NOT released (file still exists) - On hard delete (30 days after soft): quota freed ### Quota Cache (Redis) ``` GET quota:usr_123 → used_bytes (cached) INCRBY quota:usr_123 fileSize → atomic increment Sync to PostgreSQL every 5 minutes ``` --- ## 15. Storage Lifecycle ### nexgate-raw ``` Auto-delete: 24 hours (MinIO lifecycle policy) Incomplete multiparts: aborted after 24 hours ``` ### nexgate-public (Social Content) ``` All variants: keep FOREVER (never delete) Move to cold storage tier based on access frequency: Hot (< 30 days or high views) → NVMe SSD Warm (30-365 days, moderate) → HDD Cold (1yr+, rare access) → Cold object tier CDN serves from cache after first request MinIO barely hit after content warms up Smart lifecycle (access-based, not just age): < 100 views/day for 30 days → move to warm < 10 views/day for 90 days → move to cold Viral spike on old content → auto-promote back to hot ``` ### nexgate-private ``` DM attachments: 1 year Audio temp files (for Rec Engine): 24 hours OR deleted by Rec Engine after transcription KYC documents: per legal requirement ``` ### nexgate-digital ``` Original product files: FOREVER (seller's product) Preview files: FOREVER Cover images: FOREVER ``` ### Deleted Content (Soft Delete) ``` User deletes post ↓ DB: soft delete (mark deleted, keep record) ↓ 30-day grace period: User can appeal/restore Legal holds possible CDN cache still valid (purge takes time) ↓ 30 days → hard delete from MinIO Cloudflare cache purge DB record updated Quota freed ``` --- ## 16. Abandoned Upload Handling ### The Problem ``` File uploaded to nexgate-raw ✅ Client never sends POST /media/confirm ❌ Reasons: app crash, network drop, user closed app → file stuck in raw bucket = wasted storage ``` ### Three-Layer Solution **Layer 1 — Client Confirm (Primary fast path)** ``` Client sends POST /media/confirm after upload → processing triggered immediately → best UX (instant) ``` **Layer 2 — Cleanup Job (Safety net, every 30 min)** ```java // Runs every 30 minutes // Finds PENDING records > 1 hour old // Checks if file exists in MinIO File exists → RECOVER (upload done, confirm missed) → update status UPLOADED → trigger processing → Tanzania network drop scenario handled ✅ File not exists → ABANDONED (user closed app) → update status ABANDONED → release quota reservation ``` **Layer 3 — Raw Bucket Lifecycle (Final safety)** ``` nexgate-raw auto-delete: 24 hours Incomplete multiparts aborted: 24 hours = catches anything missed by layers 1 + 2 ``` ### Cleanup Job Scaling ``` Add DB index: CREATE INDEX idx_media_cleanup ON media_files (created_at) WHERE status = 'PENDING' Process in batches of 100 Parallel MinIO checks (parallelStream) Redis distributed lock (prevent overlap): SET lock:cleanup_job "running" NX EX 2100 ``` ### fileId in Object Key (Enables Recovery) ``` objectKey = uploads/{userId}/{fileId}/original.mp4 ↑ fileId encoded in path → cleanup job extracts fileId from objectKey → can recover without client confirm ``` --- ## 17. Progress Tracking (SSE) ### Why SSE (not polling, not WebSocket)
PollingSSEWebSocket
DirectionBothServer→Client onlyBoth
ConnectionNew each timePersistentPersistent
Auto-reconnectManualBuilt-in ✅Manual
ComplexityMediumSimpleComplex
Tanzania networksWastefulResilient ✅Complex
Upload progress = one direction only → SSE is perfect fit. ### Status Flow ``` Redis key: upload_status:{fileId} PENDING → "Upload requested" UPLOADING → "Uploading..." UPLOADED → "File received" SCANNING → "Checking file..." PROCESSING → "Processing video..." LIVE_PARTIAL → "Almost ready!" + { available: ["360p"] } READY → "Your post is live!" + { liveUrl: "..." } FAILED → "Something went wrong" QUARANTINED → "File failed security check" ABANDONED → "Upload timed out" ``` ### Spring Boot SSE ```java @GetMapping( value = "/media/{fileId}/progress", produces = MediaType.TEXT_EVENT_STREAM_VALUE ) SseEmitter trackProgress(@PathVariable String fileId) { SseEmitter emitter = new SseEmitter(7_200_000L); // 2hr emitterRegistry.register(fileId, emitter); // Send current status immediately emitter.send(redis.get("upload_status:" + fileId)); return emitter; } ``` ### Multi-Instance Scaling ``` File Thunder (Server A) updates Redis File Thunder (Server B) has SSE connection → Redis Pub/Sub bridges them: PUBLISH media_status:{fileId} { status: "READY" } Server B receives → finds local emitter → pushes to client ``` --- ## 18. Communication Architecture ### Rule: No Direct HTTP Between Services ``` ❌ No REST API calls between File Thunder and Main Backend ❌ No webhooks ✅ RabbitMQ for internal operations ✅ Kafka for event streaming to external services ``` ### RabbitMQ (File Thunder ↔ Main Backend) **Exchange:** `nexgate.media` (topic exchange)
Routing KeyDirectionConsumer
`media.upload.request`MB → FTFile Thunder
`media.upload.url.ready`FT → MBMain Backend
`media.ready`FT → MBMain Backend
`media.live.partial`FT → MBMain Backend
`media.failed`FT → MBMain Backend
`media.quarantined`FT → MBMain Backend
`digital.product.ready`FT → MBMain Backend
**Example MEDIA\_READY event:** ```json { "event": "MEDIA_READY", "fileId": "vid_789", "ownerId": "usr_123", "type": "SHORT_VIDEO", "status": "READY", "variants": { "360p_clean": "posts/usr_123/vid_789/360p_clean.mp4", "720p_clean": "posts/usr_123/vid_789/720p_clean.mp4", "thumbnail": "posts/usr_123/vid_789/thumb.webp", "blurhash": "LGF5]+Yk^6#M@-5c,1J5@[or[Q6.", "lqip": "data:image/webp;base64,...", "dominantColor": "#1a1a2e", "preview_3s": "posts/usr_123/vid_789/preview_3s.mp4", "og_clean": "posts/usr_123/vid_789/og_clean.webp", "og_preview_video": "og/vid_789/og_preview.mp4" }, "isReelEligible": true, "streamingFormat": "MP4", "duration": 28, "aspectRatio": "9:16" } ``` ### Kafka (File Thunder → Rec Engine + Analytics) **Topic:** `content.new` ```json { "event": "FILE_READY", "fileId": "vid_789", "type": "SHORT_VIDEO", "ownerId": "usr_123", "media": { "duration": 28, "aspectRatio": "9:16", "qualityScore": 75, "isReelEligible": true, "hasAudio": true, "streamingFormat": "MP4" }, "userProvided": { "caption": "Check this Spring Boot setup", "hashtags": ["#tech", "#java", "#springboot"], "location": null }, "audioRef": { "bucket": "nexgate-private", "objectKey": "audio/vid_789.wav", "expiresAt": "2026-05-23T10:00:00Z" }, "thumbnailUrl": "https://media.nexgate.com/...", "processedAt": "2026-05-22T10:00:00Z" } ``` Rec Engine consumes this, fetches audio.wav, runs Whisper transcription, classifies content, deletes audio.wav. --- ## 19. CDN Strategy ### How It Works ``` First request (cache miss): Client → Cloudflare edge → MinIO → Cloudflare caches → serves Every subsequent request (cache hit): Client → Cloudflare edge → serves from cache MinIO never touched ``` ### Cache-Control Headers
ContentCache-ControlTTL
Post thumbnailspublic, max-age=315360001 year
Video MP4 variantspublic, max-age=315360001 year
HLS .ts chunkspublic, max-age=315360001 year
HLS .m3u8 manifestspublic, max-age=315360001 year
OG imagespublic, max-age=315360001 year
OG preview videopublic, max-age=315360001 year
Profile picturespublic, max-age=864001 day
API responsesno-storeNever cached
Private contentprivate, no-storeNever cached
### Cloudflare Phases
PhaseSetupCost
Now (launch)Cloudflare Free$0
GrowthCloudflare Pro$20/month
ScaleMigrate to Cloudflare R2 (zero egress)Pay storage only
### MinIO — Never Publicly Exposed ``` Public access: BLOCKED Only Cloudflare IPs can reach MinIO Client → media.nexgate.com (Cloudflare) → MinIO (internal) MinIO URL never seen by clients ``` --- ## 20. Shareable Links ### Share URL Format ``` Generated client-side (no server call): nexgate.com/reels/reel_abc123?ref=whatsapp ref values: whatsapp, telegram, twitter, sms, copy, email ``` ### Follow Prompt (On Link Click) ``` Sarah taps share link → NexGate loads post → Server looks up post owner from DB (postId → authorId) → Shows follow prompt for correct creator → No uid in URL (no manipulation possible) → No token needed (public content) ``` ### Analytics from ref Parameter ``` Track per share: Which platform shared from Click-through rate per platform Follow conversion rate per platform New signups from shares Watch completion rate from shares ``` --- ## 21. Technology Stack
ComponentTechnology
LanguageJava 21 (Spring Boot 3.x)
Video processingFFmpeg (global install) via Jaffree
Image processingImageMagick (global install) via IM4Java
BlurHashblurhash-java library
Virus scanningClamAV (Docker container, TCP :3310)
Object storageMinIO (S3 compatible)
CDNCloudflare
Message queue (ops)RabbitMQ
Event streamingKafka
Cache / progressRedis
DatabasePostgreSQL
Resumable uploadsTUS protocol
SecretsHashiCorp Vault (vault.qbitspark.com)
ContainerDocker + Docker Compose
Progress deliverySSE (Server-Sent Events)
### Why FFmpeg + ImageMagick Global Install ``` Both are CLI tools (not daemons) Called on demand via ProcessBuilder Jaffree/IM4Java call /usr/bin/ffmpeg and /usr/bin/convert Same filesystem = works perfectly No network overhead Simpler than separate containers ClamAV = separate Docker container because: Runs as daemon (long-running process) Official Docker image exists Connects via TCP = natural for Docker Auto-updates virus definitions ``` --- ## 22. Database Schema ```sql -- Core media file record media_files file_id UUID PK owner_id UUID -- accountId directory ENUM -- PROFILE, POSTS, STORIES, EVENTS, SHOPS, MESSAGES, PRODUCTS original_name TEXT object_key TEXT -- raw upload path in nexgate-raw mime_type TEXT file_size BIGINT status ENUM -- PENDING, UPLOADING, UPLOADED, SCANNING, PROCESSING, -- LIVE_PARTIAL, READY, FAILED, QUARANTINED, ABANDONED, EXPIRED variants JSONB -- all processed variant paths metadata JSONB -- dimensions, duration, fps, codec scan_result TEXT -- "clean" or virus name hash TEXT -- SHA-256 for deduplication is_reel_eligible BOOLEAN streaming_format ENUM -- MP4, HLS created_at TIMESTAMPTZ ready_at TIMESTAMPTZ -- Content signals for Rec Engine media_content_signals file_id UUID FK has_audio BOOLEAN audio_ref TEXT -- nexgate-private/audio/{fileId}.wav dominant_color TEXT -- hex color blurhash TEXT lqip TEXT -- base64 aspect_ratio TEXT -- "9:16", "16:9", "1:1", "4:5" quality_score INT -- 0-100 width INT height INT duration_seconds DECIMAL -- User storage quota user_storage_quota user_id UUID PK plan ENUM -- FREE, PRO, BUSINESS quota_bytes BIGINT used_bytes BIGINT file_count INT updated_at TIMESTAMPTZ -- Per bucket breakdown user_storage_breakdown user_id UUID bucket TEXT used_bytes BIGINT file_count INT updated_at TIMESTAMPTZ -- Deduplication hash index file_hashes hash TEXT PK -- SHA-256 object_key TEXT -- canonical storage path file_size BIGINT mime_type TEXT reference_count INT created_at TIMESTAMPTZ -- Variant storage class tracking media_variants file_id UUID FK quality TEXT -- "360p", "720p", "1080p", "thumb", etc status ENUM -- HOT, WARM, COLD, FROZEN storage_class TEXT object_key TEXT file_size BIGINT last_accessed TIMESTAMPTZ access_count INT -- Digital products digital_product_files file_id UUID PK product_id UUID FK file_type ENUM -- MAIN, PREVIEW, COVER original_name TEXT mime_type TEXT file_size BIGINT checksum TEXT -- SHA-256 object_key TEXT status ENUM -- PROCESSING, READY, FAILED created_at TIMESTAMPTZ -- Digital orders digital_orders order_id UUID PK product_id UUID FK buyer_id UUID FK seller_id UUID FK amount_paid DECIMAL currency TEXT status ENUM -- PAID, REFUNDED, DISPUTED download_count INT DEFAULT 0 download_limit INT expires_at TIMESTAMPTZ purchased_at TIMESTAMPTZ -- Download audit log digital_download_logs log_id UUID PK order_id UUID FK buyer_id UUID FK file_id UUID FK downloaded_at TIMESTAMPTZ ip_address TEXT device_id TEXT user_agent TEXT country TEXT download_number INT -- which download (1st, 2nd, etc) ``` **Critical indexes:** ```sql -- Cleanup job performance CREATE INDEX idx_media_cleanup ON media_files (created_at) WHERE status = 'PENDING'; -- Hash lookup for deduplication CREATE UNIQUE INDEX idx_file_hashes_hash ON file_hashes (hash); -- Owner lookup CREATE INDEX idx_media_files_owner ON media_files (owner_id, status, created_at DESC); ``` --- ## 23. Docker & Infrastructure ### File Thunder Dockerfile ```dockerfile FROM eclipse-temurin:21-jdk-alpine # Install FFmpeg (global, called by Jaffree) RUN apk add --no-cache ffmpeg # Install ImageMagick with WebP support (called by IM4Java) RUN apk add --no-cache \ imagemagick \ imagemagick-webp \ imagemagick-heic # Copy Spring Boot jar COPY target/file-thunder.jar app.jar ENTRYPOINT ["java", "-jar", "/app.jar"] ``` ### Docker Compose (Relevant Services) ```yaml file-thunder: build: ./file-thunder ports: ["8081:8081"] environment: SPRING_DATASOURCE_URL: jdbc:postgresql://postgres:5432/nexgate SPRING_REDIS_HOST: redis RABBITMQ_HOST: rabbitmq KAFKA_BOOTSTRAP_SERVERS: kafka:9092 MINIO_ENDPOINT: http://minio:9000 CLAMAV_HOST: clamav CLAMAV_PORT: 3310 VAULT_ADDR: https://vault.qbitspark.com depends_on: [postgres, redis, rabbitmq, kafka, minio, clamav] clamav: image: clamav/clamav:stable ports: ["3310:3310"] volumes: - clamav_data:/var/lib/clamav # Auto-updates virus definitions via freshclam ``` ### VPS Resource Allocation ``` File Thunder: 4 cores, 16GB RAM (FFmpeg is CPU intensive) ClamAV: 1 core, 2GB RAM ``` --- ## 24. What File Thunder Does NOT Do ``` ❌ ML / AI content classification ❌ Speech-to-text (Whisper) — that's Rec Engine ❌ Image visual analysis (CLIP) — that's Rec Engine ❌ Feed computation or ranking ❌ Social graph operations (follow/unfollow) ❌ Payment processing ❌ User authentication / authorization ❌ Push notifications ❌ Search indexing ❌ Business logic (post creation, comments, likes) ❌ Fan-out to followers ❌ Recommendation logic ``` File Thunder extracts audio.wav and stores temporarily. The Rec Engine fetches it, runs Whisper, classifies content, and deletes the audio file. File Thunder never knows what's in the audio. --- ## Summary — File Thunder in One Sentence > File Thunder receives raw files, validates and secures them via ClamAV, processes them into optimized delivery variants using FFmpeg and ImageMagick, stores everything in MinIO across four isolated buckets, serves public content efficiently via Cloudflare CDN, tracks progress via SSE and Redis, communicates with the Main Backend via RabbitMQ, and publishes content signals to Kafka for downstream services — without ever performing any business logic, ML, or social operations. --- *File Thunder Architecture Guide v1.0 — NexGate / QBIT SPARK* *Document compiled: May 2026* # NextGate Media Delivery — Redesign Target: a safe, scalable media layer for the feed, modelled on patterns observed in TikTok's delivery (signed expiring URLs, CDN edge, variant matrix, batch minting) and mapped onto NextGate's existing media engine (MinIO + RabbitMQ + workers + Redis). --- ## 1. What's wrong today Observed in the current `GET /posts` (feed) response.
\#ProblemWhy it matters
1`originalUrl` is a raw, permanent, **unsigned** link to `s3.nexgate.co`Post visibility is enforced at the API only. A PRIVATE post's bytes stay reachable at a permanent public URL → media bypasses access control entirely.
2Storage origin exposed, no CDNEvery view hits MinIO directly. High latency from East Africa, full egress cost, zero edge cache.
3`thumbnailUrl: null`, `placeholderBase64: null`Full-res images (e.g. 1536×2048) served into feed thumbnail slots. Heavy on expensive mobile data. Blurhash field exists but unused.
4Video = raw `.mp4`, `width/height/duration` = nullNo transcode, no poster frame, no ladder, no HLS. No adaptive playback.
5Per-user path prefix doesn't match ownere.g. joshdoe's avatar stored under eddiesmith19's `stg-acc-{uuid}` prefix. Leaks UUIDs and misattributes ownership.
6`dev.s3.nexgate.co` leaking into staging dataEnvironment bleed in a live feed response.
What to KEEP (already good): batch-hydrated feed (one call, all posts + nested quotes + engagement), permanent `shareCode` short IDs, graceful deleted-post handling (`unavailable: true`), the rich post data model. The fix is scoped to the **media delivery layer**, not the data model. --- ## 2. Core principle Two URLs, two lifespans (same split TikTok uses): - **Permanent identity** — the post `id` / `shareCode` and the storage `objectKey`. Stored in DB. Shareable forever. - **Ephemeral delivery ticket** — a signed, expiring CDN URL. Minted on every feed build. Never stored. Rule: **store the `objectKey`, never store the signed URL.** --- ## 3. Three delivery tiers
TierContentAuth gate?URL signingExpiryCDN cache
1 — public assetavatars, thumbnailsnonone (or very long)n/a / longaggressive, long TTL
2 — public content, protected deliverypublic post images/videonoyes (short-ish)hourscareful (see §6)
3 — private contentprivate/followers posts, paid digitalyes (ownership/visibility check)yesminutesnone / private
The only code difference between tier 2 and tier 3 is **whether the permission check runs before minting**. Same minting call underneath. --- ## 4. Storage layer changes ### 4.1 Buckets - `public` — tier 1 assets (avatars, generated thumbnails) - `private` — tier 2 + tier 3 post media - `digital` — paid products (tier 3, strictest) Shared buckets, **owner-prefixed object keys**, prefix must match the real owner. Opaque object keys — no user UUID needed in the public URL. ``` private/{ownerId}/{mediaId}/original.jpg private/{ownerId}/{mediaId}/thumb_320.webp private/{ownerId}/{mediaId}/thumb_720.webp ``` ### 4.2 DB: store keys, not URLs Persist the storage coordinates and processing state, not a full URL. ```java // ── MediaEntity (delivery-relevant fields) ── private String bucket; // "private" private String objectKey; // "ownerId/mediaId/original.jpg" (opaque, owner-correct) private MediaType mediaType; // IMAGE | VIDEO private Integer width; private Integer height; private Double duration; // video only private String placeholder; // blurhash / LQIP — fills the empty placeholderBase64 private MediaStatus status; // PROCESSING | READY | FAILED // variants: thumb sizes, video rungs, formats — JSONB private String variantsJson; // {"thumb":["320","720"],"formats":["webp"],"rungs":[...]} ``` --- ## 5. Delivery layer changes ### 5.1 CDN in front of MinIO Serve everything through `media.nexgate.co` (Cloudflare) → MinIO origin. **Never** return `s3.nexgate.co` (or `dev.s3…`) in an API response again. ### 5.2 Minting helper (SDK does the crypto) ```java // ── Presigned URL minting (MinIO SigV4) ── public String mintUrl(String bucket, String objectKey, int seconds) { return minioClient.getPresignedObjectUrl( GetPresignedObjectUrlArgs.builder() .method(Method.GET) .bucket(bucket) .object(objectKey) .expiry(seconds, TimeUnit.SECONDS) // → X-Amz-Expires + X-Amz-Signature .build()); // returned host is rewritten to media.nexgate.co at the edge / via MinIO public endpoint } ``` ### 5.3 Feed response shape (mint at build time, batch) Mirror TikTok's batch mint: build the page, mint every media URL inline, ship once. No per-item round trip from the client. ```jsonc "media": [{ "id": "014ab4eb-...", "mediaType": "IMAGE", "url": "https://media.nexgate.co/.../original.jpg?X-Amz-Expires=3600&X-Amz-Signature=...", "thumbnailUrl": "https://media.nexgate.co/.../thumb_720.webp?...sig...", "placeholderBase64": "LEHV6nWB2yk8...", // blurhash, no longer null "width": 945, "height": 2048, "expiresIn": 3600 }] ``` ```java // ── In the feed mapper ── String url = mintUrl(m.getBucket(), m.getObjectKey(), tier2Seconds); // hours String thumb = mintUrl(m.getBucket(), thumbKey(m, 720), tier2Seconds); // tier 3 (private/paid): run accessService.canAccess(user, m) first, use short expiry ``` --- ## 6. CDN + signed-URL gotcha Do not let the CDN cache a per-user signed URL under a key that includes the signature, or one user's link can be served to another. Options: - cache key = path only (strip query/signature), short edge TTL for tier 2; or - tier 1 (avatars/thumbs) = truly public objects, cache hard; tier 2/3 originals = bypass shared cache or use very short edge TTL. Tier 1 assets are the ones you cache aggressively. Tier 3 never shares cache. --- ## 7. Variant + transcode pipeline (reuse existing media engine) On upload (presigned PUT direct to MinIO → enqueue job → workers → webhook back): - **ClamAV** scan first; reject on hit. - **Image worker**: generate `thumb_320` + `thumb_720` (WebP), compute blurhash → `placeholder`, read width/height. Write variants to JSONB, set status READY. - **Video worker**: extract poster frame (tier-1 thumbnail), read width/height/duration, transcode to 1–2 MP4 rungs now (e.g. 540p + 720p), HLS later when analytics/scale justify it. - **Redis** tracks job state; **webhook** flips `MediaEntity.status` PROCESSING → READY. Until a media row is READY, the feed returns the blurhash placeholder + a `processing` flag instead of a broken/raw link. --- ## 8. Migration sequence (non-breaking) 1. **CDN** — stand up `media.nexgate.co` → MinIO, dual-serve. Stop emitting `s3.nexgate.co`. 2. **Schema** — add `bucket`/`objectKey`/`status`/`placeholder`/`variantsJson`. Backfill `objectKey` by parsing existing `originalUrl`; fix owner prefix + drop `dev.` rows here. 3. **Mint** — change feed to mint from `objectKey` instead of returning raw `originalUrl`. Roll out tier 1 + tier 2 first (public) so nothing visibly breaks. 4. **Backfill variants** — run image/video workers over existing media; populate thumbs + blurhash. 5. **Lock the bucket** — remove public-read from MinIO; force all access through signed CDN URLs. At this point the visibility-bypass hole (problem #1) is closed. 6. **Tier 3** — gate private/followers/paid media behind `canAccess` + short expiry. Closing problem #1 happens at step 5; everything before it is safe groundwork. --- ## 9. Security checklist ("safe" part) - [ ] No storage origin host (`s3.nexgate.co`, `dev.s3…`) in any API response. - [ ] No user UUID in public URLs (opaque object keys). - [ ] Private/followers post media served tier 3 (auth gate + short signed URL). - [ ] MinIO buckets are **not** public-read once migration completes. - [ ] Owner prefix on every key matches the real owner. - [ ] Dev/staging/prod storage hosts strictly separated; no cross-env URLs. # File Thunder — Architecture & Design Guide V2 ### NexGate Media Engine | Version 2.0 > v2 integrates the media-security refinements worked out in design review. Every change from v1 is tagged inline as **\[v2 FIX\]** so it can be reviewed individually. Section 0 lists all of them in one place — read that first. --- ## 0. What changed in v2 (read this first) These are the careful corrections. Each is explained in full in its section. 1. **Signing is three tiers, not two.** "Public = unsigned, private = signed" was wrong. Correct model: trivial public assets unsigned; *public content* (full images/videos) **signed even though public**; private/paid signed + auth. → §8 2. **Public content gets signed too.** A signed URL is anti-scrape / anti-hotlink, *not* only access control. TikTok signs even public videos. → §8, §14 3. **The visibility-bypass hole must be closed.** In v1, media URLs were permanent and unsigned, so a PRIVATE post's bytes were reachable by anyone with the link — the post was hidden by the API but the file was not. Fixed by locking the bucket (no public-read) and routing everything through signed CDN URLs. → §14 4. **Backend signs, CDN validates, they never talk at sign time.** Signing is pure math on `(path + expiry + secret)`. No file, no CDN contact. CDN fetches from MinIO lazily on first view. → §9 5. **Domain is `media.nexgate.co`, not `.com`.** v1 mixed both. One domain everywhere. → §10 6. **Object keys must be owner-correct.** v1 had one user's avatar stored under another user's prefix. The owner segment in the key must match the real owner. → §3 7. **Store the `objectKey`, never the signed URL.** Signed URLs are minted per request and thrown away. Never persisted in the DB or feed cache. → §13, §22 8. **MP4 and HLS use different token strategies.** Short MP4 = one signed URL. Long HLS = dual-token (short manifest + long segments) or prefix-signing, so playback never dies mid-video. → §11 9. **Single-use is for downloads only, never streams.** A stream is many requests; single-use would break playback. Streams use short expiry + user-binding instead. → §11, §12 10. **Watermarked file is NEVER in the feed response.** Feed = clean playback URLs only. Watermarked download comes from a *separate* endpoint, minted on tap. → §12, §13 11. **Cache gotcha: never cache a signed object keyed by its signature.** A cached object can outlive its stamp; private/paid content must bypass shared cache. → §10, §11 12. **Bot gate is the missing half of anti-scraping.** Signed URLs stop *permanent* theft; the bot gate stops the *act* of mass-pulling. v1 had no bot gate section. → §14 13. **Open decision parked: NextGate's moat (content vs commerce)** drives how hard to stamp public media and how heavy the bot gate needs to be. → §30 --- ## 1. What is File Thunder? File Thunder is NexGate's dedicated media processing engine — a standalone Spring Boot microservice responsible for all file and media operations across the platform. **Core principle:** The Main Backend never touches raw files. It delegates all media operations to File Thunder and acts as a thin client. **Responsibilities:** receive uploads, validate (type/size/quota), virus scan, transcode, generate variants (thumbnails, WebP, HLS, MP4), watermark, store in MinIO, serve via CDN, mint signed delivery URLs, publish media-ready events. **Not:** AI/ML, recommendation, social graph, payments, auth, business logic. --- ## 2. The Four Wheels ``` FILE THUNDER ┌───────────┬───────────┬───────────┬───────────┐ │ FFmpeg │ ImageMagic│ ClamAV │ MinIO │ │ (video) │ (images) │ (security)│ (storage) │ └───────────┴───────────┴───────────┴───────────┘ transcode resize + virus object HLS, w/mark WebP, blur scan store + CDN ``` - **FFmpeg (Jaffree)** — transcode, HLS segmenting, thumbnail/best-frame, 3s preview, watermark overlay, blur-pad, fMP4, format conversion, audio extract/normalize. - **ImageMagick (IM4Java)** — resize variants, format→WebP, EXIF strip (privacy), auto-orient, dominant color, LQIP, OG image, GIF→WebP, PDF→preview. Plus `blurhash-java`. - **ClamAV** — virus scan every upload (TCP `clamav:3310`), hash-cache, digital products scanned twice. - **MinIO** — S3-compatible storage, presigned direct uploads, origin for Cloudflare, multipart + byte-range, lifecycle policies. **Never publicly exposed** (only Cloudflare + File Thunder reach it). --- ## 3. Storage Architecture (MinIO Buckets) **Rule: 4 buckets total. Never one bucket per user.** ``` nexgate-raw/ ← temporary upload landing (auto-delete 24h) nexgate-public/ ← social content (avatars, posts, stories, events, shops) nexgate-private/ ← DMs, documents, KYC, temp audio nexgate-digital/ ← purchase-gated products ``` ### Object Key Pattern ``` {bucket}/{domain}/{ownerId}/{entityId}/{fileId}/{variant} nexgate-public/posts/usr_123/post_456/file_789/720p_clean.mp4 nexgate-private/messages/conv_abc/file_789/original.jpg nexgate-digital/products/shop_xyz/prod_123/file_789/original/file.pdf ``` > **\[v2 FIX\] Owner-correct keys.** The `{ownerId}` segment MUST be the real owner of the file. v1 had cases where one user's avatar lived under a different user's prefix — that leaks identity and misattributes ownership. The owner in the key = the owner in the DB. Keys are otherwise opaque (no usernames, no PII). ### nexgate-public layout (unchanged from v1) ``` profiles/{userId}/{fileId}/ avatar_400.webp, avatar_150.webp, avatar_50.webp, cover.webp posts/{userId}/{fileId}/ {360,720,1080}p_clean.mp4, {360,720,1080}p_watermarked.mp4, preview_3s.mp4, hls/master.m3u8, hls/{360,720,1080}p/seg_*.ts, thumbnail.webp, og_clean.webp, og_play.webp, og_preview.mp4 stories/{userId}/{fileId}/ 720p_clean.mp4, thumbnail.webp events/{accountId}/{eventId}/{fileId}/ banner.webp, banner_mobile.webp, banner_thumb.webp shops/{shopId}/{productId}/{fileId}/ large.webp, medium.webp, thumb.webp ``` *(See §12 for the watermarked-variant storage decision — store one rung, not all three.)* ### nexgate-private / nexgate-digital As v1: messages, documents, kyc, temp audio (private); products with `original/` (never exposed) + `preview/` + `cover/` (digital). --- ## 4. Accepted File Formats - **Images** accept JPEG/PNG/HEIC/WebP/GIF; reject BMP/TIFF/RAW/SVG (SVG = security risk). Output always **WebP**. - **Videos** accept MP4/MOV/MKV/WEBM/AVI/3GP; reject WMV/FLV/VOB. Output **H.264 MP4** (short) or **HLS H.264** (long). - **Digital** PDF, DOCX/XLSX/PPTX, GLB/OBJ/FBX/STL, MP3/WAV/FLAC, ZIP/RAR, PSD/AI, PNG. --- ## 5. Upload Flow ``` [1] Client-side intelligent compression (max 1080p, CRF 18 light, HEIC→JPEG) [2] POST /media/upload-request { fileName, fileSize, mimeType, directory, clientMeta } [3] File Thunder: validate MIME + size + quota (atomic SQL) → DB record PENDING → presigned MinIO URL (nexgate-raw) → { fileId, uploadUrl, expiresIn: 1800 } [4] Client uploads directly to MinIO (TUS resumable / multipart >5MB) [5] Client confirms POST /media/confirm { fileId } (cleanup job recovers if missed) [6] Processing pipeline starts ``` ``` client compress │ ▼ POST upload-request ──▶ validate + quota ──▶ presigned URL │ ▼ upload direct to MinIO (raw) │ ▼ confirm ──▶ ClamAV ──▶ dedup ──▶ process ──▶ READY │ ▼ RabbitMQ media.ready ──▶ Main Backend ``` **Quota** checked at presigned-URL time (atomic `UPDATE ... WHERE used+size <= quota`). **Duration** checked AFTER upload via FFprobe (client value is a hint, not trusted) — over limit → delete from raw, release quota, 403 `DURATION_EXCEEDED`. ### Per-Upload Limits
PlanImageShort VideoLong VideoDigital VideoDuration
FREE20MB200MB❌❌5 min
PRO20MB500MB2GB5GB60 min
BUSINESS20MB2GB5GB20GBUnlimited
--- ## 6. Processing Pipelines (summary) All pipelines: **ClamAV scan → SHA-256 dedup check → process → store → delete raw → DB READY → publish events.** Watermarking and all placeholders are generated **here, at processing time**, never at serve time. ### 6.1 Image Auto-orient → strip EXIF (privacy) → NSFW check → dominant color + LQIP + BlurHash → size variants per content type (Post: 1600/800/300; Profile: 400/150/50; Product: 1000/500/200 square; Event: 1200×630/800×420) → WebP → OG image → store public → READY. ### 6.2 Short Video (MP4, duration < 3 min) FFprobe → adaptive transcode decision (never upscale, never inflate) → **fast lane** (quick 360p → `LIVE_PARTIAL`) → full processing: - **Clean** variants `{360,720,1080}p_clean.mp4` (faststart). Non-9:16 reel-eligible → **blur-pad** to 9:16 (scale-fill+blur background, scale-fit sharp foreground, overlay). - **Watermarked** variant(s) — **moving** watermark (jumps corners every 3s; 80×80 logo 60% opacity + username; app-based label from `clientApp`). **\[v2 FIX\] store one download rung (720p), not all three** (§12). - Extras: `thumbnail.webp` (best-frame scored) + its LQIP/BlurHash/dominantColor, `preview_3s.mp4` (360p muted watermarked), `og_clean.webp`, `og_play.webp`, `og_preview.mp4`. - fMP4 (`frag_keyframe+empty_moov+faststart`) for byte-range. - Extract `audio.wav` → nexgate-private (Rec Engine). - READY: `isReelEligible` (<3min), `streamingFormat: MP4`, variants JSONB, aspectRatio, qualityScore, duration. ### 6.3 Long Video (HLS, duration ≥ 3 min) Same as short EXCEPT: HLS adaptive (no fast lane — stays PROCESSING until done). `hls/{360,720,1080}p/` segments + `master.m3u8`. Segment = 2s, H.264 + AAC, MPEG-TS. Still makes preview\_3s, thumbnail, placeholders, OG. `isReelEligible: false`, `streamingFormat: HLS`. ### 6.4 Digital Products Double ClamAV scan → SHA-256 checksum → light preview by type (PDF p1 watermarked, video first 2min 480p watermarked, audio 30s 96kbps, 3D multi-angle thumbs, image 600px watermarked) → store `original/` (never exposed) + `preview/` + `cover/` → READY. ### 6.5 DM Attachments nexgate-private, **no CDN**, conversation-membership check, thumb+original only, no watermark, no reel pool, signed 5-min URLs. Virus scanned like everything. --- ## 7. Watermarking Strategy - **Applied at processing time, once.** Stored as separate `_watermarked` variant. Never at serve time, never client-side. - **Moving watermark** (corners every 3s) — static is croppable, moving is not. - **Served:** in-app playback → **clean** variant; download button → **watermarked** variant (§12). Direct CDN URL → clean (unavoidable; accepted, same tradeoff every platform has). - Pre-watermarked uploads (from other platforms): Phase 1 = no detection, just overlay ours. Phase 3 = AI logo detection → suppress reach (Instagram model), never block. - Username change: old videos keep old username (like TikTok). > **\[v2 FIX\] Honest scope.** A watermark does NOT prevent download — it survives download. Its job is attribution + traceability, not prevention. The clean file is reachable by a determined ripper via the playback URL; that's accepted. Goal = "not worth the effort for 99% + traceable if they do." --- ## 8. The Three Delivery Tiers **\[v2 FIX — core correction\]** v1 assumed two levels (public unsigned / private signed). Correct model is **three**:
TierContentLogin to view?URL signed?ExpiryCDN cache
1 — trivial publicavatars, thumbnails, posters, blur previewsno**no**n/aaggressive, 1y
2 — public contentfull post images, video rungsno**YES**hourscareful (§10)
3 — private / paidDM media, private posts, digital products, KYC**yes**YESminutesnone / private
``` ┌──────────────────────────────────────────────┐ │ TIER 1 trivial public │ │ avatars, thumbs → unsigned, cache 1y │ ├──────────────────────────────────────────────┤ │ TIER 2 public content │ │ full images/video → SIGNED, hrs TTL │ ├──────────────────────────────────────────────┤ │ TIER 3 private / paid │ │ DM, private, digital → SIGNED + auth + x-uid │ └──────────────────────────────────────────────┘ only diff T2 vs T3 = auth check before minting ``` The key insight: **"public to view" and "signed URL" are independent.** Tier 2 content is open to watch by anyone (no login), but the *file* is still signed — because signing is **anti-scrape / anti-hotlink**, not access control. This is exactly TikTok's model (public video, yet the raw file link expires and 403s). The only difference between tier 2 and tier 3 in code is **whether an auth/visibility check runs before minting** — the signing itself is identical. > Tier 1 is the only unsigned tier. Anything a scraper would actually *want* (full media) is tier 2+, i.e. signed. --- ## 9. Signed URLs — how they actually work **\[v2 — new, the mechanism\]** A signed URL = a normal link with a **stamp** made from three things: ``` stamp (x-sig) = HMAC(secret, path + expiry) ``` - `path` — the object key being requested - `expiry` (`x-expires`) — unix time the link dies - `secret` — a key known ONLY to File Thunder/backend AND the CDN. Never in the URL, browser, or JSON. Lives in Vault. **Who does what, and when — three separate moments:** 1. **Upload time:** File Thunder makes the variants, stores them in MinIO. CDN not involved. 2. **Feed-build time (signing):** backend computes `x-sig` from `(path + expiry + secret)`. **Pure string math — no file opened, no CDN contacted, no storage touched.** It's just writing a stamped link. (You can sign a path whose bytes don't even exist yet — the math only sees the string.) 3. **Playback time (serving):** the phone requests the stamped URL → CDN re-computes the stamp with its copy of the secret. Match + not expired → serve. First viewer: CDN lazily fetches the file from MinIO and caches it. Later viewers: served from cache. ``` upload feed build playback ┌────────┐ store ┌──────────┐ stamped ┌──────────┐ │ File │ ───────▶ │ Backend │ ─ link ──▶ │ CDN │ │ Thunder│ to MinIO │ signs │ │ validates│ └────────┘ └──────────┘ └────┬─────┘ secret ▲ │ first view Vault └──────── same ────────┘ fetch+cache secret from MinIO ``` **Tamper resistance** (why a stripped/edited link fails): - No sig → CDN rejects (paths require a valid sig) → 403. - Fake sig → recompute doesn't match → 403. - Edited expiry → sig was computed *from* the old expiry, no longer matches → 403. - Swapped path → sig bound to original path → 403. - Untouched + in time → serve. After `x-expires` → even the real link 403s. Forging anything requires the **secret**, which never leaves backend + CDN. Protect the secret (Vault) and the whole scheme holds. **You don't write the HMAC.** The MinIO SDK (or CDN signer) does it in one call: ```java // ── Mint a signed delivery URL ── String url = minioClient.getPresignedObjectUrl( GetPresignedObjectUrlArgs.builder() .method(Method.GET) .bucket(bucket) .object(objectKey) .expiry(seconds, TimeUnit.SECONDS) // → x-expires + x-sig .build()); ``` --- ## 10. URL Strategy ### Tier 1 — trivial public → permanent, unsigned, cache hard ``` https://media.nexgate.co/profiles/usr_123/avatar_400.webp https://media.nexgate.co/posts/usr_123/vid_789/thumbnail.webp ``` `Cache-Control: public, max-age=31536000`. Scraping these gains nothing. ### Tier 2 — public content → signed, short-ish expiry ``` https://media.nexgate.co/posts/usr_123/vid_789/720p_clean.mp4?x-expires=1780297200&x-sig=8f3a9c ``` ### Tier 3 — private / paid → signed + user-bound + auth check first ``` https://media.nexgate.co/private/messages/conv_abc/file_789/original.jpg ?x-expires=...&x-sig=...&x-uid=usr_xyz ``` > **\[v2 FIX\] Domain is `.co`.** All URLs use `media.nexgate.co`. v1 mixed `.com`/`.co` — that produces broken links. One domain everywhere. > **\[v2 FIX\] Cache-key gotcha.** Never let the CDN cache a signed object using a cache key that includes `x-sig`/`x-uid` — you'd fragment cache per user, and a cached object can outlive its stamp (origin checks expiry, but a cache hit never reaches origin). Rules: tier 1 cache hard; tier 2 cache by **path only** (strip query from cache key), short edge TTL; tier 3 **never** on shared cache. ### Signed URL params
ParamPurpose
`x-expires`link dies after TTL
`x-sig`HMAC tamper protection
`x-uid`bound to user (tier 3) — shared link won't work for someone else
### Who generates / validates Backend (has secret) **generates**. Cloudflare edge (same secret) **validates**. MinIO never seen by clients. --- ## 11. Streaming Token Strategy: MP4 vs HLS **\[v2 — new\]** Expiry is checked **when each request opens**, not continuously. An in-flight transfer completes even if the clock passes mid-download. The two formats differ because of how many requests a single playback makes. ``` SHORT MP4 LONG HLS one request hundreds of requests ┌──────────┐ ┌──────────┐ ┌──┬──┬──┬──┐ │ 720p.mp4 │ ← 1 signed URL │master.m3u│─▶│ts│ts│ts│..│ └──────────┘ few-hr TTL └──────────┘ └──┴──┴──┴──┘ opens once, short token long token rides to end (~1 min) (> video len) per-session one, shared ``` ### Short video (MP4) — one signed URL One file = one request (or a few byte-ranges, all opened up front). A few-hour expiry is plenty; the in-flight request completes, so mid-play expiry basically never happens. → mint one signed URL per rung, few-hour TTL. **No dual-token, no single-use.** ### Long video (HLS) — dual-token (or prefix-signed) HLS = hundreds of segment requests (one every ~2s), so expiry is re-checked constantly. A single short token would die mid-video. Industry-standard fix (Google Media CDN / CloudFront): - **`master.m3u8`** → **short** token (~1 min). Per-session entry gate. Fetched once at start. - **child manifests + `.ts` segments** → **long** token (> full video length, up to ~1 day), carried by **signed cookie** or **path-prefix signing** (sign the folder, not each segment). - Segments stay **cacheable** (same bytes for everyone); only the manifest is per-session. Preserves cache hit rate. > At our stage on Cloudflare Free/Pro, do the simpler version: backend mints a **long prefix-signed** token for the whole `hls/` folder when the post loads. Graduate to true edge dual-token (short→long exchange) if/when we move to a heavier CDN. ### Single-use — downloads ONLY, never streams A stream is many requests; making the token single-use would 403 the viewer one segment in. **Streams use short expiry + `x-uid` binding instead.** Single-use (Redis) is correct only for one-file-one-request **downloads** (§12, digital products). ### Edge case (both formats) User pauses past expiry, then seeks → player fires a fresh request with a dead URL → 403. **Client safety net:** catch the 403, silently re-fetch a fresh URL from backend, resume.
Short MP4Long HLS
Requests / playone (few byte-ranges)hundreds
Tokensingle signed URLdual-token / prefix-signed
TTLfew hours> watch duration
Single-usenono
--- ## 12. Watermark & Download Path **\[v2 FIX\]** Decision: **Option B — downloads are watermarked.** (Full build, not the launch shortcut.) **The feed NEVER contains a watermarked URL.** Feed = clean playback URLs only. Watermarked download is a **separate endpoint**, minted on tap. ``` User taps Download → GET /media/{fileId}/download?quality=720p (quality optional, default 720p) → backend: check canDownload (post setting) + tier (private/paid → auth check) → mint signed URL → {quality}_watermarked.mp4 short TTL (~10 min), x-uid bound, single-use (Redis x-once) → log download event (orderId/buyerId/ip/device/country if digital) → return { downloadUrl, expiresIn, singleUse: true } Client downloads. URL marked used in Redis → 403 on reuse. ``` **Default quality, no picker (short video).** Serve one default rung (**720p**) — matches TikTok/IG, sane on TZ data. Endpoint takes optional `quality` so a picker is an additive change later (long-form only, where the size gap matters). **Storage decision:** store **one** watermarked rung (720p), not all three. If a picker is added later, **lazy-generate** other rungs on first request and cache — only spend CPU/storage on a quality someone actually downloads. **Timing:** watermarked 720p is generated at **processing time** (pre-ready), so download is instant — no first-user FFmpeg penalty. ```java // ── Download endpoint (picker-ready, single-use, watermarked) ── @GetMapping("/media/{fileId}/download") ResponseEntity download(@PathVariable String fileId, @RequestParam(defaultValue = "720p") String quality, @AuthenticationPrincipal AccountEntity user) { MediaEntity m = mediaService.findReady(fileId); // ── access gate (tier 2 = open; tier 3 = ownership/visibility) ── if (!m.isCanDownload()) return ResponseEntity.status(403).build(); if (m.isRestricted() && !accessService.canAccess(user, m)) return ResponseEntity.status(403).build(); // ── mint watermarked, user-bound, single-use URL ── String key = m.watermarkedKey(quality); // lazy-generate if absent String url = mediaService.mintDownloadUrl(key, user.getId(), Duration.ofMinutes(10)); downloadLog.record(fileId, user, request); // audit return ResponseEntity.ok(Map.of("downloadUrl", url, "expiresIn", 600, "singleUse", true)); } ``` --- ## 13. Feed Response Shape **\[v2 — new\]** Rules baked in: - **Batch-mint inline.** While building the page (already looping posts), mint every signed URL right there — no per-item round trip from the client. - **Store `objectKey`, mint URL per request.** Never persist a signed URL (it expires). - **Tier 1 unsigned** (avatar, thumb, poster, preview3s); **tier 2 signed** (full image, video rungs). - **Placeholders filled** (blurhash, lqip, dominantColor) + `width`/`height`/`duration` present. - **No watermarked URL here** (§12). `canDownload` flag tells the client whether to show the button. ```json { "data": [{ "id": "0e67aef4-...", "author": { "id": "ad03431a-...", "userName": "juliusdev_", "verified": true, "profilePictureUrl": "https://media.nexgate.co/profiles/ad03431a-.../avatar_400.webp" }, "content": "Testing video media post 🎬", "shareCode": "waBbP82h", "media": [{ "id": "436bb62b-...", "mediaType": "VIDEO", "status": "READY", "width": 1080, "height": 1920, "aspectRatio": "9:16", "duration": 28, "isReelEligible": true, "streamingFormat": "MP4", "canDownload": true, "dominantColor": "#1a1a2e", "blurhash": "L6Pj0^i_.AyE_3t7t7R**0o#DgR4", "lqip": "data:image/webp;base64,UklGRjoAAAB...", "poster": "https://media.nexgate.co/posts/ad03431a-.../436bb62b-.../thumbnail.webp", "preview3s": "https://media.nexgate.co/posts/ad03431a-.../436bb62b-.../preview_3s.mp4", "variants": { "360p": "https://media.nexgate.co/posts/ad03431a-.../436bb62b-.../360p_clean.mp4?x-expires=1780297200&x-sig=a1f2b3", "720p": "https://media.nexgate.co/posts/ad03431a-.../436bb62b-.../720p_clean.mp4?x-expires=1780297200&x-sig=b2e3c4", "1080p": "https://media.nexgate.co/posts/ad03431a-.../436bb62b-.../1080p_clean.mp4?x-expires=1780297200&x-sig=c3f4d5" }, "order": 1 }], "privacySettings": { "visibility": "PUBLIC" } }] } ``` *(HLS video: `streamingFormat: "HLS"`, replace `variants` with a single signed/prefix-signed `master.m3u8`.)* --- ## 14. Security & Anti-Scraping **\[v2 — expanded\]** ### 14.1 Close the visibility-bypass hole v1's permanent unsigned URLs meant a PRIVATE post's bytes were reachable by anyone with the link — the API hid the post, the file did not. **Fix:** 1. Remove **public-read** from MinIO buckets (already a stated rule — enforce it). 2. Route all access through signed CDN URLs. 3. Private/followers/DM/paid → tier 3 (auth check before minting + `x-uid`). After this, a hidden post's media returns 403 to anyone but the authorized viewer. ### 14.2 Two layers, both required - **Signed URLs** stop *permanent* theft: scraped links die in hours → no permanent mirror, no hotlinking. **They do NOT stop a one-time download** — a valid link is a valid link. - **Bot gate** stops the *act* of mass-pulling: it guards the **feed API door** so a bot can't cheaply pull fresh links over and over. - **Neither alone is enough. Together** = scraping isn't worth it. (You cannot make it impossible — the goal is "not worth the effort for 99%, traceable if they do.") ### 14.3 Bot gate (the missing half) — sized to moat (§30) - **App attestation** — the real app computes a hard-to-fake token (TikTok's `X-Bogus`/`msToken` equivalent). A plain script can't produce it without running app code. - **Rate limiting** — one "user" pulling thousands of posts/min = bot → throttle/block. - **Auth + behaviour** — require login; flag inhuman scroll/pull patterns. - **Escalating friction** — suspicious traffic → CAPTCHA / slowdown / ban. > **Build depth depends on the moat decision (§30).** Commerce-moat → lighter bot gate is fine for launch. Content-moat → invest earlier. ### 14.4 Protection layers (reference) `Content-Disposition: inline` · dynamic URL via JS · signed expiring URLs · session/device binding · disable right-click (frontend) · watermark on download · HLS chunking. --- ## 15. OG (Open Graph) Strategy OG images + `og_preview.mp4` are **permanent, unsigned, tier 1** — because crawlers (WhatsApp/Telegram) fetch on share and the user may open hours later; a signed URL would expire → "Error loading." Access control for private posts is enforced **server-side** (the post page returns 403), not via the OG asset. Serve `og_clean.webp` to known platforms, `og_play.webp` (burned play button) to unknown crawlers (User-Agent detection). --- ## 16. Compression & Transcoding (reference) Client: light pre-compress (max 1080p, CRF 18, never upscale). Server: FFprobe → adaptive ladder (never upscale, never inflate → `-c:v copy` if output > input). CRF 21/23/25/28 for 1080/720/480/360, 30 for OG preview. 1080p max (all platforms serve ≤1080p anyway). `-movflags +faststart` always; `frag_keyframe+empty_moov+faststart` for fMP4. Now: H.264 + AAC. Future: AV1 + Opus + CMAF. --- ## 17. BlurHash, LQIP & Dominant Color Generated for every image + video thumbnail, at processing time. Progressive load: dominantColor (0ms) → BlurHash (0ms) → LQIP (0ms) → thumb.webp (CDN) → full on tap. All three stored in `media_content_signals` and shipped inline in the feed JSON. --- ## 18. Deduplication SHA-256 on raw file → `file_hashes`. Exists+clean → skip processing, reference existing variants, `reference_count++`. Delete → decrement; 0 → remove from MinIO. Near-dup (pHash) = future (copyright). --- ## 19. Quota Management Enforced at **presigned-URL generation** (atomic SQL). Social = original size charged; digital = all stored bytes. Plans: FREE 1GB / PRO 20GB / BUSINESS 100GB. Warn 80%/95%/100%. Release on abandoned (immediate) and hard-delete (30d after soft). Redis cache, sync to PG every 5 min. --- ## 20. Storage Lifecycle raw: auto-delete 24h. public: keep forever, access-based tiering (hot NVMe → warm HDD → cold; viral old content auto-promotes). private: DM 1y, audio 24h, KYC per law. digital: forever. Soft delete → 30d grace → hard delete + Cloudflare purge + quota free. --- ## 21. Abandoned Upload Handling Layer 1: client confirm (fast path). Layer 2: cleanup job every 30 min (PENDING >1h → file exists = recover/process; not = abandoned, release quota). Layer 3: raw lifecycle 24h. `fileId` in object key enables recovery without confirm. Redis distributed lock on the job. --- ## 22. Progress Tracking (SSE) SSE (one-way, auto-reconnect, resilient on TZ networks). Redis key `upload_status:{fileId}`: PENDING→UPLOADING→UPLOADED→SCANNING→PROCESSING→LIVE\_PARTIAL→READY (or FAILED/QUARANTINED/ ABANDONED). Multi-instance via Redis Pub/Sub bridging emitters. --- ## 23. Communication Architecture No direct HTTP between services. **RabbitMQ** (FT ↔ Main Backend, topic `nexgate.media`): `media.upload.request`, `media.upload.url.ready`, `media.ready`, `media.live.partial`, `media.failed`, `media.quarantined`, `digital.product.ready`. **Kafka** (FT → Rec Engine / Analytics, topic `content.new`). > **\[v2 FIX\] Events carry `objectKey`s + variants, not signed URLs.** `MEDIA_READY` ships storage keys (e.g. `posts/usr_123/vid_789/720p_clean.mp4`) + placeholders. The Main Backend mints signed URLs **per feed request** from those keys — it never stores a signed URL. --- ## 24. CDN Strategy Cloudflare in front of MinIO. First request → edge → MinIO → cache → serve; thereafter from cache. Cache-Control: tier 1 = 1y; tier 2 = path-keyed short edge TTL; tier 3 / API = `private, no-store`. **MinIO never publicly exposed** (only Cloudflare IPs). Phases: Free → Pro ($20) → R2 (zero egress). See §10 cache-key gotcha. --- ## 25. Shareable Links Client-side `nexgate.co/reels/{shareCode}?ref=whatsapp`. No `uid` in URL (server resolves owner from shareCode → follow prompt). `ref` drives share analytics. Public content = no token needed for the page; the **media** on the page still follows tier rules (§8). --- ## 26. Technology Stack Java 21 / Spring Boot 3.x · FFmpeg (Jaffree) · ImageMagick (IM4Java) · blurhash-java · ClamAV (Docker, TCP 3310) · MinIO · Cloudflare · RabbitMQ · Kafka · Redis · PostgreSQL · TUS · **Vault (`vault.qbitspark.com`) — holds the signing secret** · Docker Compose · SSE. --- ## 27. Database Schema (key tables + v2 notes) ```sql media_files ( file_id UUID PK, owner_id UUID, directory ENUM, original_name TEXT, object_key TEXT, -- canonical storage key [v2: keys, never signed URLs] mime_type TEXT, file_size BIGINT, status ENUM, -- PENDING..READY..QUARANTINED..ABANDONED variants JSONB, -- variant object_keys (NOT URLs) metadata JSONB, scan_result TEXT, hash TEXT, is_reel_eligible BOOLEAN, streaming_format ENUM, can_download BOOLEAN DEFAULT true, -- [v2] drives feed canDownload + download gate created_at TIMESTAMPTZ, ready_at TIMESTAMPTZ ); media_content_signals ( file_id UUID FK, has_audio BOOLEAN, audio_ref TEXT, dominant_color TEXT, blurhash TEXT, lqip TEXT, aspect_ratio TEXT, quality_score INT, width INT, height INT, duration_seconds DECIMAL ); -- user_storage_quota, file_hashes, media_variants (HOT/WARM/COLD), -- digital_product_files, digital_orders, digital_download_logs (as v1) ``` Indexes: `idx_media_cleanup (created_at) WHERE status='PENDING'`; unique `file_hashes(hash)`; `media_files(owner_id, status, created_at DESC)`. > **\[v2 FIX\] No `signed_url` column anywhere.** Signed URLs are minted per request and discarded. The DB stores only `object_key` + `variants` (keys). --- ## 28. Docker & Infrastructure (reference) `eclipse-temurin:21` + ffmpeg + imagemagick(+webp/heic). ClamAV separate container (daemon, auto-updates). FT: 4 cores / 16GB (FFmpeg CPU-heavy). ClamAV: 1 core / 2GB. Env: PG, Redis, RabbitMQ, Kafka, MinIO, ClamAV, `VAULT_ADDR`. --- ## 29. What File Thunder Does NOT Do ❌ ML/AI classification · Whisper (Rec Engine) · CLIP · feed ranking · social graph · payments · auth · push · search · business logic · fan-out · recommendations. Extracts `audio.wav` temporarily; Rec Engine fetches/transcribes/deletes. FT never knows audio content. --- ## 30. Open Decisions (parked) 1. **Moat: content vs commerce?** *This drives §14.3 (bot-gate depth) and §8 (how hard to stamp tier-2 public media).* If commerce is the moat (shops/payments/events), public media scraping hurts less → lighter bot gate, tier-2 stamping is softer/optional. If content is the moat → invest in bot gate + stamp tier-2 firmly. **Decide before finalizing anti-scraping work.** 2. **Bot gate depth for launch** — minimum: login + rate limit. Full app-attestation later, sized to (1). 3. **HLS dual-token vs prefix-sign for launch** — prefix-sign on Cloudflare now; edge dual-token if/when heavier CDN (§11). 4. **Download quality picker** — default 720p now; picker + lazy-gen later (§12). --- ## Summary — File Thunder v2 in one line > Receives raw files → ClamAV → processes into clean + watermarked variants (FFmpeg/IM) with blurhash/lqip/dominant-color → stores **object keys** across 4 buckets in MinIO (never public-read) → the Main Backend **mints signed delivery URLs per request** from those keys using a three-tier policy (trivial-public unsigned, public-content signed, private/paid signed + auth) → Cloudflare serves and caches by tier → streams use MP4-single-URL or HLS-dual-token, downloads use single-use watermarked URLs → signed URLs + a bot gate together make scraping not worth the effort — all without business logic, ML, or social ops. --- *File Thunder Architecture Guide v2.0 — NexGate / QBIT SPARK**v2 integrates media-security design review. \[v2 FIX\] tags mark every change from v1.*