File uploads look deceptively simple.

For a standard small-scale application, the architecture is almost universally:

Frontend ──► API Server ──► File Storage (or Local Disk)
                             └──► Save URL in Database

For small payloads—a 2 MB profile picture or a 500 KB PDF receipt—this is completely reasonable.

But change one requirement:

"Users must be able to reliably upload large files (5 GB to 50 GB) and track progress in real time."

Suddenly, the simple pattern falls apart. A 2 MB avatar and a 20 GB raw video archive cannot—and should not—travel through the same infrastructure path.

The Naive Approach: Why It Breaks

The most straightforward way to implement file uploads is standard multipart form data:

┌─────────┐      POST /api/files (multipart/form-data)     ┌─────────────┐
│ Browser │ ─────────────────────────────────────────────► │ Application │
│         │ ◄───────────────────────────────────────────── │ API Server  │
└─────────┘                  200 OK + URL                  └──────┬──────┘
                                                                  │
                                                          Streams entire file
                                                                  ▼
                                                           ┌─────────────┐
                                                           │ Storage / S3│
                                                           └─────────────┘

The server receives the multipart stream, buffers or writes the file to disk/object store, inserts the URL into PostgreSQL, and sends back a response:

POST /api/files HTTP/1.1
Host: api.myapp.com
Content-Type: multipart/form-data; boundary=----WebKitFormBoundary7MA4YWxkTrZu0gW

------WebKitFormBoundary7MA4YWxkTrZu0gW
Content-Disposition: form-data; name="file"; filename="archive-20gb.zip"
Content-Type: application/zip

<binary stream...>

There is nothing syntactically wrong with this code. The catastrophe begins when "large" means gigabytes.

Problem #1: The Application Server as a Data Pipe

If a user uploads a 20 GB file, your API server sits in the middle of a massive byte pipe for 45 minutes:

Browser ──── 20 GB Stream ────► API Server ──── 20 GB Stream ────► Object Storage

If 500 users do this concurrently:

Exhausted Connection Pools: Web servers (Node.js, Python, Go, Ruby) have thread or connection limits. Hanging connections tie up memory and sockets.
Reverse-Proxy Timeouts: Nginx, Cloudflare, and AWS ALB have default body size limits (e.g., client_max_body_size) and timeout limits (typically 60–120s). A 20 GB file on a home connection will trigger a 504 Gateway Timeout.
Skyrocketing Cloud Egress Costs: You pay for incoming bandwidth to your API server and outgoing bandwidth from your API server to your storage bucket (double egress/ingress).
Deployment Disruptions: If you deploy a new container or restart your server, every 2-hour upload in flight is immediately terminated.

The Architecture Shift: Direct-to-Object-Storage

The golden rule of large file engineering is: Remove the application server from the data transfer path.

The API server should manage authorization, validation, and session records, while cloud object storage (AWS S3, Google Cloud Storage, Cloudflare R2, Azure Blob) handles the raw gigabytes.

                 ┌─────────────────────────┐
                 │  Application API Server │
                 │  (Auth, Metadata, DB)   │
                 └───────────┬─────────────┘
                             │
     1. Initiate Upload      │   2. Issue Signed
        (Session Check)      │      Upload Credentials
                             ▼
┌─────────────┐   3. Stream Bytes Directly (PUT)   ┌────────────────────────┐
│   Browser   │ ─────────────────────────────────► │  Cloud Object Storage  │
│  (Client)   │                                    │  (AWS S3 / Azure / R2) │
└─────────────┘ ◄───────────────────────────────── └────────────────────────┘
                       4. ETag / Part Ack

Wait... Does the API Wait on the 20 GB Connection?

"If the API server authenticates the request, does it hold the HTTP connection open while the 20 GB transfers, and only close it when the upload completes?"

No! Absolutely not.

Holding an HTTP connection open for 45 minutes while a user uploads would defeat the entire purpose of offloading to S3.

The key conceptual breakthrough is that the API interaction lasts less than 50 milliseconds.

Here is the exact timeline:

Time	Who	Action	Connection State
0.00s	Browser ➔ API	Sends a tiny JSON request: `POST /api/uploads/initiate` with metadata (`filename`, `fileSize: 20GB`, `mimeType`).	Connection 1 Opens
0.03s	API	Authenticates user token, verifies storage quota, creates row in DB (`status: 'PENDING'`), asks AWS S3 for pre-signed URLs.	Internal computation
0.05s	API ➔ Browser	Returns JSON containing the pre-signed S3 URL and an `uploadId`.	Connection 1 CLOSES (Done in 50ms!)
0.06s to 45m00s	Browser ➔ S3	The browser streams raw chunks directly to Amazon S3’s endpoint (`https://bucket.s3.amazonaws.com/...`). The API server is doing ZERO work for this user.	Direct S3 connection pool
45m00s	Browser ➔ API	All chunks uploaded. Browser sends a tiny JSON request: `POST /api/uploads/{id}/complete`. API verifies with S3 and marks DB `status: 'UPLOADED'`.	Connection 2 Opens & Closes in 40ms

Client                      API Server                          AWS S3
  │                             │                                 │
  │── 1. POST /initiate ───────►│                                 │
  │   (filename, size)          │── Create S3 Multipart ─────────►│
  │                             │◄─ Return UploadId & URLs ───────│
  │◄─ 2. HTTP 200 OK ───────────│                                 │
  │   [Connection 1 Closed!]    │                                 │
  │   (Took 45 milliseconds)    │                                 │
  │                             │                                 │
  │                             │                                 │
  │============================= 3. UPLOAD PHASE =================│
  │ (User uploads 20 GB over 45 minutes directly to Amazon S3)   │
  │── PUT Part 1 ────────────────────────────────────────────────►│
  │── PUT Part 2 ────────────────────────────────────────────────►│
  │── PUT Part 3 ────────────────────────────────────────────────►│
  │   [API Server is 100% idle, handling other traffic]           │
  │                             │                                 │
  │                             │                                 │
  │── 4. POST /complete ───────►│                                 │
  │   (uploadId, parts list)    │── CompleteMultipartUpload ─────►│
  │                             │◄─ 200 OK ───────────────────────│
  │◄─ 5. HTTP 200 OK ───────────│                                 │
  │   [Connection 2 Closed!]    │                                 │
  │   (Took 35 milliseconds)    │                                 │

Your API server never sees a single byte of the 20 GB file. It operates strictly as a control plane.

Multipart & Chunked Uploads: The Safety Net

What happens when a user reaches 94% of a 20 GB file on a single HTTP connection and their train enters a tunnel?

[██████████████████░░]  94%  ──►  Wi-Fi Drops  ──►  FATAL ERROR 💥

With a monolithic single-request upload, the connection resets, the temporary bytes are discarded, and the user must start over from 0%.

The Solution: Chunking (S3 Multipart Upload)

Split the 20 GB file in the browser using the HTML5 File.slice() API into discrete 10 MB–50 MB chunks:

20 GB File
 ├── Part 1 (25 MB) ──► [Uploaded ✓]
 ├── Part 2 (25 MB) ──► [Uploaded ✓]
 ├── Part 3 (25 MB) ──► [Uploaded ✓]
 ├── Part 4 (25 MB) ──► [FAILED ✗] ──► Retry ONLY Part 4
 └── Part N (25 MB) ──► [Queued]

If Part 4 fails due to network jitter, only Part 4 is retried. The preceding 15 GB of verified chunks remain safely staged in object storage.

Parallelism with Guardrails

Chunking also unlocks concurrent uploads:

               ┌── PUT Part 1 ──► [S3 Worker 1]
               ├── PUT Part 2 ──► [S3 Worker 2]
Browser Client ┼── PUT Part 3 ──► [S3 Worker 3]
               └── PUT Part 4 ──► [S3 Worker 4]

!WARNING Do not spin up 50 parallel chunk uploads in the browser. Browser connection pools (usually 6 TCP connections per origin) will bottleneck, CPU will spike, and packet loss will degrade throughput. Optimal concurrency is typically 3 to 6 simultaneous parts.

Tracking Status: Transfer Progress vs. Application State

Engineers often confuse the user's progress bar with the database state. They are two completely different systems.

┌─────────────────────────────────────────────────────────────┐
│ 1. CLIENT TRANSFER PROGRESS (Local, Ephemeral)              │
│    • Measured via XMLHttpRequest.upload.onprogress          │
│    • Bytes sent: 14.2 GB / 20 GB (71%)                      │
│    • Zero API round-trips needed! Calculated locally        │
└─────────────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────────────┐
│ 2. BACKEND LIFECYCLE STATE (Persistent in DB)              │
│    CREATED ──► UPLOADING ──► UPLOADED ──► PROCESSING ──► READY
│                                                  │
│                                            (Virus scan,
│                                             Transcoding,
│                                             Thumbnails)
└─────────────────────────────────────────────────────────────┘

Don't Mistake "Uploaded" for "Ready"

When the last byte hits S3, the file is uploaded, but it is almost never ready:

Upload (100%)
     │
     ▼
Store in Bucket
     │
     ▼
[Asynchronous Worker Queue]
     ├── Malware & Virus Scanning (ClamAV)
     ├── File Integrity & Checksum Verification (SHA-256)
     ├── Extract Metadata (FFmpeg / ExifTool)
     └── Transcode 4K Video into HLS / DASH Multi-Bitrate Streams
     │
     ▼
Mark Database Record as READY

Your UI should make this transparent:

Uploading: 100% (Done transferring bytes)
Processing video: Transcoding for web playback... (Worker active)
Ready ✓ (Available to view or share)

Async Worker Processing Pipeline

Heavy processing must never block your web API. Offload processing to asynchronous background workers:

┌──────────────┐         Upload Done        ┌─────────────────────────┐
│ Cloud Storage│ ─────────────────────────► │ S3 Event / SQS / PubSub │
│ (S3 Bucket)  │                            └────────────┬────────────┘
└──────────────┘                                         │
                                                         ▼
                                            ┌─────────────────────────┐
                                            │ Worker Consumer Pool    │
                                            │ (ECS / Kubernetes / Go) │
                                            └────────────┬────────────┘
                                                         │
                                    ┌────────────────────┼────────────────────┐
                                    ▼                    ▼                    ▼
                             [Virus Scan]         [Transcoder]       [Generate Thumbs]
                                    │                    │                    │
                                    └────────────────────┼────────────────────┘
                                                         │
                                                         ▼
                                            ┌─────────────────────────┐
                                            │   Update DB: READY      │
                                            │   Notify via WebSocket  │
                                            └─────────────────────────┘

Edge Cases That Crash Production

1. Store Storage Keys, Not Public URLs

Bad:

url = "https://my-bucket.s3.us-east-1.amazonaws.com/uploads/video.mp4"

Good:

storage_key = "uploads/2026/09/user_491/video_9a8f2c.mp4"

If you switch from AWS S3 to Cloudflare R2, put a CloudFront CDN in front, or convert public URLs into expiring 15-minute signed download links, hardcoded URLs will break your database migrations. Store the immutable object key and construct URLs dynamically.

2. The Ghost Upload Epidemic (Abandoned Chunks)

A user starts uploading a 25 GB file. At 12 GB, their laptop battery dies, and they never return.

In S3, incomplete multipart parts accumulate silently and cost money every month until completed or aborted.

The Fix: Configure an S3 Bucket Lifecycle Rule:

{
  "Rules": [
    {
      "ID": "AbortIncompleteMultipartUploadsAfter7Days",
      "Status": "Enabled",
      "Filter": {},
      "AbortIncompleteMultipartUpload": {
        "DaysAfterInitiation": 7
      }
    }
  ]
}

3. Checksum Verification

Don't trust file names or reported sizes. Have the client calculate a hash (e.g. SHA-256 or MD5 for each part) and supply it in the Content-MD5 header to S3. S3 will verify the checksum on arrival and automatically reject corrupt bits before they touch your storage layer.

Architectural Summary

Dimension	Naive Approach (Proxy via API)	Production Standard (Direct to S3)
API Load	1,000 active 10 GB streams saturate RAM/bandwidth	Under 50ms per session check; 0 stream bytes
Resilience	1 connection drop kills entire upload	Failed parts retry independently
Timeouts	Reverse proxies kill long-lived requests	No proxy in transfer path; direct S3 ingress
Processing	Blocks the web request thread	Decoupled via queues (SQS/Redis) and workers
Security	Handled manually on server disk	Pre-signed short-lived (15 min) HMAC URLs

Uploading large files is fundamentally a distributed systems problem, not a file input problem.

By separating the control plane (authentication, pre-signed token generation, lifecycle states) from the data plane (direct S3 streaming, chunked retries, async worker queues), your system can ingest petabytes of user data without your API server breaking a sweat.