Handling File Uploads and Media at Scale
Key takeaway
Upload directly from the browser to object storage with a pre-signed URL, never through your application server. Then validate by content rather than by file extension, process derivatives asynchronously, and serve private files through short-lived signed URLs rather than a permanent public link.
File upload looks like a solved problem until the files get large, the volume grows, or someone uploads something malicious. Getting the shape right early avoids a painful migration later.
Don't proxy uploads through your app
Routing file bytes through your application server ties up a worker for the duration of a slow mobile upload, limits file size to your request timeout, and puts memory pressure where you least want it. Issue a pre-signed URL and let the client upload straight to object storage; your application then receives only a notification that the object exists and records it. For large files, use multipart upload so an interrupted connection resumes instead of restarting.
Validate properly
- Check the actual content type by inspecting the file, not by trusting the extension or the client-supplied header.
- Enforce size limits at the storage policy level, not only in the browser.
- Never store the user's filename as a path, and never execute anything from an upload directory.
- Run virus scanning on anything that will be shared between users, before it becomes downloadable.
- Strip metadata from images where privacy matters — photographs routinely carry location data.
Processing and derivatives
- Generate thumbnails, transcodes, and previews in background jobs, not during the upload request.
- Keep the original, and treat derivatives as regenerable — you will change dimensions or formats later.
- Make processing idempotent and retryable; media pipelines fail routinely on malformed input.
- Show honest status in the interface: uploaded, processing, ready, failed.
Serving and access control
- Serve through a CDN with long cache lifetimes and content-hashed paths.
- For private files, generate short-lived signed URLs per request rather than relying on an unguessable path — unguessable is not private once a link is shared.
- Check authorisation on every access request for regulated content, and log those accesses.
- Set lifecycle rules to move old media to cheaper storage and to delete what your retention policy says shouldn't persist.