Never Trust File Uploads: Building a Secure S3 Pipeline
File uploads look simple.
A user selects a file, your backend receives it, saves it to storage, and returns a URL.
Done, right?
Not really.
What if someone uploads a malicious file disguised as an image? What if hundreds of users upload large files at the same time? Or what if an infected PDF becomes available for everyone to download?
A simple upload feature can quickly become a security problem.
Let's look at how to build a safer file upload pipeline using S3, magic bytes, and ClamAV.
1. Stop sending files through your API server
A common approach is to receive files through your Node.js backend and then upload them to S3.
This works, but your API becomes a middleman.
Every upload uses server bandwidth and connections. If you're buffering files in memory, large uploads can also consume a lot of RAM.
A better approach is to use S3 presigned URLs.
Here's how it works:
1. The client requests an upload URL from your API.
2. Your API checks permission and generates a short-lived URL.
3. The client uploads the file directly to S3.
4. S3 stores the file in a private quarantine location.
Your API handles permissions, not the actual file transfer.
For example, using AWS SDK v3:
import { randomUUID } from "node:crypto";
import { S3Client } from "@aws-sdk/client-s3";
import { createPresignedPost } from "@aws-sdk/s3-presigned-post";
const s3 = new S3Client({ region: "us-east-1" });
export async function createUpload(userId: string) {
// userId must come from the authenticated session
const key = `quarantine/${userId}/${randomUUID()}`;
const { url, fields } = await createPresignedPost(s3, {
Bucket: "my-private-uploads",
Key: key,
Expires: 300,
Conditions: [
["content-length-range", 1, 10 * 1024 * 1024],
],
});
return { url, fields };
}
This creates a presigned POST that allows uploads up to 10 MB and expires after five minutes.
The client sends the returned fields and file as multipart/form-data directly to S3.
The bucket stays private, and the backend should record the generated key against the authenticated user.
One important thing: a successful upload doesn't mean the file is safe.
We still need to inspect it.
2. Never trust file extensions or MIME types
Imagine someone uploads a file named profile.png.
Your backend checks:
if (file.mimetype === "image/png") {
// Accept file
}
Looks fine, right?
The problem is that the MIME type usually comes from the client.
Anyone can rename a file or send a fake Content-Type header.
Instead, check the actual file content.
Check the magic bytes
Many file formats have recognizable binary signatures called magic bytes.
For example:
| File | Signature |
|---|---|
| PNG | 89 50 4E 47 0D 0A 1A 0A |
| JPEG | FF D8 FF |
| PDF | 25 50 44 46 2D |
| ZIP | 50 4B 03 04 |
In Node.js, the file-type package makes this easier.
import { fileTypeFromBuffer } from "file-type";
async function validateImage(buffer: Buffer) {
const type = await fileTypeFromBuffer(buffer);
const allowed = ["image/png", "image/jpeg"];
if (!type || !allowed.includes(type.mime)) {
throw new Error("Unsupported file format");
}
return type;
}
You don't need to download the entire file just to check its signature.
For supported formats, you can read a small part of the object using an S3 range request. The number of bytes needed depends on the format and detection library.
But remember, valid magic bytes don't mean a file is safe.
A valid PDF or image can still contain harmful content or exploit a vulnerable file parser.
That's why we need another layer.
3. Scan files before making them available
This is the part I wouldn't skip.
Never make an uploaded file available to users before it passes validation and scanning.
Instead, use a quarantine system.
Here's the flow:
Client
|
| Request upload URL
v
Node.js API
|
| Generate presigned URL
v
Client
|
| Upload directly
v
Private S3 Quarantine
|
| Storage event
v
Background Worker
|
| Check file size
| Check magic bytes
| Scan with ClamAV
|
+-- Failed --> Reject
|
+-- Passed --> Approved Storage
|
v
Available to User
For malware scanning, we can use ClamAV, a free, open-source antivirus engine.
A background worker can download or stream the quarantined object to the scanner.
Here's a simplified example using the clamscan package:
import NodeClam from "clamscan";
const clam = await new NodeClam().init({
clamdscan: {
host: "clamav-service",
port: 3310,
},
});
export async function scanFile(
fileStream: NodeJS.ReadableStream,
) {
const result = await clam.scanStream(fileStream);
if (result.isInfected !== false) {
throw new Error("File rejected");
}
return true;
}
This example only shows the scanning step. The worker still needs to retrieve the correct S3 object, validate it, handle failures, and update the database.
If the scan fails, times out, or returns an unknown result, keep the file quarantined.
Don't approve it automatically.
Also, ClamAV reduces malware risk, but it cannot detect every possible threat.
4. Let background workers handle the heavy work
Virus scanning can take time.
You don't want users waiting for it inside an HTTP request.
Instead, use a background queue like BullMQ or AWS SQS.
When S3 receives a file, a storage event can trigger a job.
The worker validates the file, scans it, and updates its status.
A simple database lifecycle could look like this:
PENDING_UPLOAD
|
v
QUARANTINED
|
v
SCANNING
/ \
v v
APPROVED REJECTED
The frontend can show "Processing your file" while scanning is in progress.
Make sure workers can safely retry jobs and handle duplicate events.
There's another important detail here.
Scan and approve the exact same object.
Otherwise, a file could be replaced after scanning but before approval.
Use unique storage keys and object versioning or other controls that prevent an unscanned replacement from being published.
5. A few security rules worth following
Even with these three layers, I'd still add a few protections:
- Limit file sizes. Don't allow unlimited uploads.
- Use random file names. Generate UUID-based storage keys instead of trusting user-provided paths.
- Keep storage private. Approved files don't always need public URLs. Use authorized downloads when appropriate.
- Limit upload requests. Add authentication, rate limits, and storage quotas.
- Protect against ZIP bombs. Limit archive extraction size, file count, and nesting depth.
- Serve files safely. Use correct response headers and avoid serving risky user content from your application's main origin.
- Monitor failures. Track rejected uploads, scanner errors, and unusual upload activity.
And one underrated point: don't expose your ClamAV service directly to the internet. Keep it accessible only to trusted workers.
Final thoughts
File uploads are easy to build, but securing them takes a little more thought.
You don't need a complicated system from day one.
Start with three things:
1. Presign: Let clients upload directly to private object storage.
2. Validate: Check file size, format, and actual content instead of trusting filenames.
3. Scan: Run malware checks in the background before making files available.
These steps won't eliminate every security risk, but they'll give you a much safer foundation.
The main thing I've learned is simple:
Uploading a file and trusting a file should be two different steps.
How do you handle file uploads in your applications? Do you scan files before making them available, or use a different approach?
---
References
- AWS S3 Presigned URLs
- AWS S3 Presigned POST
- ClamAV Documentation
- OWASP File Upload Cheat Sheet
1 reaction · 1 comment
Mentions
ok doing this with regex will be impossible so im just using the "strip-json-comments" node package cuz this is what i'd have to do otherwise: github.com/sindresorhus/strip-json-comments/blob/…
Hey all. I'm the one working on this (a human). It's still early. It's something I have been wanting for many years, but I never had the time to invest in it until now. It's already powering the frontmatter of my website: github.com/sindresorhus/sindresorhus.github.com/b…
Awesome The legendary curated lists of awesome things. ~500k+ stars. sindresorhus/awesome
SOML is already powering the frontmatter on my website: github.com/sindresorhus/sindresorhus.github.com/b…
SOML is already powering the frontmatter on my website: github.com/sindresorhus/sindresorhus.github.com/b…
Repositories as compressed state: a rate-distortion view of agent context budgets A repository is not the source code. It is a lossy compression of the intent that produced the source code, and an agent's context window is a second, much tighter compressor on top of it. Rate-distortion framing: the source is developer intent I, the reproduction is the agent's patch P, and C is the token budget you are allowed to spend. D(R, C) = min over compressions R with budget C of E[distortion] In practice retrieval is the compressor. You pick C tokens of context and the distortion is how badly the resulting patch diverges from the reference. Two repositories with the same token count can have wildly different distortion, because information density per token is not uniform: it depends on the entropy rate of that repository's history. This reframes the agent did not find the right file as a distortion problem rather than a retrieval failure, and it predicts that the most valuable context is not the most relevant file but the highest entropy-per-token region of the repo, which is usually the test suite. For anyone looking for a well-curated list of repos to apply this to: sindresorhus/awesome How would you estimate the distortion term without a reference patch?
Awesome Think of this as a library of resource lists. You'll find collections covering Python, frontend, security, machine learning and much more. Choose one topic you care about and explore its list. GitHub repo - sindresorhus/awesome
empty-trash-cli: A SuperCLI Plugin Guide empty-trash-cli Empty the trash Source sindresorhus/empty-trash-cli Tags - files - directories - deleting, - copying, - renaming - empty - trash - empty-trash-cli Installation 1. npm install -g empty-trash-cli 2. Verify: empty-trash-cli --version 3. supercli plugins install ./plugins/empty-trash-cli --on-conflict replace --json About SuperCLI This plugin is part of the SuperCLI ecosystem - a unified CLI plugin manager for AI agents and developers. Discover, install, and orchestrate thousands of command-line tools with SuperCLI.
Never Trust File Uploads: Building a Secure S3 Pipeline File uploads look simple. A user selects a file, your backend receives it, saves it to storage, and returns a URL. Done, right? Not really. What if someone uploads a malicious file disguised as an image? What if hundreds of users upload large files at the same time? Or what if an infected PDF becomes available for everyone to download? A simple upload feature can quickly become a security problem. Let's look at how to build a safer file upload pipeline using S3, magic bytes, and ClamAV. 1. Stop sending files through your API server A common approach is to receive files through your Node.js backend and then upload them to S3. This works, but your API becomes a middleman. Every upload uses server bandwidth and connections. If you're buffering files in memory, large uploads can also consume a lot of RAM. A better approach is to use S3 presigned URLs. Here's how it works: 1. The client requests an upload URL from your API. 2. Your API checks permission and generates a short-lived URL. 3. The client uploads the file directly to S3. 4. S3 stores the file in a private quarantine location. Your API handles permissions, not the actual file transfer. For example, using AWS SDK v3: import { randomUUID } from "node:crypto"; import { S3Client } from "@aws-sdk/client-s3"; import { createPresignedPost } from "@aws-sdk/s3-presigned-post"; const s3 = new S3Client({ region: "us-east-1" }); export async function createUpload(userId: string) { // userId must come from the authenticated session const key = `quarantine/${userId}/${randomUUID()}`; const { url, fields } = await createPresignedPost(s3, { Bucket: "my-private-uploads", Key: key, Expires: 300, Conditions: [ ["content-length-range", 1, 10 * 1024 * 1024], ], }); return { url, fields }; } This creates a presigned POST that allows uploads up to 10 MB and expires after five minutes. The client sends the returned fields and file as multipart/form-data directly to S3. The bucket stays private, and the backend should record the generated key against the authenticated user. One important thing: a successful upload doesn't mean the file is safe. We still need to inspect it. 2. Never trust file extensions or MIME types Imagine someone uploads a file named profile.png. Your backend checks: if (file.mimetype === "image/png") { // Accept file } Looks fine, right? The problem is that the MIME type usually comes from the client. Anyone can rename a file or send a fake Content-Type header. Instead, check the actual file content. Check the magic bytes Many file formats have recognizable binary signatures called magic bytes. For example: | File | Signature | |---|---| | PNG | 89 50 4E 47 0D 0A 1A 0A | | JPEG | FF D8 FF | | PDF | 25 50 44 46 2D | | ZIP | 50 4B 03 04 | In Node.js, the file-type package makes this easier. import { fileTypeFromBuffer } from "file-type"; async function validateImage(buffer: Buffer) { const type = await fileTypeFromBuffer(buffer); const allowed = ["image/png", "image/jpeg"]; if (!type || !allowed.includes(type.mime)) { throw new Error("Unsupported file format"); } return type; } You don't need to download the entire file just to check its signature. For supported formats, you can read a small part of the object using an S3 range request. The number of bytes needed depends on the format and detection library. But remember, valid magic bytes don't mean a file is safe. A valid PDF or image can still contain harmful content or exploit a vulnerable file parser. That's why we need another layer. 3. Scan files before making them available This is the part I wouldn't skip. Never make an uploaded file available to users before it passes validation and scanning. Instead, use a quarantine system. Here's the flow: Client | | Request upload URL v Node.js API | | Generate presigned URL v Client | | Upload directly v Private S3 Quarantine | | Storage event v Background Worker | | Check file size | Check magic bytes | Scan with ClamAV | +-- Failed --> Reject | +-- Passed --> Approved Storage | v Available to User For malware scanning, we can use ClamAV, a free, open-source antivirus engine. A background worker can download or stream the quarantined object to the scanner. Here's a simplified example using the clamscan package: import NodeClam from "clamscan"; const clam = await new NodeClam().init({ clamdscan: { host: "clamav-service", port: 3310, }, }); export async function scanFile( fileStream: NodeJS.ReadableStream, ) { const result = await clam.scanStream(fileStream); if (result.isInfected !== false) { throw new Error("File rejected"); } return true; } This example only shows the scanning step. The worker still needs to retrieve the correct S3 object, validate it, handle failures, and update the database. If the scan fails, times out, or returns an unknown result, keep the file quarantined. Don't approve it automatically. Also, ClamAV reduces malware risk, but it cannot detect every possible threat. 4. Let background workers handle the heavy work Virus scanning can take time. You don't want users waiting for it inside an HTTP request. Instead, use a background queue like BullMQ or AWS SQS. When S3 receives a file, a storage event can trigger a job. The worker validates the file, scans it, and updates its status. A simple database lifecycle could look like this: PENDING_UPLOAD | v QUARANTINED | v SCANNING / \ v v APPROVED REJECTED The frontend can show "Processing your file" while scanning is in progress. Make sure workers can safely retry jobs and handle duplicate events. There's another important detail here. Scan and approve the exact same object. Otherwise, a file could be replaced after scanning but before approval. Use unique storage keys and object versioning or other controls that prevent an unscanned replacement from being published. 5. A few security rules worth following Even with these three layers, I'd still add a few protections: - Limit file sizes. Don't allow unlimited uploads. - Use random file names. Generate UUID-based storage keys instead of trusting user-provided paths. - Keep storage private. Approved files don't always need public URLs. Use authorized downloads when appropriate. - Limit upload requests. Add authentication, rate limits, and storage quotas. - Protect against ZIP bombs. Limit archive extraction size, file count, and nesting depth. - Serve files safely. Use correct response headers and avoid serving risky user content from your application's main origin. - Monitor failures. Track rejected uploads, scanner errors, and unusual upload activity. And one underrated point: don't expose your ClamAV service directly to the internet. Keep it accessible only to trusted workers. Final thoughts File uploads are easy to build, but securing them takes a little more thought. You don't need a complicated system from day one. Start with three things: 1. Presign: Let clients upload directly to private object storage. 2. Validate: Check file size, format, and actual content instead of trusting filenames. 3. Scan: Run malware checks in the background before making files available. These steps won't eliminate every security risk, but they'll give you a much safer foundation. The main thing I've learned is simple: Uploading a file and trusting a file should be two different steps. How do you handle file uploads in your applications? Do you scan files before making them available, or use a different approach? --- References - AWS S3 Presigned URLs - AWS S3 Presigned POST - ClamAV Documentation - OWASP File Upload Cheat Sheet
Ich hab ein ganz gutes Gedächtnis… Der Code hier ist geklaut: github.com/sindresorhus/slugify/blob/main/index.js Das ist nicht nur PHP-Style, sondern das Pattern von Laravel 2013. Stutzig machte mich das Options-Assoc. Wenn ich jetzt ins illuminati-repo gehe, werde ich sicher in der History fündig.