The Secret Life of AWS: The Direct Handshake (Amazon S3 Pre-Signed URLs)

How to securely offload large file downloads from your serverless API

#AWS #S3 #Serverless #CloudArchitecture

Margaret is a senior software engineer. Timothy is her junior colleague. They work in a grand Victorian library in London — the kind of place where code quality is the unspoken objective, and craftsmanship is the only thing that matters.

Episode 68

Timothy was watching a loading spinner on his screen with a look of mounting frustration. The new feature the marketing team requested—allowing customers to download high-resolution PDF manuals for their purchased keyboards—was failing miserably in staging.

Margaret walked into the studio and noticed the red HTTP 502 Bad Gateway error flashing on his screen.

"Our decoupled architecture is choking," Timothy explained, pulling up his CloudWatch logs. "The customer requests the manual. API Gateway routes the request to our Lambda function. The Lambda function pulls the 15-megabyte PDF out of our private S3 bucket, holds it in memory, and tries to send it back through API Gateway to the browser. But it keeps timing out or throwing errors."

"You are using a sports car to haul a tractor," Margaret said, taking a seat. "API Gateway and AWS Lambda are designed to route lightweight JSON payloads, not to stream massive binary files. In fact, API Gateway has a hard payload limit of 10 megabytes. You are hitting a physical wall."

"But the S3 bucket has to be private," Timothy countered. "If I make the bucket public, anyone on the internet can steal our proprietary manuals. I have to use Lambda to verify the user's IAM token before giving them the file."

"Your security instinct is correct, but let's improve the data flow," Margaret smiled. "Just like we used WebSockets in Episode 65 to push real-time updates directly to the browser, we are now going to bypass the API layer for heavy file transfers. We want Lambda to authorize the user, but we want S3 to deliver the file directly. We need an S3 Pre-Signed URL."

The Cryptographic Handshake

Margaret opened Timothy's Node.js Download service.

"Instead of Lambda fetching the heavy PDF," Margaret explained, "Lambda is going to use its secure IAM role to forge a temporary, cryptographic VIP pass. It will hand this pass to the customer's browser, and the browser will go straight to S3 to pick up the file."

She updated the code using the AWS SDK:

const { S3Client, GetObjectCommand } = require("@aws-sdk/client-s3");
const { getSignedUrl } = require("@aws-sdk/s3-request-presigner");

const s3Client = new S3Client({ region: "us-east-1" });

exports.handler = async function generateDownloadLink(event) {
    // 1. Verify user authorization via Amazon Cognito JWT
    const userId = event.requestContext.authorizer.claims.sub;

    // 2. Create the command to get the specific file, forcing a download
    const command = new GetObjectCommand({
        Bucket: "corp-private-manuals-vault",
        Key: `manuals/mechanical-keyboard-v2.pdf`,
        ResponseContentDisposition: `attachment; filename="keyboard-manual.pdf"`
    });

    // 3. Generate the Pre-Signed URL with a 5-minute expiration
    const signedUrl = await getSignedUrl(s3Client, command, { expiresIn: 300 });

    // 4. Return the lightweight URL, NOT the heavy file
    return {
        statusCode: 200,
        body: JSON.stringify({ downloadUrl: signedUrl })
    };
}

The Offloaded Transfer

Timothy studied the updated logic. "This Lambda function finishes in milliseconds. It just generates a string of text and returns it to the frontend."

"Exactly," Margaret said. "We reduced a 15-megabyte payload down to a 200-byte JSON response. Once the frontend JavaScript receives this downloadUrl, it simply redirects the user's browser to it. And because we added ResponseContentDisposition: attachment, we are forcing the browser to download the file rather than trying to display it inline."

"And it is secure because of the expiresIn: 300 parameter," Timothy realized. "Even if the customer copies the link and emails it to a friend, the link becomes completely useless after five minutes."

"Precisely," Margaret nodded. "The URL is cryptographically signed using the Lambda function's IAM credentials. S3 verifies the signature, checks the expiration time, and serves the file. If a single second has passed beyond that five-minute window, S3 returns a 403 Forbidden error. And if we ever need customers to upload large files, like support videos, the exact same pattern works in reverse. We generate a pre-signed PUT URL, and their browser uploads straight to S3."

Timothy updated his architecture diagram. His API Gateway was no longer acting as a congested freight tunnel. By offloading heavy file transfers directly to S3, his serverless backend remained lightning fast, highly secure, and incredibly cost-effective.

Key Concepts Introduced

API Gateway Payload Limits
Amazon API Gateway has a hard maximum payload size of 10MB. Using it to route or stream large files (like high-resolution images, videos, or PDFs) is an architectural anti-pattern that leads to 502 Bad Gateway errors, latency spikes, and inflated compute costs.

S3 Pre-Signed URLs
A mechanism that grants temporary, time-limited access to specific objects stored in a private Amazon S3 bucket. A backend service uses its own IAM credentials to cryptographically sign a URL. The client can then use this URL to interact directly with S3 (either downloading via GET or uploading via PUT) without needing their own AWS credentials.

The Offloading Pattern
A performance best practice where lightweight, compute-heavy tasks (like Cognito JWT authentication and URL generation) are handled by the API layer, while heavy data transfers are offloaded directly to the storage layer (S3). This bypasses the API Gateway bottleneck entirely.

Content-Disposition & Time-To-Live (TTL)
Pre-signed URLs offer immense control. Setting ResponseContentDisposition: attachment forces a client's browser to download the file rather than rendering it. Furthermore, strict expiration windows (e.g., 5 minutes) ensure that generated links cannot be permanently shared, scraped, or abused before returning a native 403 Forbidden error.

Aaron Rose is a software engineer and technology writer at tech-reader.blog.

Catch up on the latest explainer videos, podcasts, and industry discussions below.

Search This Blog

Tech-Reader.blog