Determining the Real MIME Type of an S3 Object Without Downloading It

Determining the Real MIME Type of an S3 Object Without Downloading It

Problem

You need a fast and accurate way to determine the actual MIME type of a file uploaded to Amazon S3, even when the file has the wrong extension. You don’t control the uploads, which are done via client-side pre-signed URLs, and S3’s headObject() isn’t reliable—it just guesses MIME types from filenames.

Clarifying the Issue

Since clients can name files arbitrarily when uploading through signed URLs, relying on file extensions or Content-Type metadata is not trustworthy. Your goal is to examine the actual bytes in the file—what's known as the file signature or "magic number"—without downloading the entire object in your Lambda function.

Why It Matters

Downstream services or validations may depend on the correct MIME type. Incorrect assumptions could lead to security flaws, processing errors, or content delivery issues. Detecting the true MIME type in a time-efficient way helps maintain system integrity without burdening Lambda with large file downloads.

Key Terms

• MIME Type: The media type identifier (e.g., image/png, application/zip)
• Range Request: An HTTP feature that allows fetching a byte slice of a file
• Magic Number: A sequence of bytes at the beginning of a file that reveals its true format
• Lambda: Serverless compute that can react to S3 uploads in real time

Steps at a Glance

1. Trigger Lambda on S3 upload.
2. Use getObject() with a Range header (e.g., 'bytes=0-4095').
3. Analyze the byte buffer using a library like file-type or python-magic.
4. Determine the correct MIME type based on content.
5. Optionally update metadata or log/store the result elsewhere.

Detailed Steps

1. Trigger Lambda on S3 upload. 

Use an S3 event notification to invoke Lambda each time an object is uploaded.

2. Use getObject() with a Range header (e.g., 'bytes=0-4095'). 

Fetch only the first chunk of the object to minimize time and memory usage.

3. Analyze the byte buffer using a library like file-type or python-magic. 

These libraries detect MIME types based on file signatures.

4. Determine the correct MIME type based on content.
The library will return the most likely MIME type.

5. Optionally update metadata or log/store the result elsewhere. 

Since S3 metadata can't be modified in-place, store the result in DynamoDB, add S3 object tags, or log for later use.

Python Example (with python-magic)

Python

import boto3
import magic

s3 = boto3.client('s3')

def lambda_handler(event, context):
    bucket = event['Records'][0]['s3']['bucket']['name']
    key = event['Records'][0]['s3']['object']['key']

    response = s3.get_object(
        Bucket=bucket,
        Key=key,
        Range='bytes=0-4095'
    )

    partial_data = response['Body'].read()
    mime_type = magic.from_buffer(partial_data, mime=True)

    print(f"Detected MIME type: {mime_type}")
    # Optional: store in DynamoDB or tag the S3 object here

Requires including the libmagic binary and python-magic with your Lambda deployment package or using a Lambda layer.

Node.js Example (with file-type)

JavaScript

const AWS = require('aws-sdk');
const { fileTypeFromBuffer } = require('file-type');

const s3 = new AWS.S3();

exports.handler = async (event) => {
  const bucket = event.Records[0].s3.bucket.name;
  const key = decodeURIComponent(event.Records[0].s3.object.key.replace(/\+/g, ' '));

  const params = {
    Bucket: bucket,
    Key: key,
    Range: 'bytes=0-4095'
  };

  const s3Response = await s3.getObject(params).promise();
  const buffer = s3Response.Body;

  const fileType = await fileTypeFromBuffer(buffer);

  if (fileType) {
    console.log(`Detected MIME type: ${fileType.mime}`);
    // Optional: Tag object, log, or store the result
  } else {
    console.log("Unable to determine MIME type");
  }
};

file-type is a small and easy dependency. Just npm install file-type before packaging or bundling.

Conclusion

For fast and accurate MIME detection without full downloads, using an S3 Range request with a content-aware library like file-type or python-magic gives you the best blend of precision and performance. This method ensures you’re reading the actual file contents—not just trusting metadata or extensions—while keeping Lambda execution time and cost low.

Need AWS Expertise?

If you're looking for guidance on AWS or any cloud challenges, feel free to reach out! We'd love to help you tackle AWS projects. 🚀

Email us at: info@pacificw.com

Image: Gemini