When S3 Encryption Breaks Lambda: Recovering with an SQS DLQ (S3 → Lambda → DynamoDB)

#aws #s3 #lambda #dynamodb

When KMS policies silently block your Lambda reads, the danger isn’t the failure itself — it’s not knowing it happened.

A single encryption setting can quietly cripple your S3 → Lambda → DynamoDB workflow.

We’ll build the system, break it on purpose, observe the failure, capture the fallout with a DLQ, fix the permissions, and replay the lost events until the pipeline is whole again.

Problem

Your S3 bucket triggers Lambda to process files and write metadata into DynamoDB.

Then someone enables SSE-KMS encryption—or tightens access—and the function still runs but no longer writes data.

CloudWatch shows AccessDenied.

Clarifying the Issue

S3 delivers the event, Lambda tries to read the object, and KMS checks permissions.

If the Lambda role isn’t trusted by the KMS key or bucket, the read fails instantly.

S3 doesn’t retry. Without a DLQ, the event disappears.

Why It Matters

These are silent failures. Your dashboards look fine while data quietly drifts.

A DLQ surfaces every lost event and lets you recover them later instead of accepting bad counts and incomplete analytics.

Key Terms

AccessDenied — Missing permission or KMS grant blocking S3 read.
SSE-KMS — Server-side encryption using AWS KMS keys. (The most common way to break least-privilege Lambda functions.)
DLQ — Dead-Letter Queue for failed Lambda events.
Replay — Re-invoking stored DLQ messages after repairing permissions.

Steps at a Glance

Create the baseline pipeline.
Test the pipeline.
Introduce the failure.
Observe the error.
Add and test a DLQ while still broken.
Apply the fix.
Replay and verify recovery.
Re-Test and confirm.
Delete resources.

Step 1 – Create the Baseline Pipeline

We’ll set up an S3 bucket for uploads, a DynamoDB table for storage, a Lambda function to connect them, and a customer-managed KMS key for encryption testing later.

Create the bucket, table, and CMK

aws s3api create-bucket \
  --bucket s3-kms-demo \
  --region us-east-1
aws dynamodb create-table \
  --table-name kms-demo-table \
  --attribute-definitions AttributeName=id,AttributeType=S \
  --key-schema AttributeName=id,KeyType=HASH \
  --billing-mode PAY_PER_REQUEST
aws kms create-key \
  --description "KMS demo key" \
  --query KeyMetadata.KeyId \
  --output text

✅ Database and CMK are live.
Next, build the Lambda handler.

Create the Lambda handler

import boto3, hashlib, json

dynamodb = boto3.resource('dynamodb')
table = dynamodb.Table('kms-demo-table')

def lambda_handler(event, context):
    for record in event['Records']:
        bucket = record['s3']['bucket']['name']
        key = record['s3']['object']['key']
        s3 = boto3.client('s3')
        obj = s3.get_object(Bucket=bucket, Key=key)
        data = obj['Body'].read().decode('utf-8')
        item_id = hashlib.md5(key.encode()).hexdigest()
        table.put_item(Item={'id': item_id, 'filename': key, 'content': data})
    print("All records processed successfully.")

Next, package and deploy with least-privilege access.

Package and deploy the Lambda function

zip function.zip lambda_handler.py
aws iam create-role \
  --role-name kms-demo-role \
  --assume-role-policy-document file://trust-policy.json
aws iam put-role-policy \
  --role-name kms-demo-role \
  --policy-name kms-demo-policy \
  --policy-document file://inline-policy.json
aws iam attach-role-policy \
  --role-name kms-demo-role \
  --policy-arn arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
aws lambda create-function \
  --function-name kms-demo-func \
  --runtime python3.9 \
  --role arn:aws:iam::<account-id>:role/kms-demo-role \
  --handler lambda_handler.lambda_handler \
  --zip-file fileb://function.zip

Expected output:

{ "FunctionName": "kms-demo-func", "State": "Active" }

✅ Lambda is live with tight permissions and proper CloudWatch logging.
Next, wire S3 to Lambda.

Connect S3 to Lambda

aws lambda add-permission \
  --function-name kms-demo-func \
  --principal s3.amazonaws.com \
  --statement-id s3invoke \
  --action "lambda:InvokeFunction" \
  --source-arn arn:aws:s3:::s3-kms-demo
aws s3api put-bucket-notification-configuration \
  --bucket s3-kms-demo \
  --notification-configuration '{"LambdaFunctionConfigurations":[{"LambdaFunctionArn":"arn:aws:lambda:us-east-1:<account-id>:function:kms-demo-func","Events":["s3:ObjectCreated:*"]}]}'

✅ Pipeline connected.
Next, verify that it works.

Step 2 – Test the Pipeline

echo "hello world" > file1.txt
aws s3 cp file1.txt s3://s3-kms-demo/
aws dynamodb scan \
  --table-name kms-demo-table

Expected output:

{
  "Items": [
    { "filename": {"S":"file1.txt"}, "content": {"S":"hello world"} }
  ],
  "Count": 1
}

✅ Data flows correctly.
Next, we’ll break it.

Step 3 – Introduce the Failure

KEY_ID=$(aws kms create-key \
  --description "Demo encryption key" \
  --query KeyMetadata.KeyId \
  --output text)
aws s3api put-bucket-encryption \
  --bucket s3-kms-demo \
  --server-side-encryption-configuration '{"Rules":[{"ApplyServerSideEncryptionByDefault":{"SSEAlgorithm":"aws:kms","KMSMasterKeyID":"'$KEY_ID'"}}]}'
echo "should fail" > locked.txt
aws s3 cp locked.txt s3://s3-kms-demo/

✅ We’ve created an AccessDenied condition.
Next, verify it.

Step 4 – Observe the Failure

aws logs filter-log-events \
  --log-group-name /aws/lambda/kms-demo-func \
  --filter-pattern "AccessDenied" \
  --limit 1

Expected output:

botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the GetObject operation: Access Denied

The event failed silently.

Next, add a DLQ while the pipeline is still broken.

Step 5 – Add and Test a DLQ While Still Broken

aws sqs create-queue \
  --queue-name kms-demo-dlq
QUEUE_ARN=$(aws sqs get-queue-attributes \
  --queue-url https://sqs.us-east-1.amazonaws.com/<account-id>/kms-demo-dlq \
  --attribute-names QueueArn \
  --query 'Attributes.QueueArn' \
  --output text)
aws lambda update-function-configuration \
  --function-name kms-demo-func \
  --dead-letter-config TargetArn=$QUEUE_ARN
echo "dlq-test" > dlq.txt
aws s3 cp dlq.txt s3://s3-kms-demo/
aws sqs receive-message \
  --queue-url https://sqs.us-east-1.amazonaws.com/<account-id>/kms-demo-dlq

✅ The DLQ successfully caught the event.
Next, fix the permissions on our CMK and bucket.

Step 6 – Apply the Fix

aws kms put-key-policy \
  --key-id $KEY_ID \
  --policy-name default \
  --policy file://kms-policy.json
aws s3api put-bucket-policy \
  --bucket s3-kms-demo \
  --policy file://bucket-policy.json

✅ Lambda can now read encrypted objects.\

Here are the two policy files referenced above — add them before re-running the commands for a complete, reproducible setup.

kms-policy.json

{
  "Version": "2012-10-17",
  "Id": "kms-demo-policy",
  "Statement": [
    {
      "Sid": "EnableRootPermissions",
      "Effect": "Allow",
      "Principal": { "AWS": "arn:aws:iam::<account-id>:root" },
      "Action": "kms:*",
      "Resource": "*"
    },
    {
      "Sid": "AllowLambdaDecrypt",
      "Effect": "Allow",
      "Principal": { "AWS": "arn:aws:iam::<account-id>:role/kms-demo-role" },
      "Action": ["kms:Decrypt", "kms:DescribeKey"],
      "Resource": "*"
    }
  ]
}

bucket-policy.json

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowLambdaGetObject",
      "Effect": "Allow",
      "Principal": { "AWS": "arn:aws:iam::<account-id>:role/kms-demo-role" },
      "Action": ["s3:GetObject"],
      "Resource": ["arn:aws:s3:::s3-kms-demo/*"]
    }
  ]
}

Next, replay the failed events from the DLQ.

Step 7 – Replay and Verify Recovery

import boto3, json

lambda_client = boto3.client('lambda')
sqs = boto3.client('sqs')
queue_url = 'https://sqs.us-east-1.amazonaws.com/<account-id>/kms-demo-dlq'

while True:
    messages = sqs.receive_message(QueueUrl=queue_url, MaxNumberOfMessages=10)
    if 'Messages' not in messages:
        break
    for msg in messages['Messages']:
        payload = json.loads(msg['Body'])
        lambda_client.invoke(FunctionName='kms-demo-func', InvocationType='Event', Payload=json.dumps(payload))
        sqs.delete_message(QueueUrl=queue_url, ReceiptHandle=msg['ReceiptHandle'])
        print("Replayed message:", msg['MessageId'])

aws sqs receive-message \
  --queue-url https://sqs.us-east-1.amazonaws.com/<account-id>/kms-demo-dlq
aws dynamodb scan \
  --table-name kms-demo-table

Expected output:

{ "Count": 2 }

✅ Replay succeeded.
Next, confirm normal operation.

Step 8 – Re-Test and Confirm

echo "fixed" > fixed.txt
aws s3 cp fixed.txt s3://s3-kms-demo/
aws dynamodb scan \
  --table-name kms-demo-table

Expected output:

{ "Count": 3 }

✅ Everything works again.
Next, clean it all up.

Step 9 – Delete Resources

aws lambda delete-function \
  --function-name kms-demo-func
aws s3 rb s3://s3-kms-demo \
  --force
aws dynamodb delete-table \
  --table-name kms-demo-table
aws kms schedule-key-deletion \
  --key-id $KEY_ID \
  --pending-window-in-days 7

Expected output:

{ "ResponseMetadata": { "HTTPStatusCode": 200 } }

✅ Environment cleared — no leftover costs.

Pro Tips

Add your DLQ before fixing the pipeline —you can’t replay what you didn’t catch.
Use customer-managed CMKs for controlled demos; AWS-managed keys can’t be re-policy-ed.
Keep IAM policies least-privilege for tighter examples.
Automate DLQ replays for production durability.
Replays must be idempotent — no duplicates, no overwrites. Using PutItem with a guaranteed unique key (like an MD5 of the S3 object key) makes this automatic.

Conclusion

Encryption isn’t the enemy — invisibility is.

When KMS policies silently block your Lambda reads, the danger isn’t the failure itself — it’s not knowing it happened.

The combination of DLQs, CMKs, and clear IAM boundaries turns that silence into signal.

You see the failure, you capture it, you fix it, and you replay it — no drift, no guesswork.

Aaron Rose is a software engineer and technology writer at tech-reader.blog and the author of Think Like a Genius.

Search This Blog

Tech-Reader.blog