S3 Event Timing Drift: When Lambda Triggers Before the Object Exists (S3 → Lambda → DynamoDB)

#aws #s3 #lambda #dynamodb

When a pipeline runs faster than the storage it depends on, things get strange.

Your Lambda gets the “object created” event — but the object isn’t actually there yet.

Problem

You’ve built a simple, automated ingestion pipeline:

S3 → Lambda → DynamoDB.

Uploads to S3 trigger a Lambda function, which reads the object and writes metadata to DynamoDB.

Everything hums along until one random upload fails with a NoSuchKey error — Lambda claims the file doesn’t exist.

You check the console. The file is there.

What happened?

Lambda triggered too soon — before S3 finalized the object.

Clarifying the Issue

While Amazon S3 has offered strong read-after-write consistency for object keys since December 2020, large or multipart uploads can still exhibit short windows where the object’s data isn’t yet fully retrievable. The event fires as soon as the key is committed, even if the content hasn’t finished replicating.

During that small propagation window, the object metadata is visible, but any attempt to access the data via get_object() may fail with:

{
  "Error": {
    "Code": "NoSuchKey",
    "Message": "The specified key does not exist."
  }
}

That’s event timing drift — when signals move faster than state.

Why It Matters

At scale, even a one-in-a-thousand timing drift can cascade:

Your pipeline skips files and leaves holes in data sets.
Error alarms spike intermittently and are hard to reproduce.
Downstream analytics trust bad data because partial ingestion looks “successful.”

S3 isn’t lying — it’s just faster than your downstream consumers.

To build reliable ingestion, your Lambda must learn to verify and wait.

Key Terms

Event Timing Drift — When the trigger event fires before the data is fully available.
Strong Read-After-Write Consistency — Guaranteed visibility of object keys immediately after a successful write.
Backoff Retry — Controlled waiting period between attempts.
HEAD Request — Lightweight check for object existence without downloading content.
Idempotent Insert — Logic ensuring retries don’t create duplicates.

Steps at a Glance

Create the ingestion pipeline — S3 bucket, Lambda, and DynamoDB table.
Upload a large file to trigger an early event.
Observe the NoSuchKey error from Lambda.
Add verification logic and retry backoff in Lambda.
Re-test and confirm the function self-recovers.

Step 1 – Create the Ingestion Pipeline

Create the resources

aws s3api create-bucket --bucket s3-timing-demo --region us-east-1
aws dynamodb create-table \
  --table-name timing-demo-table \
  --attribute-definitions AttributeName=id,AttributeType=S \
  --key-schema AttributeName=id,KeyType=HASH \
  --billing-mode PAY_PER_REQUEST

✅ Verify table creation:

aws dynamodb describe-table --table-name timing-demo-table --query "Table.TableStatus"

✅ Output:

"ACTIVE"

Create the Lambda handler

cat > lambda_handler.py <<'EOF'
import boto3, json, hashlib

s3 = boto3.client('s3')
dynamodb = boto3.resource('dynamodb')
table = dynamodb.Table('timing-demo-table')

def lambda_handler(event, context):
    for record in event['Records']:
        bucket = record['s3']['bucket']['name']
        key = record['s3']['object']['key']

        # Attempt to read the object immediately
        try:
            s3.get_object(Bucket=bucket, Key=key)
            event_id = hashlib.md5(key.encode()).hexdigest()
            table.put_item(Item={'id': event_id, 'filename': key, 'status': 'processed'})
        except s3.exceptions.NoSuchKey:
            print(f"Object not found yet: {key}")
            raise
EOF

Deploy the Lambda

zip function.zip lambda_handler.py
aws lambda create-function \
  --function-name timingDemoHandler \
  --zip-file fileb://function.zip \
  --handler lambda_handler.lambda_handler \
  --runtime python3.11 \
  --role arn:aws:iam::123456789012:role/LambdaExecutionRole

Connect S3 to Lambda

aws s3api put-bucket-notification-configuration \
  --bucket s3-timing-demo \
  --notification-configuration '{
    "LambdaFunctionConfigurations":[
      {
        "LambdaFunctionArn":"arn:aws:lambda:us-east-1:123456789012:function:timingDemoHandler",
        "Events":["s3:ObjectCreated:*"]
      }
    ]
  }'

✅ Confirm configuration

aws s3api get-bucket-notification-configuration --bucket s3-timing-demo

✅ Output:

{"LambdaFunctionConfigurations":[{"Events":["s3:ObjectCreated:*"]}]}

Step 2 – Upload a Large File to Trigger an Early Event

Simulate a file that takes a moment to finalize:

dd if=/dev/zero of=bigfile.bin bs=1M count=20
aws s3 cp bigfile.bin s3://s3-timing-demo/

Lambda triggers immediately — but the file may not be readable yet.

Step 3 – Observe the `NoSuchKey` Error

Inspect CloudWatch logs for the Lambda:

aws logs filter-log-events \
  --log-group-name /aws/lambda/timingDemoHandler \
  --filter-pattern "Object not found yet"

✅ Expected output:

Object not found yet: bigfile.bin
An error occurred (NoSuchKey) when calling the GetObject operation

At this point, S3’s metadata is visible, but the object body hasn’t finished replication or multipart assembly.

Lambda tries to fetch too early, fails, and retries.

Step 4 – Add Verification Logic and Exponential Backoff

We’ll now make the Lambda patient using exponential backoff with jitter for efficient retries.

cat > lambda_handler.py <<'EOF'
import boto3, json, hashlib, time, random

s3 = boto3.client('s3')
dynamodb = boto3.resource('dynamodb')
table = dynamodb.Table('timing-demo-table')

def object_exists(bucket, key):
    try:
        s3.head_object(Bucket=bucket, Key=key)
        return True
    except s3.exceptions.ClientError as e:
        if e.response['Error']['Code'] == "404":
            return False
        raise

def lambda_handler(event, context):
    for record in event['Records']:
        bucket = record['s3']['bucket']['name']
        key = record['s3']['object']['key']
        event_id = hashlib.md5(key.encode()).hexdigest()

        # 👇 New verification logic with exponential backoff + jitter
        for attempt in range(5):
            if object_exists(bucket, key):
                print(f"Object available after {attempt} checks: {key}")
                table.put_item(Item={'id': event_id, 'filename': key, 'status': 'processed'})
                break
            delay = min(2 ** attempt, 16) + random.random()
            print(f"Waiting {delay:.1f}s before next check ({attempt+1}/5)")
            time.sleep(delay)
        else:
            print(f"Failed after retries: {key}")
EOF

✅ Redeploy and re-upload:

zip function.zip lambda_handler.py
aws lambda update-function-code \
  --function-name timingDemoHandler \
  --zip-file fileb://function.zip
aws s3 cp bigfile.bin s3://s3-timing-demo/

Step 5 – Re-Test and Confirm Self-Recovery

After the upload, check CloudWatch logs:

aws logs filter-log-events \
  --log-group-name /aws/lambda/timingDemoHandler \
  --filter-pattern "Object available"

✅ Output:

Object available after 3 checks: bigfile.bin

✅ Confirm DynamoDB entry:

aws dynamodb scan --table-name timing-demo-table

✅ Output:

{"Count":1,"Items":[{"filename":{"S":"bigfile.bin"},"status":{"S":"processed"}}]}

The function now waits for S3 consistency before committing downstream — the pipeline is reliable again.

Pro Tips

Use head_object() to verify readiness before attempting get_object(), and pair it with exponential backoff to avoid unnecessary retries.
Tune retry delay based on average object size and replication time.
For very large files, offload long polling to SQS and let Lambda pull asynchronously.
Combine backoff logic with idempotent writes to ensure duplicates never appear.

Conclusion

S3 does its job quickly — maybe too quickly.

The event pipeline fires the starter’s pistol before the data reaches the track.

By adding lightweight verification and backoff logic, you make Lambda resilient to S3’s timing drift.

Your ingestion becomes consistent, predictable, and self-healing.

In AWS, speed is easy — patience is architecture.

Aaron Rose is a software engineer and technology writer at tech-reader.blog and the author of Think Like a Genius.

Search This Blog

Tech-Reader.blog