AWS Lambda Error – Handler timed out (execution duration exceeded)

#aws#lambda#devops#cloud

A practical diagnostic guide for debugging Lambda timeout failures caused by slow dependencies, CPU starvation, VPC latency, event-loop hangs, or upstream service limits.

Problem

Your Lambda invocation fails with:

Task timed out after X.XX seconds

Lambda forcibly killed the execution because your function exceeded the configured timeout. This is a hard SIGKILL — no cleanup runs afterward.

Possible causes:

• Slow external APIs or database calls
• CPU starvation due to low memory (memory = CPU)
• Unawaited promises / blocked event loop
• Cold start overhead (large packages, VPC ENI attachments)
• VPC NAT latency or DNS slowness
• API Gateway’s strict 29-second ceiling

---

Clarifying the Issue

Timeout = Your handler did not return fast enough.
Lambda terminates execution immediately — not an exception, not recoverable.

API Gateway callers can only wait 29 seconds, while Lambda can run 15 minutes.

If Lambda continues running after the caller gives up?
You still pay for the execution.

---

Why It Matters

Timeouts cause:

• Retry storms (SQS, SDK retries, EventBridge)
• Ghost billing (Lambda continues running after API Gateway times out)
• SQS backlog growth
• Cold start amplification from repeated crashes

To fix timeouts, you must find the actual bottleneck — not simply increase the timeout.

---

Key Terms

• Timeout — Maximum execution duration allowed
• Billed Duration — Actual time billed
• INIT_DURATION — Time spent loading code before handler runs
• CPU–Memory Link — More memory = more CPU = faster execution
• X-Ray — AWS tracing that visualizes slow components

---

Diagnostic Flow

Timeout
  ↓
Check CloudWatch Logs
  ↓
INIT_DURATION high? → VPC or large packages
  ↓
Handler duration high? → X-Ray → external calls or CPU starvation

---

Steps at a Glance

1. Identify Lambda timeout & memory size
2. Inspect CloudWatch logs & X-Ray
3. Test external dependencies
4. Detect blocking / hung code
5. Check VPC latency & ENI cold starts
6. Increase memory (CPU) or raise timeout
7. Validate API Gateway constraints

---

Detailed Steps

Step 1: Identify the function configuration

aws lambda get-function-configuration \
  --function-name my-fn \
  --query '{Timeout:Timeout,Memory:MemorySize}'

Example output:

{
  "Timeout": 3,
  "Memory": 128
}

If you have a 3-second timeout but dependencies regularly take 1.5–2 seconds, your function is already running too close to the wall.

---

Step 2: Inspect CloudWatch logs & enable X-Ray

View recent logs:

aws logs tail /aws/lambda/my-fn --since 5m

Look for:

• INIT_DURATION — high = cold start problem
• Billed Duration == Timeout — Lambda killed your handler

Enable X-Ray:

aws lambda update-function-configuration \
  --function-name my-fn \
  --tracing-config Mode=Active

X-Ray reveals:

• Where time is spent (HTTP, DB, DNS, etc.)
• Whether initialization or invocation dominates
• Whether retries / DNS lookups are consuming latency

---

Step 3: Test external dependencies (HTTP/DB)

Python:

import time, requests
t0 = time.time()
requests.get(url, timeout=3)
print(f"Fetch: {(time.time() - t0)*1000:.2f} ms")

Go:

start := time.Now()
client := http.Client{ Timeout: 2 * time.Second }
_, _ = client.Get(url)
fmt.Println("Fetch:", time.Since(start))

If external calls exceed 500–800ms consistently, you’ve found your timeout root cause.

---

Step 4: Detect unawaited promises or blocking code

For Node.js:

node --trace-warnings index.js

Common slowdowns:

• Missing await
• Promise never resolved
• Large synchronous operations (JSON.parse on multi-MB payloads)
• CPU-heavy loops

These block the event loop and cause handler overruns.

---

Step 5: Check VPC networking & ENI cold starts

Two VPC killers:

1. ENI Cold Start Cost

VPC-attached Lambdas may spend 5–15 seconds attaching a network interface.

High INIT_DURATION = smoking gun.

2. NAT / DNS Latency

Test outbound speed:

time curl https://example.com

If latency > 1s, investigate:

• NAT Gateway saturation
• Route tables
• SGs
• DNS slowness

---

Step 6: Increase memory (CPU) or raise timeout

Option A (Recommended): Increase Memory (faster CPU)

Lambda CPU scales with memory:

Memory	Approx CPU
128MB	~0.125 vCPU
512MB	~0.5 vCPU
1024MB	~1 vCPU

Increasing memory often reduces cost because execution time drops.

aws lambda update-function-configuration \
  --function-name my-fn \
  --memory-size 512

Option B: Increase the Timeout

aws lambda update-function-configuration \
  --function-name my-fn \
  --timeout 30

Warning: API Gateway never waits beyond 29 seconds.

---

Step 7: Re-test using real events

aws lambda invoke \
  --function-name my-fn \
  --payload file://event.json \
  out.json

If your timeout disappears, the bottleneck you fixed was the correct one.

---

Pro Tips

• Fail fast:
  Set HTTP client timeouts below your Lambda timeout.

• Use X-Ray early:
  It is the fastest way to pinpoint slow components.

• Right-size memory:
  CPU-bound loads (PDF parsing, encryption) scale dramatically with memory.

• Synchronous hotspots:
  Any large synchronous workload freezes runtimes across Python, Node, and Go.

• Downstream capacity matters:
  High Lambda concurrency can exceed database connection limits or API rate limits, indirectly causing timeouts.

• Find slowest invocations:
  

filter @type="REPORT"
| stats max(@duration) by @requestId
| sort max(@duration) desc

---

Conclusion

Timeouts are never random — they’re signals that Lambda is waiting too long on compute, networking, or external services.

By inspecting logs, enabling X-Ray, testing dependencies, correcting VPC configuration, and sizing memory properly, you eliminate the real bottleneck and avoid retry storms and billing traps.

---

Aaron Rose is a software engineer and technology writer at tech-reader.blog and the author of Think Like a Genius.