When Persistence Breaks in LocalStack Paid Tiers: How to Diagnose and Fix Incomplete Restores
You restart LocalStack Pro, and suddenly your Lambda can’t find its API Gateway. The resources are still there — but the links between them are gone.
Aaron Rose
Software Engineer & Technology Writer
Problem
In LocalStack’s paid tiers, persistence sometimes fails to fully restore your environment. Core resources like S3 buckets and DynamoDB tables usually come back, but the links—connections between services such as Lambda, API Gateway, IAM roles, and event source mappings—can break. The result: resources exist, but the wiring between them doesn’t.
Clarifying the Issue
- Is this a basic persistence reload or a Cloud Pod restore?
- Which links are missing (Lambda code/ARN, API Gateway integration URI, IAM role attachment, event source mapping)?
- Was the stack fully deployed and healthy before saving state?
Visual Overview
Flow Sketch
[ S3 ] [ DynamoDB ]
\ /
\ /
v v
[ Lambda ] <—— role/code
|
v
[ API Gateway ]
Concept: resources connected by links.
Tree Diagram with ✅/❌
stack/
├── s3-bucket/
├── dynamodb-table/
├── lambda-function/
│ ├── iam-role/ ✅ healthy
│ ├── code-package/ ✅ healthy
│ └── (❌ broken restore: role/code missing)
├── apigateway/
│ ├── integration-URI -> lambda-function ✅ healthy
│ └── (❌ broken restore: URI missing/invalid)
└── event-source-mapping/
├── dynamodb-stream -> lambda-function ✅ healthy
└── (❌ broken restore: mapping lost)
Reality: resources survive, links can vanish.
Why It Matters
- Broken links create flaky environments and false failures in tests/CI.
- Team workflows stall if snapshots aren’t reliably reproducible.
Key Terms
- Persistence: keeps LocalStack state between restarts.
- Cloud Pods (Team): versioned, shareable snapshots (
pod save
/pod restore
). - Export/Import State: file-based save/restore for reproducibility.
Steps at a Glance
- Reproduce cleanly on paid tier
- Validate stack health before saving
- Restore and check links
- Re-attach broken links with
awslocal
- Harden workflow with a checklist and automation
Detailed Steps
1. Reproduce cleanly on paid tier
# Persistence path
LOCALSTACK_PERSISTENCE=1 localstack start
# Cloud Pods (Team)
localstack start
localstack pod save myapp:v1
localstack pod restore myapp:v1
2. Validate stack health before saving
awslocal s3 ls
awslocal dynamodb list-tables
awslocal lambda list-functions
curl -i "http://localhost:4566/restapis/$API_ID/test/_user_request_/hello"
3. Restore and check links
Common errors when links break:
ResourceNotFoundException: Function not found
Invalid ARN
The integration URI is invalid
AccessDeniedException
InternalFailure
(API Gateway → Lambda)
4. Re-attach broken links
awslocal lambda create-function
awslocal lambda create-function \
--function-name MyFn \
--runtime python3.11 \
--role arn:aws:iam::000000000000:role/lambda-ex \
--handler handler.main \
--zip-file fileb://function.zip
awslocal apigateway put-integration
awslocal apigateway put-integration \
--rest-api-id "$API_ID" \
--resource-id "$RES_ID" \
--http-method GET \
--type AWS_PROXY \
--integration-http-method POST \
--uri "arn:aws:apigateway:local:lambda:path/2015-03-31/functions/$LAMBDA_ARN/invocations"
awslocal lambda add-permission \
--function-name MyFn \
--statement-id apigw-invoke \
--action lambda:InvokeFunction \
--principal apigateway.amazonaws.com
awslocal lambda create-event-source-mapping
awslocal lambda create-event-source-mapping \
--function-name MyFn \
--event-source-arn "$DDB_STREAM_ARN" \
--starting-position LATEST
5. Harden workflow
Checklist before save
- ✅ Stack deployed, no pending changes
- ✅ Health checks pass (S3, DynamoDB, Lambda, API GW)
- ✅ IAM roles and mappings confirmed
Automation
- CI jobs run health checks →
pod save
(Team) or export state - Smoke test on restore (invoke Lambda, hit API GW endpoint)
- CI jobs run health checks →
Conclusion
LocalStack’s paid tiers provide tools, not magic. Persistence issues usually come down to saving too early or losing links across services. By visualizing the problem, using copy-paste commands to re-attach links, and locking in a simple checklist + automation, persistence stops being guesswork and becomes muscle memory.
Aaron Rose is a software engineer and technology writer at tech-reader.blog and the author of The Rose Theory series on math and physics.
Comments
Post a Comment