Solve: Fixing Poisoned Dynamic References in CloudFormation

Solve: Fixing Poisoned Dynamic References in CloudFormation

Problem

A CloudFormation update fails due to a Secrets Manager dynamic reference that no longer resolves. Even after removing the reference from the template, CFN still attempts to evaluate the broken reference — blocking further updates and leaving the stack stuck.

Clarifying the Issue

Dynamic references like this: 

yaml

!Sub '{{resolve:secretsmanager:${SecretArn}:SecretString:TOKEN:AWSCURRENT:VersionId}}'

resolve at deployment planning time, not runtime. If the referenced VersionId is no longer valid for the given VersionStage (e.g., AWSCURRENT), the stack update fails.

The trap? Even if you remove the reference from your new template, CloudFormation still evaluates the old reference during update planning. So the same error occurs — again and again — and the stack appears unrecoverable.

Before / After Snippet

❌ Before (Poisoned) 

yaml

Resources:
  HelloWorldFunction:
    Type: AWS::Lambda::Function
    Properties:
      FunctionName: hello-world
      Role: !GetAtt LambdaExecutionRole.Arn
      Handler: index.handler
      Runtime: nodejs18.x
      Environment:
        Variables:
          TOKEN: !Sub >
            {{resolve:secretsmanager:${SecretArn}:SecretString:TOKEN:AWSCURRENT:badVersionId}}
      Code:
        S3Bucket: my-bucket
        S3Key: my-code.zip
  SecretArn:
    Type: AWS::SSM::Parameter
    Properties:
      Type: String
      Name: /config/tokenArn

✅ After (Clean) 

yaml

Resources:
  HelloWorldFunctionV2:
    Type: AWS::Lambda::Function
    DeletionPolicy: Retain
    Properties:
      FunctionName: hello-world-v2
      Role: !GetAtt LambdaExecutionRole.Arn
      Handler: index.handler
      Runtime: nodejs18.x
      Environment:
        Variables:
          TOKEN: !Sub >
            {{resolve:secretsmanager:${SecretArn}:SecretString:TOKEN:AWSCURRENT}}
      Code:
        S3Bucket: my-bucket
        S3Key: my-code.zip

  SecretArn:
    Type: AWS::SSM::Parameter
    Properties:
      Type: String
      Name: /config/tokenArn

Visual Flowchart: Recovery Plan 

text

CloudFormation Update Fails
│
├──> SecretsManager dynamic ref is invalid
│     └── VersionStage ≠ VersionId
│
├── Does removing the reference from template fix it?
│     └── No — CFN still evaluates old reference during planning
│
├── Fix Plan:
│
├──> Step 1: Remove Lambda from template
│     └── Add `DeletionPolicy: Retain` to preserve it
│
├──> Step 2: Align VersionStage to correct VersionId
│     └── Use `update-secret-version-stage` to point AWSCURRENT correctly
│
├──> Step 3: Redeploy template without Lambda
│     └── Stack should now update successfully
│
├──> Step 4: Reintroduce Lambda
│     ├── Option A: Add original Lambda back
│     └── Option B: Use new logical ID to avoid poisoned history
│
└──> ✅ Stack is now healthy and deploys cleanly

Why It Matters

• Dynamic references are powerful — but brittle.
• Once broken, they can lock you out of CloudFormation updates entirely.
• Standard rollback paths may fail because the evaluation logic is baked into CFN’s planning phase.

Key Terms

• Secrets Manager Dynamic Reference – Inline syntax to pull secrets into a template at deploy time
• VersionId – Unique identifier for a specific version of a secret
• VersionStage – Label attached to one or more versions (e.g., AWSCURRENT)
• CloudFormation Update Planning – Phase where references are resolved before resources are modified
• DeletionPolicy: Retain – Instructs CFN not to delete the resource, even if removed from the template

Steps at a Glance

1. Temporarily remove the poisoned Lambda using Retain
2. Manually fix the secret’s VersionStage and VersionId
3. Update the stack without the Lambda
4. Re-introduce the Lambda or create a new logical ID
5. Confirm stack health and resume normal updates

Detailed Steps

Step 1: Remove the Lambda with a Retain Policy 

yaml

Resources:
  HelloWorldFunction:
    DeletionPolicy: Retain

Remove the Lambda from the template temporarily. This tells CloudFormation: "Don't touch this resource, even if it's gone from the template."

Step 2: Manually Align the Secret’s VersionStage and VersionId

Inspect the secret: 

bash

aws secretsmanager describe-secret --secret-id my-api-token

Fix the stage: 

bash

aws secretsmanager update-secret-version-stage \
  --secret-id my-api-token \
  --version-stage AWSCURRENT \
  --move-to-version-id correctVersionId

Step 3: Update the Stack Without the Lambda 

bash

aws cloudformation update-stack --stack-name my-stack \
  --template-body file://main-clean.yaml \
  --capabilities CAPABILITY_IAM

Once this succeeds, the poisoned reference is out of scope.

Step 4: Reintroduce or Replace the Lambda

Add it back or use a new logical ID: 

yaml

Resources:
  HelloWorldFunctionV2:
    Type: AWS::Lambda::Function
    ...

This ensures CloudFormation doesn’t tie back to the old, poisoned resource history.

Step 5: Confirm Stack Health 

bash

aws cloudformation describe-stacks --stack-name my-stack 

Ensure UPDATE_COMPLETE and verify your template: 

bash

aws cloudformation get-template --stack-name my-stack

Conclusion

Broken dynamic references can freeze a stack in place, especially when CloudFormation won’t let go of its previous template context. But with a smart use of Retain, manual Secrets Manager surgery, and a clean re-introduction of the poisoned resource, you can recover the stack without blowing it away.

Stay tuned — in a future post, we’ll explore how to prevent this class of error entirely with safe referencing patterns and automated secret rotation sanity checks.

* * *

Aaron Rose is a software engineer and technology writer.