Solve: Build-Time Drift and the Danger of ":latest" in AWS Lambda

Solve: Build-Time Drift and the Danger of :latest in AWS Lambda

What Happened?

You deployed a Golang-based AWS Lambda function, built in a Docker container based on arm64v8/amazonlinux:latest. Everything had been working fine—until it wasn’t. Suddenly, cold starts began failing with this runtime error: 

Bash

GLIBCXX_3.4.30 not found   

No code had changed. No libraries were touched. But your Lambda wouldn’t boot.

Why Did This Happen?

The :latest tag silently changed behind your back. That base image update included a newer version of libstdc++, which exposed a symbol (GLIBCXX_3.4.30) not present in the AWS Lambda runtime environment. Your Go binary was built expecting that symbol—so at cold start, it crashed.

Let's Talk About Drift

What you're seeing here is a textbook example of what we call build-time drift and runtime drift:

• Build-time drift occurs when your build environment (like a Docker base image) changes over time without coordination or visibility. You build your binaries today with one version of a system library, and tomorrow, the same build produces a subtly different binary because :latest moved.
  
  
• Runtime drift happens when your production environment doesn't evolve in sync with your build tools. In this case, AWS Lambda's runtime hadn't caught up with the library version introduced in your new build, causing incompatibility.

These terms help describe a pattern engineers have felt for years but haven’t always had the words for. It’s the same way people knew objects fell toward the earth long before we called it gravity—naming the force makes it easier to recognize and fight.

How Can I Reproduce This?

Clone this minimal repro (thanks to N Martin’s public example):
github.com/noel-ooh/amazonlinux-glib-issue

Use the following Dockerfile to simulate the problem: 

Docker

FROM arm64v8/amazonlinux:latest 

RUN yum install -y gcc-c++ glibc libstdc++ && \ 
    go build -o /app main.go 

CMD ["/app"]   

Deploy this binary to AWS Lambda using a custom runtime or container image, and you’ll likely trigger the cold-start failure.

What’s the Fix?

Pin the Docker base image to a known-safe version. In this case: 

Docker

FROM arm64v8/amazonlinux:2023.6.20250317.2   

This image doesn’t include the newer version of libstdc++, so your Go binary will avoid linking against unavailable runtime symbols.

How Do I Prevent This in the Future?

• Never use :latest in a production Dockerfile.
• Inspect your compiled Go binary using ldd or strings: 

Bash

ldd ./your_binary | grep GLIBC strings ./your_binary | grep GLIBC    

• Consider static linking with musl or xgo to remove runtime dependencies entirely.
• Align your build image with the Lambda runtime you're targeting.

TL;DR

This wasn’t your code’s fault—it was a quiet case of build-time drift and runtime drift.  Fix it by pinning your Docker base image. Avoid it by never trusting :latest.

Need AWS Expertise?

We'd love to help you with your AWS projects.  Feel free to reach out to us at info@pacificw.com.

Image: Gemini