The Secret Life of AWS: The Vault (AWS Secrets Manager)

How to securely store and retrieve API keys and credentials in a serverless architecture

#AWS #SecretsManager #APIKeys #Security

Margaret is a senior software engineer. Timothy is her junior colleague. They work in a grand Victorian library in London — the kind of place where code quality is the unspoken objective, and craftsmanship is the only thing that matters.

Episode 67

Timothy was staring intently at the code for his Payment Lambda function. After using AWS X-Ray the previous week to discover that their external payment gateway was causing latency spikes, he was reviewing his integration logic to see if he could optimize the connection timeout.

Margaret walked into the studio and glanced at his monitor. She zeroed in on the AWS Console tab he had open in the background.

"Timothy," Margaret said, her voice carrying a rare tone of strictness. "Is that the production API key for our payment gateway sitting in plain text in your Lambda environment variables?"

Timothy blinked, slightly defensive. "Yes. The environment variables are encrypted at rest by AWS, and it keeps the key out of our source code repository. Isn't that the best practice?"

"Keeping it out of source control is a good start, but environment variables are for configuration, not credentials," Margaret explained, pulling up a chair. "Anyone on our team with basic read access to the Lambda console can see that key. Furthermore, if the payment provider requires us to rotate that key every ninety days, you have to manually update the variable and redeploy the entire Lambda function. That is a security incident waiting to happen. We need to move that key into AWS Secrets Manager."

The Digital Vault

Margaret took the keyboard and navigated to the AWS Secrets Manager console.

"Secrets Manager is exactly what it sounds like—a highly secure, encrypted digital vault," Margaret said. She created a new secret named prod/payment/gateway-key and pasted Timothy's API key inside. "We can even define this secret in our CloudFormation template, just like we learned in Episode 55, keeping our infrastructure as code while injecting the actual value securely."

She opened the IAM dashboard. "Now, access to this specific secret is strictly governed by IAM. We scope the permission to the exact secret ARN—not just all secrets—following the principle of least privilege. A developer can have full access to view and edit the Lambda function, but unless their IAM role explicitly grants them secretsmanager:GetSecretValue for this exact resource, they cannot see this key."

"Okay, so the vault is locked," Timothy nodded. "But how does the Lambda function get the key when a customer tries to checkout?"

"We teach your code to unlock the vault at runtime," Margaret replied. She updated his Node.js Payment service:

const { SecretsManagerClient, GetSecretValueCommand } = require("@aws-sdk/client-secrets-manager");

const client = new SecretsManagerClient({ region: "us-east-1" });
let cachedSecret = null; // Cache the secret outside the handler

exports.handler = async function processPayment(event) {
    // 1. Check the in-memory cache first
    if (!cachedSecret) {
        console.log("Cache miss: Fetching secret from AWS Secrets Manager...");
        const command = new GetSecretValueCommand({ SecretId: "prod/payment/gateway-key" });
        const response = await client.send(command);
        cachedSecret = response.SecretString;
    }

    // 2. Use the secret to authorize the external payment gateway
    const paymentResult = await callExternalGateway(event.amount, cachedSecret);
    return paymentResult;
}

The In-Memory Cache

Timothy read the code carefully. "Wait, I recognize this pattern. You initialized the cachedSecret variable outside of the Lambda handler, just like we did with the Redis connection back in Episode 64."

"Excellent architectural memory," Margaret smiled. "Every time you call the Secrets Manager API, it costs a fraction of a cent and adds a few milliseconds of network latency. If we have ten thousand checkouts an hour, we do not want to fetch the secret ten thousand times."

"So by declaring it outside the handler," Timothy realized, "the first checkout after a cold start will experience a cache miss and a few milliseconds of latency. But after that, the execution environment holds the secret in its local memory. The next nine thousand checkouts just use the cached version instantly."

"Precisely," Margaret said. "And because the secret is isolated in Secrets Manager, we can enable automatic rotation. We can configure AWS to automatically negotiate a new key with the payment provider every thirty days, without ever touching our application code or requiring a deployment."

Timothy updated his architecture diagram. His system was not just fast, resilient, and observable; its most sensitive data was now locked in a vault, protected from both internal eyes and external threats.

Key Concepts Introduced

Environment Variables vs. Secrets: Environment variables should be used for non-sensitive configuration data (like database table names, log levels, or timeout limits). They should never be used for sensitive credentials (like API keys or database passwords) because they are easily visible in the AWS Console and require a full code redeployment to update.

AWS Secrets Manager: A fully managed service that securely stores, encrypts, and controls access to sensitive credentials. It integrates natively with AWS Identity and Access Management (IAM), ensuring that only authorized users and compute roles can retrieve the plaintext secrets. Following the principle of least privilege, IAM permissions should be scoped to the specific secret's ARN.

In-Memory Secret Caching: Retrieving a secret from AWS Secrets Manager incurs a small API cost and network latency. In a serverless environment, developers should fetch the secret once and cache it in a variable defined outside the Lambda handler. The first request will experience a cold-start cache miss, but subsequent invocations within the same warm execution environment reuse the secret instantly.

Automatic Secret Rotation: A security best practice where credentials are changed frequently to limit the blast radius if a key is ever compromised. AWS Secrets Manager can automate this process using a helper Lambda function to negotiate new credentials with third-party APIs or databases on a set schedule, completely eliminating manual human intervention.

Aaron Rose is a software engineer and technology writer at tech-reader.blog.

Catch up on the latest explainer videos, podcasts, and industry discussions below.

Search This Blog

Tech-Reader.blog