The Secret Life of AWS: The Pulse (Observability)
.jpeg)
Monitoring is not enough. How to use AWS X-Ray and CloudWatch to achieve Observability.
Part 17 of The Secret Life of AWS
Timothy stared at his dashboard. Every single widget was green.
- CPU Utilization: 12%
- Memory Usage: 40%
- HTTP Status: 200 OK
Yet, his inbox was full of complaints. "The application is slow," the emails said. "The checkout page takes ten seconds to load."
"I don't understand," Timothy said, pointing at the green dashboard. "The metrics say everything is healthy. The server isn't crashing. The CPU is bored. But the users say it is broken."
Margaret stood behind him. "You are looking at Monitoring," she said. "Monitoring tells you if the system is up. But it doesn't tell you why it is slow."
"To answer 'why'," she continued, "you need Observability."
The Blind Spot
Margaret opened the Amazon CloudWatch console.
"Right now, you are collecting Metrics," she explained. "Metrics are aggregates. They tell you that on average, your API takes 200 milliseconds. But aggregates hide the outliers."
"If 99 requests take 10 milliseconds, and 1 request takes 10 seconds, the average is still low," she noted. "But that one user is having a terrible experience."
"How do I find that one user?" Timothy asked.
"We need to trace the request," Margaret said. "We need AWS X-Ray."
Distributed Tracing (AWS X-Ray)
Margaret navigated to the AWS X-Ray service in the console.
"In a modern cloud application, a single user request might touch five different services," she said. "It hits the Load Balancer, then a Lambda function, then a DynamoDB table, and maybe a third-party API."
"If the request is slow, you need to know which step is slow."
"AWS X-Ray assigns a unique Trace ID to every incoming request," she explained. "As the request travels through your system, X-Ray records the start and end time of every hop. These are called Segments."
"Is it hard to install?" Timothy asked.
"No," Margaret said. "For your code, we just import the AWS X-Ray SDK. It automatically instruments your calls to other AWS services."
The Service Map
Margaret enabled X-Ray for Timothy's application and waited for a few minutes. Then, she opened the Service Map.
It visualized the architecture as a graph. Most nodes were green, but one circle—representing the DynamoDB database—was yellow.
"Look at the trace," Margaret said, clicking on a slow request.
The timeline view opened. Timothy could see the request lifecycle:
- Lambda Invocation: 15ms
- Processing Logic: 10ms
- DynamoDB Query: 8,000ms
"There is your latency," Margaret said. "It is not the CPU. It is not the code. It is the database query."
Timothy looked closer. "I am scanning the entire table instead of querying by index."
"Exactly," Margaret said. "Your metrics showed low CPU because the CPU was just waiting for the database to respond. Monitoring showed you the symptom. Observability showed you the root cause."
The Cost of Insight (Sampling)
"Does this generate a lot of data?" Timothy asked. "Tracing every single request sounds expensive."
"It can be," Margaret warned. "That is why we use Sampling Rules. We don't need to trace 100% of the traffic to see the patterns. We might trace only 5%, or only requests that result in an error. That gives us the insight without the cost."
The Three Pillars
"True Observability relies on three signals working together," Margaret summarized.
- Metrics: Numeric data measured over time (e.g., "CPU is at 50%"). This alerts you when something is wrong.
- Logs: Text records of discrete events (e.g., "Error: Connection timeout"). This gives you the details.
- Traces: The path of a request through the system (e.g., "The delay occurred in the database"). This shows you where the latency is.
"You had Metrics and Logs," she said. "But without Traces, you were guessing."
The Lesson
Timothy updated his code to fix the database query. He deployed the change (using the Pipeline from Episode 16). Within minutes, the emails stopped.
He looked at the X-Ray console. The yellow node turned green.
"I thought green meant 'good'," Timothy said.
"Green means 'up'," Margaret corrected. "Observability means you understand why it is green."
Aaron Rose is a software engineer and technology writer at tech-reader.blog and the author of Think Like a Genius.
Comments
Post a Comment