The Secret Life of AWS: The Fanout (Amazon SNS & Pub/Sub)
.jpg)
The Secret Life of AWS: The Fanout (Amazon SNS & Pub/Sub)
How to broadcast one event to multiple microservices
#AWS #SNS #PubSub #Microservices
🎧 Audio Edition: Prefer to listen? Check out the expanded AI podcast version of this deep dive on YouTube.
đź“ş Video Edition: Prefer to watch? Check out the 7-minute visual explainer on YouTube.
Part 47 of The Secret Life of AWS
Timothy’s Checkout service was finally stable. By routing messages through Amazon SQS, the Checkout process was completely insulated from Inventory database outages.
But success brings new requirements.
"The product team wants to expand," Timothy told Margaret, mapping out the new flow on the whiteboard. "When a customer clicks 'Buy', we still need to update Inventory. But now, we also need to notify the Fulfillment service to print a shipping label, and we need to trigger the Notification service to email a receipt."
Timothy drew three arrows pointing out of the Checkout service. "I am going to update the Checkout code. Instead of sending one message to the inventory-queue, it will loop and send three separate messages: one to inventory-queue, one to fulfillment-queue, and one to email-queue."
Margaret shook her head and handed him an eraser. "You are re-coupling the architecture, Timothy."
"How?" Timothy asked. "They are all asynchronous queues. Checkout isn't waiting for any of them to finish."
"But Checkout now has to know about all of them," Margaret explained. "If we add a Data Analytics queue next month, you have to rewrite and redeploy the Checkout code to send a fourth message. The Checkout service shouldn't care who is listening. It should only care that an order was placed."
"We need to implement the Publish/Subscribe (Pub/Sub) pattern using Amazon SNS."
The Broadcaster (Amazon SNS)
Margaret opened the AWS Console and navigated to Amazon Simple Notification Service (SNS).
"SQS is for one-to-one messaging," she explained. "A message goes into a queue, and one consumer pulls it out. SNS is for one-to-many broadcasting. We use it to announce events to the entire architecture."
She created a new SNS Topic and named it order-placed-topic.
"This is our logical access point," Margaret said. "From now on, the Checkout service does exactly one thing: it publishes a single JSON event to this SNS Topic. That is it. Its job is done."
The Fanout Pattern
"But if Checkout is publishing to a Topic, how do the messages get into the SQS queues so the other services can poll them?" Timothy asked.
"Through Subscriptions," Margaret replied.
She navigated to the new SNS Topic and clicked Create subscription. She selected Amazon SQS as the protocol and pointed it to the inventory-queue. She repeated the process for the fulfillment-queue and the email-queue.
"This is the SNS to SQS Fanout pattern," she explained. "When Checkout publishes one message to the SNS Topic, SNS instantly pushes a copy of that exact message into all three SQS queues simultaneously. SNS topics can even deliver to SQS queues in completely different AWS accounts, making this perfect for large organizations with strict account boundaries."
"The beauty of this is absolute architectural decoupling," she noted. "The publisher (Checkout) has no idea who the subscribers are. The subscribers (Inventory, Fulfillment, Email) have no idea who the publisher is. They only know about the Topic."
Message Filtering
Timothy looked at the diagram. "Wait. What if someone buys a digital gift card? The Inventory and Email services need to know, but the Fulfillment service doesn't need to print a physical shipping label. Won't Fulfillment get clogged with digital orders?"
"Excellent architectural foresight," Margaret smiled. "We handle that with SNS Message Filtering."
She opened the subscription configuration for the fulfillment-queue. She added a JSON filter policy:
{
"item_type": ["physical"]
}
"Now," Margaret explained, "SNS inspects the incoming message from Checkout. If the JSON payload says "item_type": "digital", SNS routes copies to Inventory and Email, but it silently drops the message for Fulfillment. We don't have to write custom filtering logic in our application code; the AWS network handles it for us."
Limits and Guarantees
Timothy updated the Checkout code to make a single SDK call to SNS instead of multiple calls to SQS.
"What if the order payload is huge?" Timothy asked. "Like a massive JSON blob containing dozens of custom items?"
"SNS has a strict 256KB payload limit," Margaret cautioned. "If your message exceeds that, we store the payload in an Amazon S3 bucket and just send the S3 URI through SNS. That is called the Claim-Check pattern."
"And remember our lesson on idempotency," Margaret added. "Like SQS, standard SNS guarantees at-least-once delivery. Your subscribers must still be idempotent to handle the rare duplicate message. If a workflow requires strict ordering and exactly-once processing, we would use SNS FIFO topics paired with SQS FIFO queues—but for our standard Checkout flow, this is perfect."
"So if the Analytics team wants a feed of all orders next week..." Timothy started.
"You don't touch the Checkout code," Margaret finished. "You just create an analytics-queue and subscribe it to the Topic. The architecture scales itself."
Key Concepts
- Amazon SNS (Simple Notification Service): A fully managed Pub/Sub service for A2A (application-to-application) and A2P (application-to-person) messaging. It uses a "push" model.
- Publish/Subscribe (Pub/Sub): An asynchronous messaging pattern where senders (publishers) categorize published messages into classes without knowledge of which subscribers, if any, there may be.
- Topic: A logical access point and communication channel in SNS. Publishers send messages to topics, and subscribers receive them from topics.
- Fanout Pattern: An architecture where a single message published to an SNS topic is replicated and pushed to multiple Amazon SQS queues, parallelizing asynchronous processing.
- Message Filtering: An SNS feature that allows subscribers to specify which subset of messages they want to receive from a topic, reducing downstream compute costs.
- Claim-Check Pattern: A messaging pattern used to bypass size limits by storing the actual payload in a database or object store (like S3) and passing only the reference (the claim check) through the message bus.
Aaron Rose is a software engineer and technology writer at tech-reader.blog and the author of Think Like a Genius.
Comments
Post a Comment