Tech-Reader.blog

  The Secret Life of Azure: The Basement with No Bottom (Blob Storage) Storing massive amounts of unstructured data with Azure Blob Storage. #Azure #BlobStorage #DataEngineering #SAS Data & AI The library lobby was overflowing. Timothy was standing next to a mountain of heavy crates containing thousands of high-resolution scrolls, fragile audio cylinders, and millions of archived maps. "Margaret," Timothy said, "we’ve given the library its notebooks—the  Azure SQL  ledger for our strict records and the  Cosmos DB  binder for our flexible data—but I can't fit these crates into a notebook. I’m running out of floor space, and these items aren't just heavy; there are millions of them. Where do I put the things that aren't 'text'?" Margaret walked over to a heavy oak door in the floor and pulled it open to reveal a staircase leading down into an expanse that seemed to have no end. "Timothy, you're looking for  Azure Blob Storage . In the ...

  The Secret Life of Go: Worker Pools How to Stop Crashing Your Server with 10,000 Goroutines #Golang #Concurrency # WorkerPools #SystemDesign Part 26: Controlling Concurrency and The Dispatcher Ethan was staring at a wall of red text on his monitor. "Out of memory," he muttered. "How? Go is supposed to be incredibly efficient." Eleanor walked by with her coffee. "What did you crash this time?" "The image processing service," Ethan sighed. "We had a backlog of ten thousand images to resize. Since goroutines are lightweight, I just launched one for every single image so they would all process at the same time." He showed her the code: func ProcessAll (images []string) { var wg sync.WaitGroup for _, img := range images { wg. Add ( 1 ) go func (imageName string) { defer wg. Done () resizeImage (imageName) // Opens the file, decodes, resizes, saves }(img) } wg. Wait () } ...

  LimitExceededException  When Starting Too Many Rekognition Video Jobs How to track active video jobs and design around Rekognition concurrency limits #AWS #AmazonRekognition #CloudArchitecture #DevOps Category: Service Quotas & Throttling Problem Your system submits multiple video analysis jobs using  Amazon Rekognition . You call: StartLabelDetection StartFaceDetection StartContentModeration Other async video APIs The first few requests succeed. Then new requests fail with: LimitExceededException: The number of concurrent jobs exceeds the limit. IAM permissions are correct. S3 access works. SNS role is configured. Yet new video jobs refuse to start. Clarifying the Issue Rekognition video APIs are  asynchronous . When you call a  Start*  operation, AWS: Accepts the job Queues it for processing Returns a  JobId That job remains active until it completes. Each AWS account has a  maximum number of concurrent active video jobs per region . If yo...

  The Secret Life of AWS: The Plaintext Password (AWS Secrets Manager) Why hardcoding credentials is a breach waiting to happen. #AWS #SecretsManager #Security #DevOps 🎧 Audio Edition: Prefer to listen? Check out the expanded AI podcast version of this deep dive on YouTube . Part 45 of The Secret Life of AWS Timothy successfully connected his Inventory microservice to the Checkout database across the new Transit Gateway. The network was secure. The traffic was private. He opened his Lambda function configuration to add the database connection string. In the Environment Variables section, he typed: DB_HOST :  10.0.5.12   DB_USER :  admin   DB_PASS :  SuperSecretDatabasePassword123! He clicked Deploy. The function ran, connected to the database, and returned a 200 OK. Margaret walked by and glanced at his screen. "Timothy," she said, "you just secured the entire network, but you left the master database password sitting in plain text." "It's an environment v...

  The Secret Life of Azure: The Notebook (Azure SQL and Cosmos DB) Choosing between Azure SQL and Cosmos DB for your data. #Azure #DataArchitecture #CosmosDB #AzureSQL 🎧 Audio Edition: Prefer to listen? Check out the expanded AI podcast version of this deep dive on  YouTube . Data & AI The library was bustling, but Timothy was sitting behind a mountain of loose-leaf papers, looking defeated. He was trying to find a record of a book loan from three weeks ago, but the papers were in no particular order. "Margaret," he said, "the automated assembly line is bringing in thousands of new records every hour, but I don't know where to put them. I tried keeping a ledger, but some entries are just names, while others are long descriptions and photos. The ledger is too stiff, and these loose papers are too messy. I need a notebook that knows how to organize itself." Margaret picked up two very different books from a nearby shelf. One was a strictly lined ledger; the o...

  The Secret Life of Go: The Select Statement How to Stop Fast Data from Waiting on Slow Channels #Golang #Concurrency #CodingTips #SoftwareEngineering Part 25: The Multiplexer, The Timeout, and The Non-Blocking Read Ethan was watching his terminal output drip line by line. It was agonizingly slow. "I don't understand," he said, rubbing his eyes. "I have two goroutines sending data. One is a local cache that returns in one millisecond. The other is a network call that takes five seconds. But the fast data is waiting for the slow data." Eleanor walked over and looked at his code. The Problem Code: func process (cacheChan <-chan string, netChan <-chan string) { // Read from the network (takes 5 seconds) netData := <-netChan fmt. Println ( "Received:" , netData) // Read from the cache (takes 1 millisecond) cacheData := <-cacheChan fmt. Println ( "Received:" , cacheData) } "You have created a traffic ja...