LimitExceededException When Starting Too Many Rekognition Video Jobs
LimitExceededException When Starting Too Many Rekognition Video Jobs
How to track active video jobs and design around Rekognition concurrency limits
#AWS #AmazonRekognition #CloudArchitecture #DevOps
Category: Service Quotas & Throttling
Problem
Your system submits multiple video analysis jobs using Amazon Rekognition.
You call:
StartLabelDetectionStartFaceDetectionStartContentModeration- Other async video APIs
The first few requests succeed.
Then new requests fail with:
LimitExceededException: The number of concurrent jobs exceeds the limit.
IAM permissions are correct. S3 access works. SNS role is configured.
Yet new video jobs refuse to start.
Clarifying the Issue
Rekognition video APIs are asynchronous.
When you call a Start* operation, AWS:
- Accepts the job
- Queues it for processing
- Returns a
JobId
That job remains active until it completes.
Each AWS account has a maximum number of concurrent active video jobs per region.
If you attempt to start more jobs than allowed, Rekognition rejects the request with LimitExceededException.
đź“Ś This is not throttling.
đź“Ś It is a concurrency ceiling.
Why It Matters
This commonly appears when:
- Batch-submitting many videos at once
- Replaying failed jobs without coordination
- Running multiple ingestion pipelines simultaneously
- Scaling horizontally without concurrency controls
Unlike ThrottlingException, this failure does not depend on request speed.
It depends on how many jobs are currently active.
If five jobs are allowed concurrently and five are already running, the sixth request fails instantly.
This is a concurrency-boundary issue — not a Rekognition issue.
Key Terms
Asynchronous API – Operation that starts a background job and returns immediately.
Concurrent Job Limit – Maximum number of active video analysis jobs allowed per account per region.
JobId – Unique identifier returned when starting a Rekognition video job.
SNS Notification Channel – Mechanism used by Rekognition to notify when a job completes.
Steps at a Glance
- Confirm which
Start*API call is failing. - Count currently active Rekognition video jobs.
- Check your account’s concurrent job quota.
- Implement controlled job submission.
- Validate job completion tracking via SNS.
- Request a quota increase if required.
Detailed Steps
Step 1: Identify the Failing API
Review logs to confirm which operation produces the error:
StartLabelDetectionStartFaceDetectionStartTextDetectionStartContentModeration
Only async video operations are subject to concurrent job limits.
Step 2: Determine Active Job Count
Track:
- Job submissions
- Job completions
- Outstanding
JobIdvalues
If your system does not maintain a list of active JobIds, you cannot enforce concurrency discipline.
Important nuance:
Rekognition job metadata is retained for 7 days. If your system relies on polling old job IDs beyond that window, or fails to retrieve results before expiration, your internal state machine can drift out of sync.
Concurrency control depends on accurate lifecycle tracking.
Step 3: Check Concurrent Job Quotas
Navigate to:
AWS Console → Service Quotas → Rekognition
Locate concurrent video job limits.
Important considerations:
- Quotas are region-specific.
- Different video analysis types (Label Detection, Content Moderation, etc.) may share a concurrency bucket or have separate limits depending on the region.
Always verify the exact quota name associated with the API you are using.
Step 4: Implement Controlled Job Submission
Do not submit all jobs simultaneously.
Instead:
- Maintain an internal queue
- Submit only N jobs at a time (where N is below quota)
- Start a new job only when one completes
Architectural pattern:
- Producer adds video to queue
- Worker checks current active count
- Worker starts new job only if below threshold
Retry logic alone will not solve concurrency ceilings.
Step 5: Validate Completion Handling
Rekognition signals job completion via SNS.
Ensure:
- SNS topic permissions are correct
- The consuming service reliably processes notifications
- Completed jobs are marked finished in your system
If SNS delivery fails or is ignored, your system may continue submitting jobs blindly until the limit is reached.
Concurrency safety requires reliable completion processing.
Step 6: Request a Quota Increase
If production workload requires higher concurrency:
- Submit a quota increase request
- Specify expected concurrent job volume
- Confirm region alignment
AWS commonly increases limits for legitimate production use cases.
Pro Tips
- This error is about how many jobs exist — not how fast they start.
- Retry logic without lifecycle tracking makes the problem worse.
- Always track active
JobIdvalues in durable storage. - Rekognition job metadata persists for only 7 days — design your tracking system accordingly.
- Verify which specific video quota applies to your API call.
- Concurrency ceilings require architectural control, not just configuration changes.
If you see LimitExceededException on a Start* call, count active jobs before touching IAM.
Conclusion
LimitExceededException during Rekognition video processing means your account has exceeded its allowed number of concurrent active jobs.
Rekognition is not malfunctioning.
Your workload has reached its concurrency ceiling.
The solution is architectural:
- Track job lifecycle
- Limit concurrent submissions
- Or increase your quota
This is a concurrency-boundary issue — not a Rekognition issue.
Aaron Rose is a software engineer and technology writer at tech-reader.blog and the author of Think Like a Genius.
Comments
Post a Comment