SageMaker Error – AccessDeniedException: User is not authorized to perform action

#aws #sagemaker #machinelearning #cloudcomputing

A diagnostic guide for resolving IAM permission failures in SageMaker Notebooks, Training Jobs, and Endpoints regarding Execution Roles and S3 access.

Problem

Your SageMaker Training Job, Processing Job, or Notebook fails with a permission error.

The Error:
ClientError: An error occurred (AccessDeniedException) when calling the CreateTrainingJob operation: User: arn:aws:sts::123456789:assumed-role/MySageMakerRole/Session is not authorized to perform: sagemaker:CreateTrainingJob

Or, when accessing S3 data inside a notebook:
403 Forbidden: Access Denied

This means the identity attempting to run the action does not have the required permissions attached to its IAM policy.

Potential causes:

Execution Role Mismatch: The notebook is using a role that lacks S3 access.
Bucket Policy: The S3 bucket explicitly denies external roles.
KMS Keys: The data is encrypted, and the role has access to S3 but not the decryption key.
PassRole Missing: You have permission to create a job, but not permission to pass the role to the service.

Clarifying the Issue

This error is rarely about your personal IAM user (the one you used to log into the console). It is about the Execution Role.

When you run a command in a SageMaker Notebook or launch a Training Job, AWS uses a specific "Service Role" (Execution Role) to perform actions on your behalf.

If you can log in, but your code fails to read a file: It's the Execution Role.
If you can't even open the SageMaker console page: It's Your User.

This guide focuses on the most common scenario: The code runs, but hits a wall.

Why It Matters

SageMaker operates in a highly sandboxed environment. To keep data secure, it defaults to "Zero Trust." If you don't explicitly grant your notebook permission to touch a specific S3 bucket, your data science workflow halts immediately.

Steps at a Glance

Analyze the Error ARN to identify who is being denied.
Locate the Execution Role in the SageMaker console.
Audit S3 Permissions attached to that role.
Check KMS Key Policy (If using encryption).
Verify `iam:PassRole` (If launching jobs).
Test specific access using the AWS CLI.

Detailed Steps

Step 1: Analyze the Error ARN.

Don't just gloss over the error message. It tells you exactly who the culprit is.
Look for the ARN in the error:
User: arn:aws:sts::123:assumed-role/SageMaker-Execution-Role-2023/SageMaker is not authorized...

assumed-role/: This confirms it is a service role, not a human user.
SageMaker-Execution-Role-2023: This is the exact name of the IAM Role you need to fix. Copy this name.

Step 2: Locate the Execution Role.

If you aren't sure which role your resource is using, check the configuration.

For Notebooks: Open the SageMaker Console > Notebook instances > Click your instance > Look for "IAM role ARN" in the Permissions and encryption section.
For Training Jobs: Look at the RoleArn parameter in your estimator definition in Python:

estimator = Estimator(
    role='arn:aws:iam::123:role/service-role/AmazonSageMaker-ExecutionRole-2023',
    ...
)

Step 3: Audit S3 Permissions.

Go to the IAM Console > Roles and search for the role name you found in Step 1.
Click the role and look at the Permissions tab.

The Quick Fix (Dev Only): Attach the AmazonS3FullAccess managed policy.
The Production Fix: Create an inline policy that allows specific access to your bucket:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:PutObject",
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::my-sagemaker-bucket",
                "arn:aws:s3:::my-sagemaker-bucket/*"
            ]
        }
    ]
}

Note: You must list both the bucket itself AND the contents (`/`).*

Step 4: Check KMS Key Policy (The Silent Blocker).

If you have S3FullAccess but still get AccessDenied or 403, your data is likely encrypted with a custom KMS key.
KMS requires a "Two-Way Handshake":

The IAM Role must have permission to call KMS.
The KMS Key Policy must explicitly trust the IAM Role.

The Fix: Go to KMS > Customer managed keys > Click your Key > Key Policy. Add this statement:

{
  "Sid": "Allow SageMaker Role to Use Key",
  "Effect": "Allow",
  "Principal": {
    "AWS": "arn:aws:iam::123456789012:role/YourSageMakerExecutionRole"
  },
  "Action": [
    "kms:Decrypt",
    "kms:GenerateDataKey"
  ],
  "Resource": "*"
}

Note: The KMS key must be in the same AWS Region as your SageMaker resource. Cross-region KMS access is not supported by default.

Step 5: Verify `iam:PassRole`.

If the error occurs right when you call .fit() or .deploy(), the error might say:
User is not authorized to perform: iam:PassRole on resource...
This means your personal user has permission to talk to SageMaker, but not permission to give SageMaker the Execution Role.

The Fix: Add this to YOUR (the human user's) IAM policy:

{
    "Effect": "Allow",
    "Action": ["iam:PassRole"],
    "Resource": "arn:aws:iam::123456789:role/service-role/AmazonSageMaker-ExecutionRole*"
}

Step 6: Test Access via CLI.

Before re-running a massive training job, test the fix inside the Notebook terminal:

# Try to list the bucket using the notebook's role
aws s3 ls s3://my-sagemaker-bucket/

If this lists the files, your permissions are fixed.

Pro Tips

The "GetRole" Latency
IAM changes are eventually consistent. After attaching a policy, it can take 10–60 seconds to propagate. If you fix it and immediately hit "Run," it might fail again. Wait 1 minute.

Service Control Policies (SCPs)
If you are in a large organization and have checked everything above but still get Access Denied, check for SCPs. These are master policies applied at the AWS Organization level that can override your local admin rights. If an SCP denies S3 access, no amount of local IAM tweaking will fix it.

S3 VPC Endpoints
If your SageMaker instance is in a private VPC without internet access, you must have an S3 VPC Endpoint. Without it, even S3FullAccess will result in timeouts or connection errors that look like permission failures.

Conclusion

AccessDeniedException in SageMaker is almost always a case of mistaken identity. You are looking at your own permissions, but the code is running as an Execution Role.

By isolating the ARN in the error message and verifying that specific role's access to both S3 buckets and KMS keys, you can resolve these blockers quickly. Always remember: In AWS, having the key to the library (S3) doesn't help if you can't read the language the books are written in (KMS).

Aaron Rose is a software engineer and technology writer at tech-reader.blog and the author of Think Like a Genius.

Search This Blog

Tech-Reader.blog