Problem: S3 Objects Randomly Disappear After Upload

Problem: S3 Objects Randomly Disappear After Upload

$ aws s3 cp myfile.txt s3://my-data-bucket/
upload: ./myfile.txt to s3://my-data-bucket/myfile.txt

$ aws s3 ls s3://my-data-bucket/
2025-02-09 10:05:32   51200  myfile.txt

# Wait a few minutes...

$ aws s3 ls s3://my-data-bucket/
# No output – file has disappeared!

$ aws s3api get-bucket-lifecycle-configuration --bucket my-data-bucket
{
    "Rules": [
        {
            "ID": "ExpireOldFiles",
            "Status": "Enabled",
            "Filter": {},
            "Expiration": {
                "Days": 1
            }
        }
    ]
}

Issue:

Uploaded objects randomly disappear after a short time due to an active S3 Lifecycle policy that automatically expires objects after one day (or possibly sooner if misconfigured). 

Fix: Disable or Adjust the S3 Lifecycle Expiration Policy

# Step 1: Check the existing lifecycle policy to confirm the expiration rule
$ aws s3api get-bucket-lifecycle-configuration --bucket my-data-bucket
{
    "Rules": [
        {
            "ID": "ExpireOldFiles",
            "Status": "Enabled",
            "Filter": {},
            "Expiration": {
                "Days": 1
            }
        }
    ]
}

# Step 2: Delete the lifecycle policy to stop automatic deletions
$ aws s3api delete-bucket-lifecycle --bucket my-data-bucket
# Step 2.1: Verify the lifecycle policy has been deleted 
# (No output means it's gone)
$ aws s3api get-bucket-lifecycle-configuration --bucket my-data-bucket
An error occurred (NoSuchLifecycleConfiguration) when calling 
the GetBucketLifecycleConfiguration operation: 
The lifecycle configuration does not exist

# Step 3: Re-upload the file and confirm it persists
$ aws s3 cp myfile.txt s3://my-data-bucket/
upload: ./myfile.txt to s3://my-data-bucket/myfile.txt

$ aws s3 ls s3://my-data-bucket/
2025-02-09 10:15:32   51200  myfile.txt

# Step 4: Verify object remains accessible after some time
$ sleep 300  # Wait for 5 minutes

$ aws s3 ls s3://my-data-bucket/
2025-02-09 10:15:32   51200  myfile.txt

Need AWS Expertise?

If you're looking for guidance on AWS challenges or want to collaborate, feel free to reach out! We'd love to help you tackle your AWS projects. 🚀

Email us at: info@pacificw.com

Image: Gemini