Installing and using s3cmd to copy files to AWS S3

s3cmd is a useful tool that lets you list put and get objects from an AWS S3 bucket. To install:

Install python2.7 with :

sudo apt-get install python2.7

Install setup tools with :

sudo apt-get install python-setuptools

Download and unzip the .zip distro from link here: http://s3tools.org/download

Install with:

sudo python2.7 setup.py install

To see options, run:

s3cmd --help

Before running the s3cmd setup, you need to create an AWS IAM user with programmatic access, to get a access key that will be used by the s3cmd.

First, create a new user from the Management Console, and ensure ‘Programmatic Access’ is checked:

Create a new IAM Policy and attach to this user with read, write and list actions, and restrict the resource to the ARN for this S3 bucket that you want to use the s3cmd with:

If you want to narrow down the permissions to a minimal list, a policy list like this is the minimum needed for s3cmd to work (based on answers to this question on SO):

{
 "Version": "2012-10-17",
 "Statement": [
   {
     "Sid": "Stmt123456",
     "Effect": "Allow",
     "Action": [
       "s3:ListAllMyBuckets"
     ],
     "Resource": [
     "arn:aws:s3:::*"
     ]
   },
   {
     "Sid": "VisualEditor0",
     "Effect": "Allow",
     "Action": [
       "s3:ListBucket",
       "s3:PutObject",
       "s3:PutObjectAcl"
     ],
     "Resource": [
       "arn:aws:s3:::bucketname",
       "arn:aws:s3:::bucketname/*"
     ]
   }
 ]
}

Following how-to guide here, for first time setup, run:

s3cmd --configure

and provide your IAM user api access key and secret key and other values as prompted. After configuring, when prompted to test the config, the util will attempt to list all buckets, but if the policy you created was for limited read/write on a specific bucket, this will fail, but that’s ok.

To confirm access to your bucket, try:

s3cmd ls s3://bucketname

and to put a file:

s3cmd put filename s3:/bucketname