Tag: s3

Posted on

Using Route 53 to create subdomain names for your projects

If you create and deploy your own software projects to the cloud, at some point you probably end up with a number of things deployed to various places and unless you spend time maintaining your bookmarks to all these projects, it becomes hard to keep track after a while.

One of the interesting things about Route 53 is that you can create A records that resolve to IP addresses either within AWS or hosted elsewhere. If you have you own domain setup in Route 53, you can easily create subdomains with A records pointing to where ever these projects are hosted. e.g.

example1.youdomain.com -> x.x.x.x

example2.yourdomain.com -> y.y.y.y

A while back I deployed my Sudoku Solver React app to an S3 bucket hosting the website, and I can never remember the S3 endpoint name. But using a Route 53 Alias to the S3 endpoint, you can create whatever subdomain you need to point to the target resource. Here’s what it looks like setting up an alias:

Notes:

  • when you click in the Alias Target box you should see your S3 bucket already listed (if not, check you’ve enabled Static Website Hosting)
  • the recordset name must be identical to the first part of your bucket name (e.g. ‘example’)
  • the S3 bucket name must be the subdomain name plus full domain, e.g. example.yourdomain.com

Posted on

AWS S3 error: “The bucket you are attempting to access must be addressed using the specified endpoint”

Most errors on AWS are explicit and self explanatory, but once in a while you run into something that tells you something went wrong, but not how to fix it.

When setting up an S3 bucket to serve a static website, I got this error being returned as 301s for referenced files:

<Error>
<Code>PermanentRedirect</Code>
<Message>The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.</Message>
<Endpoint>s3.amazonaws.com</Endpoint>
<Bucket>static</Bucket>
...
</Error>

If you click an object in a bucket and click the overview tab, you can get a URL direct to the object, for example:

To load a static site from S3 though, you need to use a URL referencing the bucket in the server name, so instead of this:

https://s3-us-west-1.amazonaws.com/react-sudoku-solver/index.html

You should use this:

http://react-sudoku-solver.s3-us-west-1.amazonaws.com/index.html

(I’m currently working on a React app as a Sudoku Solver – more on this later)

Posted on

Installing and using s3cmd to copy files to AWS S3

s3cmd is a useful tool that lets you list put and get objects from an AWS S3 bucket. To install:

Install python2.7 with :

sudo apt-get install python2.7

Install setup tools with :

sudo apt-get install python-setuptools

Download and unzip the .zip distro from link here: http://s3tools.org/download

Install with:

sudo python2.7 setup.py install

To see options, run:

s3cmd --help

Before running the s3cmd setup, you need to create an AWS IAM user with programmatic access, to get a access key that will be used by the s3cmd.

First, create a new user from the Management Console, and ensure ‘Programmatic Access’ is checked:

Create a new IAM Policy and attach to this user with read, write and list actions, and restrict the resource to the ARN for this S3 bucket that you want to use the s3cmd with:

If you want to narrow down the permissions to a minimal list, a policy list like this is the minimum needed for s3cmd to work (based on answers to this question on SO):

{
 "Version": "2012-10-17",
 "Statement": [
   {
     "Sid": "Stmt123456",
     "Effect": "Allow",
     "Action": [
       "s3:ListAllMyBuckets"
     ],
     "Resource": [
     "arn:aws:s3:::*"
     ]
   },
   {
     "Sid": "VisualEditor0",
     "Effect": "Allow",
     "Action": [
       "s3:ListBucket",
       "s3:PutObject",
       "s3:PutObjectAcl"
     ],
     "Resource": [
       "arn:aws:s3:::bucketname",
       "arn:aws:s3:::bucketname/*"
     ]
   }
 ]
}

Following how-to guide here, for first time setup, run:

s3cmd --configure

and provide your IAM user api access key and secret key and other values as prompted. After configuring, when prompted to test the config, the util will attempt to list all buckets, but if the policy you created was for limited read/write on a specific bucket, this will fail, but that’s ok.

To confirm access to your bucket, try:

s3cmd ls s3://bucketname

and to put a file:

s3cmd put filename s3:/bucketname

 

Posted on

Configuring AWS S3 Storage Gateway on VMware ESXi for uploading files to S3 via local NFS mount

AWS provides a VM image that can be run locally to provide a local NFS mount that transparently transfers files copied to the mount to an S3 bucket.

To setup, start by creating a Storage Gateway from the AWS Management Console:

Select ‘VMware ESXi’ host platform:

Unzip the downloaded .zip file.

From your ESXi console, create a new VM and select the ‘Deploy VM from OVF / OVA file’ option:

Click the ‘select file’ area and point to the unzipped .ova file on your local machine:

Per Storage Gateway instructions, select Thick provisioned disk:

Press Finish on next Summary screen, and wait for VM to be created.

Back in your AWS Management Console, Next through remaining setup pages, enter IP of your Storage Gateway. The instructions say the VM does not need to be accessible from the internet, and instructions here walk you logging on to the VM with the default credentials to get your IP assuming it’s not already displayed on your ESXi console for the running VM):

Powering on the VM, I get a logon prompt:

After logging on with the default credentials, the next screen gives me the assigned IP:

Entering the IP in the Management Console, setting timezone to match my local timezone, and then press Activate to continue:

At this point, AWS is telling me I didn’t create any local disks attached to my VM, which is true, I missed that step:

According to the docs you need to attach 1 disk for an upload buffer, and 1 for cache storage (files pending upload). Powering down my VM, I created 2 new disks, 2GB each (since this is just for testing):

Pressing the refresh icon, the disks are now detected (interesting that AWS is able to communicate with my VM?),  and it tells me the cache disk needs to be at least 150GB:

Powering down the VM again and increasing one of the disks to 150GB, but thinly provisioned (not sure I have too much spare disk on my server for 150GB thickly provisioned):

Powering back on, pressing refresh in the AWS Console:

Ok, maybe it needs to be thick after all. Powering off and provisioning as thick:

I allocated the 150GB drive as the Cache, and left the other drive unallocated for now. Next to allocate a share:

At this point you need to configure the file share to point to an existing S3 bucket, so make sure you have one created at this point, if not open another Console and create one then enter it here:

By default, any client that’s able to mount my share on my VM locally is allowed to upload to this bucket. This can be configured by pressing edit. I’ll leave as the default for now. Press the Create File Share button to complete, and we’re done!

 

Next following the instructions here, let’s mount the Storage Gateway file share and test uploading a file:

For Linux:

sudo mount -t nfs -o nolock [Your gateway VM IP address]:/[S3 bucket name] [mount path on your client]

For MacOS:

sudo mount_nfs -o vers=3,nolock -v [Your gateway VM IP address]:/[S3 bucket name] [mount path on your client]:

Note: from the AWS docs there’s a missing space between the nfs drive and the mount point in both of these examples, and the Linux example has a trailing ‘:’ which is also not needed.

For Ubuntu, if you haven’t installed nfs-common, you’ll need to do that first with

sudo apt-get install install nfs-common

… otherwise you’ll get this error when attempting to mount:

mount: wrong fs type, bad option, bad superblock on ...

For Ubuntu, here’s my working mount statement (after installing nfs-common):

sudo mount -t nfs -o nolock vm-ip-address:/s3bucket-name /media/s3gw

… where /media/s3gw is my mount point (created earlier).

To test, I create a file, copied to the mount dir, and then took a look at my bucket contents via the Console:

My file is already there, everything is working!

 

Proudly powered by WordPress