Deploying Docker containers to AWS ECS Fargate

The interesting feature of AWS ECS Fargate is that it’s ‘serverless for containers’. Serverless broadly means you don’t need to be concerned with the provisioning and maintenance of the servers or compute that are running your code. With Fargate, you don’t have to provision compute for your Docker Containers, AWS manages the compute for you.

If you’re working with Docker containers, AWS have multiple runtime options, each with their own pros and cons:

  • running Docker on your own EC2 instances – the roll your own approach, you provision instances and manage everything yourself
  • AWS ECS with EC2 launch type – you still need to provision a pool of available EC2 instances on which AWS will run your containers
  • AWS EKS – managed Kubernetes
  • AWS ECS with Fargate launch type – you don’t need to provision any compute (e.g. EC2), AWS manages the compute for you

I’m taking a look at AWS ECS Fargate to see what it takes to deploy a Docker container.

An ECS cluster needs a VPC in which your container instances will run, with at least 1 public or private subnet. Steps to create a new VPC with subnets is covered here.

Following these steps from the VPC section in ECS tutorials using the AWS Console I created:

  • an Elastic IP to associate with my cluster for public access
  • a new VPC with 1 private subnet and 1 public subnet

I created these with the VPC Wizard using this option:

Apparently your public subnet doesn’t get assigned a public IP by default, so follow these steps in the guide to change this default behavior:

When you select your public subnet, this option is under Actions here:

Select this option:

My public subnet was created in AZ us-west-2a and my private subnet is also in the same AZ. The guide recommends creating 1 additional public and private subnets in a different AZ high for availability.

To create a ECS Fargate cluster you can use the AWS CLI like this:

aws ecs create-cluster --cluster-name your-fargate-cluster-name

This will return some stats about your newly created cluster, like:

"clusterName": "fargate-cluster1",
"status": "ACTIVE",
"registeredContainerInstancesCount": 0,
"runningTasksCount": 0,
"pendingTasksCount": 0,
"activeServicesCount": 0,
"statistics": [],
"tags": [],
"settings": [
{
"name": "containerInsights",
"value": "disabled"
}
],
"capacityProviders": [],
"defaultCapacityProviderStrategy": []

However, I’m not sure at this point how to configure the new cluster to specify the VPC and subnets I just created, so for my first cluster I’m going to use the ECS wizard in the AWS Console first, and then come back to the CLI later.

Using the wizard I selected the Networking Only option with Fargate:

I don’t need to select the ‘Create VPC’ option because I’ve already created one:

Turns out there aren’t any options to associate the VPC at this point, the tasks are associated to your VPC and subnets when you create them next. So using the CLI step earlier would create the cluster exactly the same.

You need to define an ECS task definition that defines the task that will run on the ECS cluster. Following the tutorial here, the example JSON file provided as an example looks like this:

{
"family": "sample-fargate",
"networkMode": "awsvpc",
"containerDefinitions": [
{
"name": "fargate-app",
"image": "httpd:2.4",
"portMappings": [
{
"containerPort": 80,
"hostPort": 80,
"protocol": "tcp"
}
],
"essential": true,
"entryPoint": [
"sh",
"-c"
],
"command": [
"/bin/sh -c \"echo ' Amazon ECS Sample App
body {margin-top: 40px; background-color: #333;}
Amazon ECS Sample App
Congratulations!
Your application is now running on a container in Amazon ECS.
' > /usr/local/apache2/htdocs/index.html && httpd-foreground\""
]
}
],
"requiresCompatibilities": [
"FARGATE"
],
"cpu": "256",
"memory": "512"
}

Since we’re deploying a Docker container, we need to specify a Docker image to pull some somewhere. This example provides the name of a Docker container to pull from Docker Hub, in this case httpd:2.4. To. deploy your own apps, you configure your own dockerfile for your app, and publish it to a Docker repo like Docker Hub, or AWS ECR.

Register this task definition with:

aws ecs register-task-definition --cli-input-json file://task-def.json

When –cli-input-json reads your config file, it will open is whatever is your default editor in your shell. On my Mac in zsh it appears to open the file in vim with a ‘:’ prompt at the bottom of the screen, and pressing ‘q’ quits the editor and continues registering the Task Def.

You can list registered Task Definitions with:

aws ecs list-task-definitions

By default, your ECS service will only have a private IP, and would typically be exposed publicly via an ELB. You can configure the task to get allocated it’s own public IP by adding this config:

"networkConfiguration": {
"awsvpcConfiguration": {
"assignPublicIp": "ENABLED",
"securityGroups": [ "sg-12345678" ],
"subnets": [ "subnet-12345678" ]
}
}

This is where we we specify the subnets that were created earlier. I’m going to publicly expose this container, so I’m associating it with the 2 public subnets I created (added to the above config snippet).

I also need a Security Group for the config, so I’ll create that too and allow incoming traffic on port 80.

It’s not obvious from the docs where this NetworkConfiguration section gets specified, but it doesn’t go in the Task Definition json, it gets passed when you create the Service using the Task Definition.

To create a Service, use this cli command:

aws ecs create-service --cluster fargate-cluster --service-name fargate-service --task-definition sample-fargate:1 --desired-count 1 --launch-type "FARGATE" --network-configuration "awsvpcConfiguration={subnets=[subnet-abcd1234],securityGroups=[sg-abcd1234],assignPublicIp=ENABLED}"

Using this command to plug in the subnet ids and Security Group id, from the ECS Console you’ll now see you have service running! If you drill down to the task you can find the assigned public IP. Hit the IP to call the service! Since we’re running an httpd container with a sample web page, we see:

Awesome, up and running!

Interrelated personal side projects can be a goldmine of learning opportunities

Some people struggle to find ideas for working on side/personal software development projects for learning (and fun!). I’ve kept a running list of ideas in an online notebook and if I’m struggling for an idea for a new project I refer back to this list.

Over the past couple of years I stumbled across a series of inter-related projects that’s been a goldmine of learning opportunities, from frontend to backend to serverless.

This series of projects started with a question “Can you solve a problem without understanding the problem?”. tl;dr? No. The problem for this activity was solving Sudoku puzzles, and you can read more about this here:

This lead to the next step, researching more about exact cover problems, and how they can be solved with well established algorithms:

After building an implementation of Donald Knuth’s Algorithm X, I packaged it up as an AWS Lambda, and then built a React frontend for it:

I have one other project in progress for the frontend, replacing Flux with Redux. I’m still working on that one.

After building my solver and the frontend, I starting thinking about how to generate new puzzles. This is in progress here: https://github.com/kevinhooke/sudoku-generator

If you’re generating new puzzles you also need a way to grade the difficulty of puzzles. The unusual thing about this is there’s no established algorithm for grading the complexity of Sudoku puzzles, they’re typically graded by applying human solving techniques, so this led to this: https://github.com/kevinhooke/sudoku-human-grader

This string of related projects has kept me busy of the past couple of years. Not every idea will lead to a series of related projects like this, but if you can find an idea that does, it will keep you busy with plenty of problems to solve and many learning opportunities.

Using Route 53 to create subdomain names for your projects

If you create and deploy your own software projects to the cloud, at some point you probably end up with a number of things deployed to various places and unless you spend time maintaining your bookmarks to all these projects, it becomes hard to keep track after a while.

One of the interesting things about Route 53 is that you can create A records that resolve to IP addresses either within AWS or hosted elsewhere. If you have you own domain setup in Route 53, you can easily create subdomains with A records pointing to where ever these projects are hosted. e.g.

example1.youdomain.com -> x.x.x.x

example2.yourdomain.com -> y.y.y.y

A while back I deployed my Sudoku Solver React app to an S3 bucket hosting the website, and I can never remember the S3 endpoint name. But using a Route 53 Alias to the S3 endpoint, you can create whatever subdomain you need to point to the target resource. Here’s what it looks like setting up an alias:

Notes:

  • when you click in the Alias Target box you should see your S3 bucket already listed (if not, check you’ve enabled Static Website Hosting)
  • the recordset name must be identical to the first part of your bucket name (e.g. ‘example’)
  • the S3 bucket name must be the subdomain name plus full domain, e.g. example.yourdomain.com

node.js, node-oracledb and Oracle Instant Client

To access an Oracle DB from an AWS Lambda function developed with node.js, you need to package you Lambda with shared libraries from Oracle’s Instant Client. The install instructions are here ( http://oracle.github.io/node-oracledb/INSTALL.html#quickstart ) but the only part that is really needed is the download location (since there’s no specific instructions for bundling the libs with an AWS Lambda): https://www.oracle.com/database/technologies/instant-client/linux-x86-64-downloads.html

Not all the Oracle Instant Client files are needed. From this older npm module to automate the packaging of the required libraries, I used this same list of required libraries:

libclntshcore.so.19.1
libclntsh.so.19.1
libmql1.so
libipc1.so
libnnz19.so
libons.so (not packaged in current Instant Client)
libociicus.so
libaio.so (from separate download - see next step)

libaio – if you’re on a Linux platform you can ‘apt-get install libaio’ or similar, but building my Lambda on a Mac I had to manually download the package and extract just the .so file from here (download the Arch Linux x64 package): https://pkgs.org/download/libaio

Put these in a /lib dir and zip up the folder and files. Use this to create a Lambda Layer.

For the Lambda itself install the node.js module for the api:

npm install –save node-oracledb

For examples in api usage, see the examples here: https://github.com/oracle/node-oracledb/tree/master/examples