Articles, notes and random thoughts on Software Development and Technology
To connect to an EC2 instance over SSH, if the permissions on your .pem file are too broad then you’ll see this error:
Permissions 0644 for ‘keypair.pem’ are too open.
It is required that your private key files are NOT accessible by others.
This private key will be ignored.
chmod the .pem file to 0400 and then you should be good. This is described here.
When an AWS EC2 User Data script fails, you’ll see something like this in /var/log/cloud-init.log in your instance:
2018-02-03 06:08:16,536 - util.py[DEBUG]: Failed running /var/lib/cloud/instance/scripts/part-001 [127] Traceback (most recent call last): File "/usr/lib/python3/dist-packages/cloudinit/util.py", line 806, in runparts subp(prefix + [exe_path], capture=False) File "/usr/lib/python3/dist-packages/cloudinit/util.py", line 1847, in subp cmd=args) cloudinit.util.ProcessExecutionError: Unexpected error while running command. Command: ['/var/lib/cloud/instance/scripts/part-001'] Exit code: 127 Reason: - Stdout: - Stderr: - 2018-02-03 06:08:16,541 - cc_scripts_user.py[WARNING]: Failed to run module scripts-user (scripts in /var/lib/cloud/instance/scripts) 2018-02-03 06:08:16,541 - handlers.py[DEBUG]: finish: modules-final/config-scripts-user: FAIL: running config-scripts-user with frequency once-per-instance
It tells you something failed, but not what. The trouble seems that output from your user data script does not go to the cloud-init.log by default.
One of the answers in this post suggests to pipe your script commands and output to logger into a separate log file like this:
set -x
exec > >(tee /var/log/user-data.log|logger -t user-data ) 2>&1
echo BEGIN
date '+%Y-%m-%d %H:%M:%S'Now running my script with a ‘apt-get update -y’ looks like:
+ echo BEGIN BEGIN + date '+%Y-%m-%d %H:%M:%S' 2018-02-03 23:37:55 + apt-get update -y ... output continues here
And further down, here’s my specific error I was looking for:
+ java -Xmx1024M -Xms1024M -jar minecraft_server.1.12.2.jar nogui /var/lib/cloud/instance/scripts/part-001: line 11: java: command not found
My EC2 running the Ubuntu AMI does not have Java installed by default, so I need to install it with (adding to my User Data script):
apt-get install openjdk-8-jre-headless -y
… and now my script runs as expected.
EC2 pricing at first look seems simple: for on-demand pricing the current prices are listed here, as fractions of a dollar per hour, Here’s a quick look (below) at how these prices are listed for the first few t2 instance types. The list of different instance types is pretty extensive (check out the link for the complete list):
The page says the price is:
by the hour or second (minimum of 60 seconds) with no long-term commitments
and further down:
Pricing is per instance-hour consumed for each instance, from the time an instance is launched until it is terminated or stopped. Each partial instance-hour consumed will be billed as a full hour or per-second depending on which Amazon EC2 instances you run
I added the emphasis as this part is not entirely clear. So to summarize:
or
The last part of this last statement is (I think) the key. It refers to which AMI image you’re running, since some images have an hourly charge (if I’m wrong, please leave me a comment and let me know!) :
If you click the Marketplace link, you’ll notice AMI images with commercial products, along with an hourly charge for usage. Ahah, there you go!
With EC2, you have a huge variety of instance types to chose from, and with each type having a range of sizes small to large:
For each of the types you have a further range of options for how you chose to provision an instance, which has an impact on how the instance is priced:
I’ve never created a Spot Instance before, and I’m curious what the steps are. As with every service on AWS, there’s more than one approach, and I’ve going to look at using a Launch Template:
Bu creating a Launch Template you can configure a number of settings for you instance (AMI image, EC2 instance type, etc).
From the Request Spot Instance page in the EC2 Management Console, you can now use your Launch Template which prepopulates most of the settings for your requested Spot Instance:
Further down on this page is where you request your pricing – it defaults to buy at cheapest available, and is capped at the current On-Demand price (if the spot price rises to match the On-Demand price then there’s no savings from the Spot pricing and you might as well get On-Demand instead):
After submitted, it shows the request in a submitted state:
So here’s my first error:
Repeated errors have occured processing the launch specification “t2.small, ami-41e0b93b, Linux/UNIX (Amazon VPC), us-east-1d”. It will not be retried for at least 13 minutes. Error message: com.amazonaws.services.ec2.model.AmazonEC2Exception: Network interfaces and an instance-level subnet ID may not be specified on the same request (Service: AmazonEC2; Status Code: 400; Error Code: InvalidParameterCombination)
Network instances and a instance-level subnet – I did add an interface because I wanted to select the public ip option. Let’s create a new version of the template and try again.
Now my request is in ‘pending fulfillment’ status:
Same error. One of the answers here suggests this is because I don’t have a Security Group. Ok, let’s add a Security Group to the launch template and try again.
Same error even though I added a Security Group to my template, but I noticed this is the Request Spot Instance options – when you select your Template, if you’ve made version updates to your template, make sure you select the latest version as it defaults to 1, i.e. I was restarting with the original template that I know doesn’t work:
Next error:
com.amazonaws.services.ec2.model.AmazonEC2Exception: The security group ‘sg-825bfe14’ does not exist in VPC ‘vpc-058f5d7c’ (Service: AmazonEC2; Status Code: 400; Error Code: InvalidGroup.NotFound)
Hmm. So I’m interpreting this as my Security Group is not for the default VPC that my instance was assigned, so let’s create a new VPC, and then a new Security Group for this VPC:
Now create a new Template version with this new VPC and SG.
Next error:
Error message: com.amazonaws.services.ec2.model.AmazonEC2Exception: Security group sg-b756adc0 and subnet subnet-f756c7bf belong to different networks. (Service: AmazonEC2; Status Code: 400; Error Code: InvalidParameter)
SG and subnet belong to different networks. Ok, getting close. Let’s take a look.
On the VPC for my SG I have: 10.0.0.0/16
On my subnet for us-east-1d I have: 10.0.0.0/24
Ah, ok. let’s add a new subnet for us-east-1d with the same CIDR block and try again.
When creating your spot request, make sure you select your VPC and subnet to match:
Ahah! Now we’re looking good and my Spot Instance is being provisioned:
Ugh, next error:
Looks like in my template I didn’t give a device name (‘missing device name’) for my EBS volume, e.g. /dev/sdb. New template version, trying again.
Next error:
Error message: com.amazonaws.services.ec2.model.AmazonEC2Exception: The parameter iops is not supported for gp2 volumes. (Service: AmazonEC2; Status Code: 400; Error Code: InvalidParameterCombination)
Geesh. Ok, removing the iops value in the template and trying again (would help to have some validation on the template form)
And now:
Fulfilled, we made it, a Spot Instance provisioned!
At this point though my instance was started without a public IP, so now I’ve got the Security Group and Subnet issue sorted, I’ll go back to the template and add a network interface and select ‘assign public IP’. Rather than assigning this on the network interface though, it looks like it’s also an option from the subnet config, so I edited and added it here:
And now we’re up, with a public IP! Whether my User Data init script actually did what it was supposed to is the next thing to check, but I’ll look at that next!