AWS CLI error “The provided token is malformed or otherwise invalid”

Starting getting this error when running a simple ‘aws s3 ls’

An error occurred (InvalidToken) when calling the ListBuckets operation: The provided token is malformed or otherwise invalid.

My current version:

% aws --version
aws-cli/2.0.59 Python/3.7.4 Darwin/20.1.0 exe/x86_64

Downloaded latest version and upgraded and problem fixed. New version is now:

% aws --version aws-cli/2.1.3 Python/3.7.4 Darwin/20.1.0 exe/x86_64

Java Developer’s Journal Resource CD from 2004

I’ve got some weird stuff on my office bookshelf. While looking for something else, I came across this CD from 2004 which claims to have every issue of JDJ published, up until 2004. That can’t be that may so far, but apparently there was 85 issues up until this point.

“The Complete Works”!

Time to crack out the USB CD drive and see exactly what’s on this CD. I’ll post a follow up later with some examples what’s on the CD!

Using Serverless framework and AWS sts:assume-role to cross deploy to different AWS accounts

In order to assume a role in another account, the owning account needs to grant a ‘trust relationship’ to those allowed to assume the role. This can be done by referencing an IAM username or role for those in the other account that are allowed to assume this role.

You can do this in the Console using the Trust Relationship tab:

A Policy to grant access to to a specific IAM user looks like:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::ACCOUNT-ID:user/USER-ID"
      },
    "Action": "sts:AssumeRole"
    }
  ]
}

To assume this role, use the ‘aws sts assume-role’ cli command:

aws sts assume-role --role-arn arn:aws:iam::ACCOUNT-ID:role/ROLE-NAME --role-session-name SESSION-NAME

If this is successful, you’ll see a response that grants temporary values for the following AWS credentials that can be used after this point:

  • AccessKeyId
  • SecretAccessKey
  • SessionToken

The returned values can be used to set env vars to use with the CLI and other AWS SDK apps:

  • export AWS_ACCESS_KEY_ID=
  • export AWS_SECRET_ACCESS_KEY=
  • export AWS_SESSION_TOKEN=
  • export AWS_DEFAULT_REGION=

For Servlerless to deploy into another account, if you attempt a Serverless deploy at this point, you’ll see errors like:

User: arn:aws:sts::ACCOUNT-ID:assumed-role/ServerlessLambdaDeployRole/lambdadeploy is not authorized to perform: cloudformation:CreateStack on resource: arn:aws:cloudformation:us-east-1:TARGET-ACCOUNT-ID:stack/deploy-demo/*

In this case cloudformation:CreateStack is missing from the assumed role. If you incrementally attempt to find what additional permissions you’ll need to deploy, you’ll also need to add:

  • cloudformation:DescribeStackEvents
  • cloudformation:DescribeStackResource
  • cloudformation:ValidateTemplate
  • cloudformation:UpdateStack
  • cloudformation:DeleteStack
  • apigateway:POST
  • iam:CreateRole
  • iam:PutRolePolicy

ValidateTemplate appears to throw an error unless the Resource is for a wildcard of ‘*’ and not anything more specific, otherwise you’ll see this error:

Error: The CloudFormation template is invalid: User: arn:aws:sts::ACOUNT-ID:assumed-role/ServerlessLambdaDeployRole/lambdadeploy is not authorized to perform: cloudformation:ValidateTemplate

To grant permissions for ValidateTemplate specify a Resource of “*”

{
"Sid": "CreateCloudFormationStackValidate",
"Effect": "Allow",
"Action": [
"cloudformation:ValidateTemplate"
],
"Resource": "*"
}

The STS temporary credentials will expire after 1 hour, so if you see this error:

An error occurred (ExpiredToken) when calling the AssumeRole operation: The security token included in the request is expired

then you’ll need to rerun the ‘aws sts assume-role’ command again. If you previously set the session token in AWS_SESSION_TOKEN, you’ll need to set it back to blank (along with AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY) before you run the command again. When you get the refreshed values, remember to set the env vars with the updated values.

At this point, if you’ve run ‘aws sts assume-role’ and you’ve set the env vars for the returned temp credentials, you’ll be able to run a ‘serverless deploy’ and deploy into the other account where you’ve assumed this new role with the permissions to deploy. This should include permissions for creating a new Lambda and API Gateway, if you’re deploying anything else from your serverless config, you’ll need to add those permissions to the role you’re assuming.

Using different profiles with aws cli

With the AWS CLI you can configure a number of named profiles, with credentials for different accounts. These are stored in ~/.aws/configuration

To show the currently in use profile, use

aws configure list

To view all configured profiles:

aws configure list-profiles

The default profile is used when you don’t specify a named profile. The others are used when you pass the –profile profilename parameter