GitLab CI – docker compose to remote host fails with: Host key verification failed

I’m getting this error when a GitLab CI job is attempting to ‘docker compose up’ to a remote context:

$ docker --context remote compose -f docker-compose-remote-db.yml up -d --pull always
unable to get image '10.0.10.3:5000/my-image:latest': error during connect: Get "http://docker.example.com/v1.51/images/10.0.10.3:5000/adsb-dashboard:latest/json": command [ssh -l gitlab-runner -o ConnectTimeout=30 -T -- 10.0.10.3 docker system dial-stdio] has exited with exit status 255, make sure the URL is valid, and Docker 18.09 or later is installed on the remote host: stderr=Host key verification failed.

According to answers here, this could be because when the job runs for the first time the host signature has not been added to known_hosts yet. You can avoid this by:

echo "StrictHostKeyChecking no" >> ~/.ssh/config

base64 encoding Kubernetes Secrets includes newline char when exported as an env variable (if you don’t use -n option with echo)

I’m deploying an app to Kubernetes that references a Kubernetes Secret that is exported as an env var on the pod. I couldn’t work out why I kept getting this error when the pod was starting up:

FATAL: password authentication failed for user "admin"

but if I exec’d into the pod to check the value of the env var, it was the correct value that I expected.

Eventually I did stumble across this clue – ‘printenv’ inside the pod shows:

DB_PASSWORD=[value here]

KUBERNETES_SERVICE_PORT_HTTPS=443
[... other values here]

Between DB_PASSWORD and the next value there’s a blank line, followed by a long list of other env var values, with no other blank lines.

From this question, the issue is how I originally encoded the base64 value with:

echo your-value-here | base64

which is not the same as:

echo -n your-value-here | base64

echo apparently includes a newline by default, so you need to use it as above with the -n option