GitLab CI – docker compose to remote host fails with: Host key verification failed

I’m getting this error when a GitLab CI job is attempting to ‘docker compose up’ to a remote context:

$ docker --context remote compose -f docker-compose-remote-db.yml up -d --pull always
unable to get image '10.0.10.3:5000/my-image:latest': error during connect: Get "http://docker.example.com/v1.51/images/10.0.10.3:5000/adsb-dashboard:latest/json": command [ssh -l gitlab-runner -o ConnectTimeout=30 -T -- 10.0.10.3 docker system dial-stdio] has exited with exit status 255, make sure the URL is valid, and Docker 18.09 or later is installed on the remote host: stderr=Host key verification failed.

According to answers here, this could be because when the job runs for the first time the host signature has not been added to known_hosts yet. You can avoid this by:

echo "StrictHostKeyChecking no" >> ~/.ssh/config

GitLab Runners stale after SSL cert updates

I’m using Let’s Encrypt SSL certs on my homelab servers which are periodically re-issued using CertBot. This causes my VMs running GitLab Runners to go stale, and when attempting to reverify, they get ‘x509: certificate signed by unknown authority’ errors.

After a cert re-issue, the steps on the GitLab Runner to update it’s certs again I use are:

openssl s_client -showcerts -connect my-homelab-service-name:443 /dev/null|openssl x509 -outform PEM > gitlab.crt

sudo cp gitlab.crt /usr/local/share/ca-certificates/
sudo update-ca-certificates
sudo gitlab-runner restart
sudo gitlab-runner verify

After those steps, the Runner status on the GitLab server is back to active and starts picking up pipeline jobs as normal.

GitLab Runner with Docker executor: “client version 1.43 is too old”

Setting up a Docker image build in my GitLab ci I got this error:

ERROR: Error response from daemon: client version 1.43 is too old. Minimum supported API version is 1.44, please upgrade your client to a newer version: driver not connecting

Docker version on the VM running the GitLab Runner:

$ docker --version
Docker version 29.1.3, build f52814d

Versions in my .gitlab-ci.yml:

image: docker:24.0.5-cli
services:
  - docker:24.0.5-dind

Updated both to match the Docker version on the VM resolved the issue.