Tag: homelab

Posted on

Site update: Migrating hosting providers – automating deployment with Terraform, Ansible and GitLab CI Pipelines

Over the past couple of years I’ve been working on and off on a personal project to migrate and update a GitLab CI pipeline on my self-hosted GitLab for building and deploying this site. Unfortunately my self-hosted GitLab used to be on a e-waste HP DL380 G7 rack server that I no longer have after moving house, so I’ve gone back to using my old 2008 MacPro 3,1 as a Proxmox server, where I now run GitLab (which oddly is what I first used this Mac for several years ago).

As part of the update, I wanted to achieve a couple of goals:

  • update the GitLab pipeline to deploy to a staging server for testing, and then deploy to the live server
  • template any deployment files that are server/domain specific
  • update my Docker images for WordPress, updating the plugins, and anything that needs to be in the image to support the runtime, e.g. nginx, php plugins for nginx etc.
  • move to a new cloud provider that would allow me to provision VMs with Terraform
  • automate updating SSL certs with Let’s Encrypt certbot

I won’t share my completed pipeline because I don’t want to share specifics about how my WordPress site is configured, but I’ll give an overview of what I used to automate various parts of it:

While I’ve ended up with a working solution that meets my goals (I can run the pipeline to deploy to my test server or deploy latest to my new live server), I still have a few areas I could improve:

  • GitLab CI Environments, and parameterization – I don’t feel I’ve taken enough advantage of these yet. The jobs that deploy to my test server run automatically, but the deploy to my live set is the same set of jobs that I manually run, and configured to deploy to a different server – I feel there’s more I can parameterize here and need to do some more experimentation in this area

Although this effort was spread over a couple of years before I got to a point of completion, it was a great opportunity to gain some more experience across all these tools.

Posted on

docker-compose (v1.29.2) to remote host with ssh fails

I have a personal project that is docker-compose based which I’ve deployed to remote servers before in the past (a few years ago, using steps here), and recently attempting to redeploy it from a more recent self-hosted GitLab pipeline on Ubuntu 24.0.4 I get this error:

docker.errors.DockerException: Install paramiko package to enable ssh:// support

This issue is exactly as described on this ticket. The issue also seems to be OS specific as well as docker-compose version specific – I have docker-compose 1.29.2 on MacOS Sequoia and it works fine, but 1.29.2 on Ubuntu 24.04 or 22.04 fails with the above error.

The workaround as described by multiple comments on the ticket is to not use the version installed by apt-get, instead install a specific older/working version with pip instead:

pip3 install docker-compose==1.28.2
Posted on

GitLab Runner “Runner has never contacted this instance”

I just installed and configured a GitLab Runner on a separate Ubuntu 22.04 server. Following tips here and here, I’ve started the runner with ‘sudo gitlab-runner start’ and I’ve verified the config with ‘sudo gitlab-runner verify’, but the GitLab server is still showing the ‘never contacted this instance’ error:

Following the troubleshooting steps here, I used journalctl to view the logs:

sudo journalctl --unit=gitlab-runner.service -n 100 --no-pager

and this gave me a clue:

Oct 19 14:36:24 proxmox-ubuntu2204-server systemd[1]: Started GitLab Runner.
Oct 19 14:36:25 proxmox-ubuntu2204-server gitlab-runner[1264]: FATAL: failed to get user home dir: $HOME is not defined
Oct 19 14:36:25 proxmox-ubuntu2204-server systemd[1]: gitlab-runner.service: Main process exited, code=exited, status=1/FAILURE
Oct 19 14:36:25 proxmox-ubuntu2204-server systemd[1]: gitlab-runner.service: Failed with result 'exit-code'.

To set a home dir for the gitlab-runner user, I checked /etc/passwd, found the line for gitlab-runner and it has an entry towards the end of the line for /home/gitlab-runner, so that seems ok.

Following tips here, I edited the service config with:

sudo systemctl edit gitlab-runner

which edits
/etc/systemd/system/gitlab-runner.service.d/override.conf and added this section:

[Service]
Environment="HOME=/home/gitlab-runner"

Restarted the service with:

sudo systemctl restart gitlab-runner

checked the logs again with jornalctl and now we’re up and running, and the runner is reported as available on the Runners page. Done!

Oct 19 14:51:48 proxmox-ubuntu2204-server gitlab-runner[1400]: Runtime platform                                    arch=amd64 os=linux pid=1400 revision=66a723c3 version=17.5.0
Oct 19 14:51:48 proxmox-ubuntu2204-server gitlab-runner[1400]: Starting multi-runner from /etc/gitlab-runner/config.toml...  builds=0 max_builds=0
Oct 19 14:51:48 proxmox-ubuntu2204-server gitlab-runner[1400]: Running in system-mode.                           
Oct 19 14:51:48 proxmox-ubuntu2204-server gitlab-runner[1400]:                                                   
Oct 19 14:51:48 proxmox-ubuntu2204-server gitlab-runner[1400]: Configuration loaded                                builds=0 max_builds=1
Posted on

GitLab Runner unable to run Docker commands

I have a GitLab Runner using a Shell Executor that needs to build a Docker container. When it executes the first Docker command it gets this error:

docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', PermissionError(13, 'Permission denied'))

If I logon as the gitlab-runnner user and try to execute docker commands manually I get this error:

$ docker ps
permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get "http://%2Fvar%2Frun%2Fdocker.sock/v1.47/containers/json": dial unix /var/run/docker.sock: connect: permission denied

A quick Google and I need to add the gitlab-runner to the Docker group to grant it permission to execute Docker:

sudo usermod -a -G docker $USER
Proudly powered by WordPress