Towards the end of 2023 I was looking into a new way to setup my homelab. My main goal was to define my whole infrastructure as code. Once everything is defined as code, it’s relatively simple to build on top of that and expand where necessary or deploy somewhere else if that is needed. This also gave me a new way to learn even more about the software tools I already use for automation on smaller scale.

After some research and a little bit of planning, I decided on the following:

Practice Tool
Version control Gitea
CI/CD Jenkins
Provisioning Terraform (Telmate proxmox provider) + cloud-init
Config management Puppet + Ansible

Starting out

Before starting to define everything as code, I decided to setup a node which will be my Puppetserver. Using an Almalinux 8 template (created in an earlier post) it was really simple to clone the template and create a fresh node. Once the node was created, I utilized the puppet-role made by Jeff Geerling! With a few adjustments I could easily setup the new node as a puppetserver and start utilizing it.

Next up was creating a new node for the Jenkins + Gitea stack (mainly combining these to save on some resources). Pretty much the same here, cloning the template Almalinux 9 in this case. Now that I had a puppetserver running, I could define the installation en configuration of these services in code. The old Jenkins node was still running so I could utilize that one to run some pipelines and deploy the code on the puppetserver.

Great, my 2 most important nodes are now running.

Repositories

I setup 2 repositories for puppet which will copy over all the necessary code to the right location on the puppetserver:

  • homelab_puppet -> main puppet profiles

Jenkinsfile:

// Jenkinsfile
pipeline {
    agent any

    environment {
        SERVER_HOST = 'x'
        SERVER_PORT = 'x'
        SERVER_USER = 'x'
        REPO_URL    = 'x'
    }

    stages {
        stage('Checkout') {
            steps {
                script {
                    git branch: 'master', credentialsId: 'jenkins_ssh', url: "${env.REPO_URL}"
                }
            }
        }

        stage('Rsync to Server') {
            steps {
                script {
                    sshagent(['jenkins_ssh']) {
                        sh "rsync -crtvz --delete -e 'ssh -o StrictHostKeyChecking=no -p ${env.SERVER_PORT}' . ${env.SERVER_USER}@${env.SERVER_HOST}:/etc/puppetlabs/code/homelab_puppet/"
                    }
                }
            }
        }
    }
}
  • homelab_hiera -> puppet hieradata

Jenkinsfile:

// Jenkinsfile
pipeline {
    agent any

    environment {
        SERVER_HOST = 'x'
        SERVER_PORT = 'x'
        SERVER_USER = 'x'
        REPO_URL    = 'x'
    }

    stages {
        stage('Checkout') {
            steps {
                script {
                    git branch: 'master', credentialsId: 'jenkins_ssh', url: "${env.REPO_URL}"
                }
            }
        }

        stage('Rsync to Server') {
            steps {
                script {
                    sshagent(['jenkins_ssh']) {
                        sh "rsync -crtvz --delete -e 'ssh -o StrictHostKeyChecking=no -p ${env.SERVER_PORT}' nodes/ ${env.SERVER_USER}@${env.SERVER_HOST}:/etc/puppetlabs/code/environments/production/data/nodes/"
                        sh "rsync -crtvz --delete -e 'ssh -o StrictHostKeyChecking=no -p ${env.SERVER_PORT}' os/ ${env.SERVER_USER}@${env.SERVER_HOST}:/etc/puppetlabs/code/environments/production/data/os/"
                        sh "rsync -crtvz --delete -e 'ssh -o StrictHostKeyChecking=no -p ${env.SERVER_PORT}' common.yaml ${env.SERVER_USER}@${env.SERVER_HOST}:/etc/puppetlabs/code/environments/production/data/common.yaml"
                        sh "rsync -crtvz --delete -e 'ssh -o StrictHostKeyChecking=no -p ${env.SERVER_PORT}' site.pp ${env.SERVER_USER}@${env.SERVER_HOST}:/etc/puppetlabs/code/environments/production/manifests/site.pp"
                    }
                }
            }
        }
    }
}

Puppet code

Next up is the setup of the puppet profiles. I started by building a profile_base which has the basic setup I want for every one of my nodes (users, services, configs, files, …) followed up by distro specific stuff.

profile_base

This is the init.pp for now:

# @summary
#   Base class for my nodes
# 
class profile_base {
  include profile_base::os::common
  include profile_base::motd
  include profile_base::ssh

  if $facts['os'] != undef {
    case "${facts['os']['family']}_${facts['os']['release']['major']}" {
      'RedHat_8': {
        include profile_base::os::alma8
      }
      'RedHat_9': {
        include profile_base::os::alma9
      }
      'Debian_11': {
        include profile_base::os::debian11
      }
      default: {
        include profile_base::os::alma8
      }
    }
  }
}

Once the profile_base was created, I created a testnode by cloning the template Almalinux 9 again. The first thing I did after creating this node, was installing a puppet-agent on the node using the puppet-role made by Jeff Geerling again. After that I created a snapshot of the new machine so that I had a clean point in time to return to after testing every puppet profile. I first made sure the agents that are being installed are configured correctly right away. This is done by adding a task to the role tasks:

- name: Set puppet server
  lineinfile:
    path: /etc/puppetlabs/puppet/puppet.conf
    line: |
      [agent]
      environment=production
      server={{ fqdn_puppetserver }}
      ca_server={{ fqdn_puppetserver }}      
    state: present

One small adjustment I made to the puppetserver is the ability to autosign new connections. This makes it easier to just create new nodes with a puppet-agent and start running it against the server. I added the following tasks to the puppet-role:

- name: Autosigning
  copy:
    path: /etc/puppetlabs/puppet/autosign.conf
    content: "{{ domain_puppetserver }}"
    state: present

Puppetizing

After setting up the puppetserver, a node with Gitea + Jenkins and a testnode; the testing and puppetizing could begin. I started out service by service and did a lot of hacky things (that I will fix in the future… I swear…).

Jenkins

The installation steps for the Jenkins installation were fresh in my mind, so I decided to start with this. I created a profile profile_jenkins and just wrote the whole installation in puppet code. This was pretty straight forward as I just needed to add the repository, install dependencies and the Jenkins package itself. I also installed some extra tools that I will need for future automatic deployment using Opentofu (Terraform) and Ansible.

Some snippets from the profile:

class profile_jenkins::server (
  Stdlib::Httpurl $repo_base_url    = 'https://pkg.jenkins.io',
  String          $gpg_key_filename = 'jenkins.io-2023.key',
) {
  # General
  user { 'jenkins':
    ensure     => present,
    managehome => true,
    shell      => '/usr/bin/false',
  }

  yumrepo { 'jenkins-repo':
    ensure  => present,
    descr   => 'Jenkins',
    enabled => 1,
    baseurl => "${repo_base_url}/redhat-stable/",
    gpgkey  => "${repo_base_url}/redhat-stable/${gpg_key_filename}",
  }

  yumrepo { 'opentofu-repo':
    ensure  => present,
    descr   => 'Opentofu',
    enabled => 1,
    baseurl => 'https://packages.opentofu.org/opentofu/tofu/rpm_any/rpm_any/$basearch',
    gpgkey  => 'https://get.opentofu.org/opentofu.gpg',
  }

  $packages = ['fontconfig', 'java-17-openjdk', 'jenkins','ansible-core', 'tofu']
  $packages.each | $package | {
    package { $package:
      ensure => present,
    }
  }

  service { 'jenkins':
    ensure  => 'running',
    enable  => true,
    require => Package['jenkins'],
  }
}

Gitea

The Gitea installation was the same process. The key difference here, is that I’m not using a package provided by the dnf/yum package managers. I made a hacky way to download the binary (which is version-pinned in my code) and install it with the right service file and configs. Once I change the version of the package, it will automatically pull that version, replace the binary and restart the service.

First the database needed to be setup:

class profile_gitea::database (
  String  $db_name      = undef,
  String  $db_user      = undef,
  String  $db_pass      = undef,
) {
  package { 'mariadb-server':
    ensure => 'latest',
  }

  service { 'mariadb':
    ensure  => 'running',
    enable  => true,
    require => Package['mariadb-server'],
  }

  mysql::db { 'gitea':
    user     => $db_user,
    password => $db_pass,
    host     => 'localhost',
    grant    => ['ALL'],
  }
}

The version change of the Gitea binary is checked by a custom puppet fact created with a ruby script:

# Check if the binary is present
def binary_present?(binary)
    Facter::Core::Execution.which(binary)
end

# Only run the fact if the binary is present
if binary_present?('/usr/local/bin/gitea')
    Facter.add('gitea_version') do
        setcode do
            gitea_version = Facter::Core::Execution.execute('sudo -u git /usr/local/bin/gitea --version 2>&1 | grep "version " | cut -d " " -f 3')
            Facter.debug("Gitea version from fact: #{gitea_version}")
            gitea_version.strip if $?.exitstatus == 0
        end
    end
end

A snippet of the installation:

class profile_gitea::server (
  String  $version      = undef,
  String  $package_type = 'linux-amd64',
) {
  user { 'git':
    ensure     => present,
    managehome => true,
    shell      => '/usr/bin/bash',
  }

  if $::facts['gitea_version'] != $version {
    # Stop gitea if it exists
    exec { 'gitea_stop':
      command => '/bin/systemctl stop gitea',
      onlyif  => '/usr/bin/test -e /usr/local/bin/gitea',
    }

    # Backup gitea if it exists
    exec { 'backup_gitea':
      command => '/bin/bash /home/git/backup.sh',
      user    => 'git',
      onlyif  => '/usr/bin/test -e /usr/local/bin/gitea',
    }

    file { "/tmp/gitea-${version}":
      ensure => file,
      source => "https://github.com/go-gitea/gitea/releases/download/v${version}/gitea-${version}-${package_type}",
      owner  => 'root',
      group  => 'root',
      mode   => '0755',
    }
  }

  file { '/usr/local/bin/gitea':
    ensure => file,
    source => "/tmp/gitea-${version}",
    owner  => 'root',
    group  => 'root',
    mode   => '0755',
  }
}

I also created a backup script that runs every night and backs up all the important data to my NAS:

#!/bin/bash

"/usr/local/bin/gitea" dump -c "/etc/gitea/app.ini" -f "/tmp/gitea_backup_$(date +%Y-%m-%d).zip"
rsync -rav -e "ssh -o StrictHostKeyChecking=no -i /home/rein/.ssh/id_rsa" "/tmp/gitea_backup_$(date +%Y-%m-%d).zip" "{USER}@{NAS}:/x/y/gitea/"
rm "/tmp/gitea_backup_$(date +%Y-%m-%d).zip"

Once all the puppet code was tested, I could deploy it to the previously created Jenkins + Gitea node. The puppet agent made a few changes but the main setup and configuration were unchanged. The first piece of the infrastructure as code is now created.