Jenkins Operator Kubernetes Install

Jenkins stateless deployments

Mon, 01 Jan 2024

I hear a lot of people complain about maintaining a Jenkins instance with all the updates, plugins as well as setting up jobs. What if I told you that you could deploy an entire Jenkins instance with just a few yaml files?

This will be a quick tutorial on how you can do that. I’ll go over importing jobs, setting up build agents and authentication. The way that this works is that the operator will manage the lifecycle of our instance as well as back up and restore build logs. Our instance is stateless after all and gets rebuilt every single time the pod restarts.

The Configuration As Code plugin is what will be setting all of our settings once the instance gets started. Do note, the documentation is terrible. It took a few days of trial and error to get this working like I wanted.

Lets start for the top

Jenkins Operator

The Helm chart that will allow us to create jenkins instances is Jenkins Operator. I’m not going to jump into too much details about the project. The operator itself will supply a few things for us

The Instance

We can deploy an operator that will manage a single Jenkins instance. You can deploy multiple operators in order to create multiple instances. The Operator will instance plugins needed for bootstrapping the instance via Jenkins plugins CLI and then install what they call user plugins after the instance is all set up such as popular Blue Ocean plugin

Seeding Jobs

The next thing that it does create a few seed jobs that will create our build structure. This part is rather straight forward if you use keep a Jenkinsfile in each repository that you want to build. This is done with the job dsl

Backup and Restore

By nature of it being stateless, once the instance is gone you lose all build logs and history. If you want to keep this then the operator does regular scheduled backups which it will restore on the creation of a new instance. Do be warned that it truly only backs up build logs and not any settings

Installing the operator

Before we install our operator we want to setup all of our config for our instance. I’ll give you a full working copy and then go over the individual pieces

jenkins:
  namespace: jenkins
  image: jenkins/jenkins:lts # can be any official jenkins image
  env:
    - name: TZ # Timezone
      value: America/New_York # TZ took forever to figure out
  backup: # Backup Settings
    enabled: true
    image: virtuslab/jenkins-operator-backup-pvc
    interval: 3600
    makeBackupBeforePodDeletion: true
    pvc: # PVC you want to use
      enabled: true
      className: nfs-client # This is my pvc class for my nfs fileshare
  # Optional, this is my configure as code configmap    
  configuration:
    configurationAsCode:
      configurations:
        configMapName: jenkins-config
      secretRefName: jenkins-config-secrets # Secret storing secrets for my jenkins-config
  latestPlugins: true
  basePlugins: # Plugins don't auto update, you have to specify exact versions and there is not a forced version scheme
    - name: kubernetes
      version: "4054.v2da_8e2794884"
    - name: workflow-job
      version: "1360.vc6700e3136f5"
    - name: workflow-aggregator
      version: "596.v8c21c963d92d"
    - name: git
      version: "5.2.0"
    - name: job-dsl
      version: "1.87"
    - name: configuration-as-code
      version: "1714.v09593e830cfa"
    - name: kubernetes-credentials-provider
      version: "1.258.v95949f923a_a_e"
  # This section is optional, this is for user defined plugins.  I'll add a few just to get you going
  plugins:
    - name: github-branch-source
      version: "1741.va_3028eb_9fd21"
    - name: blueocean
      version: "1.27.9"
  # Resource limits.  IMO cpu should never be limited and memory requests should always equal limit
  limits:
    memory: 4Gi
  requests:
    memory: 4Gi
  # My seed jobs are stored in a gitea repository.  I'll go over the seed jobs after.  Similar syntax can be used for githubm just pick an appropiate credentialID
  seedJobs:
  - id: jenkins-org-scans
    credentialType: usernamePassword
    credentialID: gitea-login
    targets: "jobs/*.jenkins"
    description: "Jenkins org repository"
    repositoryBranch: main
    repositoryUrl: https://git.mydomain.com/devops/jenkins-configuration.git
  # Backup volume mentioned earlier in the code
  volumes:
   - name: backup
     persistentVolumeClaim:
       claimName: jenkins-backup
   - name: jenkins-secrets
     persistentVolumeClaim:
       claimName: jenkins-secrets
  volumeMounts:
   - mountPath: /var/lib/jenkins/secrets
     name: jenkins-secrets
# The amount of operators we want to depoy
operator:
  replicaCount: 1

Configuration as code

One thing to note, all of the secrets listed in this file as saved in the secret we referenced in the configureAsCode section of the operators value file

# Stores all of the credentials we want to add to the instance
credentials:
  system:
    domainCredentials:
      - credentials:
          - string:
              description: "Github PAT for Github access"
              id: "github-pat"
              scope: SYSTEM
              secret: ${GITHUB_PAT}
          - gitHubApp:
              appID: "428894"
              scope: GLOBAL
              description: "GitHub app"
              id: "github-app"
              privateKey: ${GITHUB_APP_KEY}
jenkins:
  systemMessage: ${SYSTEM_MESSAGE}
  labelAtoms:
    - name: "Linux"
  nodes: # Adds a linux node from the cluster.  I'll go over this when we add the node to the cluster
    - permanent:
        labelString: "Linux"
        launcher:
          inbound:
            webSocket: true
            workDirSettings:
              disabled: false
              failIfWorkDirIsMissing: false
              internalDir: "remoting"
        mode: EXCLUSIVE
        name: "Arthas"
        remoteFS: "/home/jenkins/agent"
        retentionStrategy: "always"
  # This is my config for Authentik Open ID Connect
  securityRealm:
    oic:
      authorizationServerUrl: "https://auth.mydomain.com/application/o/authorize/"
      automanualconfigure: "auto"
      clientId: ${AUTHENTIK_CLIENT_ID}
      clientSecret: ${AUTHENTIK_CLIENT_SECRET}
      disableSslVerification: false
      endSessionEndpoint: "https://auth.mydomain.com/application/o/jenkins/end-session/"
      overrideScopes: "email profile openid"
      overrideScopesDefined: true
      scopes: "openid profile email"
      tokenAuthMethod: "client_secret_post"
      tokenServerUrl: "https://auth.mydomain.com/application/o/token/"
      userInfoServerUrl: "https://auth.mydomain.com/application/o/userinfo/"
      wellKnownOpenIDConfigurationUrl: "https://auth.mydomain.com/application/o/jenkins/.well-known/openid-configuration"
      emailFieldName: "email"
      fullNameFieldName: "given_name"
      groupsFieldName: "groups"

Building your seed job

I’ll give you a sample seed job. This one is fairly straightforward. It will scan the github org or username’s repos and add any repo to Jenkins that contains a Jenkinsfile

organizationFolder('my-github-username') {
    organizations {
        github {
            repoOwner('my-github-username')
            credentialsId('github-app')
            traits {
                gitHubBranchDiscovery {
                    strategyId(0)
                }
                gitHubPullRequestDiscovery {
                    strategyId(1)
                }
                gitHubExcludeArchivedRepositories()
                gitHubTagDiscovery()
            }
        }
    }
    orphanedItemStrategy {
        discardOldItems {
            daysToKeep(60)
        }
    }
    triggers {
        periodicFolderTrigger {
            interval('15m')
        }
    }
}

Adding a build agent

You can dabble with Kubernetes agents and to be honest thats most likely the best approach since you would then be running a full stateless infrastructure. I’m just a guy and I used what was already there which was my old pod deployment. Do note that I did something hacky and saved the keys that Jenkins creates for agents and as part of my redeployment add those keys back to the instance. Instead of going with that hack you’re most likely better off just using the built in Kubernetes plugin. The operator already sets this up for and it is how its seeding the jobs

apiVersion: apps/v1
kind: Deployment
metadata:
  name: jenkins-arthas
  namespace: jenkins
  labels:
    app.kubernetes.io/name: jenkins-agent
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: jenkins-agent
  template:
    metadata:
      labels:
        app.kubernetes.io/name: jenkins-agent
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchExpressions:
                  - key: app.kubernetes.io/name
                    operator: In
                    values:
                      - jenkins-agent
              topologyKey: "kubernetes.io/hostname"
      containers:
        - name: jenkins-arthas
          image: jenkins/inbound-agent@sha256:c4df2a7c2779f185a2d1014d0a4d684fbca9dbc26f5425714af7423b57e7e3f2
          imagePullPolicy: Always
          resources:
            limits:
              memory: "1Gi"
            requests:
              memory: "1Gi"

Start the deployment

All thats left is to deploy!

Deploy your build agent
Deploy the configure as code config map
Create your seed job files and save it to a repo that you can access with the credentials you set up in your configure as code
Run the following below to deploy!

Save that file to your current directory as ./values/jenkins-operator.yaml

helm repo add jenkins https://raw.githubusercontent.com/jenkinsci/kubernetes-operator/master/chart
helm repo update
helm upgrade --install --namespace jenkins-operator jenkins-operator jenkins/jenkins-operator --values=.\values\jenkins-operator.yaml

Closing

This is a lot, this wasn’t easy. This should get you pretty far, if you have any questions feel free to reach out. I did not use this setup for an extended period of time. I ended up moving to BuildKite. Its fun, has an amazing test suite analyzer and is designed for me to provide compute so I don’t have to do all of this hacky things that I’ve done with TeamCity, Github Actions and Jenkins

Jenkins Operator

The Instance

Seeding Jobs

Backup and Restore

Installing the operator

Configuration as code

Building your seed job

Adding a build agent

Start the deployment

Closing

Postgres Kubernetes backup cron job and restore

Dump your entire postgres db as part of your backup procedure

ArgoCD Custom Resource Definition too long

When CRD's get too lengthy, we need to use server side apply

Deploy Redis Operator in kubernetes

Deploy and manager a simple redis instance using Redis Operator

Gitea act runner setup

Setup some quick gitea act runners with docker support

Homebridge Docker Swarm setup

Deploy Homebridge with Docker Swarm

Gitea Renovate Setup

Create a cron job in a kubernetes cluster to run renovate

Gitea Kubernetes Deployment

Deploy Gitea in your kubernetes cluster

Urbackup Docker Deployment

Deploy urbackup with docker swarm

Kubernetes Traefik Setup with MetalLB

Deploy traefik with MetalLb in your cluster

Logging and monitoring with Grafana, Prometheus and Loki on Docker Swarm

Deploy grafana and loki to monitor your docker swarm cluster

Cron jobs with Docker Swarm

Prune images and update container images on a schedule with cron jobs in Docker Swarm

ArgoCD Discord Webhooks

Send application status to discord with argo cd

Docker buildx build cache with Github actions

Save your build cache from docker with Github Actions to improve your deployment speeds

Creating and using Kubernetes service accounts

Create the appropriate service accounts and import into your kube.config

Flutter Firebase auth testing

Test your authentication functions with firebase

YouTrack Docker Swarm and Kubernetes manifests

Deploy YouTrack to docker swarm or kubernetes

TeamCity docker swarm and kubernetes manifest

Deploy TeamCity to docker and kubernetes

Host Android Sign in with Apple function on firebase

Host your server for sign in with apple using Android on Firebase

Setting up Flutter ODM with Firestore and Freezed

Generate types for your firebase collections

Firebase functions Unit Testing

Let's walk through a basic unit tests of a function trigger

CNPG Postgres Operator cluster install

Deploy postgres clusters with ease using CNPG

Firebase Emulators easy start with backup and restore

Use this docker-compose setup to easily run your firebase suite locally with saved data

Authentik Kubernetes and Docker swarm setup

Deploy kubernetes with authentik and docker swarm

TeamCity Flutter Test Apps Config

A simple config to test flutter apps

Jenkins Operator Kubernetes Install

Jenkins stateless deployments

Time to review, reflect and move forward

Recap of what I worked on in my spare time in 2023 and plans for 2024

Fill DDC for all maps inside of a project for multiple platforms

Finally was able to figure this out without my old hack

Companion script for ue4-docker image builds

This powershell script will help clean, build and push your Unreal Engine docker images

Slim down Unreal Engine 5 docker build images

With this configuration I was able to slim down my images from 120GB to 40GB

One liner to set up ue4docker on a fresh Windows server

Powershell script to install and configure ue4docker

Unique way to backup Perforce Streams

TeamCity config to back up Perforce streams

Sync configuration parameters in a build chain in TeamCity

Use the same parameter values for dependency builds

Project Rising 7/2023 Update

This month I worked on my perforce setup and TeamCity build configurations

Unreal Engine 5 Test template

TeamCity UE5 test config that I use for Project Rising

Unreal Engine 5 Windows build template

TeamCity UE5 build config that I use for Project Rising