Jenkins Operator Kubernetes Install

Jenkins stateless deployments

Mon, 01 Jan 2024

I hear a lot of people complain about maintaining a Jenkins instance with all the updates, plugins as well as setting up jobs. What if I told you that you could deploy an entire Jenkins instance with just a few yaml files?

This will be a quick tutorial on how you can do that. I’ll go over importing jobs, setting up build agents and authentication. The way that this works is that the operator will manage the lifecycle of our instance as well as back up and restore build logs. Our instance is stateless after all and gets rebuilt every single time the pod restarts.

The Configuration As Code plugin is what will be setting all of our settings once the instance gets started. Do note, the documentation is terrible. It took a few days of trial and error to get this working like I wanted.

Lets start for the top

Jenkins Operator

The Helm chart that will allow us to create jenkins instances is Jenkins Operator. I’m not going to jump into too much details about the project. The operator itself will supply a few things for us

The Instance

We can deploy an operator that will manage a single Jenkins instance. You can deploy multiple operators in order to create multiple instances. The Operator will instance plugins needed for bootstrapping the instance via Jenkins plugins CLI and then install what they call user plugins after the instance is all set up such as popular Blue Ocean plugin

Seeding Jobs

The next thing that it does create a few seed jobs that will create our build structure. This part is rather straight forward if you use keep a Jenkinsfile in each repository that you want to build. This is done with the job dsl

Backup and Restore

By nature of it being stateless, once the instance is gone you lose all build logs and history. If you want to keep this then the operator does regular scheduled backups which it will restore on the creation of a new instance. Do be warned that it truly only backs up build logs and not any settings

Installing the operator

Before we install our operator we want to setup all of our config for our instance. I’ll give you a full working copy and then go over the individual pieces

  namespace: jenkins
  image: jenkins/jenkins:lts # can be any official jenkins image
    - name: TZ # Timezone
      value: America/New_York # TZ took forever to figure out
  backup: # Backup Settings
    enabled: true
    image: virtuslab/jenkins-operator-backup-pvc
    interval: 3600
    makeBackupBeforePodDeletion: true
    pvc: # PVC you want to use
      enabled: true
      className: nfs-client # This is my pvc class for my nfs fileshare
  # Optional, this is my configure as code configmap    
        configMapName: jenkins-config
      secretRefName: jenkins-config-secrets # Secret storing secrets for my jenkins-config
  latestPlugins: true
  basePlugins: # Plugins don't auto update, you have to specify exact versions and there is not a forced version scheme
    - name: kubernetes
      version: "4054.v2da_8e2794884"
    - name: workflow-job
      version: "1360.vc6700e3136f5"
    - name: workflow-aggregator
      version: "596.v8c21c963d92d"
    - name: git
      version: "5.2.0"
    - name: job-dsl
      version: "1.87"
    - name: configuration-as-code
      version: "1714.v09593e830cfa"
    - name: kubernetes-credentials-provider
      version: "1.258.v95949f923a_a_e"
  # This section is optional, this is for user defined plugins.  I'll add a few just to get you going
    - name: github-branch-source
      version: "1741.va_3028eb_9fd21"
    - name: blueocean
      version: "1.27.9"
  # Resource limits.  IMO cpu should never be limited and memory requests should always equal limit
    memory: 4Gi
    memory: 4Gi
  # My seed jobs are stored in a gitea repository.  I'll go over the seed jobs after.  Similar syntax can be used for githubm just pick an appropiate credentialID
  - id: jenkins-org-scans
    credentialType: usernamePassword
    credentialID: gitea-login
    targets: "jobs/*.jenkins"
    description: "Jenkins org repository"
    repositoryBranch: main
  # Backup volume mentioned earlier in the code
   - name: backup
       claimName: jenkins-backup
   - name: jenkins-secrets
       claimName: jenkins-secrets
   - mountPath: /var/lib/jenkins/secrets
     name: jenkins-secrets
# The amount of operators we want to depoy
  replicaCount: 1

Configuration as code

One thing to note, all of the secrets listed in this file as saved in the secret we referenced in the configureAsCode section of the operators value file

# Stores all of the credentials we want to add to the instance
      - credentials:
          - string:
              description: "Github PAT for Github access"
              id: "github-pat"
              scope: SYSTEM
              secret: ${GITHUB_PAT}
          - gitHubApp:
              appID: "428894"
              scope: GLOBAL
              description: "GitHub app"
              id: "github-app"
              privateKey: ${GITHUB_APP_KEY}
  systemMessage: ${SYSTEM_MESSAGE}
    - name: "Linux"
  nodes: # Adds a linux node from the cluster.  I'll go over this when we add the node to the cluster
    - permanent:
        labelString: "Linux"
            webSocket: true
              disabled: false
              failIfWorkDirIsMissing: false
              internalDir: "remoting"
        mode: EXCLUSIVE
        name: "Arthas"
        remoteFS: "/home/jenkins/agent"
        retentionStrategy: "always"
  # This is my config for Authentik Open ID Connect
      authorizationServerUrl: ""
      automanualconfigure: "auto"
      clientId: ${AUTHENTIK_CLIENT_ID}
      clientSecret: ${AUTHENTIK_CLIENT_SECRET}
      disableSslVerification: false
      endSessionEndpoint: ""
      overrideScopes: "email profile openid"
      overrideScopesDefined: true
      scopes: "openid profile email"
      tokenAuthMethod: "client_secret_post"
      tokenServerUrl: ""
      userInfoServerUrl: ""
      wellKnownOpenIDConfigurationUrl: ""
      emailFieldName: "email"
      fullNameFieldName: "given_name"
      groupsFieldName: "groups"

Building your seed job

I’ll give you a sample seed job. This one is fairly straightforward. It will scan the github org or username’s repos and add any repo to Jenkins that contains a Jenkinsfile

organizationFolder('my-github-username') {
    organizations {
        github {
            traits {
                gitHubBranchDiscovery {
                gitHubPullRequestDiscovery {
    orphanedItemStrategy {
        discardOldItems {
    triggers {
        periodicFolderTrigger {

Adding a build agent

You can dabble with Kubernetes agents and to be honest thats most likely the best approach since you would then be running a full stateless infrastructure. I’m just a guy and I used what was already there which was my old pod deployment. Do note that I did something hacky and saved the keys that Jenkins creates for agents and as part of my redeployment add those keys back to the instance. Instead of going with that hack you’re most likely better off just using the built in Kubernetes plugin. The operator already sets this up for and it is how its seeding the jobs

apiVersion: apps/v1
kind: Deployment
  name: jenkins-arthas
  namespace: jenkins
  labels: jenkins-agent
  replicas: 1
    matchLabels: jenkins-agent
      labels: jenkins-agent
            - labelSelector:
                  - key:
                    operator: In
                      - jenkins-agent
              topologyKey: ""
        - name: jenkins-arthas
          image: jenkins/inbound-agent@sha256:c4df2a7c2779f185a2d1014d0a4d684fbca9dbc26f5425714af7423b57e7e3f2
          imagePullPolicy: Always
              memory: "1Gi"
              memory: "1Gi"

Start the deployment

All thats left is to deploy!

  1. Deploy your build agent
  2. Deploy the configure as code config map
  3. Create your seed job files and save it to a repo that you can access with the credentials you set up in your configure as code
  4. Run the following below to deploy!

Save that file to your current directory as ./values/jenkins-operator.yaml

helm repo add jenkins
helm repo update
helm upgrade --install --namespace jenkins-operator jenkins-operator jenkins/jenkins-operator --values=.\values\jenkins-operator.yaml


This is a lot, this wasn’t easy. This should get you pretty far, if you have any questions feel free to reach out. I did not use this setup for an extended period of time. I ended up moving to BuildKite. Its fun, has an amazing test suite analyzer and is designed for me to provide compute so I don’t have to do all of this hacky things that I’ve done with TeamCity, Github Actions and Jenkins

Buy Me A CoffeeDigitalOcean Referral Badge
Edward Beazer

Edward Beazer - I just like to build shit. Sometimes I get stuck for hours, even days while trying to figure out how to solve an issue or implement a new feature. Hope my tips and tutorials can save you some time.

DigitalOcean Referral Badge