The Instance#

We can deploy an operator that will manage a single Jenkins instance. You can deploy multiple operators in order to create multiple instances. The Operator will instance plugins needed for bootstrapping the instance via Jenkins plugins CLI and then install what they call user plugins after the instance is all set up such as popular Blue Ocean plugin

Seeding Jobs#

The next thing that it does create a few seed jobs that will create our build structure. This part is rather straight forward if you use keep a Jenkinsfile in each repository that you want to build. This is done with the job dsl

Backup and Restore#

By nature of it being stateless, once the instance is gone you lose all build logs and history. If you want to keep this then the operator does regular scheduled backups which it will restore on the creation of a new instance. Do be warned that it truly only backs up build logs and not any settings

Installing the operator#

Before we install our operator we want to setup all of our config for our instance. I’ll give you a full working copy and then go over the individual pieces

jenkins:
  namespace: jenkins
  image: jenkins/jenkins:lts # can be any official jenkins image
  env:
    - name: TZ # Timezone
      value: America/New_York # TZ took forever to figure out
  backup: # Backup Settings
    enabled: true
    image: virtuslab/jenkins-operator-backup-pvc
    interval: 3600
    makeBackupBeforePodDeletion: true
    pvc: # PVC you want to use
      enabled: true
      className: nfs-client # This is my pvc class for my nfs fileshare
  # Optional, this is my configure as code configmap    
  configuration:
    configurationAsCode:
      configurations:
        configMapName: jenkins-config
      secretRefName: jenkins-config-secrets # Secret storing secrets for my jenkins-config
  latestPlugins: true
  basePlugins: # Plugins don't auto update, you have to specify exact versions and there is not a forced version scheme
    - name: kubernetes
      version: "4054.v2da_8e2794884"
    - name: workflow-job
      version: "1360.vc6700e3136f5"
    - name: workflow-aggregator
      version: "596.v8c21c963d92d"
    - name: git
      version: "5.2.0"
    - name: job-dsl
      version: "1.87"
    - name: configuration-as-code
      version: "1714.v09593e830cfa"
    - name: kubernetes-credentials-provider
      version: "1.258.v95949f923a_a_e"
  # This section is optional, this is for user defined plugins.  I'll add a few just to get you going
  plugins:
    - name: github-branch-source
      version: "1741.va_3028eb_9fd21"
    - name: blueocean
      version: "1.27.9"
  # Resource limits.  IMO cpu should never be limited and memory requests should always equal limit
  limits:
    memory: 4Gi
  requests:
    memory: 4Gi
  # My seed jobs are stored in a gitea repository.  I'll go over the seed jobs after.  Similar syntax can be used for githubm just pick an appropiate credentialID
  seedJobs:
  - id: jenkins-org-scans
    credentialType: usernamePassword
    credentialID: gitea-login
    targets: "jobs/*.jenkins"
    description: "Jenkins org repository"
    repositoryBranch: main
    repositoryUrl: https://git.mydomain.com/devops/jenkins-configuration.git
  # Backup volume mentioned earlier in the code
  volumes:
   - name: backup
     persistentVolumeClaim:
       claimName: jenkins-backup
   - name: jenkins-secrets
     persistentVolumeClaim:
       claimName: jenkins-secrets
  volumeMounts:
   - mountPath: /var/lib/jenkins/secrets
     name: jenkins-secrets
# The amount of operators we want to depoy
operator:
  replicaCount: 1

Configuration as code#

One thing to note, all of the secrets listed in this file as saved in the secret we referenced in the configureAsCode section of the operators value file

# Stores all of the credentials we want to add to the instance
credentials:
  system:
    domainCredentials:
      - credentials:
          - string:
              description: "Github PAT for Github access"
              id: "github-pat"
              scope: SYSTEM
              secret: ${GITHUB_PAT}
          - gitHubApp:
              appID: "428894"
              scope: GLOBAL
              description: "GitHub app"
              id: "github-app"
              privateKey: ${GITHUB_APP_KEY}
jenkins:
  systemMessage: ${SYSTEM_MESSAGE}
  labelAtoms:
    - name: "Linux"
  nodes: # Adds a linux node from the cluster.  I'll go over this when we add the node to the cluster
    - permanent:
        labelString: "Linux"
        launcher:
          inbound:
            webSocket: true
            workDirSettings:
              disabled: false
              failIfWorkDirIsMissing: false
              internalDir: "remoting"
        mode: EXCLUSIVE
        name: "Arthas"
        remoteFS: "/home/jenkins/agent"
        retentionStrategy: "always"
  # This is my config for Authentik Open ID Connect
  securityRealm:
    oic:
      authorizationServerUrl: "https://auth.mydomain.com/application/o/authorize/"
      automanualconfigure: "auto"
      clientId: ${AUTHENTIK_CLIENT_ID}
      clientSecret: ${AUTHENTIK_CLIENT_SECRET}
      disableSslVerification: false
      endSessionEndpoint: "https://auth.mydomain.com/application/o/jenkins/end-session/"
      overrideScopes: "email profile openid"
      overrideScopesDefined: true
      scopes: "openid profile email"
      tokenAuthMethod: "client_secret_post"
      tokenServerUrl: "https://auth.mydomain.com/application/o/token/"
      userInfoServerUrl: "https://auth.mydomain.com/application/o/userinfo/"
      wellKnownOpenIDConfigurationUrl: "https://auth.mydomain.com/application/o/jenkins/.well-known/openid-configuration"
      emailFieldName: "email"
      fullNameFieldName: "given_name"
      groupsFieldName: "groups"

Building your seed job#

I’ll give you a sample seed job. This one is fairly straightforward. It will scan the github org or username’s repos and add any repo to Jenkins that contains a Jenkinsfile

organizationFolder('my-github-username') {
    organizations {
        github {
            repoOwner('my-github-username')
            credentialsId('github-app')
            traits {
                gitHubBranchDiscovery {
                    strategyId(0)
                }
                gitHubPullRequestDiscovery {
                    strategyId(1)
                }
                gitHubExcludeArchivedRepositories()
                gitHubTagDiscovery()
            }
        }
    }
    orphanedItemStrategy {
        discardOldItems {
            daysToKeep(60)
        }
    }
    triggers {
        periodicFolderTrigger {
            interval('15m')
        }
    }
}

Adding a build agent#

You can dabble with Kubernetes agents and to be honest thats most likely the best approach since you would then be running a full stateless infrastructure. I’m just a guy and I used what was already there which was my old pod deployment. Do note that I did something hacky and saved the keys that Jenkins creates for agents and as part of my redeployment add those keys back to the instance. Instead of going with that hack you’re most likely better off just using the built-in Kubernetes plugin. The operator already sets this up for, and it is how it’s seeding the jobs

apiVersion: apps/v1
kind: Deployment
metadata:
  name: jenkins-arthas
  namespace: jenkins
  labels:
    app.kubernetes.io/name: jenkins-agent
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: jenkins-agent
  template:
    metadata:
      labels:
        app.kubernetes.io/name: jenkins-agent
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchExpressions:
                  - key: app.kubernetes.io/name
                    operator: In
                    values:
                      - jenkins-agent
              topologyKey: "kubernetes.io/hostname"
      containers:
        - name: jenkins-arthas
          image: jenkins/inbound-agent@sha256:c4df2a7c2779f185a2d1014d0a4d684fbca9dbc26f5425714af7423b57e7e3f2
          imagePullPolicy: Always
          resources:
            limits:
              memory: "1Gi"
            requests:
              memory: "1Gi"