TeamCity docker swarm and kubernetes manifest

Usage#

I loved TeamCity! It was my favorite platform when it came down to running CI builds. I tried Github Actions, Gitlab Runners, Gitea Actions, Circle Ci and Jenkins.

One of my favorite things which ironically became the pain point of the system for me was the ability to define everything inside the web console and export those settings to my repo. TeamCity also had a custom Perforce plugin that was top notch. It works better than the plugin the Perforce team made for Jenkins!

Overall I ended up moving away from TeamCity because the workflow was a bit slow for me, the pricing model didn’t work well for me and the UI felt a little outdated. TeamCity will always have a special place in my heart and I will always check out what they’re doing simply because that application taught me a lot about DevOps and how I could optimize my pipelines

At the moment I’m currently using BuildKite and loving it. Their test suites are top notch, the UI is minimalistic, they bill by minutes rather than agents which is awesome and finally, it just works without having to install any crazy plugins that I have to hope will be continued to be maintained

Docker Swarm#

I am a firm believer that CPU should be left to unlimited on every service that is meant to run 24/7 and that you should limit your memory to prevent OOM issues. Your memory limit should never exceed your requests limit. This is set up with traefik on a traefik network I named “traefik-public”. My certresolver is a lets encrypt resolver I named le. I also use loki for logging. If you want to deploy just the image without traefik and loki, you can delete all of those lines as well as update the volumes section. I’m running a postgres db in the same cluster

version: "3.9"
services:
  teamcity:
    image: jetbrains/teamcity-server:2023.05.4
    volumes:
    - /shares/docker/tds/teamcity/data:/data/teamcity_server/datadir
    - /shares/docker/tds/teamcity/logs:/opt/teamcity/logs
    environment:
      TZ: America/New_York
      PUID: 1026
      PGID: 100
    networks:
    - traefik-public
    - databases
    deploy:
      resources:
        reservations:
          memory: 8000M
        limits:
          memory: 8000M
      labels:
      # Traefik Config
      - traefik.enable=true
      - traefik.docker.network=traefik-public
      - traefik.constraint-label=traefik-public
      # HTTPS Rules
      - traefik.http.routers.teamcity.rule=Host(`teamcity.mydomain.com`)
      - traefik.http.routers.teamcity.entrypoints=https
      - traefik.http.routers.teamcity.tls=true
      - traefik.http.routers.teamcity.tls.certresolver=le
      # Services
      - traefik.http.services.teamcity.loadbalancer.server.port=8111
    logging:
      driver: loki
      options:
        loki-url: http://192.168.50.95:3100/loki/api/v1/push

Kubernetes#

Very similar to my docker swarm setup. This uses traefik, and exposes my service to all ip’s on my local subnet. The shared storage is still on my nas. I also included a deployment for a pod that can be used to run builds. The pod has docker in docker support so you can use docker build commands

apiVersion: v1
kind: Namespace
metadata:
name: dev-ops
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: teamcity
namespace: dev-ops
labels:
  app: teamcity
  app.kubernetes.io/name: teamcity
spec:
replicas: 1
selector:
  matchLabels:
    app: teamcity
template:
  metadata:
    labels:
      app: teamcity
  spec:
    containers:
    - name: teamcity
      image: jetbrains/teamcity-server:2023.11
      ports:
        - name: web
          containerPort: 8111
      env:
      - name: TZ
        value: America/New_York
      - name: PUID
        value: "1026"
      - name: PGID
        value: "100"
      resources:
        requests:
          memory: 6Gi
      volumeMounts:
      - name: nfs-teamcity-vol
        mountPath: /data/teamcity_server/datadir
        subPath: data
      - name: nfs-teamcity-vol
        mountPath: /opt/teamcity/logs
        subPath: logs
    volumes:
    - name: nfs-teamcity-vol
      nfs:
        server: 192.168.50.227
        path: /volume1/docker/teamcity
---
apiVersion: v1
kind: Service
metadata:
name: teamcity
namespace: dev-ops
labels:
  app: teamcity
  app.kubernetes.io/name: teamcity
spec:
ports:
- port: 8111
  targetPort: web
  name: web
selector:
  app: teamcity
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: teamcity
namespace: dev-ops
annotations:
  cert-manager.io/cluster-issuer: le-prod
  kubernetes.io/ingress.class: traefik
  traefik.ingress.kubernetes.io/router.entrypoints: websecure
  traefik.ingress.kubernetes.io/router.tls: "true"
  traefik.ingress.kubernetes.io/router.middlewares: network-internal-whitelist@kubernetescrd
spec:
tls:
  - hosts:
      - teamcity.mydomain.com
    secretName: teamcity-tls
rules:
  - host: teamcity.mydomain.com
    http:
      paths:
        - path: /
          pathType: Prefix
          backend:
            service:
              name: teamcity
              port:
                name: web
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
labels:
  app: teamcity-agent
name: teamcity-agent
namespace: dev-ops
spec:
selector:
  matchLabels:
    app: teamcity-agent
template:
  metadata:
    labels:
      app: teamcity-agent
  spec:
    volumes:
      - name: docker-certs
        emptyDir: {}
    securityContext:
      fsGroup: 1000
    containers:
      - name: teamcity-agent
        image: jetbrains/teamcity-agent:latest
        resources:
          limits:
            memory: "1Gi"
          requests:
            memory: "1Gi"
        env:
          - name: DOCKER_HOST
            value: tcp://localhost:2376
          - name: DOCKER_CERT_PATH
            value: /certs/client
          - name: DOCKER_TLS_VERIFY
            value: "1"
        volumeMounts:
          - name: docker-certs
            mountPath: /certs
      - name: daemon
        image: docker:24.0.7-dind
        env:
          - name: DOCKER_TLS_CERTDIR
            value: /certs
        securityContext:
          privileged: true
        volumeMounts:
          - name: docker-certs
            mountPath: /certs

Extra Services#

If you search my website for traefik and loki you should find posts that go through a basic setup for Traefik in Kubernetes and Docker Swarm as well as Loki with Docker Swarm. I also have one for postgres