TeamCity docker swarm and kubernetes manifest
Deploy TeamCity to docker and kubernetes
Usage
I loved TeamCity! It was my favorite platform when it came down to running CI builds. I tried Github Actions, Gitlab Runners, Gitea Actions, Circle Ci and Jenkins.
One of my favorite things which ironically became the pain point of the system for me was the ability to define everything inside the web console and export those settings to my repo. TeamCity also had a custom Perforce plugin that was top notch. It works better than the plugin the Perforce team made for Jenkins!
Overall I ended up moving away from TeamCity because the workflow was a bit slow for me, the pricing model didn’t work well for me and the UI felt a little outdated. TeamCity will always have a special place in my heart and I will always check out what they’re doing simply because that application taught me a lot about DevOps and how I could optimize my pipelines
At the moment I’m currently using BuildKite and loving it. Their test suites are top notch, the UI is minimalistic, they bill by minutes rather than agents which is awesome and finally, it just works without having to install any crazy plugins that I have to hope will be continued to be maintained
Docker Swarm
I am a firm believer that CPU should be left to unlimited on every service that is meant to run 24/7 and that you should limit your memory to prevent OOM issues. Your memory limit should never exceed your requests limit. This is set up with traefik on a traefik network I named “traefik-public”. My certresolver is a lets encrypt resolver I named le. I also use loki for logging. If you want to deploy just the image without traefik and loki, you can delete all of those lines as well as update the volumes section. I’m running a postgres db in the same cluster
version: "3.9"
services:
teamcity:
image: jetbrains/teamcity-server:2023.05.4
volumes:
- /shares/docker/tds/teamcity/data:/data/teamcity_server/datadir
- /shares/docker/tds/teamcity/logs:/opt/teamcity/logs
environment:
TZ: America/New_York
PUID: 1026
PGID: 100
networks:
- traefik-public
- databases
deploy:
resources:
reservations:
memory: 8000M
limits:
memory: 8000M
labels:
# Traefik Config
- traefik.enable=true
- traefik.docker.network=traefik-public
- traefik.constraint-label=traefik-public
# HTTPS Rules
- traefik.http.routers.teamcity.rule=Host(`teamcity.mydomain.com`)
- traefik.http.routers.teamcity.entrypoints=https
- traefik.http.routers.teamcity.tls=true
- traefik.http.routers.teamcity.tls.certresolver=le
# Services
- traefik.http.services.teamcity.loadbalancer.server.port=8111
logging:
driver: loki
options:
loki-url: http://192.168.50.95:3100/loki/api/v1/pushKubernetes
Very similar to my docker swarm setup. This uses traefik, and exposes my service to all ip’s on my local subnet. The shared storage is still on my nas. I also included a deployment for a pod that can be used to run builds. The pod has docker in docker support so you can use docker build commands
apiVersion: v1
kind: Namespace
metadata:
name: dev-ops
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: teamcity
namespace: dev-ops
labels:
app: teamcity
app.kubernetes.io/name: teamcity
spec:
replicas: 1
selector:
matchLabels:
app: teamcity
template:
metadata:
labels:
app: teamcity
spec:
containers:
- name: teamcity
image: jetbrains/teamcity-server:2023.11
ports:
- name: web
containerPort: 8111
env:
- name: TZ
value: America/New_York
- name: PUID
value: "1026"
- name: PGID
value: "100"
resources:
requests:
memory: 6Gi
volumeMounts:
- name: nfs-teamcity-vol
mountPath: /data/teamcity_server/datadir
subPath: data
- name: nfs-teamcity-vol
mountPath: /opt/teamcity/logs
subPath: logs
volumes:
- name: nfs-teamcity-vol
nfs:
server: 192.168.50.227
path: /volume1/docker/teamcity
---
apiVersion: v1
kind: Service
metadata:
name: teamcity
namespace: dev-ops
labels:
app: teamcity
app.kubernetes.io/name: teamcity
spec:
ports:
- port: 8111
targetPort: web
name: web
selector:
app: teamcity
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: teamcity
namespace: dev-ops
annotations:
cert-manager.io/cluster-issuer: le-prod
kubernetes.io/ingress.class: traefik
traefik.ingress.kubernetes.io/router.entrypoints: websecure
traefik.ingress.kubernetes.io/router.tls: "true"
traefik.ingress.kubernetes.io/router.middlewares: network-internal-whitelist@kubernetescrd
spec:
tls:
- hosts:
- teamcity.mydomain.com
secretName: teamcity-tls
rules:
- host: teamcity.mydomain.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: teamcity
port:
name: web
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
labels:
app: teamcity-agent
name: teamcity-agent
namespace: dev-ops
spec:
selector:
matchLabels:
app: teamcity-agent
template:
metadata:
labels:
app: teamcity-agent
spec:
volumes:
- name: docker-certs
emptyDir: {}
securityContext:
fsGroup: 1000
containers:
- name: teamcity-agent
image: jetbrains/teamcity-agent:latest
resources:
limits:
memory: "1Gi"
requests:
memory: "1Gi"
env:
- name: DOCKER_HOST
value: tcp://localhost:2376
- name: DOCKER_CERT_PATH
value: /certs/client
- name: DOCKER_TLS_VERIFY
value: "1"
volumeMounts:
- name: docker-certs
mountPath: /certs
- name: daemon
image: docker:24.0.7-dind
env:
- name: DOCKER_TLS_CERTDIR
value: /certs
securityContext:
privileged: true
volumeMounts:
- name: docker-certs
mountPath: /certsExtra Services
If you search my website for traefik and loki you should find posts that go through a basic setup for Traefik in Kubernetes and Docker Swarm as well as Loki with Docker Swarm. I also have one for postgres
