This blog post is meant to be more of a personal journey, migrating from a server running docker & traefik, to a server running Kubernetes (k3s to be exact). I hope this might help other people that already know the basics of docker, but want to transfer that knowledge into the realm of Kubernetes.

So let’s start with what we already have:

Screenshot from Portainer, showing the containers I will migrate to Kubernetes

Now, I’m not going to go though the migration for each application, but I am going to give a few examples.

Setting up k3s⌗

For this I chose to run k3s, instead of the full k8s (kubernetes) stack. K3s is designed to be lightweight and performant.

Once you have your server up and running, you can run the following command to install k3s simply. In my case, my server is running Arch Linux, because why not, but the process should be similar or exactly the same for other Linux distributions.

Now, normally I would tell you not to just pipe any old content from the internet into a shell, and this is no exception! Review the contents of the installer if you value security!

curl -sfL https://get.k3s.io | sh -

We’ll then copy the kubeconfig file from it’s default location (/etc/rancher/k3s/k3s.yaml) to our local machine in case we ever need to run manual commands on it.

sudo cp /etc/rancher/k3s/k3s.yaml ~/k3s.yaml
sudo chown $USER k3s.yaml

We copy it to our home directory first, and change the owner, so we can scp it to our local machine easily.

mkdir -p ~/.kube
scp user@my.host:~/k3s.yaml  ~/.kube/config

After that, you can delete the k3s.yaml file in your server’s home directory.

Make sure to change the ~/.kube/config file locally, and edit the server url to your public dns/server ip where you installed k3s! If you have a firewall set up on your server, make sure to allow tcp/6443.

apiVersion: v1
clusters:
- cluster:
    ...
    server: https://your.tld:6443
  	...

You should now be able to run the following command and see your server name show up.

$ kubectl get nodes             
NAME   STATUS   ROLES                  AGE   VERSION
alyx   Ready    control-plane,master   10m   v1.20.7+k3s1

At this point, you would join any worker nodes to the cluster, however, I’m just going to use this one node as my entire cluster.

Flux and GitOps⌗

You could do everything manually, by applying yaml files using kubectl, or helm commands, but that seems like a lot of effort. Especially bad if something goes wrong, or if you need to change something 3 months down the line. Or maybe you want to add another environment to your setup, like a staging env, or a development env. You’d have to re-do, and probably re-google, all the commands you did.

Luckely, Flux and GitOps saves our bacon.

GitOps is Ops on Git, as the name implies. It’s coding all your infrastructure and putting it in git.

Flux is a way to automatically deploy those config files from our GitOps process to our Kubernetes cluster. It will take care of watching for changes to our git repo, as well as look for updates for our running software, and update the cluster with those details.

Let’s get started with flux⌗

Following their installation guide, set the following environment variables. If you’re using something other than GitHub, look to their documentation for how to do that. You can get a personal access token here.

export GITHUB_USER=<username>
export GITHUB_TOKEN=<personal access token>

Next you need to get the flux cli, on Arch Linux, that’s as simple as installing flux-bin from the AUR. If you’re on another distribution, you can install it manually.

Make sure everything is alright, and then continue.

flux check --pre
► checking prerequisites
✔ kubectl 1.21.0 >=1.18.0-0
✔ Kubernetes 1.20.7+k3s1 >=1.16.0-0
✔ prerequisites checks passed

You can then run the bootstrap command. If you didn’t create the repository already, a private one will be created for you.

flux bootstrap github \
  --owner=$GITHUB_USER \
  --repository=fleet-infra \
  --branch=main \
  --path=./cluster \
  --personal

After a bunch of output, you should see

✔ all components are healthy

Next, we’ll clone the repo, and make some changes!

git clone ssh://git@github.com:$GITHUB_USER/fleet-infra
cd fleet-infra

Testing it out⌗

To just test that everything works, let’s install podinfo like they suggest.

Adding the source (where to find the helm charts) & installing the charts located

flux create source helm podinfo \ 
  --url https://stefanprodan.github.io/podinfo \
  --export > ./cluster/podinfo-source.yaml

flux create helmrelease podinfo \
  --chart podinfo \
  --source HelmRepository/podinfo
  --export > ./cluster/podinfo-release.yaml

git add .
git commit -m "added podinfo"
git push

After a little bit, you should be able to run kubectl get pods and see the pod(s) are Running.

$ kubectl get pods
NAME                       READY   STATUS    RESTARTS   AGE
podinfo-7c9fd56cb4-2985h   1/1     Running   0          81

“But hold on,” you might say, “if I go to my server’s domain, I get a nasty 404 page not found”. And that’s right! The example above doesn’t configure an ingress for podinfo, rendering it kindof useless as a web-app! Let’s fix that:

Edit the podinfo-release.yaml file, adding the values section, enabling the ingress and, setting a path it will listen on:

---
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: podinfo
spec:
  chart:
    spec:
      chart: podinfo
      sourceRef:
        kind: HelmRepository
        name: podinfo
      version: '>5.2.0'
  interval: 1m0s
  values:
    ingress:
      enabled: true
      path: /

Then commit & push your changes. You can watch for changes with

kubectl get pods --watch

Less than a minute later, you should be able to go to your server’s ip/domain and see podinfo:

PodInfo

Wait, what just happend?⌗

How did I know what I needed to add to make that work?

You can think of helm being a shortcut to applying templated yaml files. The Ingress template for PodInfo is located here. As you can see, it’s just a normal kubernetes Ingress yaml file, with some templating thrown in. The values we passed in above, makes its way to this template and inserts the values where needed.

You can also see all the configurations you can add on artifacthub. Note that even though you can put other paths than /, the pretty screen above only shows up on /.

Tear it down⌗

Now that we know everything is set up correctly, we can just delete the two yaml files we created, commit and push, and flux should automatically remove podinfo. Pretty cool!

The Lounge⌗

The Lounge is a web-based irc client. It’s pretty cool if you’re into irc still! I am going to use it as an example of a stateful application. To learn more about what a stateful-, and stateless applications are, see the workloads page on kubernete’s website. Applying this to any application with a docker-image should be straight forward.

We’re going to create a couple of files for this process, though you can technically put them all in one file.

The Kubernetes way⌗

The Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: the-lounge
  labels:
    app: the-lounge
spec:
  replicas: 1
  selector:
    matchLabels:
      app: the-lounge
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: the-lounge
    spec:
      containers:
      - image: thelounge/thelounge:latest
        name: the-lounge
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 9000
          name: http
        livenessProbe:
          httpGet:
            path: /
            port: http
        readinessProbe:
          httpGet:
            path: /
            port: http
        volumeMounts:
        - name: the-lounge-data
          mountPath: /var/opt/thelounge
        - name: the-lounge-config
          mountPath: /var/opt/thelounge/config.js
          subPath: config.js
      volumes:
      - name: the-lounge-data
        persistentVolumeClaim:
          claimName: the-lounge-pvc
      - name: the-lounge-config
        configMap:
          name: the-lounge-config

Mounting a file to the container (config - get the full config file here)

apiVersion: v1
kind: ConfigMap
metadata:
  name: the-lounge-config
data:
  config.js: |
  	"use strict";  

	module.exports = {
		// ## Server settings
	
		// ### `public`
		//
        ...

The Persistent Volume Claim for writing logs & uploaded content to.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: the-lounge-pvc
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: "5Gi"

The Service

apiVersion: v1
kind: Service
metadata:
  name: the-lounge
spec:
  type: ClusterIP
  ports:
    - port: 9000
      targetPort: http
      protocol: TCP
      name: http
  selector:
    app: the-lounge

And finially, the Ingress

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: the-lounge
spec:
  rules:
    - host: YOUR.HOST.HERE
      http:
        paths:
            path: /
            pathType: Prefix
            backend:
              serviceName: the-lounge
              servicePort: http

You should now see The Lounge running on your host. But this requires us to hard-code things, like the host into our ingress. Wouldn’t it be great if we could put that in a separate file and just tell kubernetes “hey, go look in this file for values?” That’s exactly where Kustomize comes in!

Kustomize works on a “patching” system, where you can define resources, and patch them to add, remove, or change certain parameters on them. This is especially useful if you want to run the same apps on multiple clusters, but with slightly different configurations – you just need to define the core app, and then patch them for each environment.

I only have one cluster, but I’m going to follow the principals anyway.

The Kustomization way⌗

Let’s start with moving the contents of the the-lounge folder out of the cluster folder and into the root of the git repo apps/the-lounge/base. Then, in there, create a kustomization.yaml file, with the follwing contents:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - deployment.yaml
  - ingress.yaml
  - pvc.yaml
  - service.yaml

Notice how I didn’t include the config-map.yaml? We’re going to generate that from a file, instead of putting the file in one giant string. So copy the config.js content into apps/the-lounge/config.js, and delete the config-map.yaml file. Also, remove the host in the ingress.yaml file. We’ll specify that in just a moment outside of the “app template.”

In apps/the-lounge/ingress-patch.json, paste the following:

[
  {
    "op": "replace",
    "path": "/spec/rules/0/host",
    "value": "YOUR.HOST.HERE"
  }
]

And in apps/the-lounge/kustomization.yaml (new file), add the following:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
metadata:
  name: the-lounge

resources:
  - ./base

patches:
  - path: ingress-patch.json
    target:
      kind: Ingress
      name: the-lounge

configMapGenerator:
  - name: the-lounge-config
    files:
      - config.js

And finally, in cluster/apps.yaml paste the following:

apiVersion: kustomize.toolkit.fluxcd.io/v1beta1
kind: Kustomization
metadata:
  name: apps
  namespace: flux-system
spec:
  interval: 1m0s
  sourceRef:
    kind: GitRepository
    name: flux-system
  path: ./apps
  prune: true
  validation: client

You should now have a folder structure like this:

.
├── apps
│   └── the-lounge
│       ├── base
│       │   ├── deployment.yaml
│       │   ├── ingress.yaml
│       │   ├── kustomization.yaml
│       │   ├── pvc.yaml
│       │   └── service.yaml
│       ├── config.js
│       ├── ingress-patch.json
│       └── kustomization.yaml
└── cluster
    ├── apps.yaml
    └── flux-system
        ├── gotk-components.yaml
        ├── gotk-sync.yaml
        └── kustomization.yaml

Explanation time:

Flux will scan our respository, in the cluster directory, for files to apply. It’ll find the Kustomization we made, refrencing the repo, but in the apps directory. It will then find all the kustomization.yaml files, and apply them.

In the main the-lounge kustomization, we tell it to find the resources in the base directory, but apply the following on top of that:

configMapGenerator, we generate the config map from the file, patches we patch (in this case add) a host to the ingress.

For more on how a multi-cluster system would look like with this system, you can check out this git repo.

Adding a user to the lounge⌗

The Lounge requires us to run a command to create users, so let’s see how that can be done:

$ kubectl get pods
NAME                          READY   STATUS    RESTARTS   AGE
the-lounge-6695b6bfd5-kmtv9   1/1     Running   0          33m

$ kubectl exec -it the-lounge-6695b6bfd5-kmtv9 -- bash

root@the-lounge-6695b6bfd5-kmtv9:/# su -u node

node@the-lounge-6695b6bfd5-kmtv9:/$ thelounge add polarizedions
2021-05-27 20:09:50 [WARN] Unknown key "prefetchMaxSearchSize", please verify your config.
2021-05-27 20:09:50 [WARN] Config file owner does not match the user you are currently running The Lounge as.
2021-05-27 20:09:50 [WARN] To prevent any issues, please run thelounge commands as the correct user that owns the config folder.
2021-05-27 20:09:50 [WARN] See https://thelounge.chat/docs/usage#using-the-correct-system-user for more information.
2021-05-27 20:09:51 [PROMPT] Enter password: 
2021-05-27 20:10:04 [PROMPT] Save logs to disk? (yes) yes
2021-05-27 20:10:06 [INFO] User polarizedions created.
2021-05-27 20:10:06 [INFO] User file located at /var/opt/thelounge/users/polarizedions.json.

node@the-lounge-6695b6bfd5-kmtv9:/$ exit
root@the-lounge-6695b6bfd5-kmtv9:/# exit

First I got the pods (right now there was just one), then I exec‘ed into the pod with bash. I changed my user to node, because that’s the user they recommend running commands form, and then I ran the command to add the user.

We’re not secure!⌗

You may have noticed we’re not serving a valid ssl certificate over https. Let’s change that.

Cert-manager⌗

In the root of the repo, create a folder infrastructure, and inside that, cert-manager and sources.

Inside sources, we’re going to put these two files:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
metadata:
  name: infra-sources

resources:
  - jetstack.yaml

apiVersion: source.toolkit.fluxcd.io/v1beta1
kind: HelmRepository
metadata:
  name: jetstack
  namespace: flux-system
spec:
  interval: 1m0s
  url: https://charts.jetstack.io

This adds the helm repository we need for cert-manager.

Next, we’ll install cert-manager: add a kustomization.yaml file to the cert-manager folder:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
metadata:
  name: cert-manager

resources:
  - https://github.com/jetstack/cert-manager/releases/download/v1.3.1/cert-manager.yaml

This kustomization applies v1.3.1’s yaml file to our cluster.

Next up, put cert-issuer.yaml inside our cluster folder:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-staging
spec:
  acme:
    email: YOUR@EMAIL.HERE
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: cert-manager-account-key-secret
    solvers:
    - http01:
        ingress:
          class: traefik

Update our ingress, by adding an annotation & tls section:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: the-lounge
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-staging
spec:
  tls:
    - secretName: the-lounge-tls
      hosts: []

  rules:
    - http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: the-lounge
                port:
                  number: 9000

Update our patch file, adding the host to the section:

[
    {
        "op": "replace",
        "path": "/spec/rules/0/host",
        "value": "YOUR.HOST.HERE"
    },
    {
        "op": "add",
        "path": "/spec/tls/0/hosts/0",
        "value": "YOUR.HOST.HERE"
    }
]

And finally, point to our infrastructure folder:

apiVersion: kustomize.toolkit.fluxcd.io/v1beta1
kind: Kustomization
metadata:
  name: infrastructure
  namespace: flux-system
spec:
  interval: 1m0s
  sourceRef:
    kind: GitRepository
    name: flux-system
  path: ./infrastructure
  prune: true
  validation: client

Now, when you commit and push this, it won’t immediately work! This is because that cert-issuer.yaml file is using a custom resource that is registered by cert-manager. However, flux may try to apply it before cert-manager, in which case, it will fail.

But, it will continue with the infrastructure kusomization, where it will register it, and the next time it tries to reconcile it (1min later in this case) it will succeed.

If you watch the ingesses carefully, you’ll notice a temporary one pop up. This is cert-manager doing its thing and verifying that you own the domain. As soon as it’s done, it’ll clean up after itself, and you should have a staging certificate on your domain. Exciting!

DNS challenges⌗

But what if I don’t want to do http challenges, but DNS challenges instead (like if I’m behind cloudflare?) Good thing that is also doable.

Update our cert-issuer.yaml file so it looks like:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-staging
spec:
  acme:
    email: YOUR.EMAIL.HERE
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: cert-manager-account-key-secret
    solvers:
    - dns01:
        cloudflare:
          email: YOUR.EMAIL.HERE
          apiKeySecretRef:
            name: cloudflare-apikey-secret
            key: apiKey

But, we’re not going to commit our secret to the git repo, after all, it’s reversible! Instead, we’re going to use sealed secrets.

Sealed Secrets⌗ 

flux create source helm sealed-secrets \
--interval=1h \
--url=https://bitnami-labs.github.io/sealed-secrets \
--export > infrastructure/sources/sealed-secrets.yaml

Don’t forget to add sealed-secrets.yaml to the sources’ kustomization.yaml file!

flux create helmrelease sealed-secrets \
--interval=1h \
--release-name=sealed-secrets \
--target-namespace=flux-system \
--source=HelmRepository/sealed-secrets \
--chart=sealed-secrets \
--crds=CreateReplace \
--export > infrastructure/sealed-secrets.yaml

Next, we need to install the kubeseal command line tool. For Arch Linux, it was as simple as sudo pacman -S kubeseal.

Commit and push at this point to get kubeseal on your cluster.

Next we run this command to fetch the public key, and put it into our repo:

kubeseal --fetch-cert \
--controller-name=sealed-secrets \
--controller-namespace=flux-system \
> pub-sealed-secrets.pem

Create our plaintext secret file:

mkdir -p infrastructure/secrets
kubectl -n default create secret generic cloudflare-apikey-secret \
    --from-literal=apiKey=keyboardcat \
    --namespace=cert-manager
    --dry-run=client \
	-o yaml > infrastructure/secrets/cloudflare-apikey-secret.yaml

Then seal it:

kubeseal --format=yaml --cert=pub-sealed-secrets.pem \
< infrastructure/secrets/cloudflare-apikey-secret.yaml > infrastructure/secrets/cloudflare-apikey-secret-sealed.yaml

You can now delete the plaintext cloudflare-apikey-secret.yaml file.

Goodbye Staging⌗

Now that all our testing is done, let’s switch to the production ACME servers - edit cert-issuer.yaml:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    email: YOUR@EMAIL.HERE
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: cert-manager-account-key-secret
    solvers:
    - dns01:
        cloudflare:
          email: YOUR@EMAIL.HERE
          apiKeySecretRef:
            name: cloudflare-apikey-secret
            key: apiKey

And your ingress’s annotation:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: the-lounge
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
...

Prometheus & Grafana⌗

Instead of going the route of applying a lot of yaml files, we’ll use the helm charts provided by the prometheus-community.

apiVersion: source.toolkit.fluxcd.io/v1beta1
kind: HelmRepository
metadata:
  name: prometheus-community
  namespace: flux-system
spec:
  interval: 1m0s
  url: https://prometheus-community.github.io/helm-charts

Don’t forget to reference it in the kusomization.yaml file!

Then we install it with helm, with a kustomization referencing it:

---
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: kube-prometheus-stack
  namespace: flux-system
spec:
  chart:
    spec:
      chart: kube-prometheus-stack
      sourceRef:
        kind: HelmRepository
        name: prometheus-community
  interval: 1h0m0s
  releaseName: kube-prometheus-stack
  targetNamespace: monitoring

apiVersion: v1
kind: Namespace
metadata:
  name: monitoring

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
metadata:
  name: prometheus-stack

resources:
  - namespace.yaml
  - prometheus-stack.yaml

You should then be able to see the pods coming up;

$ kubectl get pods -n monitoring -w 
NAME                                                        READY   STATUS    RESTARTS   AGE
kube-prometheus-stack-operator-55c67b68f5-lfgz4             1/1     Running   0          4m33s
kube-prometheus-stack-kube-state-metrics-6bd9c9779f-strt4   1/1     Running   0          4m33s
kube-prometheus-stack-prometheus-node-exporter-77g97        1/1     Running   0          4m34s
alertmanager-kube-prometheus-stack-alertmanager-0           2/2     Running   0          4m25s
prometheus-kube-prometheus-stack-prometheus-0               2/2     Running   1          4m24s
kube-prometheus-stack-grafana-766c75cfbc-hsn72              2/2     Running   0          4m33s

Access the dashboard via port-forwarding:

$ kubectl port-forward -n monitoring service/prometheus-operated 9090
Forwarding from 127.0.0.1:9090 -> 9090
Forwarding from [::1]:9090 -> 9090

Then you can visit localhost:9090 to view prometheus’s dashboard.

To get to Grafana, we do the following:

$ kubectl port-forward -n monitoring service/kube-prometheus-stack-grafana 9090:80
Forwarding from 127.0.0.1:9090 -> 3000
Forwarding from [::1]:9090 -> 3000

And visit localhost:9090 again. The default user, and password we can get via:

kubectl get secret -n monitoring kube-prometheus-stack-grafana -o jsonpath='{.data.admin-user}' | base64 --decode; echo

kubectl get secret -n monitoring kube-prometheus-stack-grafana -o jsonpath='{.data.admin-password}' | base64 --decode; echo

Note, that you probably want to enable a Persistent Volume Claim for grafana, since it defaults settings and dashboards to in-memory storage:

---
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: kube-prometheus-stack
  namespace: flux-system
spec:
  chart:
    spec:
      chart: kube-prometheus-stack
      sourceRef:
        kind: HelmRepository
        name: prometheus-community
  interval: 1h0m0s
  releaseName: kube-prometheus-stack
  targetNamespace: monitoring
  values:
    grafana:
      persistence:
        enabled: true

And if you want to enable an ingress for it:

---
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: kube-prometheus-stack
  namespace: flux-system
spec:
  chart:
    spec:
      chart: kube-prometheus-stack
      sourceRef:
        kind: HelmRepository
        name: prometheus-community
  interval: 1h0m0s
  releaseName: kube-prometheus-stack
  targetNamespace: monitoring
  values:
    grafana:
      persistence:
        enabled: true
      ingress:
        annotations:
          kubernetes.io/ingress.class: traefik
          cert-manager.io/cluster-issuer: letsencrypt-prod
        enabled: true
        hosts:
          - YOUR.HOST.HERE
        tls: 
          - secretName: grafana-tls
            hosts:
            - YOUR.HOST.HERE

Dependencies between pods⌗

So what do I do if there’s a dependency between pods? For example, a database pod and an application pod? Kubernetes provides the solution: init-containers.

Let’s create a small deployment that depends on mysql, with a accompanying kustomization file.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: test-mysql
  labels:
    app: test-mysql
spec:
  replicas: 1
  selector:
    matchLabels:
      app: test-mysql
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: test-mysql
    spec:
      containers:
      - image: busybox
        name: test-mysql
        imagePullPolicy: IfNotPresent
        command: ['sh', '-c', 'echo Hello world! Mysql Is Running!']
      initContainers:
      - name: init-test-mysql
        image: busybox
        command: ['sh', '-c', "until nslookup mysql.$(cat /var/run/secrets/kubernetes.io/serviceaccount/namespace).svc.cluster.local; do echo waiting for mysql; sleep 2; done"]

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
metadata:
  name: test-mysql

resources:
  - deployment.yaml

You’ll see a new section in the deployment: initContainers.

Kubernetes will start this list of containers before the main list of conatiners in the pod, and wait for them to finish before starting the main list of conatiners.

In this example, we put a init container, which loops and checks for the existence of a mysql service in the current namespace. Effectively doing: nslookup mysql.default.svc.cluster.local until it succeeds, then exits.

If you get the list of pods now, you’ll see it’s “pending” init:

$ kubectl get pods
NAME                          READY   STATUS     RESTARTS   AGE
the-lounge-7bcbd99f4b-q9lh9   1/1     Running    2          2d6h
test-mysql-6b5858b649-8pz97   0/1     Init:0/1   0          3m41s

Let’s add a mysql container then:

Add the bitnami helm sources:

apiVersion: source.toolkit.fluxcd.io/v1beta1
kind: HelmRepository
metadata:
  name: bitnami
  namespace: flux-system
spec:
  interval: 1m0s
  url: https://charts.bitnami.com/bitnami

And add a refrence to it in the kustomization.yaml file.

apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: mysql
  namespace: flux-system
spec:
  chart:
    spec:
      chart: mysql
      sourceRef:
        kind: HelmRepository
        name: bitnami
  interval: 1h0m0s
  releaseName: mysql
  targetNamespace: default

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
metadata:
  name: mysql

resources:
  - mysql.yaml

Commit and push, and watch the pods:

$ kubectl get pods -w
NAME                          READY   STATUS     RESTARTS   AGE
the-lounge-7bcbd99f4b-q9lh9   1/1     Running    2          2d6h
test-mysql-6b5858b649-8pz97   0/1     Init:0/1   0          15m

After a bit, you should see this output:

$ kubectl get pods -w
NAME                          READY   STATUS     RESTARTS   AGE
the-lounge-7bcbd99f4b-q9lh9   1/1     Running    2          2d6h
test-mysql-6b5858b649-8pz97   0/1     Init:0/1   0          15m
mysql-0                       0/1     Pending    0          0s
mysql-0                       0/1     Pending    0          3s
mysql-0                       0/1     ContainerCreating   0          3s
test-mysql-6b5858b649-8pz97   0/1     PodInitializing     0          17m
test-mysql-6b5858b649-8pz97   0/1     Completed           0          17m
test-mysql-6b5858b649-8pz97   0/1     Completed           1          17m
test-mysql-6b5858b649-8pz97   0/1     CrashLoopBackOff    1          18m
test-mysql-6b5858b649-8pz97   0/1     Completed           2          18m
mysql-0                       0/1     Running             0          26s
test-mysql-6b5858b649-8pz97   0/1     CrashLoopBackOff    2          18m

Mysql came up, and test-mysql started Crashing! Whoops!

This is to be expected though, since we didn’t tell k8s it’s a job, but a normal container. It will automatically attempt to keep the amount we specified (one in this case) containers running. Since it exits after printing “Hello world”, k8s assumes it crashed, and tries to create another container in that pod. If it fails a bunch of times, it enters a state known as CrashLoopBackOff, where it will wait longer and longer before provisioning a new container to the pod.

$ kubectl logs test-mysql-6b5858b649-8pz97
Hello world! Mysql Is Running!

To get the generated root password, you can execute the following:

$ kubectl get secret mysql -o jsonpath='{.data.mysql-root-password}' | base64 --decode; echo 
AdUoJURdmi

And to access it, you can port-forward to it, and access it on localhost:3306.

$ kubectl port-forward service/mysql 3306

You can now kill the test container (delete the folder & commit and push), and install whatever you need to.

Backups⌗

I decided on my own, custom implementation for backing up my MySQL database, using this script as a docker image, ghcr.io/polarizedions/backup-mysql-b2, I can backup a MySQL db to Backblaze’s B2 platform.

All I then need to do is create a cron job for it, and specify the secrets for B2:

apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: backup-mysql
  namespace: default
spec:
  schedule: "0 1 * * *"
  successfulJobsHistoryLimit: 1
  failedJobsHistoryLimit: 1
  jobTemplate:
    spec:
      template:
        spec:
          containers:
            - name: backup-mysql
              image: ghcr.io/polarizedions/backup-mysql-b2
              imagePullPolicy: IfNotPresent
              env:
              - name: DB_HOST
                value: mysql.default.svc.cluster.local
              - name: DB_USER
                value: root
              - name: DB_PASS
                valueFrom:
                  secretKeyRef:
                    name: mysql
                    key: mysql-root-password
              - name: B2_USER
                valueFrom:
                  secretKeyRef:
                    name: b2
                    key: user
              - name: B2_KEY
                valueFrom:
                  secretKeyRef:
                    name: b2
                    key: key
              - name: B2_BUCKET_NAME
                valueFrom:
                  secretKeyRef:
                    name: b2
                    key: bucket
          restartPolicy: Never

This cron schedule runs at 1am every day. I then just put a lifecycle setting on the bucket to delete files after 7 days.

Wrapping up⌗

I’ve covered, and learnt, a lot during this blog post, so hopefully you did too! Happy Kubernetting!