k3s + Flux: Building a Proper GitOps Pipeline

Part four. Dokploy crashed my server once (turned out to be a laptop suspend issue, not Dokploy's fault, but the timing was enough to make me want something with fewer moving parts I don't control). This is the full rebuild: k3s, Flux, a self hosted runner, and a domain pointed at the result. Writing this one straight through, correct config first, with enough explanation that I still understand it in six months.

1. The Plan
2. Variables Used in This Guide
3. Installing k3s
4. Bootstrapping Flux
5. Self Hosted GitHub Actions Runner
6. CI: Build and Push
7. image.yaml: Watching for New Images
8. app.yaml: Running the App and Exposing It
9. Known Issues
10. Now What If I Want to Deploy a New App?
11. Final Checklist
12. Closing Thoughts

---

The Plan

git push
  -> self hosted runner builds an image, pushes to a registry
  -> image.yaml notices the new image, writes the new tag into app.yaml, pushes that
  -> Flux applies app.yaml, pod restarts
  -> Cloudflare Tunnel + k3s's own Traefik serve it at the domain

Two manifest files do almost all the conceptual work here, image.yaml (watches, decides, writes) and app.yaml (what's actually running). Worth understanding that split before copying anything, it's explained properly further down.

Variables Used in This Guide

Every file below reuses the same four placeholders. Swap each one in everywhere it appears rather than per-file, a mismatch between files was the actual cause of more than one bug while building this (see Known Issues).

<APP_NAME> doesn't have to match <GH_REPO>, Kubernetes doesn't know or care what the app is called on GitHub, but keeping them the same removes one more thing to keep track of.

Installing k3s

ss -tulnp | grep -E ':80 |:443 '

Make sure nothing else (an old Caddy, an old Dokploy) is still sitting on those ports first. Then:

curl -sfL https://get.k3s.io | sh -

k3s bundles Traefik, a storage provisioner, and CoreDNS in that single install, nothing extra to set up for a single node.

kubectl won't find the cluster by default, since k3s writes its kubeconfig somewhere kubectl doesn't look automatically:

echo 'export KUBECONFIG=/etc/rancher/k3s/k3s.yaml' >> ~/.bashrc
source ~/.bashrc
kubectl get nodes

One node, status Ready, and that's the whole install.

Bootstrapping Flux

curl -s https://fluxcd.io/install.sh | sudo bash

Bootstrapping with image automation enabled from the start matters here: it's possible to add it later, but the deploy key Flux creates on the first run is read only by default, and Flux never upgrades an existing key's permissions on a second bootstrap. Doing it up front avoids a redo.

export GITHUB_TOKEN=<your-token>

flux bootstrap github \
  --owner=<GH_USERNAME> \
  --repository=homelab-k8s \
  --branch=main \
  --path=clusters/proxmox \
  --personal \
  --components-extra=image-reflector-controller,image-automation-controller \
  --read-write-key=true

--read-write-key=true is the flag most tutorials skip, it's what lets Flux push commits back to the repo, which the entire automated deploy loop depends on.

flux check
kubectl get pods -n flux-system

Everything Running means the bootstrap actually landed.

Self Hosted GitHub Actions Runner

Self hosted instead of GitHub's own runners because the repo is private (2,000 free minutes a month on GitHub's side, generous for occasional pushes, but no reason to have a ceiling at all when there's spare CPU sitting right here). A dedicated non-root user runs it, since the runner's config script refuses to run as root by default, a workflow job inherits whatever privileges the runner has, so root would mean any job can touch anything on the box.

useradd -m -s /bin/bash gharunner
su - gharunner

Everything from here runs as gharunner, not root:

mkdir ~/actions-runner && cd ~/actions-runner
curl -o actions-runner-linux-x64-<version>.tar.gz -L <url-from-github>
tar xzf ./actions-runner-linux-x64-<version>.tar.gz
./config.sh --url https://github.com/<GH_USERNAME>/<GH_REPO> --token <token>

The registration token comes from the repo, Settings → Actions → Runners → New self hosted runner, and expires within about an hour.

svc.sh only exists after config.sh finishes successfully, it isn't part of the tarball. Once it's there:

exit
# back as root, service install needs to write to systemd
cd /home/gharunner/actions-runner
./svc.sh install gharunner
./svc.sh start

usermod -aG docker gharunner
cd /home/gharunner/actions-runner
./svc.sh stop
./svc.sh start

That last restart matters, group membership only applies to processes started after the change, not the one already running, so docker build will fail with a permission error until the runner service itself is restarted.

CI: Build and Push

.github/workflows/build.yml in the app repo:

name: Build and Push
on:
  push:
    branches: [main]

jobs:
  build:
    runs-on: self-hosted
    permissions:
      contents: read
      packages: write
    steps:
      - uses: actions/checkout@v4
      - uses: docker/login-action@v3
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}
      - uses: docker/build-push-action@v6
        with:
          context: .
          push: true
          tags: ghcr.io/<GH_USERNAME>/<GH_REPO>:${{ github.run_number }}-${{ github.sha }}

Tags use run_number-sha, not latest. image.yaml needs an ordering it can actually compare, a mutable latest tag gives it nothing to watch for.

Since the package is private, the cluster needs its own credentials to pull it back down, one copy per namespace that needs it:

kubectl create namespace <APP_NAME>

kubectl create secret docker-registry ghcr-auth \
  --docker-server=ghcr.io \
  --docker-username=<GH_USERNAME> \
  --docker-password=<token-with-read:packages> \
  --namespace=flux-system

kubectl create secret docker-registry ghcr-auth \
  --docker-server=ghcr.io \
  --docker-username=<GH_USERNAME> \
  --docker-password=<token-with-read:packages> \
  --namespace=<APP_NAME>

image.yaml: Watching for New Images

clusters/proxmox/<APP_NAME>/image.yaml

This file doesn't run anything and doesn't know what a port or a domain is. Its only job is watching the registry and, when something new shows up, writing the new tag into app.yaml and pushing that change to git. Three resources, each one link in that chain:

apiVersion: image.toolkit.fluxcd.io/v1
kind: ImageRepository
metadata:
  name: <APP_NAME>
  namespace: flux-system
spec:
  image: ghcr.io/<GH_USERNAME>/<GH_REPO>
  interval: 1m
  secretRef:
    name: ghcr-auth
---
apiVersion: image.toolkit.fluxcd.io/v1
kind: ImagePolicy
metadata:
  name: <APP_NAME>
  namespace: flux-system
spec:
  imageRepositoryRef:
    name: <APP_NAME>
  policy:
    numerical:
      order: asc
  filterTags:
    pattern: '^(?P<num>[0-9]+)-.*'
    extract: '$num'
---
apiVersion: image.toolkit.fluxcd.io/v1
kind: ImageUpdateAutomation
metadata:
  name: <APP_NAME>
  namespace: flux-system
spec:
  interval: 1m
  sourceRef:
    kind: GitRepository
    name: flux-system
  git:
    checkout:
      ref:
        branch: main
    commit:
      author:
        email: fluxcdbot@users.noreply.github.com
        name: fluxcdbot
      messageTemplate: |
        chore: automated image update
        {{ range .Changed.Changes -}}
        - {{ .OldValue }} -> {{ .NewValue }}
        {{ end -}}
    push:
      branch: main
  update:
    path: ./clusters/proxmox/<APP_NAME>
    strategy: Setters

ImageRepository polls ghcr.io/<GH_USERNAME>/<GH_REPO> every interval and keeps a list of every tag it finds there. It doesn't decide anything, it just watches.

ImagePolicy looks at that list and picks the one tag that counts as "latest," based on a rule. Here that rule is numerical ascending order, matched against the leading number in tags shaped like 12-a1b2c3d, the GitHub Actions run number, which always increases. Without this, Flux has no way to tell a new build from an old one, tags alone don't carry an order.

ImageUpdateAutomation takes whatever ImagePolicy picked, finds the marker comment inside app.yaml, rewrites the tag on that exact line, commits, and pushes. This is the only piece of the three that actually writes to git.

app.yaml: Running the App and Exposing It

clusters/proxmox/<APP_NAME>/app.yaml

The opposite half of the pair. This file doesn't watch anything, it describes what should currently be running and how it's reached, both from inside the cluster and from the internet. image.yaml's only interaction with it is rewriting one line.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: <APP_NAME>
  namespace: <APP_NAME>
spec:
  replicas: 1
  selector:
    matchLabels:
      app: <APP_NAME>
  template:
    metadata:
      labels:
        app: <APP_NAME>
    spec:
      imagePullSecrets:
        - name: ghcr-auth
      containers:
        - name: <APP_NAME>
          image: ghcr.io/<GH_USERNAME>/<GH_REPO>:1-init # {"$imagepolicy": "flux-system:<APP_NAME>"}
          ports:
            - containerPort: 3000
---
apiVersion: v1
kind: Service
metadata:
  name: <APP_NAME>
  namespace: <APP_NAME>
spec:
  selector:
    app: <APP_NAME>
  ports:
    - port: 80
      targetPort: 3000
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: <APP_NAME>
  namespace: <APP_NAME>
spec:
  ingressClassName: traefik
  rules:
    - host: <DOMAIN>
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: <APP_NAME>
                port:
                  number: 80

Deployment is what actually keeps a pod running: which image, how many replicas, what port the container listens on. If the pod crashes, this is what restarts it. The comment on the image: line is the marker ImageUpdateAutomation searches for, that exact comment has to survive, everything else on the line can change.

Service exists because pods don't have a stable address, they get a new internal IP every time they restart. Service gives them one fixed name that always points at whichever pod currently matches the label selector. Nothing outside the cluster can reach a Service directly though.

Ingress is the entry point from outside the cluster, a routing rule saying requests for <DOMAIN> get forwarded to the <APP_NAME> Service on port 80. Without it, the app is only reachable from inside the cluster, no external traffic, including the Cloudflare Tunnel, has anywhere to land. ingressClassName: traefik tells k3s to route this through its own bundled Traefik rather than some other controller.

No tls block on the Ingress. Cloudflare Tunnel terminates HTTPS at the edge, Traefik only ever sees plain HTTP arriving from the tunnel:

Visitor --HTTPS--> Cloudflare edge --tunnel--> cloudflared --HTTP--> Traefik --> Service

The existing tunnel config already points at localhost:80, which k3s's Traefik now owns, so nothing there needs to change:

# ~/.cloudflared/config.yml
ingress:
  - hostname: <DOMAIN>
    service: http://localhost:80
  - service: http_status:404

cd homelab-k8s
mkdir -p clusters/proxmox/<APP_NAME>
# save both files into that folder, then:
git add -A
git commit -m "add <APP_NAME> app and image automation"
git push

Known Issues

Quick reference for every real error hit building this, so future-me doesn't have to re-diagnose the same wall twice.

Now What If I Want to Deploy a New App?

Splitting this by what's genuinely one-time versus what has to happen again for every new app, since most of the pain the first time around was infrastructure that never needs touching again.

Never repeats, already done:

• k3s itself
• Flux plus the image automation controllers, bootstrapped once with --read-write-key
• cloudflared running as a systemd service
• the homelab-k8s repo as the GitOps source

Repeats per app:

1. A new self hosted runner, since runners are currently registered per-repo, not per-organization:

su - gharunner
mkdir ~/actions-runner-<APP_NAME> && cd ~/actions-runner-<APP_NAME>
# same tarball, just extract it again into this new folder
./config.sh --url https://github.com/<GH_USERNAME>/<GH_REPO> --token <token>
exit
cd /home/gharunner/actions-runner-<APP_NAME>
./svc.sh install gharunner
./svc.sh start

If registering a new runner for every app starts feeling tedious, that's the point to revisit organization-level runners instead of repository-level ones, register once, every repo in the org shares it.

2. The workflow file, copied from an existing app's .github/workflows/build.yml, <GH_REPO> swapped to match.

3. A ghcr-auth secret in the new namespace only. The one in flux-system is shared, it's scoped to the GitHub account, not any single app, so it doesn't get recreated:

kubectl create namespace <APP_NAME>
kubectl create secret docker-registry ghcr-auth \
  --docker-server=ghcr.io \
  --docker-username=<GH_USERNAME> \
  --docker-password=<token> \
  --namespace=<APP_NAME>

4. A new manifest folder, copied from an existing app and placeholders swapped:

clusters/proxmox/<APP_NAME>/
├── app.yaml
└── image.yaml

This is the easiest place to reintroduce the portofolio vs portfolio typo from earlier, go slow through the placeholder swap.

5. A new Ingress route on the tunnel, only if the app needs its own subdomain. Remember the <DOMAIN> value here follows the same rule as everywhere else: yourdomain.com if the app sits on the root domain, subdomain.yourdomain.com if it's getting its own subdomain instead:

nano ~/.cloudflared/config.yml

ingress:
  - hostname: yourdomain.com
    service: http://localhost:80
  - hostname: subdomain.yourdomain.com
    service: http://localhost:80
  - service: http_status:404

cloudflared tunnel route dns your-cloudfare-tunnel-name subdomain.yourdomain.com
systemctl restart cloudflared

6. Push:

cd homelab-k8s
git add -A
git commit -m "add <APP_NAME>"
git push

Of the six steps, only the first two (runner and workflow) involve anything outside homelab-k8s. Everything else is a folder copy and a placeholder swap.

Final Checklist

• kubectl get nodes shows the node Ready
• kubectl get pods -n flux-system shows all controllers Running, including the two image automation ones
• flux get image update <APP_NAME> reports Ready: True
• A git push to the app repo results in a new commit from fluxcdbot in homelab-k8s within a couple minutes
• kubectl get pods -n <APP_NAME> shows a pod running the freshly built tag
• https://<DOMAIN> serves the app

Closing Thoughts

Every real problem here traced back to one of two things: an API that moved on since whatever guide I first read, or a permission that defaults to the safer, more restrictive option. Neither is a flaw in k3s or Flux exactly, it's what happens working against a fast moving ecosystem. Worth reading the actual error message twice before assuming the setup itself is wrong.