Part Five: A Rust Backend, a Database, and Two Very Quiet Bugs

Part five. The pipeline from the last post works, it builds, it ships, it deploys. What it hadn't done yet was run anything with a database behind it, or anything that needed a build-time-vs-runtime distinction in its environment variables. Both turned out to have failure modes that produce zero error output, which is a special kind of annoying to debug. This is that backend, wired all the way through.

1. The Plan
2. Variables Used in This Guide
3. Setting Up Axum + PostgreSQL
4. Talking to Postgres: sqlx and Offline Mode
5. Environment Variables: Server-Side vs Client-Side
6. Dockerizing the Backend
7. Postgres Inside the Cluster
8. app.yaml, Extended: Env Vars and a Database
9. Connecting the Service to Cloudflare
10. Secrets: Creating, Updating, and the Idempotency Trap
11. Useful kubectl Commands
12. Known Issues
13. Final Checklist
14. Closing Thoughts

---

The Plan

git push
  -> same pipeline as before: runner builds an image, pushes it, Flux updates the tag
  -> pod restarts, this time it also needs to reach a database
  -> Postgres lives in the same cluster, its own Deployment + PVC + Service
  -> app env vars come from a Secret, injected at pod start
  -> anything the frontend needs baked into its JS bundle does NOT come from that Secret

That last line is the part that cost the most time. A Kubernetes Secret is a runtime thing, injected when the container starts. Some frontend frameworks bake certain env vars into the JS bundle at build time instead, before the container running that bundle even exists. Same-looking envFrom block, two completely different points in time. More on this in Environment Variables.

Variables Used in This Guide

Same placeholders as the last post, plus one new one for the database:

Setting Up Axum + PostgreSQL

Cargo.toml:

[dependencies]
axum = "0.8"
tokio = { version = "1", features = ["full"] }
serde = { version = "1", features = ["derive"] }
dotenvy = "0.15"
sqlx = { version = "0.9", features = ["postgres", "runtime-tokio", "macros"] }
tower-http = { version = "0.7", features = ["cors"] }

The connection pool lives in a shared AppState, cloned into every handler through axum's State extractor. A PgPool is already internally reference-counted, so cloning it is cheap, it's not duplicating the actual connections:

use sqlx::{PgPool, postgres::PgPoolOptions};

#[derive(Clone)]
struct AppState {
    db: PgPool,
}

#[tokio::main]
async fn main() {
    dotenvy::dotenv().ok();

    let db_url = std::env::var("DATABASE_URL").expect("DATABASE_URL must be set");
    let port = std::env::var("PORT").expect("PORT must be set");

    let pool = PgPoolOptions::new()
        .max_connections(5)
        .connect(&db_url)
        .await
        .expect("failed to connect to database");

    let state = AppState { db: pool };

    let app = Router::new()
        // .route(...) etc
        .with_state(state);

    let listener = tokio::net::TcpListener::bind(&format!("0.0.0.0:{}", port))
        .await
        .unwrap();
    axum::serve(listener, app).await.unwrap();
}

That 0.0.0.0 is not decoration, it's load-bearing. See Known Issues, it's responsible for one of the two silent bugs this post is named after.

Connection string, per environment

The same DATABASE_URL format, postgres://<user>:<password>@<host>:<port>/<db>, but the <host> changes depending on what's connecting to what:

localhost inside a container always means the container itself, never the host or another container. Forgetting this once is enough to never forget it again.

Talking to Postgres: sqlx and Offline Mode

sqlx::query! and sqlx::query_as! are compile-time checked, they connect to a real database while compiling to verify column names and types. That's great locally, where Postgres is one docker compose up away, and useless inside a CI build container, where no database exists at all:

error: set `DATABASE_URL` to use query macros offline, or run `cargo sqlx prepare`

The fix is offline mode: generate a cache of the verified queries once, locally, commit that cache, and let the build read from it instead of a live database.

cargo install sqlx-cli --no-default-features --features postgres,rustls
# with a local Postgres running and DATABASE_URL set:
cargo sqlx prepare

This writes a .sqlx/ folder to the project root. It has to be committed to git, it's not a .gitignore candidate, the CI build depends on it being there. Every time a query changes or a new one is added, cargo sqlx prepare needs to run again and the updated .sqlx/ needs to be committed, or the next CI build fails with the exact same error.

Environment Variables: Server-Side vs Client-Side

Every env var so far has been server-side: read with std::env::var(...) at runtime, whenever the process actually needs it. A Secret injected through envFrom works perfectly for this, because the value only needs to exist by the time the process reads it, which is well after the container has started.

Frontend frameworks with a "public" env var convention (Next.js's NEXT_PUBLIC_*, Vite's VITE_*, Create React App's REACT_APP_*) work differently. Those values are meant to end up in the browser, so the framework can't wait until a request comes in to look them up, it bakes the literal value into the built JS bundle during the build step. By the time a container starts and a Secret gets injected, that bundle is already frozen. The env var technically exists in the container's environment, and is technically never read by anything, because the code that would have read it doesn't exist anymore, it was replaced with a literal string before the container image was even built.

The fix is passing the value as a Docker build argument, so it's present during docker build, not after:

Dockerfile:

ARG NEXT_PUBLIC_SOME_VALUE
ENV NEXT_PUBLIC_SOME_VALUE=$NEXT_PUBLIC_SOME_VALUE

RUN npm run build

.github/workflows/build.yml, the build step:

- uses: docker/build-push-action@v6
  with:
    context: .
    push: true
    build-args: |
      NEXT_PUBLIC_SOME_VALUE=${{ vars.NEXT_PUBLIC_SOME_VALUE }}
    tags: ghcr.io/<GH_USERNAME>/<GH_REPO>:${{ github.run_number }}-${{ github.sha }}

Multiple build args are just multiple lines under build-args: |, one KEY=VALUE per line, both there and in the matching ARG/ENV pairs in the Dockerfile. vars. rather than secrets. is intentional when the value isn't actually sensitive, a public client ID meant to ship to every visitor's browser doesn't need to be treated like a credential.

The envFrom: secretRef pattern from app.yaml still applies normally to whatever server-side env vars that same frontend needs, an API route reading a value that never reaches the browser, for instance. The split is by where the value is read, not by which app it belongs to.

Dockerizing the Backend

FROM rust:1.94-slim AS builder

WORKDIR /app

RUN apt-get update && apt-get install -y pkg-config libssl-dev && rm -rf /var/lib/apt/lists/*

COPY Cargo.toml Cargo.lock ./
COPY .sqlx ./.sqlx

ENV SQLX_OFFLINE=true

RUN mkdir src && echo "fn main() {}" > src/main.rs && \
    cargo build --release && \
    rm -rf src

COPY src ./src

RUN touch src/main.rs && cargo build --release

FROM debian:bookworm-slim AS runtime

RUN apt-get update && apt-get install -y libssl3 ca-certificates && rm -rf /var/lib/apt/lists/*

WORKDIR /app

COPY --from=builder /app/target/release/<APP_NAME> ./<APP_NAME>

EXPOSE 3000

CMD ["./<APP_NAME>"]

Two things worth calling out:

ENV SQLX_OFFLINE=true plus COPY .sqlx ./.sqlx is what makes the query macros work without a live database, covered above.

The dummy src/main.rs build before copying the real source is a caching trick: dependencies get compiled once and cached as a Docker layer, and only get rebuilt when Cargo.toml/Cargo.lock actually change, not on every source edit. touch src/main.rs after copying the real source forces cargo to see it as changed, since the dummy build already produced an binary with the same timestamp logic cargo relies on.

Postgres Inside the Cluster

Running the database as another Deployment in the same namespace, rather than reaching out to something external:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: db
  namespace: <APP_NAME>
spec:
  replicas: 1
  selector:
    matchLabels:
      app: db
  template:
    metadata:
      labels:
        app: db
    spec:
      containers:
        - name: db
          image: postgres:17-alpine
          ports:
            - containerPort: 5432
          env:
            - name: POSTGRES_USER
              value: postgres
            - name: POSTGRES_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: db-secret
                  key: password
            - name: POSTGRES_DB
              value: <DB_NAME>
          volumeMounts:
            - name: pgdata
              mountPath: /var/lib/postgresql/data
      volumes:
        - name: pgdata
          persistentVolumeClaim:
            claimName: db-pvc
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: db-pvc
  namespace: <APP_NAME>
spec:
  accessModes: ["ReadWriteOnce"]
  resources:
    requests:
      storage: 1Gi
---
apiVersion: v1
kind: Service
metadata:
  name: db
  namespace: <APP_NAME>
spec:
  selector:
    app: db
  ports:
    - port: 5432
      targetPort: 5432

The Service name (db) is what becomes the connection host, not the Deployment name or the label. Those two can differ without consequence, only metadata.name on the Service is what gets registered in cluster DNS.

The PersistentVolumeClaim is what survives a pod restart, k3s's bundled local-path provisioner backs it with a directory on the node's own disk. On a single-node cluster this is fine, restarting the pod re-mounts the same volume and the data is untouched. It only becomes a real constraint with multiple nodes, where a pod could get scheduled somewhere that isn't the node holding the data.

None of this creates the actual schema. A fresh PersistentVolumeClaim is an empty database, tables need to be created manually the first time (or via a proper migration tool, not covered here):

kubectl exec -it <db-pod-name> -n <APP_NAME> -- psql -U postgres -d <DB_NAME> -c "
CREATE TABLE example (
    id TEXT PRIMARY KEY
);
"

app.yaml, Extended: Env Vars and a Database

The app.yaml from the last post, with the addition that actually matters here, envFrom:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: <APP_NAME>
  namespace: <APP_NAME>
spec:
  replicas: 1
  selector:
    matchLabels:
      app: <APP_NAME>
  template:
    metadata:
      labels:
        app: <APP_NAME>
    spec:
      imagePullSecrets:
        - name: ghcr-auth
      containers:
        - name: <APP_NAME>
          image: ghcr.io/<GH_USERNAME>/<GH_REPO>:1-init # {"$imagepolicy": "flux-system:<APP_NAME>"}
          ports:
            - containerPort: 3000
          envFrom:
            - secretRef:
                name: <APP_NAME>-env
---
apiVersion: v1
kind: Service
metadata:
  name: <APP_NAME>
  namespace: <APP_NAME>
spec:
  selector:
    app: <APP_NAME>
  ports:
    - port: 80
      targetPort: 3000
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: <APP_NAME>
  namespace: <APP_NAME>
spec:
  ingressClassName: traefik
  rules:
    - host: <DOMAIN>
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: <APP_NAME>
                port:
                  number: 80

containerPort and Service.targetPort both need to match whatever port the app actually binds to at runtime, not whatever port an earlier example happened to use. And the app needs to bind 0.0.0.0, not 127.0.0.1 or localhost. Binding to loopback means the process only accepts connections from inside its own container, which is exactly what kube-proxy (routing traffic from the Service) and Traefik (routing from the Ingress) are not. The symptom is Connection refused at the Service level and a 502 at the Ingress level, with nothing at all in the app's own logs, since the connection never got far enough to reach application code. kubectl port-forward will misleadingly keep working through this, it tunnels straight into the pod's network namespace rather than going through kube-proxy, so it doesn't exercise the same path real traffic takes.

Connecting the Service to Cloudflare

The Ingress from the previous section makes the app reachable from inside the cluster's own Traefik, on whatever <DOMAIN> was set. Nothing outside the server knows that domain exists yet, that's the tunnel's job:

Visitor --HTTPS--> Cloudflare edge --tunnel--> cloudflared --HTTP--> Traefik --> Service

If <DOMAIN> reuses a hostname the tunnel already routes (the root domain from an earlier app, say), nothing further is needed, every app behind the same tunnel shares localhost:80, and Traefik is what decides which Service a given hostname actually reaches, based on each app's own Ingress. A brand new subdomain, on the other hand, has to be added to the tunnel explicitly, cloudflared doesn't infer it from a k8s manifest it has no visibility into.

nano ~/.cloudflared/config.yml

ingress:
  - hostname: yourdomain.com
    service: http://localhost:80
  - hostname: <DOMAIN>
    service: http://localhost:80
  - service: http_status:404

Every entry maps to the same localhost:80, since every app shares one tunnel and one Traefik in front of it. The catch-all http_status:404 has to stay last, anything placed below it never gets evaluated.

cloudflared tunnel list
cloudflared tunnel route dns <tunnel-name> <DOMAIN>
systemctl restart cloudflared

tunnel route dns is the step that actually creates the DNS record, editing config.yml alone only changes how the tunnel routes traffic it already receives, it doesn't make Cloudflare aware the hostname exists in the first place.

Two things worth knowing before assuming this didn't work:

dig <DOMAIN> won't show a cfargotunnel.com record, not even once the route is live, if the record is proxied (orange cloud) through Cloudflare. Proxied records resolve to Cloudflare's own edge IPs from the outside, the tunnel mapping underneath is intentionally hidden. curl -i https://<DOMAIN> is the reliable check, not dig.

cloudflared running as a systemd service doesn't necessarily read ~/.cloudflared/config.yml. Editing that file has no effect if the service was set up to read from somewhere else, commonly /etc/cloudflared/config.yml instead:

systemctl cat cloudflared

The --config flag on the ExecStart= line is the actual path in use. Edit that file, not the one assumed by habit, then restart.

Secrets: Creating, Updating, and the Idempotency Trap

kubectl create secret generic <APP_NAME>-env \
  --namespace=<APP_NAME> \
  --from-literal=DATABASE_URL=postgres://postgres:<password>@db:5432/<DB_NAME> \
  --from-literal=PORT=3000

kubectl create secret is not idempotent. Running it again against a Secret that already exists fails with AlreadyExists, it doesn't update it. Two ways around that:

# delete, then recreate
kubectl delete secret <APP_NAME>-env --namespace=<APP_NAME>
kubectl create secret generic <APP_NAME>-env --namespace=<APP_NAME> \
  --from-literal=DATABASE_URL=... \
  --from-literal=PORT=...

# or, actually idempotent: generate the manifest, apply it
kubectl create secret generic <APP_NAME>-env --namespace=<APP_NAME> \
  --from-literal=DATABASE_URL=... \
  --from-literal=PORT=... \
  --dry-run=client -o yaml | kubectl apply -f -

Either way, a running pod does not pick up a changed Secret on its own, env vars are injected once at container start, not watched afterward:

kubectl rollout restart deployment/<APP_NAME> -n <APP_NAME>

Useful kubectl Commands

Inspecting state:

kubectl get pods -n <APP_NAME>                    # status, restart count
kubectl get pods -n <APP_NAME> -o wide             # + pod IP, node
kubectl get pods -n <APP_NAME> --show-labels       # labels, for matching against a Service selector
kubectl describe pod <pod-name> -n <APP_NAME>      # events: why a pod is stuck/failing
kubectl get endpoints <service-name> -n <APP_NAME> # which pod IPs a Service actually resolves to
kubectl get ingress -n <APP_NAME>
kubectl describe ingress <name> -n <APP_NAME>      # confirm host + backend Service/port match

Logs:

kubectl logs <pod-name> -n <APP_NAME>              # current container
kubectl logs -f <pod-name> -n <APP_NAME>           # follow, live
kubectl logs <pod-name> -n <APP_NAME> --previous   # the instance before the last restart
kubectl logs -f -l app=<APP_NAME> -n <APP_NAME>    # by label, don't need the exact pod name

Reading a Secret back:

kubectl get secret <name> -n <APP_NAME> -o jsonpath='{.data.<KEY>}' | base64 -d

Checking what actually landed inside a running container:

kubectl exec -it <pod-name> -n <APP_NAME> -- env | grep <KEY>

Isolating a networking problem, testing the same DNS+port path that Traefik uses (unlike port-forward, which bypasses it):

kubectl run debug --rm -it --restart=Never --image=busybox -n <APP_NAME> -- wget -qO- http://<service-name>:80/health

Forcing things instead of waiting:

kubectl rollout restart deployment/<name> -n <APP_NAME>
kubectl delete pod -l app=<label> -n <APP_NAME>    # safe if managed by a Deployment, it gets recreated
flux reconcile source git flux-system              # don't wait for the next sync interval
flux reconcile kustomization flux-system

Known Issues

Final Checklist

• .sqlx/ is committed, not gitignored
• Dockerfile sets ENV SQLX_OFFLINE=true before cargo build
• The app binds 0.0.0.0, not 127.0.0.1 or localhost
• DATABASE_URL's host matches the Postgres Service name, not localhost
• containerPort / targetPort in app.yaml match the port the app actually listens on
• <APP_NAME>-env Secret exists and kubectl exec ... -- env confirms the values actually landed in the container
• Any client-side "public" env var is passed as a build arg, not just present in the runtime Secret
• The database schema exists (checked manually, it isn't created automatically)
• kubectl get endpoints <service> -n <APP_NAME> shows a real pod IP, not empty
• https://<DOMAIN>/health (or equivalent) returns a real response, not a 502

Closing Thoughts

Both real bugs in this post had the same shape: nothing crashed, nothing logged an error, the failure was just silence at the exact layer that would've explained it. 127.0.0.1 doesn't throw, it just quietly refuses connections from anywhere that isn't itself. A build-time env var doesn't throw either, it just isn't there anymore by the time anything could have complained about it. Neither shows up in kubectl logs, because by the definition of the bug, the code never got far enough to log anything. Worth remembering next time something "just doesn't work" with zero error output: that's often not a sign there's nothing to find, it's a sign the problem is one layer earlier than wherever's currently being checked.