Kubernetes Deployment

This guide covers deploying ubTrace on Kubernetes using the official Helm chart. The chart supports vanilla Kubernetes, AWS EKS, and OpenShift 4+.

For Docker Compose deployments (including air-gapped / offline), see Installation.

Prerequisites

• Kubernetes 1.25+ cluster (1.28+ recommended)

• Helm 3.12+

• kubectl configured for your cluster. Install via your distro’s package manager (brew install kubectl, sudo apt install kubectl, sudo pacman -S kubectl, …) or follow the upstream guide at kubernetes.io/docs/tasks/tools.

• A StorageClass that supports ReadWriteOnce PVCs

• (Optional) An Ingress controller (NGINX, ALB, etc.)

Note

The chart bundles PostgreSQL, Elasticsearch, Redis, and Keycloak as sub-deployments. For production, consider using managed services (RDS, OpenSearch, ElastiCache, etc.) and pointing the chart at them via the external.* values.

Quick Start

# Clone the repository (or obtain the chart archive)
git clone https://github.com/useblocks/ubtrace.git
cd ubtrace

# Install with default values
helm install ubtrace ./deploy/helm/ubtrace -n ubtrace --create-namespace

# Watch pods come up
kubectl get pods -n ubtrace -w

The default values.yaml starts all infrastructure locally in the cluster with sensible defaults. All services should be ready within 2-5 minutes, depending on image pull times.

Platform-Specific Overlays

The chart ships with overlay files for common platforms. Use the -f flag to layer them on top of the defaults.

Minikube (Local Development)

minikube start --cpus=4 --memory=8192
minikube addons enable ingress

helm install ubtrace ./deploy/helm/ubtrace \
  -f deploy/helm/ubtrace/values-minikube.yaml \
  -n ubtrace --create-namespace

# Resolve ingress hostnames to the minikube node.
echo "$(minikube ip)  ubtrace.local ubtrace-api.local admin.ubtrace.local keycloak.local" \
  | sudo tee -a /etc/hosts

The minikube overlay uses reduced resource requests and the standard StorageClass. Ingress is enabled with the NGINX controller, and all OIDC / frontend URLs are pre-wired to the *.local ingress hosts — no --set overrides are needed for the happy path.

Note

--cpus and --memory only apply when creating a new profile. An existing profile with smaller resources will keep its old size silently — Keycloak or Elasticsearch will then fail to schedule with no error from minikube start itself. Either run minikube delete first, or use a dedicated profile:

minikube start -p ubtrace --cpus=4 --memory=8192

Note

Keycloak 26 runs a one-time Quarkus augmentation on first start (Updating the server image.) that briefly exceeds 1 GiB of RAM. The overlay sets the Keycloak memory limit to 3 GiB for that reason — lowering it causes OOMKilled loops during the first install.

Minikube (NodePort, no /etc/hosts)

Alternative overlay for environments where editing /etc/hosts requires admin rights you do not have (managed laptop, CI runner). Each service is reachable at http://$(minikube ip):<port> directly — no Ingress controller, no DNS entries, no host-file edits.

minikube start --cpus=4 --memory=8192

export minikube_ip=$(minikube ip)
helm install ubtrace ./deploy/helm/ubtrace \
  -n ubtrace --create-namespace \
  -f deploy/helm/ubtrace/values-minikube-nodeport.yaml \
  --set-string api.publicUrl="http://${minikube_ip}:30150" \
  --set-string api.config.frontendUbtUrl="http://${minikube_ip}:30155" \
  --set-string api.config.adminUbtUrl="http://${minikube_ip}:30156" \
  --set-string oidc.issuer="http://${minikube_ip}:30181/realms/ubtrace" \
  --set-string keycloak.hostname="http://${minikube_ip}:30181" \
  --set-string frontend.config.apiUrl="http://${minikube_ip}:30150/api" \
  --set-string frontend.config.apiLoginUrl="http://${minikube_ip}:30150/api" \
  --set-string frontend.config.oidcIssuer="http://${minikube_ip}:30181/realms/ubtrace" \
  --set-string frontend.config.authRedirectUrl="http://${minikube_ip}:30155/auth/callback" \
  --set-string frontend.config.adminUrl="http://${minikube_ip}:30156" \
  --set-string admin.config.apiUrl="http://${minikube_ip}:30150/api" \
  --set-string admin.config.apiLoginUrl="http://${minikube_ip}:30150/api" \
  --set-string admin.config.oidcIssuer="http://${minikube_ip}:30181/realms/ubtrace" \
  --set-string admin.config.authRedirectUrl="http://${minikube_ip}:30156/auth/callback" \
  --set-string admin.config.mainAppUrl="http://${minikube_ip}:30155"

Service	URL
Frontend	http://$(minikube ip):30155
Admin	http://$(minikube ip):30156
API	http://$(minikube ip):30150
Keycloak	http://$(minikube ip):30181

The minikube addons enable ingress step is not required for this overlay. Tradeoffs vs the Ingress path:

• Four distinct URLs instead of four *.local hostnames behind a single ingress.

• Every public URL still has to be passed via --set-string because $(minikube ip) is only known at install time. The Ingress overlay needs zero --set flags.

• NodePorts must be free on the node. If a colliding release already uses 30150/30155/30156/30181, either uninstall it or override the relevant <svc>.service.nodePort values.

Note

Both minikube overlays bake COOKIE_SECURE=false into api.config because neither path terminates TLS. Browsers drop cookies that carry the Secure flag when the response arrives over plain HTTP, which would make the OIDC login appear to succeed while every subsequent request returns 401. Production overlays (EKS, on-prem behind a TLS-terminating ingress) keep the default COOKIE_SECURE=true — do not copy this override into a deployment that actually serves HTTPS.

Note

macOS / Apple Silicon (Docker driver): Both minikube snippets above are written for Linux and hit three gotchas on Mac. Apply all three or neither overlay will be reachable from your browser.

1. Docker Desktop memory. minikube start --memory=8192 fails unless Docker Desktop itself has ≥10 GB allocated. Bump it in Docker Desktop → Settings → Resources → Memory and restart Docker Desktop.

2. Elasticsearch seccomp fix. The bundled amd64 ES image fails under Rosetta with seccomp unavailable: CONFIG_SECCOMP not compiled into kernel. Overwrite the loaded image with the native arm64 variant immediately after offline-load.sh:

   eval $(minikube docker-env)
   docker pull docker.elastic.co/elasticsearch/elasticsearch:9.3.0
   kubectl delete pod -n ubtrace ubtrace-elasticsearch-0
   eval $(minikube docker-env -u)
   

   Requires network access for the one-time pull; persists until minikube delete.

3. ``$(minikube ip)`` is not reachable from the host browser. The Docker driver runs the minikube node inside Docker Desktop’s VM; 192.168.49.2 routes only from inside that VM, so both overlays’ advertised URLs hang on the host. Fixes per overlay:

   • Ingress overlay: use 127.0.0.1 (NOT $(minikube ip)) in /etc/hosts, then kubectl port-forward the ingress-nginx controller to localhost:80 in a separate terminal. minikube tunnel is the path the Linux-focused minikube docs push, but on Mac + Docker driver it often fails to claim an external IP (the ingress-nginx addon ships as NodePort, and even after patching it to LoadBalancer the tunnel is fragile). The port-forward approach is one reliable command:

     echo "127.0.0.1  ubtrace.local ubtrace-api.local admin.ubtrace.local keycloak.local" | \
       sudo tee -a /etc/hosts
     # Separate terminal, leave running (prompts for sudo to bind :80):
     sudo kubectl port-forward -n ingress-nginx svc/ingress-nginx-controller 80:80
     

   • NodePort overlay: retarget every URL in the --set-string block above from ${minikube_ip} to 127.0.0.1, then run one kubectl port-forward per service in the background:

     kubectl port-forward -n ubtrace svc/ubtrace-frontend 30155:3000 &
     kubectl port-forward -n ubtrace svc/ubtrace-admin    30156:3000 &
     kubectl port-forward -n ubtrace svc/ubtrace-api      30150:3000 &
     kubectl port-forward -n ubtrace svc/ubtrace-keycloak 30181:8080 &
     

   The Ingress path is easier to maintain (one tunnel, no dangling background jobs); use it unless you truly cannot edit /etc/hosts.

AWS EKS

helm install ubtrace ./deploy/helm/ubtrace \
  -f deploy/helm/ubtrace/values-eks.yaml \
  --set api.ingress.annotations."alb\.ingress\.kubernetes\.io/certificate-arn"=arn:aws:acm:REGION:ACCOUNT:certificate/CERT-ID \
  --set api.ingress.hosts[0].host=ubtrace-api.example.com \
  --set frontend.ingress.hosts[0].host=ubtrace.example.com \
  --set keycloak.ingress.hosts[0].host=keycloak.example.com \
  -n ubtrace --create-namespace

The EKS overlay pre-configures:

• StorageClass: gp3 (EBS gp3 volumes)

• Ingress: AWS ALB Ingress Controller with HTTPS termination

• Annotations: ALB scheme, target type, health check paths

Before deploying, ensure:

1. The AWS Load Balancer Controller is installed

2. An ACM certificate exists for your domain(s)

3. DNS records point to the ALB (created automatically by the controller)

Tip

For production EKS deployments, consider using managed services:

# In your custom values file
postgresql:
  enabled: false
  external:
    host: "mydb.cluster-xxxxx.us-east-1.rds.amazonaws.com"
    port: 5432
    database: "ubtrace"

elasticsearch:
  enabled: false
  external:
    url: "https://search-ubtrace-xxxxx.us-east-1.es.amazonaws.com"

redis:
  enabled: false
  external:
    host: "ubtrace.xxxxx.ng.0001.use1.cache.amazonaws.com"
    port: 6379

OpenShift 4+

helm install ubtrace ./deploy/helm/ubtrace \
  -f deploy/helm/ubtrace/values-openshift.yaml \
  -n ubtrace --create-namespace

The OpenShift overlay:

• Sets securityContext.runAsUser: null so OpenShift assigns UIDs from the namespace range (required by SCCs)

• Disables Ingress objects (OpenShift uses Routes)

All ubTrace images are built with GID 0 (root group) permissions on their data directories (chmod g=u), so they work without additional SCCs under the default OpenShift restricted policy. The entrypoints automatically detect OpenShift mode (non-root UID + GID 0) and skip operations that require root.

After deployment, expose services with OpenShift Routes:

oc expose svc/ubtrace-frontend -n ubtrace
oc expose svc/ubtrace-api -n ubtrace
oc expose svc/ubtrace-keycloak -n ubtrace

# For TLS-terminated Routes:
oc create route edge ubtrace-frontend \
  --service=ubtrace-frontend \
  --hostname=ubtrace.example.com \
  -n ubtrace

Configuration

All configuration is done via values.yaml overrides. The chart follows a consistent pattern for each component.

Image Configuration

global:
  imageRegistry: "ghcr.io/useblocks"    # Registry for ubTrace images
  imagePullSecrets: []                   # Pull secrets for private registries
  imagePullPolicy: "IfNotPresent"        # IfNotPresent, Always, or Never

# Per-component overrides (optional)
api:
  image:
    registry: ""          # Falls back to global.imageRegistry
    repository: ub-backend
    tag: ""               # Falls back to Chart.AppVersion

Infrastructure Toggles

Each infrastructure component can be disabled and replaced with an external service:

# Use external PostgreSQL (e.g., AWS RDS)
postgresql:
  enabled: false
  external:
    host: "mydb.example.com"
    port: 5432
    database: "ubtrace"

# Use external Elasticsearch (e.g., AWS OpenSearch)
elasticsearch:
  enabled: false
  external:
    url: "https://search.example.com"

# Use external Redis (e.g., AWS ElastiCache)
redis:
  enabled: false
  external:
    host: "redis.example.com"
    port: 6379

Passwords & Secrets

The chart generates a Kubernetes Secret with default passwords from values.yaml. For production, either:

Option A: Override values at install time

helm install ubtrace ./deploy/helm/ubtrace \
  --set postgresql.auth.password=<strong-password> \
  --set postgresqlKeycloak.auth.password=<strong-password> \
  --set keycloak.adminPassword=<strong-password> \
  --set redis.auth.password=<strong-password> \
  -n ubtrace --create-namespace

Option B: Use an existing Secret

# Create the secret manually
kubectl create secret generic ubtrace-secrets -n ubtrace \
  --from-literal=postgresql-password=<pw> \
  --from-literal=postgresql-keycloak-password=<pw> \
  --from-literal=keycloak-admin-password=<pw> \
  --from-literal=redis-password=<pw> \
  --from-literal=oidc-client-secret=<secret>

# Reference it during install
helm install ubtrace ./deploy/helm/ubtrace \
  --set api.existingSecret=ubtrace-secrets \
  -n ubtrace --create-namespace

Important

OIDC Client Secret: The bundled Keycloak realm ships with a pre-configured client secret (FnZuseq02pwaNKqsvDxL3jq4HhzPey2b). Do not change the oidc.clientSecret value unless you also regenerate the secret in Keycloak. See Keycloak / OIDC Hardening.

OIDC / Authentication

Configure the OIDC URLs to match your deployment’s public hostnames:

oidc:
  issuer: "https://keycloak.example.com/realms/ubtrace"
  clientId: "nestjs-app"
  clientSecret: "FnZuseq02pwaNKqsvDxL3jq4HhzPey2b"

keycloak:
  hostname: "https://keycloak.example.com"

api:
  publicUrl: "https://ubtrace-api.example.com"
  config:
    frontendUbtUrl: "https://ubtrace.example.com"

frontend:
  config:
    apiUrl: "https://ubtrace-api.example.com/api"
    oidcIssuer: "https://keycloak.example.com/realms/ubtrace"
    authRedirectUrl: "https://ubtrace.example.com/auth/callback"
    adminUrl: "https://admin.example.com"    # Required for "Go to Admin" button

Note

frontend.config.adminUrl maps to the UBTRACE_ADMIN_URL environment variable on the frontend pod. It controls the “Go to Admin” button in the user menu for users with the ubtrace-admin Keycloak realm role. If not set, the button will not appear regardless of the user’s roles. Set this to the public URL of the admin app when deploying it alongside the main frontend.

Each frontend.config.* Helm key maps to a UBTRACE_* environment variable (e.g. apiUrl becomes UBTRACE_API_URL). For non-Helm deployments, set these env vars directly on the frontend container.

Persistence

The chart creates PVCs for shared pipeline volumes. Customize sizes and StorageClass per volume:

persistence:
  output:
    size: 20Gi
    storageClass: ""       # Uses global.storageClass if empty
    accessMode: ReadWriteOnce
    existingClaim: ""      # Use a pre-existing PVC

  inputSrcBuild:
    size: 10Gi

  inputBuild:
    size: 10Gi

  inputSrc:
    size: 10Gi

  keycloakThemeJar:
    size: 128Mi

Resource Requests & Limits

Each component has configurable resource requests and limits:

Component	CPU Request	CPU Limit	Memory Request	Memory Limit
API	250m	1	512Mi	1Gi
Frontend	100m	500m	256Mi	512Mi
Worker	500m	2	1Gi	4Gi
Builder	250m	1	512Mi	2Gi
Keycloak	250m	1	512Mi	1Gi
PostgreSQL	250m	1	256Mi	1Gi
Elasticsearch	500m	2	1Gi	2Gi
Redis	100m	500m	128Mi	512Mi

Override in your values file:

api:
  resources:
    requests:
      cpu: 500m
      memory: 1Gi
    limits:
      cpu: "2"
      memory: 2Gi

Importing Artifacts

ubTrace uses a worker pipeline that automatically detects new or changed artifacts. The pipeline reads from Persistent Volume Claims (PVCs). Since the worker and builder mount these PVCs as read-only, you need a temporary helper pod to write data into them.

Note

This is a key difference from Docker Compose deployments, where you can copy files directly into host-mounted directories. On Kubernetes, PVC access requires a pod with the volume mounted.

Option A: Pre-Built CI Output (Recommended)

If your CI pipeline already builds with ubt_sphinx, copy the output into the input-build PVC.

Step 1: Start a temporary import pod

kubectl run ubtrace-import --image=busybox --restart=Never \
  --overrides='{
    "spec": {
      "containers": [{
        "name": "import",
        "image": "busybox",
        "command": ["sleep", "3600"],
        "volumeMounts": [{
          "name": "input-build",
          "mountPath": "/data/input_build"
        }]
      }],
      "volumes": [{
        "name": "input-build",
        "persistentVolumeClaim": {
          "claimName": "ubtrace-input-build"
        }
      }]
    }
  }' -n ubtrace

# Wait for the pod to be ready
kubectl wait --for=condition=Ready pod/ubtrace-import -n ubtrace --timeout=30s

Step 2: Create the directory structure and copy artifacts

The required path inside the PVC is {org}/{project}/{version}/. These values must match the fields in config/ubtrace_project.toml.

# Create directories
kubectl exec ubtrace-import -n ubtrace -- \
  mkdir -p /data/input_build/mycompany/my-project/v1

# Copy pre-built artifacts
kubectl cp ./ci-output/mycompany/my-project/v1 \
  ubtrace/ubtrace-import:/data/input_build/mycompany/my-project/v1 -n ubtrace

Step 3: Verify and clean up

# Verify the structure
kubectl exec ubtrace-import -n ubtrace -- \
  find /data/input_build -maxdepth 4 -type d

# Delete the temporary pod
kubectl delete pod ubtrace-import -n ubtrace

The worker polls input_build/ every 30 seconds and processes new versions automatically. No restart is needed.

Tip

If you have the PVC_NAME from a custom existingClaim, replace ubtrace-input-build in the pod spec with your claim name.

Option B: Sphinx Source Files

To have ubTrace build your Sphinx projects, copy source files into the input-src PVC using the same temporary pod pattern:

# Start a temporary import pod with the input-src PVC
kubectl run ubtrace-import-src --image=busybox --restart=Never \
  --overrides='{
    "spec": {
      "containers": [{
        "name": "import",
        "image": "busybox",
        "command": ["sleep", "3600"],
        "volumeMounts": [{
          "name": "input-src",
          "mountPath": "/data/input_src"
        }]
      }],
      "volumes": [{
        "name": "input-src",
        "persistentVolumeClaim": {
          "claimName": "ubtrace-input-src"
        }
      }]
    }
  }' -n ubtrace

kubectl wait --for=condition=Ready pod/ubtrace-import-src -n ubtrace --timeout=30s

# Create directory and copy Sphinx source
kubectl exec ubtrace-import-src -n ubtrace -- \
  mkdir -p /data/input_src/mycompany/my-project/v1
kubectl cp ./my-sphinx-project/ \
  ubtrace/ubtrace-import-src:/data/input_src/mycompany/my-project/v1 -n ubtrace

# Clean up
kubectl delete pod ubtrace-import-src -n ubtrace

The builder polls input_src/ every 60 seconds, builds with Sphinx, and passes output to the worker.

See Installation for details on directory structure, supported Sphinx layouts, and the required ubtrace_project.toml format.

First Login

The bundled Keycloak realm includes a pre-created test user:

Username	test
Password	Test1234!
Email	test-keycloak-user@useblocks.com

Open the ubTrace frontend and log in with these credentials. Before going to production, delete this user and rotate the OIDC client secret in Keycloak (see Keycloak / OIDC Hardening).

Day-to-Day Operations

Upgrading

# Update chart values (e.g., new image tag)
helm upgrade ubtrace ./deploy/helm/ubtrace \
  -f my-values.yaml \
  -n ubtrace

# Check rollout status
kubectl rollout status deployment/ubtrace-api -n ubtrace

Rolling Back

# List releases
helm history ubtrace -n ubtrace

# Rollback to a previous revision
helm rollback ubtrace <REVISION> -n ubtrace

Checking Status

# All pods
kubectl get pods -n ubtrace

# Service endpoints
kubectl get svc -n ubtrace

# PVC usage
kubectl get pvc -n ubtrace

# Logs for a specific component
kubectl logs -l app.kubernetes.io/component=api -n ubtrace --tail=100

Scaling

The API and frontend support horizontal scaling:

# Scale API replicas
helm upgrade ubtrace ./deploy/helm/ubtrace \
  --set api.replicaCount=3 \
  -n ubtrace

Note

The worker and builder are single-instance by design (they process shared PVC data). Do not scale them beyond 1 replica.

Uninstalling

helm uninstall ubtrace -n ubtrace

Warning

This removes all Kubernetes resources but preserves PVCs by default. To delete data volumes:

kubectl delete pvc -l app.kubernetes.io/instance=ubtrace -n ubtrace

Troubleshooting

Pods stuck in Pending

Usually caused by insufficient resources or missing StorageClass:

kubectl describe pod <pod-name> -n ubtrace

# Check if the StorageClass exists
kubectl get sc

Keycloak not ready

Keycloak requires its database to be running first and takes 1-3 minutes for the initial startup (realm import). Check the startup probe:

kubectl logs -l app.kubernetes.io/component=keycloak -n ubtrace --tail=50

# The startup probe allows up to 5 minutes (60 attempts x 5s)
kubectl describe pod -l app.kubernetes.io/component=keycloak -n ubtrace

API / Worker crash-restarts on fresh install

On a fresh helm install, the api and worker pods typically restart 2–3 times before stabilising. This happens because both pods attempt to connect to PostgreSQL and Keycloak immediately, while those services are still initialising (Keycloak in particular takes 1–3 min for realm import on first start).

This is expected behaviour — Kubernetes’ restart policy (Always) re-launches the containers, and the startup probe (30 attempts × 5 s = 150 s for the api) gives them enough headroom to succeed once the dependencies are ready.

# Watch pod status — you will see a few restarts before Running / Ready
kubectl get pods -n ubtrace -w

# Check events if a pod is stuck in CrashLoopBackOff for more than 5 min
kubectl describe pod -l app.kubernetes.io/component=api -n ubtrace

If restarts persist beyond 5 minutes, check that PostgreSQL and Keycloak pods are healthy first (see sections above).

Note

A future release will add initContainers that gate the api and worker pods until their dependencies are reachable, eliminating the initial crash-restarts entirely.

OIDC redirect errors

The most common cause is mismatched URLs. Ensure oidc.issuer, keycloak.hostname, api.publicUrl, and frontend.config.* all use the correct public hostnames for your deployment.

For port-forwarding setups, all URLs must use localhost with the forwarded ports (see the minikube section above).

Database authentication failures (stale PVCs)

If PostgreSQL pods fail with FATAL: password authentication failed after a re-install, this is almost always caused by stale PVCs from a previous run. helm uninstall preserves StatefulSet-owned PVCs by design, so the next install creates a new secret but re-attaches the old volume whose superuser password was set on the original initdb.

Replace <release> with your actual helm install name — PVC names are prefixed with it (data-<release>-postgresql-0 and so on), which is why the label-selector form below is safer than hard-coded names.

release=ubtrace   # set to your release name
ns=ubtrace

helm uninstall "$release" -n "$ns"

# StatefulSet-owned PVCs survive the uninstall. The label selector
# catches every PVC Helm created for this release regardless of name.
kubectl delete pvc -n "$ns" -l "app.kubernetes.io/instance=$release" --wait=false

# Wait for Terminating to finish before re-installing, otherwise the
# new pod schedules while the old PVC is still being deleted and you
# get "persistentvolumeclaim ... is being deleted" events.
kubectl wait --for=delete pvc -n "$ns" \
  -l "app.kubernetes.io/instance=$release" --timeout=60s || true

# Re-install from scratch.
helm install "$release" ./deploy/helm/ubtrace \
  -f deploy/helm/ubtrace/values-minikube.yaml \
  -n "$ns" --create-namespace

Private Registry

To pull images from a private registry:

# Create a pull secret
kubectl create secret docker-registry ubtrace-pull-secret \
  --docker-server=harbor.corp.example \
  --docker-username=robot\$ubtrace \
  --docker-password=<token> \
  -n ubtrace

# Install with private registry settings
helm install ubtrace ./deploy/helm/ubtrace \
  --set global.imageRegistry=harbor.corp.example/ubtrace \
  --set global.imagePullSecrets[0].name=ubtrace-pull-secret \
  -n ubtrace --create-namespace

Values Reference

For the complete list of configurable values, see:

# Print all default values
helm show values ./deploy/helm/ubtrace

Key files in the chart:

File	Purpose
values.yaml	Default configuration
values-minikube.yaml	Minikube overlay (reduced resources, NGINX ingress)
values-minikube-nodeport.yaml	Minikube NodePort overlay (no Ingress, no /etc/hosts)
values-eks.yaml	AWS EKS overlay (ALB, gp3, TLS)
values-openshift.yaml	OpenShift 4+ overlay (SCC-compatible, no Ingress)