Kubernetes Deployment¶

This guide covers deploying ubTrace on Kubernetes using the official Helm chart. The chart supports vanilla Kubernetes, AWS EKS, and OpenShift 4+.

For Docker Compose deployments (including air-gapped / offline), see Installation.

Prerequisites¶

Kubernetes 1.25+ cluster (1.28+ recommended)
Helm 3.12+
kubectl configured for your cluster. Install via your distro’s package manager (brew install kubectl, sudo apt install kubectl, sudo pacman -S kubectl, …) or follow the upstream guide at kubernetes.io/docs/tasks/tools.
A StorageClass that supports ReadWriteOnce PVCs
(Optional) An Ingress controller (NGINX, ALB, etc.)

Note

The chart bundles PostgreSQL, Elasticsearch, Redis, and Keycloak as sub-deployments. For production, consider using managed services (RDS, OpenSearch, ElastiCache, etc.) and pointing the chart at them via the external.* values.

Quick Start¶

The Helm chart and overlay files are shipped inside the offline bundle (ubtrace-offline-bundle-*.tar.gz) you received as part of your ubTrace delivery. All commands below assume your shell is sitting at the bundle root after tar xzf / cd.

# Extract the bundle and cd into its root
tar xzf ubtrace-offline-bundle-*.tar.gz
cd ubtrace-offline-bundle-*

# Make the bundled images reachable by the cluster.
# Option A — load directly on each node (fine for single-node /
#            evaluation clusters and for minikube):
./offline-load.sh ubtrace-images-*.tar.gz
# Option B — push to a private registry your cluster can pull from
#            (see "Private Registry" below).

# Install with default values + the bundled license file.
helm install ubtrace ./charts/ubtrace-*.tgz \
  --set-file api.license.fileContent=licenses/license.skm \
  -n ubtrace --create-namespace

# Watch pods come up
kubectl get pods -n ubtrace -w

The default values.yaml starts all infrastructure locally in the cluster with sensible defaults. All services should be ready within 2-5 minutes, depending on image pull times.

Note

The chart’s defaults (values.yaml) live inside charts/ubtrace-*.tgz — they are not loose files in the bundle. To inspect or copy them, run helm show values ./charts/ubtrace-*.tgz (see Values Reference below). Production deployments layer charts/values-production.yaml on top, which is shipped loose in the bundle alongside the platform overlays.

Note

The Docker Compose path covered in Installation and the Helm path covered here both consume the same offline bundle. Pick one — they are not meant to run side by side.

Online Install (OCI Chart)¶

If your cluster has outbound internet access, you can pull the chart directly from the GitHub Container Registry instead of unpacking it from the offline bundle. Every ubTrace release publishes the chart as an OCI artifact at ghcr.io/useblocks/charts/ubtrace:<version>, where <version> matches the ubTrace release you are deploying.

# Inspect the chart's default values without installing
helm show values oci://ghcr.io/useblocks/charts/ubtrace --version <version>

# Install straight from the registry (license file from your delivery)
helm install ubtrace oci://ghcr.io/useblocks/charts/ubtrace \
  --version <version> \
  --set-file api.license.fileContent=licenses/license.skm \
  -n ubtrace --create-namespace

The OCI install needs registry connectivity at install and upgrade time. Air-gapped clusters must keep using the bundled ./charts/ubtrace-*.tgz archive shown above — the OCI path is an online convenience, not a replacement for the offline bundle.

Note

For provisioning the underlying AWS / EKS infrastructure — the Terraform stack and the eks-tunnel helper, both shipped as GitHub Release assets — and the optional Google Workspace single sign-on, see the Cloud Deployment Guide.

Platform-Specific Overlays¶

The chart ships with overlay files for common platforms. Use the -f flag to layer them on top of the defaults.

Minikube (Local Development)¶

# Start a profile with enough headroom for Keycloak's first-boot
# spike, Elasticsearch, and the rest of the stack.
minikube start --cpus=4 --memory=8192
minikube addons enable ingress

# Point the Docker CLI at minikube's node runtime BEFORE loading
# images, so offline-load.sh imports into the cluster's daemon (not
# your host's).
eval $(minikube docker-env)
./offline-load.sh ubtrace-images-*.tar.gz

helm install ubtrace ./charts/ubtrace-*.tgz \
  -f charts/values-minikube.yaml \
  --set-file api.license.fileContent=licenses/license.skm \
  -n ubtrace --create-namespace

# Resolve ingress hostnames to the minikube node.
echo "$(minikube ip)  ubtrace.local ubtrace-api.local admin.ubtrace.local keycloak.local" \
  | sudo tee -a /etc/hosts

The minikube overlay uses reduced resource requests and the standard StorageClass. Ingress is enabled with the NGINX controller, and all OIDC / frontend URLs are pre-wired to the *.local ingress hosts — no --set overrides are needed for the happy path.

Note

--cpus and --memory only apply when creating a new profile. An existing profile with smaller resources will keep its old size silently — Keycloak or Elasticsearch will then fail to schedule with no error from minikube start itself. Either run minikube delete first, or use a dedicated profile:

minikube start -p ubtrace --cpus=4 --memory=8192

Note

Keycloak 26 runs a one-time Quarkus augmentation on first start (Updating the server image.) that briefly exceeds 1 GiB of RAM. The overlay sets the Keycloak memory limit to 3 GiB for that reason — lowering it causes OOMKilled loops during the first install.

Minikube (NodePort, no `/etc/hosts`)¶

Alternative overlay for environments where editing /etc/hosts requires admin rights you do not have (managed laptop, CI runner). Each service is reachable at http://$(minikube ip):<port> directly — no Ingress controller, no DNS entries, no host-file edits.

minikube start --cpus=4 --memory=8192

# Load bundled images into minikube's Docker daemon.
eval $(minikube docker-env)
./offline-load.sh ubtrace-images-*.tar.gz

export minikube_ip=$(minikube ip)
helm install ubtrace ./charts/ubtrace-*.tgz \
  -n ubtrace --create-namespace \
  -f charts/values-minikube-nodeport.yaml \
  --set-file api.license.fileContent=licenses/license.skm \
  --set-string api.publicUrl="http://${minikube_ip}:30150" \
  --set-string api.config.frontendUbtUrl="http://${minikube_ip}:30155" \
  --set-string api.config.adminUbtUrl="http://${minikube_ip}:30156" \
  --set-string oidc.issuer="http://${minikube_ip}:30181/realms/ubtrace" \
  --set-string keycloak.hostname="http://${minikube_ip}:30181" \
  --set-string frontend.config.apiUrl="http://${minikube_ip}:30150/api" \
  --set-string frontend.config.apiLoginUrl="http://${minikube_ip}:30150/api" \
  --set-string frontend.config.oidcIssuer="http://${minikube_ip}:30181/realms/ubtrace" \
  --set-string frontend.config.authRedirectUrl="http://${minikube_ip}:30155/auth/callback" \
  --set-string frontend.config.adminUrl="http://${minikube_ip}:30156" \
  --set-string admin.config.apiUrl="http://${minikube_ip}:30150/api" \
  --set-string admin.config.apiLoginUrl="http://${minikube_ip}:30150/api" \
  --set-string admin.config.oidcIssuer="http://${minikube_ip}:30181/realms/ubtrace" \
  --set-string admin.config.authRedirectUrl="http://${minikube_ip}:30156/auth/callback" \
  --set-string admin.config.mainAppUrl="http://${minikube_ip}:30155"

Service	URL
Frontend	`http://$(minikube ip):30155`
Admin	`http://$(minikube ip):30156`
API	`http://$(minikube ip):30150`
Keycloak	`http://$(minikube ip):30181`

The minikube addons enable ingress step is not required for this overlay. Tradeoffs vs the Ingress path:

Four distinct URLs instead of four *.local hostnames behind a single ingress.
Every public URL still has to be passed via --set-string because $(minikube ip) is only known at install time. The Ingress overlay needs zero --set flags.
NodePorts must be free on the node. If a colliding release already uses 30150/30155/30156/30181, either uninstall it or override the relevant <svc>.service.nodePort values.

Note

Both minikube overlays bake COOKIE_SECURE=false into api.config because neither path terminates TLS. Browsers drop cookies that carry the Secure flag when the response arrives over plain HTTP, which would make the OIDC login appear to succeed while every subsequent request returns 401. Production overlays (EKS, on-prem behind a TLS-terminating ingress) keep the default COOKIE_SECURE=true — do not copy this override into a deployment that actually serves HTTPS.

Note

macOS / Apple Silicon (Docker driver): Both minikube snippets above are written for Linux and hit three gotchas on Mac. Apply all three or neither overlay will be reachable from your browser.

Docker Desktop memory. minikube start --memory=8192 fails unless Docker Desktop itself has ≥10 GB allocated. Bump it in Docker Desktop → Settings → Resources → Memory and restart Docker Desktop.
Elasticsearch seccomp fix. The bundled amd64 ES image fails under Rosetta with seccomp unavailable: CONFIG_SECCOMP not compiled into kernel. Overwrite the loaded image with the native arm64 variant immediately after offline-load.sh:
```
eval $(minikube docker-env)
docker pull docker.elastic.co/elasticsearch/elasticsearch:9.3.0
kubectl delete pod -n ubtrace ubtrace-elasticsearch-0
eval $(minikube docker-env -u)
```
Requires network access for the one-time pull; persists until minikube delete.
``$(minikube ip)`` is not reachable from the host browser. The Docker driver runs the minikube node inside Docker Desktop’s VM; 192.168.49.2 routes only from inside that VM, so both overlays’ advertised URLs hang on the host. Fixes per overlay:
- Ingress overlay: use 127.0.0.1 (NOT $(minikube ip)) in /etc/hosts, then kubectl port-forward the ingress-nginx controller to localhost:80 in a separate terminal. minikube tunnel is the path the Linux-focused minikube docs push, but on Mac + Docker driver it often fails to claim an external IP (the ingress-nginx addon ships as NodePort, and even after patching it to LoadBalancer the tunnel is fragile). The port-forward approach is one reliable command:
```
echo "127.0.0.1  ubtrace.local ubtrace-api.local admin.ubtrace.local keycloak.local" | \
  sudo tee -a /etc/hosts
# Separate terminal, leave running (prompts for sudo to bind :80):
sudo kubectl port-forward -n ingress-nginx svc/ingress-nginx-controller 80:80
```
- NodePort overlay: retarget every URL in the --set-string block above from ${minikube_ip} to 127.0.0.1, then run one kubectl port-forward per service in the background:
```
kubectl port-forward -n ubtrace svc/ubtrace-frontend 30155:3000 &
kubectl port-forward -n ubtrace svc/ubtrace-admin    30156:3000 &
kubectl port-forward -n ubtrace svc/ubtrace-api      30150:3000 &
kubectl port-forward -n ubtrace svc/ubtrace-keycloak 30181:8080 &
```
The Ingress path is easier to maintain (one tunnel, no dangling background jobs); use it unless you truly cannot edit /etc/hosts.

AWS EKS¶

helm install ubtrace ./charts/ubtrace-*.tgz \
  -f charts/values-eks.yaml \
  -f charts/values-production.yaml \
  --set api.ingress.annotations."alb\.ingress\.kubernetes\.io/certificate-arn"=arn:aws:acm:REGION:ACCOUNT:certificate/CERT-ID \
  --set api.ingress.hosts[0].host=ubtrace-api.example.com \
  --set frontend.ingress.hosts[0].host=ubtrace.example.com \
  --set keycloak.ingress.hosts[0].host=keycloak.example.com \
  -n ubtrace --create-namespace

The EKS overlay pre-configures:

StorageClass: gp3 (EBS gp3 volumes)
Ingress: AWS ALB Ingress Controller with HTTPS termination
Annotations: ALB scheme, target type, health check paths

Before deploying, ensure:

The AWS Load Balancer Controller is installed
An ACM certificate exists for your domain(s)
DNS records point to the ALB (created automatically by the controller)

Tip

For production EKS deployments, consider using managed services:

# In your custom values file
postgresql:
  enabled: false
  external:
    host: "mydb.cluster-xxxxx.us-east-1.rds.amazonaws.com"
    port: 5432
    database: "ubtrace"

elasticsearch:
  enabled: false
  external:
    url: "https://search-ubtrace-xxxxx.us-east-1.es.amazonaws.com"

redis:
  enabled: false
  external:
    host: "ubtrace.xxxxx.ng.0001.use1.cache.amazonaws.com"
    port: 6379

OpenShift 4+¶

helm install ubtrace ./charts/ubtrace-*.tgz \
  -f charts/values-openshift.yaml \
  -f charts/values-production.yaml \
  -n ubtrace --create-namespace

The OpenShift overlay:

Sets securityContext.runAsUser: null so OpenShift assigns UIDs from the namespace range (required by SCCs)
Disables Ingress objects (OpenShift uses Routes)

All ubTrace images are built with GID 0 (root group) permissions on their data directories (chmod g=u), so they work without additional SCCs under the default OpenShift restricted policy. The entrypoints automatically detect OpenShift mode (non-root UID + GID 0) and skip operations that require root.

After deployment, expose services with OpenShift Routes:

oc expose svc/ubtrace-frontend -n ubtrace
oc expose svc/ubtrace-api -n ubtrace
oc expose svc/ubtrace-keycloak -n ubtrace

# For TLS-terminated Routes:
oc create route edge ubtrace-frontend \
  --service=ubtrace-frontend \
  --hostname=ubtrace.example.com \
  -n ubtrace

Configuration¶

All configuration is done via values.yaml overrides. The chart follows a consistent pattern for each component.

Image Configuration¶

global:
  imageRegistry: "ghcr.io/useblocks"    # Registry for ubTrace images
  imagePullSecrets: []                   # Pull secrets for private registries
  imagePullPolicy: "IfNotPresent"        # IfNotPresent, Always, or Never

# Per-component overrides (optional)
api:
  image:
    registry: ""          # Falls back to global.imageRegistry
    repository: ub-backend
    tag: ""               # Falls back to Chart.AppVersion

Infrastructure Toggles¶

Each infrastructure component can be disabled and replaced with an external service:

# Use external PostgreSQL (e.g., AWS RDS)
postgresql:
  enabled: false
  external:
    host: "mydb.example.com"
    port: 5432
    database: "ubtrace"

# Use external Elasticsearch (e.g., AWS OpenSearch)
elasticsearch:
  enabled: false
  external:
    url: "https://search.example.com"

# Use external Redis (e.g., AWS ElastiCache)
redis:
  enabled: false
  external:
    host: "redis.example.com"
    port: 6379

Passwords & Secrets¶

The chart generates a Kubernetes Secret with default passwords from values.yaml. For production, either:

Option A: Override values at install time

helm install ubtrace ./charts/ubtrace-*.tgz \
  --set postgresql.auth.password=<strong-password> \
  --set postgresqlKeycloak.auth.password=<strong-password> \
  --set keycloak.adminPassword=<strong-password> \
  --set redis.auth.password=<strong-password> \
  -n ubtrace --create-namespace

Option B: Use an existing Secret

# Create the secret manually
kubectl create secret generic ubtrace-secrets -n ubtrace \
  --from-literal=postgresql-password=<pw> \
  --from-literal=postgresql-keycloak-password=<pw> \
  --from-literal=keycloak-admin-password=<pw> \
  --from-literal=redis-password=<pw> \
  --from-literal=oidc-client-secret=<secret>

# Reference it during install
helm install ubtrace ./charts/ubtrace-*.tgz \
  --set api.existingSecret=ubtrace-secrets \
  -n ubtrace --create-namespace

Important

OIDC Client Secret: The bundled Keycloak realm ships with a pre-configured client secret (FnZuseq02pwaNKqsvDxL3jq4HhzPey2b). Do not change the oidc.clientSecret value unless you also regenerate the secret in Keycloak. See Keycloak / OIDC Hardening.

OIDC / Authentication¶

Configure the OIDC URLs to match your deployment’s public hostnames:

oidc:
  issuer: "https://keycloak.example.com/realms/ubtrace"
  clientId: "nestjs-app"
  clientSecret: "FnZuseq02pwaNKqsvDxL3jq4HhzPey2b"

keycloak:
  hostname: "https://keycloak.example.com"

api:
  publicUrl: "https://ubtrace-api.example.com"
  config:
    frontendUbtUrl: "https://ubtrace.example.com"

frontend:
  config:
    apiUrl: "https://ubtrace-api.example.com/api"
    oidcIssuer: "https://keycloak.example.com/realms/ubtrace"
    authRedirectUrl: "https://ubtrace.example.com/auth/callback"
    adminUrl: "https://admin.example.com"    # Required for "Go to Admin" button

Note

frontend.config.adminUrl maps to the UBTRACE_ADMIN_URL environment variable on the frontend pod. It controls the “Go to Admin” button in the user menu for users with the ubtrace-admin Keycloak realm role. If not set, the button will not appear regardless of the user’s roles. Set this to the public URL of the admin app when deploying it alongside the main frontend.

Each frontend.config.* Helm key maps to a UBTRACE_* environment variable (e.g. apiUrl becomes UBTRACE_API_URL). For non-Helm deployments, set these env vars directly on the frontend container.

Note

The chart can optionally configure a Google Workspace identity provider and an email-domain sign-up policy on the bundled Keycloak — both off by default — applied post-install by a keycloak-config-cli Job. Enable them through the keycloak.google.* and keycloak.userProfile.* values; see Configuration Reference for the full key reference.

Persistence¶

The chart creates PVCs for shared pipeline volumes. Customize sizes and StorageClass per volume:

persistence:
  output:
    size: 20Gi
    storageClass: ""       # Uses global.storageClass if empty
    accessMode: ReadWriteOnce
    existingClaim: ""      # Use a pre-existing PVC

  inputSrcBuild:
    size: 10Gi

  inputBuild:
    size: 10Gi

  inputSrc:
    size: 10Gi

  keycloakThemeJar:
    size: 128Mi

Resource Requests & Limits¶

Each component has configurable resource requests and limits:

Component	CPU Request	CPU Limit	Memory Request	Memory Limit
API	250m	1	512Mi	1Gi
Frontend	100m	500m	256Mi	512Mi
Worker	500m	2	1Gi	4Gi
Builder	250m	1	512Mi	2Gi
Keycloak	250m	1	512Mi	1Gi
PostgreSQL	250m	1	256Mi	1Gi
Elasticsearch	500m	2	1Gi	2Gi
Redis	100m	500m	128Mi	512Mi

Override in your values file:

api:
  resources:
    requests:
      cpu: 500m
      memory: 1Gi
    limits:
      cpu: "2"
      memory: 2Gi

Importing Artifacts¶

ubTrace uses a worker pipeline that automatically detects new or changed artifacts. The pipeline reads from Persistent Volume Claims (PVCs). Since the worker and builder mount these PVCs as read-only, you need a temporary helper pod to write data into them.

Note

This is a key difference from Docker Compose deployments, where you can copy files directly into host-mounted directories. On Kubernetes, PVC access requires a pod with the volume mounted.

Option A: Pre-Built CI Output (Recommended)¶

If your CI pipeline already builds with ubt_sphinx, copy the output into the input-build PVC.

Step 1: Start a temporary import pod

kubectl run ubtrace-import --image=busybox --restart=Never \
  --overrides='{
    "spec": {
      "containers": [{
        "name": "import",
        "image": "busybox",
        "command": ["sleep", "3600"],
        "volumeMounts": [{
          "name": "input-build",
          "mountPath": "/data/input_build"
        }]
      }],
      "volumes": [{
        "name": "input-build",
        "persistentVolumeClaim": {
          "claimName": "ubtrace-input-build"
        }
      }]
    }
  }' -n ubtrace

# Wait for the pod to be ready
kubectl wait --for=condition=Ready pod/ubtrace-import -n ubtrace --timeout=30s

Step 2: Create the directory structure and copy artifacts

The required path inside the PVC is {org}/{project}/{version}/. These values must match the fields in config/ubtrace_project.toml.

# Create directories
kubectl exec ubtrace-import -n ubtrace -- \
  mkdir -p /data/input_build/mycompany/my-project/v1

# Copy pre-built artifacts
kubectl cp ./ci-output/mycompany/my-project/v1 \
  ubtrace/ubtrace-import:/data/input_build/mycompany/my-project/v1 -n ubtrace

Step 3: Verify and clean up

# Verify the structure
kubectl exec ubtrace-import -n ubtrace -- \
  find /data/input_build -maxdepth 4 -type d

# Delete the temporary pod
kubectl delete pod ubtrace-import -n ubtrace

The worker polls input_build/ every 30 seconds and processes new versions automatically. No restart is needed.

Tip

If you have the PVC_NAME from a custom existingClaim, replace ubtrace-input-build in the pod spec with your claim name.

Option B: Sphinx Source Files¶

To have ubTrace build your Sphinx projects, copy source files into the input-src PVC using the same temporary pod pattern:

# Start a temporary import pod with the input-src PVC
kubectl run ubtrace-import-src --image=busybox --restart=Never \
  --overrides='{
    "spec": {
      "containers": [{
        "name": "import",
        "image": "busybox",
        "command": ["sleep", "3600"],
        "volumeMounts": [{
          "name": "input-src",
          "mountPath": "/data/input_src"
        }]
      }],
      "volumes": [{
        "name": "input-src",
        "persistentVolumeClaim": {
          "claimName": "ubtrace-input-src"
        }
      }]
    }
  }' -n ubtrace

kubectl wait --for=condition=Ready pod/ubtrace-import-src -n ubtrace --timeout=30s

# Create directory and copy Sphinx source
kubectl exec ubtrace-import-src -n ubtrace -- \
  mkdir -p /data/input_src/mycompany/my-project/v1
kubectl cp ./my-sphinx-project/ \
  ubtrace/ubtrace-import-src:/data/input_src/mycompany/my-project/v1 -n ubtrace

# Clean up
kubectl delete pod ubtrace-import-src -n ubtrace

The builder polls input_src/ every 60 seconds, builds with Sphinx, and passes output to the worker.

See Installation for details on directory structure, supported Sphinx layouts, and the required ubtrace_project.toml format.

First Login¶

The bundled Keycloak realm includes a pre-created test user:

Username	`test`
Password	`Test1234!`
Email	`test-keycloak-user@useblocks.com`

Open the ubTrace frontend and log in with these credentials. Before going to production, delete this user and rotate the OIDC client secret in Keycloak (see Keycloak / OIDC Hardening).

Day-to-Day Operations¶

Upgrading¶

# Re-apply with whichever overlay you originally installed (replace
# the value-file path with your platform overlay from ``charts/``).
helm upgrade ubtrace ./charts/ubtrace-*.tgz \
  -f charts/values-production.yaml \
  -n ubtrace

# Check rollout status
kubectl rollout status deployment/ubtrace-api -n ubtrace

Rolling Back¶

# List releases
helm history ubtrace -n ubtrace

# Rollback to a previous revision
helm rollback ubtrace <REVISION> -n ubtrace

Checking Status¶

# All pods
kubectl get pods -n ubtrace

# Service endpoints
kubectl get svc -n ubtrace

# PVC usage
kubectl get pvc -n ubtrace

# Logs for a specific component
kubectl logs -l app.kubernetes.io/component=api -n ubtrace --tail=100

Scaling¶

The API and frontend support horizontal scaling:

# Scale API replicas
helm upgrade ubtrace ./charts/ubtrace-*.tgz \
  --set api.replicaCount=3 \
  -n ubtrace

Note

The worker and builder are single-instance by design (they process shared PVC data). Do not scale them beyond 1 replica.

Uninstalling¶

helm uninstall ubtrace -n ubtrace

Warning

This removes all Kubernetes resources but preserves PVCs by default. To delete data volumes:

kubectl delete pvc -l app.kubernetes.io/instance=ubtrace -n ubtrace

Troubleshooting¶

Pods stuck in Pending¶

Usually caused by insufficient resources or missing StorageClass:

kubectl describe pod <pod-name> -n ubtrace

# Check if the StorageClass exists
kubectl get sc

Keycloak not ready¶

Keycloak requires its database to be running first and takes 1-3 minutes for the initial startup (realm import). Check the startup probe:

kubectl logs -l app.kubernetes.io/component=keycloak -n ubtrace --tail=50

# The startup probe allows up to 5 minutes (60 attempts x 5s)
kubectl describe pod -l app.kubernetes.io/component=keycloak -n ubtrace

API / Worker crash-restarts on fresh install¶

On a fresh helm install, the api and worker pods typically restart 2–3 times before stabilising. This happens because both pods attempt to connect to PostgreSQL and Keycloak immediately, while those services are still initialising (Keycloak in particular takes 1–3 min for realm import on first start).

This is expected behaviour — Kubernetes’ restart policy (Always) re-launches the containers, and the startup probe (30 attempts × 5 s = 150 s for the api) gives them enough headroom to succeed once the dependencies are ready.

# Watch pod status — you will see a few restarts before Running / Ready
kubectl get pods -n ubtrace -w

# Check events if a pod is stuck in CrashLoopBackOff for more than 5 min
kubectl describe pod -l app.kubernetes.io/component=api -n ubtrace

If restarts persist beyond 5 minutes, check that PostgreSQL and Keycloak pods are healthy first (see sections above).

Note

A future release will add initContainers that gate the api and worker pods until their dependencies are reachable, eliminating the initial crash-restarts entirely.

OIDC redirect errors¶

The most common cause is mismatched URLs. Ensure oidc.issuer, keycloak.hostname, api.publicUrl, and frontend.config.* all use the correct public hostnames for your deployment.

For port-forwarding setups, all URLs must use localhost with the forwarded ports (see the minikube section above).

Database authentication failures (stale PVCs)¶

If PostgreSQL pods fail with FATAL: password authentication failed after a re-install, this is almost always caused by stale PVCs from a previous run. helm uninstall preserves StatefulSet-owned PVCs by design, so the next install creates a new secret but re-attaches the old volume whose superuser password was set on the original initdb.

Replace <release> with your actual helm install name — PVC names are prefixed with it (data-<release>-postgresql-0 and so on), which is why the label-selector form below is safer than hard-coded names.

release=ubtrace   # set to your release name
ns=ubtrace

helm uninstall "$release" -n "$ns"

# StatefulSet-owned PVCs survive the uninstall. The label selector
# catches every PVC Helm created for this release regardless of name.
kubectl delete pvc -n "$ns" -l "app.kubernetes.io/instance=$release" --wait=false

# Wait for Terminating to finish before re-installing, otherwise the
# new pod schedules while the old PVC is still being deleted and you
# get "persistentvolumeclaim ... is being deleted" events.
kubectl wait --for=delete pvc -n "$ns" \
  -l "app.kubernetes.io/instance=$release" --timeout=60s || true

# Re-install from scratch.
helm install "$release" ./charts/ubtrace-*.tgz \
  -f charts/values-minikube.yaml \
  --set-file api.license.fileContent=licenses/license.skm \
  -n "$ns" --create-namespace

Private Registry¶

For multi-node clusters where loading images on each node with offline-load.sh is impractical, mirror the bundled images into a private registry that the cluster can pull from. The bundle ships scripts/offline-mirror.sh for this — see the “Private Registry Mirror” section of README-offline.md (also in the bundle) for the full procedure.

Once images are in your registry, configure the chart to pull from it:

# Create a pull secret
kubectl create secret docker-registry ubtrace-pull-secret \
  --docker-server=harbor.corp.example \
  --docker-username=robot\$ubtrace \
  --docker-password=<token> \
  -n ubtrace

# Install with private registry settings
helm install ubtrace ./charts/ubtrace-*.tgz \
  --set global.imageRegistry=harbor.corp.example/ubtrace \
  --set global.imagePullSecrets[0].name=ubtrace-pull-secret \
  -n ubtrace --create-namespace

Values Reference¶

For the complete list of configurable values, see:

# Print all default values
helm show values ./charts/ubtrace-*.tgz

# Or inspect the unpacked overlay files directly
ls charts/

Key files shipped in the bundle:

File	Purpose
`charts/ubtrace-*.tgz`	Helm chart archive (pass to `helm install`)
`charts/values.yaml`	Default configuration
`charts/values-production.yaml`	Production hardening overlay (replicas, resources, PDBs)
`charts/values-minikube.yaml`	Minikube overlay (reduced resources, NGINX ingress)
`charts/values-minikube-nodeport.yaml`	Minikube NodePort overlay (no Ingress, no `/etc/hosts`)
`charts/values-eks.yaml`	AWS EKS overlay (ALB, gp3, TLS)
`charts/values-eks-terraform.yaml`	AWS EKS with Terraform-managed infrastructure (add-on to `charts/values-eks.yaml`)
`charts/values-openshift.yaml`	OpenShift 4+ overlay (SCC-compatible, no Ingress)
`offline-load.sh`	Loads bundled images into the local Docker daemon
`licenses/license.skm`	ubTrace license key (referenced via `--set-file api.license.fileContent`)

Kubernetes Deployment¶

Prerequisites¶

Quick Start¶

Online Install (OCI Chart)¶

Platform-Specific Overlays¶

Minikube (Local Development)¶

Minikube (NodePort, no /etc/hosts)¶

AWS EKS¶

OpenShift 4+¶

Configuration¶

Image Configuration¶

Infrastructure Toggles¶

Passwords & Secrets¶

OIDC / Authentication¶

Persistence¶

Resource Requests & Limits¶

Importing Artifacts¶

Option A: Pre-Built CI Output (Recommended)¶

Option B: Sphinx Source Files¶

First Login¶

Day-to-Day Operations¶

Upgrading¶

Rolling Back¶

Checking Status¶

Scaling¶

Uninstalling¶

Troubleshooting¶

Pods stuck in Pending¶

Keycloak not ready¶

API / Worker crash-restarts on fresh install¶

OIDC redirect errors¶

Database authentication failures (stale PVCs)¶

Private Registry¶

Values Reference¶

Minikube (NodePort, no `/etc/hosts`)¶