Version Upgrade & Data Migration¶
Use this runbook for Helm upgrades on AWS EKS with RDS and EFS. It explains what to back up, when to rebuild data, and how to recover a failed upgrade. For other deployments, use equivalent database and file backups before applying the same checks.
Read the target release’s migration notes for backfills, reindexing, and configuration changes. See Per-release migration notes.
Principles¶
Two things are always preserved across an upgrade, no matter how any store is classified:
Sphinx-Needs source documents. The ingested source is the input the whole pipeline is derived from. It is never wiped as part of an upgrade.
Customer configuration. Keep the Helm values, secrets, license files, and Keycloak realm configuration. Back them up separately from the databases. Keep secrets in protected storage, outside version control.
Keep each store unless the release requires a rebuild. Before deleting derived data, prove that it can be rebuilt from the saved source and configuration.
Store classification¶
Store |
Contains |
Class |
Handling |
|---|---|---|---|
PostgreSQL (application) |
The project catalogue and version index, need graph nodes and links, coverage, schema violations, ASPICE data, dashboards, permissions, API tokens, and worker state. Organizations, projects, and versions are identifiers on these records, not separate entity tables. |
Keep + migrate |
System of record. Schema migrations run automatically as a pre-upgrade hook job inside the Helm release, before the new pods roll out. Take a snapshot before upgrading. |
PostgreSQL (Keycloak) |
Users, roles, realm, sessions |
Keep + migrate |
Keycloak manages its own schema. A Helm redeploy does not wipe users. Take a snapshot before upgrading. |
Elasticsearch |
The full need and document bodies, in per-project indexes
( |
Keep; rebuild only when required |
The full documents live here, not in PostgreSQL; rebuilt by re-ingesting the source (PostgreSQL keeps only minimal node records). Do not change the immutable storage-class field on the StatefulSet volume claim during an upgrade. |
Redis |
Cache, worker queues, and coordination locks |
Keep; clear only selected cache keys |
Preserve queues and locks. Let queued and active jobs finish before stopping processing. Clear only the cache keys named in the release notes, with writers stopped. Do not flush the database or instance. |
EFS pipeline volumes |
Source ( |
Split |
Back up all four volumes. Keep source and uploaded builds. An uploaded build may be the only copy on the server. Rebuild derived files only when the matching source, build configuration, and toolchain are available. |
Rebuild time and manual work¶
Use these to decide whether a wipe is acceptable inside the upgrade window.
Store |
Rebuild trigger |
Cost |
|---|---|---|
Elasticsearch |
Reindex or re-ingest the affected projects |
Measure ingestion time for the affected versions in a rehearsal. |
Redis cache keys |
Populated lazily after selective invalidation |
No data import; measure cache warm-up during a rehearsal. |
EFS build/output |
Rebuild from source or upload the matching CI build again |
Measure build and upload time; include any manual configuration changes. |
Rehearse the target upgrade on a restored copy of the customer data. Time each required rebuild from its start until version health and search checks pass. Record the data size, version count, and manual work, including configuration changes and uploads. Use these measured times for the maintenance window. Include backup, restore, and verification time, and set a deadline for starting rollback. Do not use a fixed estimate from a smaller deployment.
Pre-upgrade¶
Confirm the current release is healthy. Record its chart version and deployed revision. Use the same namespace for every command below:
export UBTRACE_NAMESPACE=ubtrace helm history ubtrace -n "$UBTRACE_NAMESPACE" kubectl get pods -n "$UBTRACE_NAMESPACE" \ -l app.kubernetes.io/instance=ubtrace
Save the healthy revision as
UBTRACE_PREVIOUS_REVISION. Check that the readiness and customer smoke tests below pass before starting.Pause writes and back up files and configuration. Block customer uploads and edits and pause publishing pipelines. Let queued and active jobs finish, then stop builder and worker processing. Preserve Redis queue and lock state. Take file backups and database snapshots while processing is stopped. Keep customer writes paused until verification passes, including during recovery. Helm may restart the builder and worker during the upgrade.
After jobs finish, stop the two deployments and wait for their pods to exit:
kubectl scale deployment/ubtrace-builder deployment/ubtrace-worker \ -n "$UBTRACE_NAMESPACE" --replicas=0 kubectl get pods -n "$UBTRACE_NAMESPACE" \ -l app.kubernetes.io/instance=ubtrace
Back up the EFS file systems used by all four pipeline volumes with AWS Backup. See Operations Guide for the backup command and the EFS restore procedure for recovery. Record the recovery points and wait for the backup jobs to complete. Rehearse restoring the files before the production upgrade.
Copy the customer values, realm configuration, and license files into protected backup storage. Also save the release configuration and the Secrets it references. Set
UBTRACE_SECRET_NAMEto each referenced Secret:umask 077 export UBTRACE_BACKUP_DIR="$(mktemp -d)" helm get values ubtrace -n "$UBTRACE_NAMESPACE" -o yaml \ > "$UBTRACE_BACKUP_DIR/release-values.yaml" helm get manifest ubtrace -n "$UBTRACE_NAMESPACE" \ > "$UBTRACE_BACKUP_DIR/release-manifest.yaml" kubectl get secret "$UBTRACE_SECRET_NAME" -n "$UBTRACE_NAMESPACE" -o yaml \ > "$UBTRACE_BACKUP_DIR/secret.yaml"
Use a different filename for each Secret. These files contain credentials; move them into encrypted backup storage. Record the destination alongside the EFS recovery points. Keep these original files intact during recovery.
Snapshot both databases. Use the actual RDS instance identifiers for the application and Keycloak databases. Record their subnet groups, security groups, parameter groups, and connection settings for a restore.
export UBTRACE_APP_DB='<application-rds-instance-id>' export UBTRACE_KEYCLOAK_DB='<keycloak-rds-instance-id>' export UBTRACE_SNAPSHOT_PREFIX='pre-<version-with-hyphens>-<timestamp>' aws rds create-db-snapshot \ --db-instance-identifier "$UBTRACE_APP_DB" \ --db-snapshot-identifier "$UBTRACE_SNAPSHOT_PREFIX-app" aws rds create-db-snapshot \ --db-instance-identifier "$UBTRACE_KEYCLOAK_DB" \ --db-snapshot-identifier "$UBTRACE_SNAPSHOT_PREFIX-keycloak"
Replace the placeholders and use a unique prefix for each attempt. Wait until both snapshots report
availablebefore proceeding. Snapshot identifiers cannot contain dots, so1.9.0becomes1-9-0.Record baseline counts. Follow Verification / integrity checks and save the results in the backup directory. Use the same accounts, project scope, and queries after the upgrade. Record Keycloak user counts per realm in its admin console, together with a test account’s roles.
Review the release-specific chapter. If the target version has one (see Per-release migration notes), read it now; it may add a manual backfill or reindex that must run at a specific point in this procedure.
Upgrade procedure¶
Dry-run the target chart. Use a chart package pinned to the target release and the customer’s saved values. Inspect the result against the current release. Keep manifests in protected storage because they can contain secrets.
export UBTRACE_TARGET_CHART='./charts/ubtrace-<target-version>.tgz' helm upgrade ubtrace "$UBTRACE_TARGET_CHART" \ -n "$UBTRACE_NAMESPACE" -f values-customer.yaml \ --dry-run=client --hide-secret
Stop if a ConfigMap disappears, a Secret changes unexpectedly, an Ingress loses annotations, or a persistent volume changes storage class or access mode. Ensure the target values include the customer’s existing EKS settings.
Apply the upgrade atomically. Run the Helm upgrade with
--atomicand a timeout. The application database migration runs as a pre-upgrade hook job inside this release, before the rollout: a failed migration fails the upgrade while the previous pods keep serving.--atomicrolls back the Helm release on failure; it does not undo database migrations or file changes. Use the rollback procedure below if the previous application cannot run against the changed data.helm upgrade ubtrace "$UBTRACE_TARGET_CHART" \ -n "$UBTRACE_NAMESPACE" -f values-customer.yaml \ --atomic --timeout 10m
Rebuild only when the release requires it. Follow its migration notes:
For Elasticsearch or build changes, use the release’s rebuild procedure. Ingest Documentation via API describes uploading a rebuilt archive with
overwrite=true. Preserve the original organization, project, version, and variant dimensions. An accepted upload is not proof that ingestion finished; wait for the version checks below. The release notes must explain how unchanged artifacts are forced through ingestion when needed.For Redis, clear only the cache keys named in the release notes. Keep worker queues and locks. Do not use
FLUSHDBorFLUSHALL.Do not delete indexes or files until their replacement and recovery paths have passed the rehearsal.
Warm and smoke-test. Resume only the processing needed for any required rebuild. Keep customer writes paused. Run the checks below, then resume normal traffic and publishing only when all checks pass.
Verification / integrity checks¶
Run these checks before and after the upgrade. Save each result separately.
Set UBTRACE_API_URL to the customer’s API base URL, including /api, and
use a user access token with permission to read the checked projects. An ingest
token cannot perform these reads.
curl --fail --silent --show-error \
"$UBTRACE_API_URL/v1/health/readiness"
curl --fail --silent --show-error \
-H "Authorization: Bearer $UBTRACE_ACCESS_TOKEN" \
"$UBTRACE_API_URL/v1/artifacts/organizations"
curl --fail --silent --show-error \
-H "Authorization: Bearer $UBTRACE_ACCESS_TOKEN" \
"$UBTRACE_API_URL/v1/artifacts/organizations/$UBTRACE_ORG/projects"
curl --fail --silent --show-error \
-H "Authorization: Bearer $UBTRACE_ACCESS_TOKEN" \
"$UBTRACE_API_URL/v1/artifacts/organizations/$UBTRACE_ORG/projects/$UBTRACE_PROJECT/versions"
curl --fail --silent --show-error \
-H "Authorization: Bearer $UBTRACE_ACCESS_TOKEN" \
"$UBTRACE_API_URL/v1/artifacts/organizations/$UBTRACE_ORG/projects/$UBTRACE_PROJECT/versions/$UBTRACE_VERSION/health"
Repeat the project and version reads for each organization and project. Use the
stored version ID from the versions response. Compare the returned lists and
components.needs.summary.totalNeeds with the baseline. Compare Keycloak
users and roles in the same realms used before the upgrade.
Check Elasticsearch need counts for each version from a host with access to the search service. Use the deployment’s Elasticsearch authentication. The query must filter by version because each project index holds several versions. Use lowercase organization and project identifiers in the index name:
POST /ubtrace-needs-<org>-<project>/_count
{"query":{"term":{"versionId":"<stored-version-id>"}}}
Compare count with that version’s recorded total needs. Document-section
counts are separate from need counts; compare those with their own baseline.
Accept the upgrade only when all of these pass:
Entity counts match. Organizations, projects, versions, needs, and Keycloak users match the pre-upgrade baseline (or the release-specific chapter explains any expected change).
Search index integrity. Each version’s Elasticsearch need count matches its application count. Document-section counts match their baseline or the change explained in the release notes.
Cache rebuilt. Post-upgrade requests succeed with no stale or missing keys after Redis has warmed.
Readiness green. The endpoint succeeds and reports
database,cache, andsearchas up. It also checksidProviderwhen an OIDC health-check URL is configured. Test login separately.Customer smoke tests pass. Sign in, open a known project and version, view a document, search for a known need, and open its graph and dashboard. Check that the test account retains its roles and cannot access a restricted project. Once read checks pass, publish a test version through the customer’s normal pipeline and confirm it appears with the expected needs.
Rollback¶
Roll back when a verification check fails and cannot be resolved in place:
A schema migration failed. Check both the Helm release status and the database state before retrying. A release rollback does not revert the schema.
Entity counts do not match the baseline and the difference is unexplained.
Health endpoints stay degraded, or smoke tests fail, after caches have warmed.
Allow cache warm-up or a rebuild to continue only within the rehearsed time budget. Start rollback at the agreed deadline if verification still fails.
Rollback procedure¶
Keep writes paused and choose the recovery path. If the previous release can use the current schema and files, roll it back to the recorded healthy revision. If it cannot, stop the API, worker, builder, and affected Keycloak workloads before restoring data. Ensure they stay stopped while reconnecting storage; a Helm operation can start them again.
helm rollback ubtrace "$UBTRACE_PREVIOUS_REVISION" \ -n "$UBTRACE_NAMESPACE" --wait --timeout 10m
Restore incompatible databases before restarting workloads. In the RDS console, select the pre-upgrade snapshot and choose Restore snapshot. Give the restored instance a new name. Select the saved subnet group, security groups, and parameter group; retain the required instance settings. Wait until its status is
available. Repeat for Keycloak if its database changed incompatibly. See Restoring an RDS snapshot.RDS restores to a new instance, with a new endpoint. Update
postgresql.external.hostin the recovery values. If the application uses an existing Secret, update itsdatabase-urlhost too, preserving the database name, credentials, and connection options. For Keycloak, updatepostgresqlKeycloak.external.host. Keep the original values and Secrets intact; make these changes in recovery copies.Keep workloads stopped until file recovery is complete. Keep the original RDS instances until recovery is accepted.
Restore changed files and configuration. Restore the EFS recovery points if source or uploaded builds were lost. AWS Backup places restored files in a recovery directory; they do not replace the live paths automatically. With processing stopped, copy the required files back to their original volume paths and preserve ownership and permissions. If restoring to a new file system, reconnect the persistent volumes before starting workloads. Restore the saved realm configuration and license files if they changed.
Keep the original archives until the recovered versions pass verification.
Start the previous release and verify recovery. Deploy its chart with the recovery values and Secrets. Restart affected workloads so they read the new connections. Use that version’s build and ingest procedure to recover derived files and Elasticsearch. Do not assume restarting the worker re-ingests unchanged files.
Re-run the checks, including readiness and login. Resume customer writes only when they pass. Any writes accepted after the backup need a separate recovery plan; restoring a snapshot does not include them.
Per-release migration notes¶
For each release, add a short chapter with any extra steps, or state that none are needed. Include the exact rebuild commands, expected count changes, and whether the previous release can use the migrated schema and files.
Version X.Y.0
- What changed that affects data or configuration
- Steps additive to the general procedure, in order
- Any verification specific to this release