Ingest Documentation via API

Upload a Sphinx-Needs documentation build to your ubTrace instance over HTTP, straight from a CI pipeline (GitHub Actions, GitLab CI, Azure Pipelines, Jenkins, or any tool that can run curl).

This is the recommended path when:

  • Your CI runs in a different network than the ubTrace server and cannot mount a shared folder.

  • You want every push to a branch or tag to publish a new version automatically.

  • You prefer a single HTTP call over copying files into input_build/.

The round trip looks like this:

  1. An admin creates an API token in the ubTrace admin panel.

  2. The token is stored as a CI secret (for example a GitHub Actions secret).

  3. The CI job builds the Sphinx-Needs documentation with ubt_sphinx.

  4. The CI job packages the output into a tar.gz (or zip) archive.

  5. The CI job uploads the archive to the /v1/ingest endpoint with curl.

  6. The ubTrace worker picks up the archive and publishes the new version.

Prerequisites

Before you start, make sure you have:

  • A running ubTrace instance that you can reach from your CI network. The API base URL is the same URL you use in the browser, with /api appended – for example https://ubtrace.example.com/api. In the bundled offline deployment this defaults to http://<host>:7150/api.

  • Admin access to the ubTrace admin panel. Only users with the ubtrace-admin role can create or revoke API tokens. Regular editors and viewers cannot.

  • A local Sphinx project that builds successfully with ubt_sphinx. The build must produce docs/ubtrace/needs.json – this is the file ubTrace uses to detect a valid artifact.

Step 1 – Create an API Token

An API token is a long, random secret that lets a machine call ubTrace without a username or password. You create one token per CI pipeline.

  1. Open the ubTrace admin panel in your browser. In the default offline deployment this is http://<host>:7156.

  2. Sign in with an account that has the ubtrace-admin role.

  3. Go to Settings → API Tokens.

  4. Click Create token.

  5. Fill in the form:

    • Name – something you will recognize later, for example GitHub Actions CI or gitlab-docs-pipeline. This label is only for you; it is never sent as part of the token.

    • Scopes – select Ingest. This is currently the only scope available; it allows uploads to POST /v1/ingest/... and nothing else. A leaked ingest token cannot read user data, change settings, or delete projects. Additional scopes may ship in future releases – when they do, pick only the ones the pipeline needs.

    • Expiry date (optional) – pick a date after which the token stops working. If you leave this empty, the token is valid until you revoke it. For CI pipelines a 6- or 12-month expiry is a good default – it forces a rotation without being disruptive.

  6. Click Create. ubTrace shows the full token exactly once, in a dialog like ubt_abcd1234....

  7. Copy the token immediately. Once you close the dialog you cannot see the full value again; you can only see the prefix (ubt_abcd...) in the list. If you lose the token, revoke it and create a new one.

Note

Where is the token stored on the server? ubTrace stores only a one-way hash of the token (argon2id) plus the short prefix. The server cannot reconstruct the original value, so even a database dump does not leak working tokens. This is the same pattern GitHub and GitLab use for personal access tokens.

Note

Token limits. Each admin can own up to 25 active tokens. If you hit the limit, revoke unused tokens from the list before creating a new one.

Step 2 – Store the Token as a CI Secret

Never paste the token into a workflow file or commit it to a repository. Store it as a CI secret and read it from an environment variable at runtime.

The examples below all use the name UBTRACE_INGEST_TOKEN. You can pick any name; just be consistent.

GitHub Actions

  1. In your repository on GitHub, go to Settings → Secrets and variables → Actions.

  2. Click New repository secret.

  3. Set Name to UBTRACE_INGEST_TOKEN and Secret to the token value you copied in Step 1.

  4. Save. The secret is now available to workflows in this repository as ${{ secrets.UBTRACE_INGEST_TOKEN }}.

Other CI systems

The same pattern works on every major CI platform – only the UI differs:

  • GitLab CI: Settings → CI/CD → Variables. Add a masked, protected variable named UBTRACE_INGEST_TOKEN. Read it in .gitlab-ci.yml with $UBTRACE_INGEST_TOKEN.

  • Azure Pipelines: Project Settings → Pipelines → Library. Add a secret variable to a variable group, then reference it with $(UBTRACE_INGEST_TOKEN).

  • Jenkins: Manage Jenkins → Credentials. Add a “Secret text” credential, then bind it inside your pipeline with withCredentials([string(credentialsId: 'ubtrace-ingest-token', variable: 'UBTRACE_INGEST_TOKEN')]) { ... }.

  • CircleCI / Bitbucket Pipelines / TeamCity / Drone: each ships a built-in secrets store. The contract is always the same – expose the token as an environment variable named UBTRACE_INGEST_TOKEN.

Step 3 – Build, Package, and Upload (GitHub Actions)

The workflow below does the full round trip. Copy it into .github/workflows/publish-docs.yml in your documentation repository and adjust the three env: values at the top.

name: Publish docs to ubTrace

on:
  push:
    branches: [main]
    tags: ['v*']

env:
  # The URL of your ubTrace API. Do NOT include a trailing slash.
  UBTRACE_URL: https://ubtrace.example.com/api
  # The organization and project identifiers you want to publish into.
  # Must already exist or be creatable on first upload (see "Identifiers" below).
  UBTRACE_ORG: mycompany
  UBTRACE_PROJECT: my-product

jobs:
  publish:
    runs-on: ubuntu-latest
    steps:
      - name: Check out the repository
        uses: actions/checkout@v5

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: '3.12'

      - name: Install ubt_sphinx
        run: |
          pip install --upgrade pip
          # Pin to a known-good release so a future ubt_sphinx version cannot
          # silently break the pipeline. Bump deliberately after testing.
          pip install "ubt_sphinx==0.6.0" --extra-index-url https://pypi.useblocks.com

      - name: Determine the version identifier
        id: version
        run: |
          # Use the git tag when the workflow is triggered by a tag (e.g. "v1.2.3"),
          # otherwise use the short commit SHA. Pick whatever fits your release flow.
          if [[ "${GITHUB_REF_TYPE}" == "tag" ]]; then
            echo "version=${GITHUB_REF_NAME}" >> "$GITHUB_OUTPUT"
          else
            echo "version=${GITHUB_SHA::7}" >> "$GITHUB_OUTPUT"
          fi

      - name: Build the documentation with ubt_sphinx
        env:
          # Read by conf.py so the builder's output path contains the
          # version we want to publish (see note below the workflow).
          UBTRACE_VERSION: ${{ steps.version.outputs.version }}
        run: |
          # Adjust "docs/" if your Sphinx source lives somewhere else.
          # The ubtrace builder writes its output to:
          #   docs/_build/ubtrace/<ubtrace_organization>/<ubtrace_project>/<ubtrace_version>/
          # -- the values on the right come from conf.py, not from this step.
          sphinx-build -b ubtrace docs docs/_build/ubtrace

      - name: Package the build output as build.tar.gz
        env:
          UBTRACE_VERSION: ${{ steps.version.outputs.version }}
        run: |
          # The archive MUST contain "docs/ubtrace/needs.json" at its root.
          # Tar from inside the version directory so the three top-level
          # entries of the archive are "config/", "docs/", and (optionally) metadata.
          BUILD_DIR="docs/_build/ubtrace/${UBTRACE_ORG}/${UBTRACE_PROJECT}/${UBTRACE_VERSION}"
          test -f "${BUILD_DIR}/docs/ubtrace/needs.json" \
            || { echo "needs.json missing -- check that conf.py values match UBTRACE_ORG/PROJECT/VERSION"; exit 1; }
          tar -czf "${GITHUB_WORKSPACE}/build.tar.gz" -C "${BUILD_DIR}" .

      - name: Upload to ubTrace
        env:
          UBTRACE_INGEST_TOKEN: ${{ secrets.UBTRACE_INGEST_TOKEN }}
          UBTRACE_VERSION: ${{ steps.version.outputs.version }}
        run: |
          curl --fail-with-body --show-error \
            -H "Authorization: Bearer ${UBTRACE_INGEST_TOKEN}" \
            -F "file=@build.tar.gz" \
            "${UBTRACE_URL}/v1/ingest/${UBTRACE_ORG}/${UBTRACE_PROJECT}/${UBTRACE_VERSION}?overwrite=true"

--fail-with-body makes curl exit with a non-zero status on any HTTP error (4xx/5xx) and still print the response body, so your CI log shows exactly what went wrong.

Important

The three identifiers in the URL (${UBTRACE_ORG}/${UBTRACE_PROJECT}/${UBTRACE_VERSION}) must match the values the Sphinx build produces. The builder takes its output path from ubtrace_organization, ubtrace_project, and ubtrace_version in conf.py. The simplest way to keep them in sync is to read the version from an environment variable in conf.py:

# conf.py
import os
ubtrace_organization = "mycompany"
ubtrace_project = "my-product"
ubtrace_version = os.environ.get("UBTRACE_VERSION", "1")

If the archive also contains config/ubtrace_project.toml (which ubt_sphinx produces automatically), its organization, project_id, and version fields must equal the URL parameters – otherwise the upload is rejected with 400. See TOML identity check below.

Step 4 – Manual Upload with curl

You can run the same upload from your laptop whenever you need to publish a version by hand (for example during testing or a one-off migration):

# Replace the placeholders with your own values first.
export UBTRACE_URL=https://ubtrace.example.com/api
export UBTRACE_INGEST_TOKEN=ubt_your_token_here
export UBTRACE_ORG=mycompany
export UBTRACE_PROJECT=my-product
export UBTRACE_VERSION=v1

# Build the docs. ubt_sphinx writes its output under
#   docs/_build/ubtrace/<ubtrace_organization>/<ubtrace_project>/<ubtrace_version>/
# so make sure those values in conf.py match UBTRACE_ORG/PROJECT/VERSION above.
sphinx-build -b ubtrace docs docs/_build/ubtrace

# Package the version directory (the one that directly contains docs/ubtrace/needs.json):
BUILD_DIR="docs/_build/ubtrace/${UBTRACE_ORG}/${UBTRACE_PROJECT}/${UBTRACE_VERSION}"
tar -czf /tmp/build.tar.gz -C "${BUILD_DIR}" .

# Upload it:
curl --fail-with-body --show-error \
  -H "Authorization: Bearer ${UBTRACE_INGEST_TOKEN}" \
  -F "file=@/tmp/build.tar.gz" \
  "${UBTRACE_URL}/v1/ingest/${UBTRACE_ORG}/${UBTRACE_PROJECT}/${UBTRACE_VERSION}"

The overwrite Flag

By default, ubTrace rejects an upload for a version that already exists (HTTP 409 Conflict). This stops you from accidentally replacing a published release.

Append ?overwrite=true to the URL to allow replacement:

POST /v1/ingest/mycompany/my-product/v1?overwrite=true

When to use it:

  • Recommended: pipelines that publish a moving pointer such as main or latest – you want every push to replace the previous build.

  • Recommended: the ${GITHUB_SHA::7} pattern from the workflow above – the same commit should always produce the same artifact.

  • Not recommended: signed releases (v1.2.3). Treat them as immutable – publish each release exactly once.

Archive Layout

ubTrace accepts two archive formats:

  • tar.gz (recommended, produced by tar -czf)

  • zip

The format is detected automatically from the file’s magic bytes. If you prefer to be explicit you can pass ?format=tar.gz or ?format=zip.

The archive must contain the ubt_sphinx output of one version at its root – that is, docs/ubtrace/needs.json must sit directly inside the archive, not nested in extra parent folders:

build.tar.gz
├── docs/
│   └── ubtrace/
│       ├── needs.json     ← required
│       └── *.fjson        ← page fragments
└── config/
    └── ubtrace_project.toml   ← optional but recommended

ubt_sphinx writes this exact structure under _build/ubtrace/<org>/<project>/<version>/. The simplest way to get the right archive layout is to tar from inside that version directory – see the BUILD_DIR line in the workflow above.

TOML identity check

If the archive contains config/ubtrace_project.toml (ubt_sphinx produces this file by default), ubTrace validates each field that is set in it:

  • organization must equal the organizationId in the URL.

  • project_id must equal the projectId in the URL.

  • version must equal the versionId in the URL.

A mismatch returns 400 Bad Request with a message listing which fields differ. Fields that are not set in the TOML file are not checked. If the TOML file itself is absent from the archive, no identity check is performed – but we recommend shipping it, because it catches “wrong identifier” mistakes before they corrupt the database.

Maximum archive size

  • Default limit: 500 MB per upload.

  • Configurable per instance via the INGEST_MAX_FILE_SIZE environment variable (value in bytes). See Installation for where to set it.

  • The absolute hard cap enforced by the upload library is 1 GB. Archives larger than that are rejected before they reach the application layer.

If your archive is close to the limit, first check whether _images or generated artifacts got included by accident. A typical ubt_sphinx output for a large multi-project documentation is well under 100 MB.

Identifiers (organizationId, projectId, versionId)

The three segments in the URL POST /v1/ingest/<organizationId>/<projectId>/<versionId> name the location of the artifact inside ubTrace and map 1-to-1 to folder names in input_build/.

Allowed characters

  • organizationId and projectId: letters (a-z, A-Z), digits (0-9), underscore (_), and hyphen (-).

  • versionId: the same set plus dot (.) – so v1.2.3 and 1.0 are valid version identifiers.

Length

  • 1 to 255 characters per segment.

Consistency with the Sphinx build

  • The three identifiers should match ubtrace_organization, ubtrace_project, and ubtrace_version in your conf.py so that the folder the builder writes to matches the URL you upload to.

  • If the archive contains config/ubtrace_project.toml, mismatched fields cause a 400 Bad Request. See TOML identity check.

Choosing a ``versionId``

  • Free-form within the character set above. Common patterns: v1, 1.2.3, main, latest, or the short commit SHA (${GITHUB_SHA::7}).

Error Responses

Every error response is JSON and includes a message field with the reason.

HTTP

Meaning

What to check

401 Unauthorized

The token is missing, malformed, or the Authorization header is wrong.

Confirm the header reads Authorization: Bearer ubt_... (note the Bearer prefix and the single space). Re-check the secret value.

403 Forbidden

The token is valid but does not have the ingest scope, has been revoked, or has expired.

Create a fresh token in the admin panel and update the CI secret.

400 Bad Request

The archive layout is wrong (docs/ubtrace/needs.json missing), the archive format cannot be detected, or an identifier contains illegal characters.

Unpack the archive locally with tar -tzf build.tar.gz | head – the first entries should include docs/ubtrace/needs.json. Also double-check your organization/project/version identifiers.

409 Conflict

The version already exists and overwrite=true was not set.

Add ?overwrite=true if replacement is intentional, or publish under a new version ID.

413 Payload Too Large

The archive is larger than INGEST_MAX_FILE_SIZE (default 500 MB).

Trim unused assets from the Sphinx output or raise the limit in the ubTrace deployment.

429 Too Many Requests

The CI pipeline is uploading faster than the rate limiter allows.

Retry with exponential backoff. A single production pipeline should never hit this in practice.

5xx

ubTrace is unhealthy.

Check make status on the server and the ubtrace-api container logs.

Step 5 – Verify the Upload

After the curl call returns 201 Created, the worker scans the input_build/ folder (default: every 30 seconds) and imports the new version. Import usually finishes within a minute for a typical project.

Verify in the UI:

  1. Open the ubTrace web app and sign in.

  2. Navigate to your organization → project.

  3. The new version should appear in the version switcher at the top of the page.

Verify via the API – useful for ad-hoc inspection from a developer machine:

# Lists all versions known for the project. Your new version should be in the list.
curl -H "Authorization: Bearer ${UBTRACE_USER_TOKEN}" \
  "${UBTRACE_URL}/v1/artifacts/organizations/${UBTRACE_ORG}/projects/${UBTRACE_PROJECT}/versions"

Note

The list-versions endpoint uses user authentication (Keycloak/OIDC JWT), not the ingest API token. UBTRACE_USER_TOKEN here is a JWT access token obtained by signing in as a normal user, not the ubt_... token from Step 1. The ingest scope only grants upload access – it cannot read artifacts. For CI pipelines, treat the 201 Created response from /v1/ingest as confirmation that the archive was accepted; use the UI path above for end-to-end verification.

If the version is accepted by /v1/ingest but never appears in the list, check the ub-worker container logs – it will report the reason (for example a malformed needs.json).

Revoking a Token

If a token is leaked, or a CI pipeline is retired:

  1. Open Settings → API Tokens in the admin panel.

  2. Find the token by its name or prefix.

  3. Click Revoke.

Revocation is immediate. The next request with that token returns 403 Forbidden. Deleting and re-creating a token in your CI secret store rotates it without any server change.