# kgpu — GPU rentals for researchers

`https://api.kgpu.net/v1` — rent a GPU, run your code, pull your results.

kgpu is a **compute provider**: you rent a GPU-backed container by the
minute, get a shell (or one-shot exec) into it, and move files in and out.
We don't host your data — the container's disk is **ephemeral** and is
destroyed when the rental ends, so copy results out before you release.

Two ways to drive it:

- **HTTPS REST** under `/v1` — rent / inspect / release pods and run
  one-shot commands. Plain `curl` + `jq` (or any HTTP client).
- **SSH bastion** at `ssh.kgpu.net:2222` — an interactive shell into your
  pod plus native `scp` / `rsync` / `sshfs` for file transfer.

---

## Quick start

```bash
# 1. Sign in at https://kgpu.net and copy your token from the dashboard.
export KGPU_API_TOKEN=kgpu_xxxxxxxxxxxxx
export KGPU_API_BASE=https://api.kgpu.net/v1
T=$KGPU_API_TOKEN; B=$KGPU_API_BASE

# 2. See what's available (no auth needed) and rent one.
curl -sS $B/gpu-models | jq
RESP=$(curl -sS -X POST $B/pods -H "Authorization: Bearer $T" \
  -H "Content-Type: application/json" -d '{"gpu_model":"rtx4090"}')
RENT=$(echo "$RESP" | jq -r .pod_id)
# POST mints your SSH key — save the private half (shown once). Keep rental
# keys in their own dir so ~/.ssh doesn't fill up with dead per-pod keys.
mkdir -p ~/.ssh/kgpu; K=~/.ssh/kgpu/$RENT
echo "$RESP" | jq -r .ssh_access.private_key > "$K" && chmod 600 "$K"

# 3. Wait until it's ready (cold start is mostly image load, ~10-60s).
until [ "$(curl -sS $B/pods/$RENT -H "Authorization: Bearer $T" | jq -r .ready)" = "true" ]; do
  sleep 10
done

# 4. Use it — shell in over the bastion with the key POST issued (step 2, $K).
ssh -i "$K" -p 2222 kgpu@ssh.kgpu.net nvidia-smi
scp -O -i "$K" -P 2222 mydata.zip kgpu@ssh.kgpu.net:/workspace/   # push data
ssh -i "$K" -p 2222 kgpu@ssh.kgpu.net 'setsid -f python /workspace/train.py < /dev/null > /tmp/train.log 2>&1'
scp -O -i "$K" -P 2222 kgpu@ssh.kgpu.net:/workspace/model.pt ./   # pull results

# 5. Release (pod + disk destroyed) and drop the now-dead key (it dies with the rental).
curl -sS -X DELETE $B/pods/$RENT -H "Authorization: Bearer $T" && rm -f "$K"
```

Every authenticated request carries `Authorization: Bearer $KGPU_API_TOKEN`.

**New accounts start with trial credits.** Maintainers can top up. Pricing
is per-minute; see [Pricing](#pricing).

---

## Tokens

- **Account token** — copy it from the dashboard at <https://kgpu.net>.
  It's your credential for everything. One active token per account: the
  dashboard's **Reissue** button revokes the old one and mints a fresh one
  (shown once — copy it then). We store only a salted hash, so a lost token
  can't be recovered, only reissued.
- **In-pod token** — inside a rented container, `$KGPU_API_TOKEN` and
  `$KGPU_API_BASE` are auto-injected, but the in-pod token is a
  **per-rental, scope-limited** bearer (`kgpu_pod_<id>.<sig>`), *not* your
  account token. It can read `/me`, read/release its own pod, and use
  `/objects` for the owning account. It **cannot** rent more pods, touch
  other pods, rotate its SSH key, or exec into itself. It dies when the
  rental ends. Use your account token for everything else.

---

## API reference

| Resource | Method · Path | What |
|---|---|---|
| **gpu-models** | `GET /gpu-models` | **public** — GPU classes available now, with `free`/`total`/price. No auth. |
| **me** | `GET /me` | credits, quota, role |
| **balance** | `GET /balance` | just the credit balance (pre-flight before renting) |
| **cluster** | `GET /cluster` | per-node availability (auth'd) |
| **pods** | `GET /pods` | your pods, newest first |
| | `POST /pods` | rent (body below; issues an SSH key) |
| | `GET /pods/{id}` | pod state — phase, ready, idle clock, end_reason |
| | `PATCH /pods/{id}` | re-issue (rotate) the rental's SSH key |
| | `DELETE /pods/{id}` | release |
| | `POST /pods/{id}/exec` | one-shot command (synchronous) |
| **objects** | `POST /objects` | mint a 24 h transit object (presigned PUT, ≤5 GB) |
| | `POST /objects/multipart` · `…/{id}/multipart/complete` | multipart upload (>5 GB, to ~5 TB) |
| | `GET /objects` · `GET /objects/{id}` | list / fetch (presigned GET) |
| | `DELETE /objects/{id}` | delete (idempotent) |

All paths are relative to `$KGPU_API_BASE`. Plus the SSH bastion at
`ssh.kgpu.net:2222`.

**Machine-readable spec:** the full OpenAPI 3.1 schema is served at
[`/openapi.json`](/openapi.json) — point your client/codegen at it. (A
human-browsable version is at [`/docs`](/docs).)

`GET /me` returns `{user_id, email, name, role, status, credits, quota}`.

---

## GPU models & pricing

`GET /v1/gpu-models` is **public** (no Bearer) so you can check the menu
before signing in. It aggregates by `gpu_model` and is the source of truth
for what's rentable *right now*:

```bash
$ curl -sS https://api.kgpu.net/v1/gpu-models | jq
[
  {"gpu_model":"rtx3080","total":1,"free":1,"price_per_hour_credits":300},
  {"gpu_model":"rtx3090","total":2,"free":2,"price_per_hour_credits":600},
  {"gpu_model":"rtx4090","total":1,"free":0,"price_per_hour_credits":1000},
  {"gpu_model":"rtx5090","total":1,"free":1,"price_per_hour_credits":1400},
  {"gpu_model":"v100",   "total":4,"free":4,"price_per_hour_credits":600}
]
```

Use the returned `gpu_model` strings as the `gpu_model` value on
`POST /pods`. For per-node detail (hostname, busy count) use the auth'd
`GET /cluster`.

### Pricing

`$1 = 1500 credits`. Per-minute deduction; balance hitting zero
auto-releases the pod. N-GPU rentals bill at N × the per-GPU rate.

| `gpu_model` | hardware | credits/hr | ≈ $/hr |
|---|---|---|---|
| `rtx3080` | RTX 3080 | 300 | 0.20 |
| `rtx3090` | RTX 3090 | 600 | 0.40 |
| `v100` | Tesla V100 32 GB | 600 | 0.40 |
| `rtx4090` | RTX 4090 | 1 000 | 0.67 |
| `rtx5090` | RTX 5090 | 1 400 | 0.93 |

More classes may come online — `GET /gpu-models` always reflects the live
fleet and prices.

---

## LLM inference — OpenAI-compatible

Besides renting GPUs, kgpu serves open models behind an **OpenAI-compatible**
API. Point any OpenAI client at `$KGPU_API_BASE` and authenticate with the
**same `KGPU_API_TOKEN`** you use for rentals — no separate key:

```python
from openai import OpenAI
client = OpenAI(base_url="https://api.kgpu.net/v1", api_key="$KGPU_API_TOKEN")
r = client.chat.completions.create(
    model="qwen3.5-122b",
    messages=[{"role": "user", "content": "Hello"}],
)
```

```bash
curl -sS https://api.kgpu.net/v1/chat/completions \
  -H "Authorization: Bearer $KGPU_API_TOKEN" -H "Content-Type: application/json" \
  -d '{"model":"qwen3.5-122b","messages":[{"role":"user","content":"Hello"}]}'
```

Standard OpenAI routes: `/v1/chat/completions`, `/v1/completions`,
`/v1/models`. `stream: true` (SSE) is supported. **`GET /v1/models` is the
source of truth** for what's servable right now — the table below is a
snapshot.

| model | quantization | context | credits / 1M in | credits / 1M out |
|---|---|---|---|---|
| `qwen3.5-122b` | NVFP4 | 32K | 390 | 3120 |
| `gpt-oss-120b` | MXFP4 | 32K | 59 | 270 |

More models are rolling out (small fast models on idle rental GPUs, more big
models on the Spark nodes); check `GET /v1/models`.

**Billing.** Pay-as-you-go from the same credit balance as rentals. Each call
is metered from the response's `usage` and charged
`prompt_tokens × in + completion_tokens × out` (rates above, per 1M tokens),
rounded up to a whole credit — so a non-empty call costs at least 1 credit.
Rates track [OpenRouter](https://openrouter.ai)'s $/token at **$1 = 1500
credits**. With `stream: true` the charge still lands (usage is captured on the
final chunk). A request made with a zero balance is refused with **402
`insufficient_credits`**.

---

## Renting a pod — `POST /pods`

All body fields are optional; defaults shown.

| Field | Default | Notes |
|---|---|---|
| `name` | `"rent"` | human label — lowercase `a-z` / `0-9` / hyphens, no leading/trailing hyphen, ≤ 30 chars (it becomes part of the pod's DNS-1123 name; the gateway also normalizes it) |
| `image` | `kgpu-pytorch:latest` | the default image is preloaded on every worker (no pull wait). Use a `ghcr.io/...` / `docker.io/...` ref for anything else. |
| `gpu_model` | `"rtx4090"` | live menu via `GET /gpu-models` |
| `gpu_count` | `1` | 0–4 (0 = CPU-only pod, still bills at the class rate) |
| `cpu` | `"2"` | CPU cores, ≤ 16 |
| `memory` | `"16Gi"` | RAM, ≤ 128 GiB |
| `ephemeral_storage_gib` | `100` | scratch disk, 1–2000 GiB |
| `env` | `null` | extra pod env vars |

Returns:

```jsonc
{
  "pod_id": "gpu-rent-1779745883-ab12cd",
  "gpu_model": "rtx4090",
  "price_per_hour_credits": 1000,
  "balance_after_precheck": 99000
}
```

### SSH access — the gateway issues your key

kgpu is **token-first**: it no longer accepts user-supplied public keys.
`POST /pods` mints a fresh per-rental ed25519 keypair and returns the
private half **once** in `ssh_access` (we never persist it):

```jsonc
{ "ssh_access": {
    "private_key": "-----BEGIN OPENSSH PRIVATE KEY-----\n...\n",
    "fingerprint": "SHA256:abc123...",
    "ssh_host": "ssh.kgpu.net", "ssh_port": 2222, "ssh_user": "kgpu" } }
```

Save it and `ssh -i <pem> -p 2222 kgpu@ssh.kgpu.net`. Lost it (or want to
rotate)? `PATCH /v1/pods/{id}` (empty body) **re-issues** a fresh keypair
and returns its private half — the old key dies the instant the new
fingerprint lands. (`PATCH` is account-token only — a compromised pod
can't rotate its key and lock you out.) Prefer not to manage a `.pem` at
all? Use `POST /pods/{id}/exec` (token-only one-shot commands).

**Key hygiene.** A per-rental key is useless the moment the pod is released,
so don't let them pile up in `~/.ssh`. Keep them in a dedicated dir and delete
on release (as the quick-start does):

```bash
mkdir -p ~/.ssh/kgpu; K=~/.ssh/kgpu/$RENT   # one dir for all rentals
# ... use $K ...
curl -sS -X DELETE $B/pods/$RENT -H "Authorization: Bearer $T" && rm -f "$K"
# sweep leftovers from crashed runs (all pods you no longer hold):
#   for f in ~/.ssh/kgpu/*; do id=$(basename "$f"); \
#     [ "$(curl -sS $B/pods/$id -H "Authorization: Bearer $T" | jq -r .phase)" = Ended ] && rm -f "$f"; done
```

For throwaway one-shots, skip the file entirely with an ephemeral key:
`K=$(mktemp); trap 'rm -f "$K"' EXIT` — it's gone when your script exits.

---

## Getting into the pod

### One-shot command — `POST /pods/{id}/exec`

Synchronous, no setup, no SSH key. Ideal for CI, dashboards, quick checks.

```bash
curl -sS -X POST $B/pods/$RENT/exec -H "Authorization: Bearer $T" \
     -H "Content-Type: application/json" \
     -d '{"cmd":["nvidia-smi"]}' | jq
# { "exit_code": 0, "stdout": "...", "stderr": "", "duration_ms": 1240, "truncated": false }
```

| Field | Default | Notes |
|---|---|---|
| `cmd` | required | argv list, e.g. `["bash","-c","ls /workspace"]` |
| `stdin` | `null` | piped to the command, ≤ 1 MiB |
| `timeout_seconds` | `60` | 1–300; killed on timeout, partial output returned |

Caps: combined stdout+stderr ≤ **1 MiB** (excess truncated, `truncated:true`);
timeout ≤ **300 s**; no PTY. Bigger output or longer jobs → use SSH.
Returns `409 pod_not_ready` if the pod isn't ready yet.

### Interactive shell + file transfer — SSH bastion

`ssh.kgpu.net:2222`, user always `kgpu`, authenticated by the key pinned to
your rental. The bastion looks up your key's fingerprint and routes you to
**exactly that one pod** — you can't land in someone else's.

```bash
ssh -p 2222 kgpu@ssh.kgpu.net              # interactive shell
ssh -p 2222 kgpu@ssh.kgpu.net nvidia-smi   # one-shot
```

A `~/.ssh/config` block per pod is handy:

```
Host kgpu-myrun
    HostName ssh.kgpu.net
    Port 2222
    User kgpu
    IdentityFile ~/.ssh/id_ed25519
    IdentitiesOnly yes
```

Locked down by design: **no port forwarding** (`-L`/`-R`/`-D`), no agent
or X11 forwarding, no tunnels. Don't enable `ControlMaster` — the session
wrapper is single-channel and multiplexing breaks it. The key dies the
moment the rental ends (`Permission denied (publickey)` thereafter).

---

## Moving files in and out

The pod disk is ephemeral — **pull your artifacts before releasing.**

### scp / rsync (over the bastion)

```bash
scp -O -P 2222 mit-bih.zip kgpu@ssh.kgpu.net:/tmp/             # push
rsync -av -e "ssh -p 2222" kgpu@ssh.kgpu.net:/workspace/out/ ./out/   # pull
```

`scp -O` (capital O) forces the legacy SCP protocol the bastion speaks —
without it OpenSSH 9.x tries SFTP and fails with `subsystem request
failed`. Transfers run over a direct byte stream, so they're reliable for
large files.

### Live mount — sshfs

Edit pod files with local tools (your editor, an LLM CLI, linters):

```bash
sshfs kgpu@ssh.kgpu.net:/workspace ./mnt -p 2222 \
  -o IdentityFile=~/.ssh/id_ed25519 \
  -o sftp_server=/usr/lib/openssh/sftp-server \
  -o reconnect,ServerAliveInterval=15
# ...edit ./mnt — changes hit the pod immediately...
fusermount -u ./mnt    # Linux   (umount on macOS)
```

The `-o sftp_server=...` flag is **required**. On Windows install
**WinFsp + SSHFS-Win** once (`winget install WinFsp.WinFsp
SSHFS-Win.SSHFS-Win`) and use a Windows-native `IdentityFile` path
(`C:/Users/<you>/.ssh/...`). The default image ships
`/usr/lib/openssh/sftp-server`; custom images need
`apt install openssh-sftp-server`.

### Large transfers & multi-pod fanout — transit objects (`/v1/objects`)

For anything over a few hundred MB, or the same dataset to many pods, push
once to Cloudflare R2 and pull directly in each pod — bytes go client → R2
→ pod over HTTPS, no bastion in the path.

```bash
# mint (returns a presigned PUT, valid 1 h) and upload once
RESP=$(curl -sS -X POST $B/objects -H "Authorization: Bearer $T" \
  -H "Content-Type: application/json" -d '{"label":"mit-bih"}')
OBJ=$(echo "$RESP" | jq -r .object_id)
curl -X PUT --upload-file mit-bih.zip "$(echo "$RESP" | jq -r .put_url)"

# fetch in each pod via a fresh GET url
GET=$(curl -sS -H "Authorization: Bearer $T" $B/objects/$OBJ | jq -r .get_url)
for POD in podA podB podC; do ssh kgpu-$POD "curl -L -o /tmp/data.zip '$GET'" & done; wait

curl -X DELETE -H "Authorization: Bearer $T" $B/objects/$OBJ   # tidy up
```

**A single PUT is capped at 5 GB by R2.** For larger objects (checkpoints,
datasets) use **multipart**:

1. `POST /objects/multipart` with `{"size_bytes": <total>}` → returns
   `object_id`, `part_size`, and `part_urls` (one presigned PUT per part).
2. PUT chunk *i* (`part_size` bytes; the last part is the remainder) to
   `part_urls[i-1]`, and read the `ETag` response header off each PUT.
3. `POST /objects/{id}/multipart/complete` with
   `{"parts":[{"part_number":i,"etag":"<etag>"}, ...]}`.

R2 stitches the parts into one object (up to ~5 TB); GET / fanout exactly as
above. `POST /objects/{id}/multipart/abort` cancels an unfinished upload.

**This is scratch, not storage.** Objects auto-expire **24 h** after
creation (`expires_at` is absolute). Per-user quota 100 GB live; per-object
cap 50 GB. PHI / patient data must not outlive the active rental.

Traps: use `--upload-file <path>` (not `-d @-`) so curl sends a
Content-Length; don't add custom headers to the presigned URL (breaks the
signature); the `get_url` is reusable by N pods within its 1 h window.

---

## Inside the container

The default `kgpu-pytorch:latest` ships PyTorch (CUDA 12.9, nv25.05) with
prebuilt kernels for **sm_75 / 80 / 86 / 89 / 90 / 100 / 120** — every GPU
in the fleet, including Blackwell (RTX 5090 / B200). Also bundled: `wfdb`,
`vitaldb`, `scipy`, `scikit-learn`, `pandas`, `matplotlib`, `seaborn`,
`duckdb`, `pyarrow`, `tmux`, `uv`, `zstd`, `openssh-sftp-server`.

Both `python` and `python3` run the bundled interpreter. `exec` commands
start in **`/workspace`** (the image's `WORKDIR`); bastion SSH sessions land
in `/` (the shell is spawned non-login via the pod namespace), so reference
**absolute `/workspace` paths** — as every example here does. `/workspace` is
scratch on the pod's ephemeral disk — put your code + checkpoints there;
`KGPU_*` env and the bundled tools are on `PATH`.

Auto-injected env:

| Var | Value |
|---|---|
| `KGPU_API_TOKEN` | per-rental scoped bearer (not your account token). Stripped from interactive SSH sessions; read it from `/proc/1/environ` if a script needs it. |
| `KGPU_API_BASE` | `https://api.kgpu.net/v1` |
| `KGPU_GPU_ID` / `KGPU_RENT_ID` | this rental's id |
| `NVIDIA_VISIBLE_DEVICES` | `void` — CDI handles GPU allocation; don't override |

Inside the container `nvidia-smi -L` and `/dev/nvidia*` expose only your
rental's GPU(s).

### Detaching long-running jobs

`nohup ... &` doesn't fully release the SSH session (OpenSSH waits on the
job's fds). Use `setsid -f`, which forks a new session and closes the
parent fds so the `ssh` call returns in milliseconds:

```bash
ssh kgpu-myrun 'setsid -f python /workspace/train.py < /dev/null > /tmp/train.log 2>&1'
ssh kgpu-myrun 'tail -f /tmp/train.log'   # check progress any time
```

`tmux new -d -s train 'python ...'` is the other safe option.

---

## Limits

- **GPU cap**: 4 concurrent GPUs across all your rentals.
- **Per-rental**: ≤ 16 CPU cores, ≤ 128 GiB RAM, 1–2000 GiB ephemeral disk.
- **Idle**: 12 h of no SSH activity → `idle_warning`; 24 h → auto-release
  (`auto_idle`). **Override**: past the warn threshold we probe pod load —
  any GPU > 5 % or process > 10 % CPU resets the clock, so a long training
  run with no API calls is *not* reaped.
- **Wall clock**: no hard ceiling. Rentals live until `DELETE`, idle-kill,
  or out-of-credits.
- **Disk eviction**: exceeding `ephemeral_storage_gib` terminates the pod
  (`evicted_by_kubelet:OutOfephemeral-storage`). Checkpoints, pip cache,
  dataset extracts and `torch.compile` cache all add up — bump the value or
  rotate checkpoints for multi-epoch runs.
- **Network isolation**: **no ingress** (pods accept nothing inbound).
  **Egress** to the public internet works (pip/git/HF) but is blocked into
  private ranges (RFC1918, link-local/metadata `169.254`, Tailscale CGNAT)
  and to other pods. No lateral scanning, no pod-to-pod.
- **Credits**: per-minute deduction; zero → auto-release
  (`out_of_credits`).

---

## Pod lifecycle

`GET /v1/pods/{id}.phase`:

| `phase` | meaning |
|---|---|
| `Pending` | scheduling / image load / container creating. Carries `pending_reason` (e.g. `starting`, `ImagePullBackOff: …`). `ready=false`. |
| `Running` | container up. `ready=true` once the readiness probe passes. |
| `Terminating` | released and tearing down; `ended_at` / `end_reason` already set. |
| `Ended` | pod object gone; the historical row remains so you can still read `end_reason`. |
| `Failed` | rare — kubelet failed it; see `end_reason`. |

**Polling**: poll `GET /pods/{id}` every 10–15 s only until `ready=true`,
then stop — the cold start is dominated by image load, sub-5 s polling buys
nothing. For training liveness, `ssh ... tail -f /tmp/train.log` or
`nvidia-smi` rather than hammering the API.

**Cost so far**: every `GET /pods/{id}` carries `credits_charged` (credits
spent on this rental to date), `price_per_hour_credits`, and `started_at`
— so per-pod spend and runtime (`now − started_at`) are visible at a glance,
including on the historical row after the pod is released.

`GET /pods/{id}` works even after the pod is gone (falls back to the
historical row), so you can always read why it ended:

| `end_reason` | meaning |
|---|---|
| `returned` | you called `DELETE` |
| `auto_idle` | idle timeout |
| `out_of_credits` | balance hit zero |
| `evicted_by_kubelet:OutOfephemeral-storage` | disk exceeded — bump it or rotate checkpoints |
| `evicted_by_kubelet:OOMKilled` | memory limit hit |
| `start_failed:<reason>` | never escaped Pending (bad image tag, etc.); auto-released ~5 min after creation, quota refunded |

---

## Windows / PowerShell

PowerShell's `curl` is an alias for `Invoke-WebRequest` and mangles JSON
bodies. Prefer **`Invoke-RestMethod`**:

```powershell
$T = $env:KGPU_API_TOKEN
$B = "https://api.kgpu.net/v1"
$H = @{ Authorization = "Bearer $T" }

$rent = Invoke-RestMethod -Method Post -Uri "$B/pods" -Headers $H `
    -ContentType "application/json" -Body (@{ gpu_model = "rtx4090" } | ConvertTo-Json)
$RENT = $rent.pod_id
# POST mints your SSH key — save it in its own dir (keeps ~/.ssh tidy).
$null = New-Item -ItemType Directory -Force "$HOME\.ssh\kgpu"
$K = "$HOME\.ssh\kgpu\$RENT"; $rent.ssh_access.private_key | Out-File -Encoding ascii $K

do { Start-Sleep 10; $p = Invoke-RestMethod -Uri "$B/pods/$RENT" -Headers $H } until ($p.ready)

ssh -i $K -p 2222 kgpu@ssh.kgpu.net nvidia-smi
# release + drop the now-dead key
Invoke-RestMethod -Method Delete -Uri "$B/pods/$RENT" -Headers $H; Remove-Item $K
```

Or call the real `curl.exe` with a single-quoted here-string to copy the
bash examples verbatim (`@'...'@`, closing `'@` at column 0). `jq` works
the same once installed (`winget install jqlang.jq`).

---

## Status codes & errors

`200` ok · `400` bad request · `401` invalid/revoked token or account not
approved · `402` insufficient credits · `403` forbidden scope (in-pod token
tried to rent/exec/rotate) · `404` not found · `409` not ready / quota
exceeded / duplicate · `410` object expired · `422` validation failed ·
`500` server error.

Bodies carry a machine-readable `error` **code** + `message` (auth /
middleware errors at the top level — `{"error":"token_revoked","message":…}`;
endpoint errors nest the same under `detail`). Codes worth handling:
**`token_revoked`** — you reissued your token in the dashboard, so the old
one is dead **immediately, no grace period**; a stale cached token 401s with
this, so re-read `KGPU_API_TOKEN` from the dashboard / env on a 401 instead
of guessing. Others: `invalid_token`, `missing_token`, `forbidden_scope`,
`insufficient_credits`, `object_quota_exceeded`.

Every response carries `X-Request-ID` (8-byte hex); 5xx bodies echo it as
`trace_id`. Quote it when filing a bug.