# kgpu — GPU rentals for researchers

`https://api.kgpu.net/v1` — rent a GPU, run your code, pull your results.

kgpu is a **compute provider**: you rent a GPU-backed container by the
minute, run commands in it via one-shot exec, and move files in and out
through your persistent `~/files` mount. We don't host ephemeral data —
the container's own disk is destroyed when the rental ends, so anything
that should survive belongs in `~/files` (or `/v1/files`) before you
release.

One way to drive it: **HTTPS REST** under `/v1` — rent / inspect /
release pods and run one-shot commands. Plain `curl` + `jq` (or any HTTP
client).

---

## Quick start

```bash
# 1. Sign in at https://kgpu.net and copy your token from the dashboard.
export KGPU_API_TOKEN=kgpu_xxxxxxxxxxxxx
export KGPU_API_BASE=https://api.kgpu.net/v1
T=$KGPU_API_TOKEN; B=$KGPU_API_BASE

# 2. See what's available (no auth needed) and rent one.
curl -sS $B/gpu-models | jq
RESP=$(curl -sS -X POST $B/pods -H "Authorization: Bearer $T" \
  -H "Content-Type: application/json" -d '{"gpu_model":"rtx4090"}')
RENT=$(echo "$RESP" | jq -r .pod_id)

# 3. Wait until it's ready (cold start is mostly image load, ~10-60s).
until [ "$(curl -sS $B/pods/$RENT -H "Authorization: Bearer $T" | jq -r .ready)" = "true" ]; do
  sleep 10
done

# 4. Use it — exec runs commands, ~/files (also GET/PUT /v1/files) moves data.
curl -sS -X PUT $B/files/mydata.zip --data-binary @mydata.zip -H "Authorization: Bearer $T"
RUN=$(curl -sS -X POST $B/pods/$RENT/exec -H "Authorization: Bearer $T" \
  -H "Content-Type: application/json" \
  -d '{"cmd":["python","/root/files/train.py"],"background":true}')
LOG=$(echo "$RUN" | jq -r .log_path)   # returns immediately — training runs in the background
curl -sS -X POST $B/pods/$RENT/exec -H "Authorization: Bearer $T" \
  -H "Content-Type: application/json" -d "{\"cmd\":[\"tail\",\"-20\",\"$LOG\"]}"   # check progress any time
curl -sS $B/files/model.pt -H "Authorization: Bearer $T" -o model.pt   # pull results

# 5. Release (pod + ephemeral disk destroyed; ~/files survives — it's yours).
curl -sS -X DELETE $B/pods/$RENT -H "Authorization: Bearer $T"
```

Every authenticated request carries `Authorization: Bearer $KGPU_API_TOKEN`.

**New accounts start with trial credits.** Maintainers can top up. Pricing
is per-minute; see [Pricing](#pricing).

---

## Tokens

- **Account tokens** — issue them from the dashboard at <https://kgpu.net>.
  You can hold several at once, each independently revocable, and each
  optionally **scoped**: a resource (`all` / `pods` / `files`) × an access
  level (**read/write** or **read-only**, where write implies read). E.g. a
  `files: read-only` token can list/download your files but can't upload,
  rent pods, or exec. Each token is shown once at issue time — copy it then;
  we store only a salted hash.
- **In-pod token** — inside a rented container, `$KGPU_API_TOKEN` and
  `$KGPU_API_BASE` are auto-injected, but the in-pod token is a
  **per-rental, scope-limited** bearer (`kgpu_pod_<id>.<sig>`), *not* your
  account token. It can read `/me` and read/release its own pod. It **cannot**
  rent more pods, touch other pods, or exec into itself. It dies when the
  rental ends. Your persistent files are already mounted at
  `~/files` — use your account token for the `/v1/files` API and everything else.

---

## API reference

| Resource | Method · Path | What |
|---|---|---|
| **gpu-models** | `GET /gpu-models` | **public** — GPU classes available now, with `free`/`total`/price. No auth. |
| **me** | `GET /me` | credits, quota, role, and your `storage` block (files quota/usage/cost) |
| **balance** | `GET /balance` | just the credit balance (pre-flight before renting) |
| **cluster** | `GET /cluster` | per-node availability (auth'd) |
| **pods** | `GET /pods` | your pods, newest first |
| | `POST /pods` | rent (body below) |
| | `GET /pods/{id}` | pod state — phase, ready, idle clock, end_reason |
| | `DELETE /pods/{id}` | release |
| | `POST /pods/{id}/exec` | one-shot command (synchronous) |
| **files** | `GET /files[/{path}]` | list a dir, or download a file |
| | `PUT /files/{path}` | upload/overwrite (whole-file, or resumable via `Content-Range`) |
| | `DELETE /files/{path}` | delete a file or dir (`?recursive=true`) |

All paths are relative to `$KGPU_API_BASE`.

**Machine-readable spec:** the full OpenAPI 3.1 schema is served at
[`/openapi.json`](/openapi.json) — point your client/codegen at it. (A
human-browsable version is at [`/docs`](/docs).)

`GET /me` returns `{user_id, email, name, role, status, credits, quota,
login_name, storage}`. `login_name` is your canonical SFTP/WebDAV username
(also used for VM accounts). The `storage` block: `{allocated, quota_gb, used_bytes, used_gb,
mount_path, files_api, webdav_url, price_per_gb_month_credits,
monthly_cost_estimate_credits}` (usage refreshed ~once a minute).

---

## GPU models & pricing

`GET /v1/gpu-models` is **public** (no Bearer) so you can check the menu
before signing in. It aggregates by `gpu_model` and is the source of truth
for what's rentable *right now*:

```bash
$ curl -sS https://api.kgpu.net/v1/gpu-models | jq
[
  {"gpu_model":"rtx3080","total":1,"free":1,"price_per_hour_credits":300},
  {"gpu_model":"rtx3090","total":2,"free":2,"price_per_hour_credits":600},
  {"gpu_model":"rtx4090","total":1,"free":0,"price_per_hour_credits":1000},
  {"gpu_model":"rtx5090","total":1,"free":1,"price_per_hour_credits":1400},
  {"gpu_model":"v100",   "total":4,"free":4,"price_per_hour_credits":600}
]
```

Use the returned `gpu_model` strings as the `gpu_model` value on
`POST /pods`. For per-node detail (hostname, busy count) use the auth'd
`GET /cluster`.

### Pricing

`$1 = 1500 credits`. Per-minute deduction; balance hitting zero
auto-releases the pod. N-GPU rentals bill at N × the per-GPU rate.

| `gpu_model` | hardware | credits/hr | ≈ $/hr |
|---|---|---|---|
| `rtx3080` | RTX 3080 | 300 | 0.20 |
| `rtx3090` | RTX 3090 | 600 | 0.40 |
| `v100` | Tesla V100 32 GB | 600 | 0.40 |
| `rtx4090` | RTX 4090 | 1 000 | 0.67 |
| `rtx5090` | RTX 5090 | 1 400 | 0.93 |

More classes may come online — `GET /gpu-models` always reflects the live
fleet and prices.

---

## LLM inference — OpenAI-compatible

Besides renting GPUs, kgpu serves open models behind an **OpenAI-compatible**
API. Point any OpenAI client at `$KGPU_API_BASE` and authenticate with the
**same `KGPU_API_TOKEN`** you use for rentals — no separate key:

```python
from openai import OpenAI
client = OpenAI(base_url="https://api.kgpu.net/v1", api_key="$KGPU_API_TOKEN")
r = client.chat.completions.create(
    model="qwen3.5-122b",
    messages=[{"role": "user", "content": "Hello"}],
)
```

```bash
curl -sS https://api.kgpu.net/v1/chat/completions \
  -H "Authorization: Bearer $KGPU_API_TOKEN" -H "Content-Type: application/json" \
  -d '{"model":"qwen3.5-122b","messages":[{"role":"user","content":"Hello"}]}'
```

Standard OpenAI routes: `/v1/chat/completions`, `/v1/completions`,
`/v1/models`. `stream: true` (SSE) is supported. **`GET /v1/models` is the
source of truth** for what's servable right now — the table below is a
snapshot.

| model | quantization | context | credits / 1M in | credits / 1M out |
|---|---|---|---|---|
| `qwen3.5-122b` | NVFP4 | 32K | 390 | 3120 |
| `gpt-oss-120b` | MXFP4 | 32K | 59 | 270 |
| `qwen3.6-35b-a3b` | NVFP4 | 256K | 225 | 1500 |
| `qwen3.6-27b` | — | 256K | 450 | 3000 |
| `qwen3.5-9b` | Q8_0 | 16K | 150 | 225 |
| `qwen3.5-4b` | Q8_0 | 16K | 75 | 120 |
| `qwen3.5-2b` | Q8_0 | 16K | 45 | 75 |
| `qwen3.5-0.8b` | Q8_0 | 16K | 22 | 38 |

The four `qwen3.5-*` sizes above are **multimodal** — pass images the usual
OpenAI way (`content: [{type: "image_url", image_url: {url: "data:image/png;base64,..."}}]`).

More models are rolling out (small fast models on idle rental GPUs, more big
models on the Spark nodes); check `GET /v1/models`.

**Billing.** Pay-as-you-go from the same credit balance as rentals. Each call
is metered from the response's `usage` and charged
`prompt_tokens × in + completion_tokens × out` (rates above, per 1M tokens),
rounded up to a whole credit — so a non-empty call costs at least 1 credit.
Rates track [OpenRouter](https://openrouter.ai)'s $/token at **$1 = 1500
credits**. With `stream: true` the charge still lands (usage is captured on the
final chunk). A request made with a zero balance is refused with **402
`insufficient_credits`**.

---

## Renting a pod — `POST /pods`

All body fields are optional; defaults shown.

| Field | Default | Notes |
|---|---|---|
| `name` | `"rent"` | human label — lowercase `a-z` / `0-9` / hyphens, no leading/trailing hyphen, ≤ 30 chars (it becomes part of the pod's DNS-1123 name; the gateway also normalizes it) |
| `image` | `kgpu-pytorch:latest` | the default image is preloaded on every worker (no pull wait). Use a `ghcr.io/...` / `docker.io/...` ref for anything else. |
| `gpu_model` | `"rtx4090"` | live menu via `GET /gpu-models` |
| `gpu_count` | `1` | 0–4 (0 = CPU-only pod, still bills at the class rate) |
| `cpu` | `"2"` | CPU cores, ≤ 16 |
| `memory` | `"16Gi"` | RAM, ≤ 128 GiB |
| `ephemeral_storage_gib` | `100` | scratch disk, 1–2000 GiB |
| `env` | `null` | extra pod env vars |

Returns:

```jsonc
{
  "pod_id": "gpu-rent-1779745883-ab12cd",
  "gpu_model": "rtx4090",
  "price_per_hour_credits": 1000,
  "balance_after_precheck": 99000
}
```

---

## Getting into the pod — `POST /pods/{id}/exec`

The only way in. Synchronous, no setup, no keypair. Ideal for CI,
dashboards, quick checks — and long-running jobs too (see below).

```bash
curl -sS -X POST $B/pods/$RENT/exec -H "Authorization: Bearer $T" \
     -H "Content-Type: application/json" \
     -d '{"cmd":["nvidia-smi"]}' | jq
# { "exit_code": 0, "stdout": "...", "stderr": "", "duration_ms": 1240, "truncated": false }
```

| Field | Default | Notes |
|---|---|---|
| `cmd` | required | argv list, e.g. `["bash","-c","ls /root/files"]` |
| `stdin` | `null` | piped to the command, ≤ 1 MiB. Not allowed with `background: true`. |
| `timeout_seconds` | `60` | 1–300; killed on timeout, partial output returned. Ignored when `background: true`. |
| `background` | `false` | launch detached and return immediately — see below |

Caps: combined stdout+stderr ≤ **1 MiB** (excess truncated, `truncated:true`);
timeout ≤ **300 s**; no PTY. Returns `409 pod_not_ready` if the pod isn't
ready yet.

**Long-running jobs — `"background": true`.** Without it, `timeout_seconds`
bounds how long the exec call blocks — no good for a job that runs for
hours. With `background: true`, `cmd` is launched detached (stdin from
`/dev/null`, output to a gateway-picked log file) and the call returns
as soon as it's launched, not once it finishes:

```bash
curl -sS -X POST $B/pods/$RENT/exec -H "Authorization: Bearer $T" \
     -H "Content-Type: application/json" \
     -d '{"cmd":["python","/root/files/train.py"],"background":true}'
# { "background": true, "pid": 4213, "log_path": "/tmp/kgpu-exec-92db5709.log",
#   "launch_exit_code": 0, "launch_stderr": null, "duration_ms": 87 }

# check progress any time — either a short follow-up exec:
curl -sS -X POST $B/pods/$RENT/exec -H "Authorization: Bearer $T" \
     -H "Content-Type: application/json" -d '{"cmd":["tail","-20","/tmp/kgpu-exec-92db5709.log"]}'
# ...or, once the run has copied its log into ~/files, read it without an exec:
curl -sS $B/files/run1/train.log -H "Authorization: Bearer $T"
```

`log_path` defaults to a gateway-picked path under `/tmp` (always
writable, no PVC needed) that catches whatever `cmd` prints — that
default is the right place for a live log, because `/tmp` is local disk.
If you need the log to survive the pod, copy it to `~/files` when the run
ends (`... ; cp /tmp/train.log ~/files/run1/`) rather than writing every
line straight to `~/files` — see "What belongs here" under `files` for
why. `pid` is the real, running process
id (useful for e.g. a later `kill -0 $pid` exec check); it's `null` if
the launch itself failed — see `launch_exit_code`/`launch_stderr`.

Under the hood this is `setsid cmd </dev/null >log_path 2>&1 &` — a
plain `&` (or `nohup cmd &`, or backgrounding by hand in your own `cmd`)
does **not** work: it stays attached to the exec stream and blocks the
call for the job's full duration, same as if you hadn't backgrounded it
at all. `background: true` exists so you don't have to get that shell
incantation right yourself.

---

## Moving files in and out

The pod's own disk is ephemeral — **anything that should survive a
release goes through `~/files` / `/v1/files`.** See `files` below.

**But `files` is shared network storage, so it takes what's worth keeping,
not everything a run produces.** Small files in bulk are the thing that
actually breaks it — generate those on the pod's own disk and archive them
before they land here. Details in "What belongs here" below; please read it
before a job writes thousands of files.

### Persistent files — `files` (`~/files` · `/v1/files` · WebDAV)

You get a durable **private** folder that **persists across pods** (kgpu is
otherwise compute-only) — only you can see it, and it's the *same* folder in
every rental. Reachable three ways:

1. **In a pod** — auto-mounted at `~/files`. Just read/write; anything you
   leave there is still there in your next rental.
2. **REST API** `/v1/files` — from your laptop or scripts, with your account
   token.
3. **WebDAV** — mount it in Finder / Explorer (see below).

```bash
# list a directory (root, or a subpath)
curl -sS -H "Authorization: Bearer $T" $B/files
curl -sS -H "Authorization: Bearer $T" $B/files/checkpoints

# download a file
curl -sS -H "Authorization: Bearer $T" $B/files/model.pt -o model.pt

# upload / overwrite (whole file)
curl -sS -X PUT -H "Authorization: Bearer $T" --upload-file model.pt \
  $B/files/checkpoints/model.pt

# delete (a dir needs ?recursive=true)
curl -sS -X DELETE -H "Authorization: Bearer $T" $B/files/old.zip
```

**Large / flaky uploads — resumable.** Send `Content-Range: bytes <start>-<end>/<total>`
per chunk; if the connection drops, probe the committed offset with
`Content-Range: bytes */<total>` and resume from there (an out-of-order chunk
returns `409` with `expected_offset`). The file is atomically finalized once
`<total>` bytes land.

**WebDAV** — mount your `files` as a drive on your own machine. **Username:
anything. Password: your KGPU API token.**

- **Windows** (maps as drive `M:`, no extra software):
  ```powershell
  net use M: https://api.kgpu.net/dav/ <YOUR_KGPU_API_TOKEN> /user:kgpu
  dir M:\           # your files; add /persistent:yes to survive reboots
  ```
  RaiDrive / Mountain Duck give a smoother experience than the native client.
- **Mac**: Finder → *Connect to Server* (⌘K) → `https://api.kgpu.net/dav/`.

**SFTP (beta)** — the same folder over SFTP, for scp/rsync/FileZilla/rclone.
**Username: the `login_name` from `GET /me`** — normally the part of your
email before the `@` (`jihee.ha@example.com` → `jihee-ha`); if that handle is
already taken, your full address in dashed form (`jihee-ha-example-com`).
**Password: your API token.** (Your plain email works too, in clients with a
separate username field.)

```bash
sftp -P 2022 jihee-ha@sftp.kgpu.net
scp -P 2022 model.pt jihee-ha@sftp.kgpu.net:checkpoints/
rclone sync ./data :sftp,host=sftp.kgpu.net,port=2022,user=jihee-ha,pass=$KGPU_API_TOKEN:datasets/
```

`https://dav.kgpu.net/` (beta) serves the identical WebDAV with the same
login. A `files: read-only` token gets browse/download only over both.

Anything you drop here is the same folder your pods see at `~/files` and the
`/v1/files` API serves — edit a checkpoint from your laptop, read it in a pod.

**What belongs here — and what doesn't.** `files` is *network* storage shared
by the whole cluster, not a local disk. Keep it for data that has to outlive
the pod: datasets, checkpoints you'd hate to regenerate, final results,
anything you'll open from your laptop. Keep **churn** out of it — training
logs written line by line, `tmp/` scratch, per-step metric dumps,
dataloader caches, `pip`/`conda` caches. Those are thousands of tiny
synchronous round-trips over the network: your job runs slower, and everyone
else's storage gets slower with it.

**Hard rule: do not create large numbers of small files in `files`.** This is
the one that actually breaks things. A run once left ~1,000,000 files of about
200 bytes each in one user's folder — 200 MB of data, but a million objects.
That is what the backup replication chokes on, and it degrades the metadata
server for *every* user on the cluster, not just the account that made them.
**Generate small files on the pod's own disk. If they have to be kept, pack
them first** — one `tar`/`zip`/`parquet`/`.npz`/WebDataset shard instead of
thousands of loose files:

```bash
# NOT this — a million tiny files straight into the network share
python make_samples.py --out ~/files/samples/     # ✗

# this — build locally, archive, then keep the archive
python make_samples.py --out /root/samples/       # ✓ pod-local disk
tar czf ~/files/samples.tar.gz -C /root samples   # ✓ one object
```

Accounts that keep doing this may have their `files` access rate-limited or
suspended to protect the shared storage — so treat it as a real constraint,
not a style note.

The pattern: **write hot output to the pod's own disk (`/root`, `/tmp`), copy
the keepers to `~/files` when the run finishes.**

```bash
# in your training script / cmd
python train.py --out /root/run1 > /root/run1/train.log 2>&1   # fast local disk
cp /root/run1/best.pt /root/run1/train.log ~/files/run1/       # keep what matters
```

Same rule for reading: if you'll touch a dataset repeatedly, copy it to
`/root` once at the start of the run and read it from there — don't stream
the same file over the network every epoch. (Pod-local disk is ephemeral and
sized by `ephemeral_storage_gib`, default 100 GiB, so it's yours to burn.)

Two more reasons beyond speed: you're **billed for what sits in `files`**, so
abandoned scratch quietly costs credits; and on some nodes `files` arrives
over NFS rather than a direct mount, where chatty writes hurt more.

**Billing.** You're charged for what you actually store (not the quota),
usage-metered per minute. `GET /me` (the `storage` block) shows your usage and
monthly estimate.

---

## Published datasets — `GET /v1/datasets`

Large research corpora (e.g. VitalDB) are published with **parquet
manifests** so you never have to crawl millions of files:

```bash
# 1. discover (public, no auth)
curl -sS $B/datasets | jq
# → [{"name": "vitaldb", "files": ..., "bytes": ...,
#     "manifest_parquet": "/v1/datasets/vitaldb/manifest.parquet",
#     "manifest_meta":    "/v1/datasets/vitaldb/manifest.json", ...}]

# 2. cheap freshness probe before touching the big file
curl -sS $B/datasets/vitaldb/manifest.json | jq .generated_at
```

The manifest lists every file (`path`, `size`, `mtime`). It's served with
HTTP Range support, so query it **remotely** — no full download:

```python
import duckdb
duckdb.sql("""
  SELECT path, size FROM read_parquet(
    'https://api.kgpu.net/v1/datasets/vitaldb/manifest.parquet')
  WHERE path LIKE 'vital/2023%' LIMIT 20""")
```

Plan your sync by diffing the manifest against what you have, then fetch
only what changed. Datasets marked `"access": "restricted"` need any valid
Bearer (scoped tokens: `files: read`); `"public"` ones are token-free.
Dated snapshots (`manifest-YYYYMMDD.parquet`) are kept for reproducibility.

---

## Inside the container

The default `kgpu-pytorch:latest` ships PyTorch (CUDA 12.9, nv25.05) with
prebuilt kernels for **sm_75 / 80 / 86 / 89 / 90 / 100 / 120** — every GPU
in the fleet, including Blackwell (RTX 5090 / B200). Also bundled: `wfdb`,
`vitaldb`, `scipy`, `scikit-learn`, `pandas`, `matplotlib`, `seaborn`,
`duckdb`, `pyarrow`, `tmux`, `uv`, `zstd`.

Both `python` and `python3` run the bundled interpreter. `exec` commands
start in **`/workspace`** (the image's `WORKDIR`) — use absolute paths for
anything under `~/files` (mounted at `/root/files`). `/workspace` is
scratch on the pod's ephemeral disk — put your code + checkpoints there,
or under `/root/files` if it needs to survive the release;
`KGPU_*` env and the bundled tools are on `PATH`.

Auto-injected env:

| Var | Value |
|---|---|
| `KGPU_API_TOKEN` | per-rental scoped bearer (not your account token) |
| `KGPU_API_BASE` | `https://api.kgpu.net/v1` |
| `KGPU_GPU_ID` / `KGPU_RENT_ID` | this rental's id |
| `NVIDIA_VISIBLE_DEVICES` | `void` — CDI handles GPU allocation; don't override |

Inside the container `nvidia-smi -L` and `/dev/nvidia*` expose only your
rental's GPU(s).

### Detaching long-running jobs

Use `"background": true` on `POST /pods/{id}/exec` (see "Getting into the
pod" above) — it launches `cmd` detached and returns immediately,
instead of you having to get the backgrounding shell incantation right
yourself.

`tmux new -d -s train 'python ...'` (as a plain, non-background exec) is the
other safe option — exec has no PTY so you can't `tmux attach` over it,
but `tmux capture-pane -p -t train` (via a follow-up exec) dumps the
pane's current output on demand.

---

## Limits

- **GPU cap**: 4 concurrent GPUs across all your rentals.
- **Per-rental**: ≤ 16 CPU cores, ≤ 128 GiB RAM, 1–2000 GiB ephemeral disk.
- **Idle**: 12 h of no `exec` activity → `idle_warning`; 24 h → auto-release
  (`auto_idle`). **Override**: past the warn threshold we probe pod load —
  any GPU > 5 % or process > 10 % CPU resets the clock, so a long training
  run with no API calls is *not* reaped.
- **Wall clock**: no hard ceiling. Rentals live until `DELETE`, idle-kill,
  or out-of-credits.
- **Disk eviction**: exceeding `ephemeral_storage_gib` terminates the pod
  (`evicted_by_kubelet:OutOfephemeral-storage`). Checkpoints, pip cache,
  dataset extracts and `torch.compile` cache all add up — bump the value or
  rotate checkpoints for multi-epoch runs.
- **Network isolation**: **no ingress** (pods accept nothing inbound).
  **Egress** to the public internet works (pip/git/HF) but is blocked into
  private ranges (RFC1918, link-local/metadata `169.254`, Tailscale CGNAT)
  and to other pods. No lateral scanning, no pod-to-pod.
- **Credits**: per-minute deduction; zero → auto-release
  (`out_of_credits`).

---

## Pod lifecycle

`GET /v1/pods/{id}.phase`:

| `phase` | meaning |
|---|---|
| `Pending` | scheduling / image load / container creating. Carries `pending_reason` (e.g. `starting`, `ImagePullBackOff: …`). `ready=false`. |
| `Running` | container up. `ready=true` once the readiness probe passes. |
| `Terminating` | released and tearing down; `ended_at` / `end_reason` already set. |
| `Ended` | pod object gone; the historical row remains so you can still read `end_reason`. |
| `Failed` | rare — kubelet failed it; see `end_reason`. |

**Polling**: poll `GET /pods/{id}` every 10–15 s only until `ready=true`,
then stop — the cold start is dominated by image load, sub-5 s polling buys
nothing. For training liveness, `exec` a `tail`/`nvidia-smi` check every
few minutes rather than hammering the API.

**Cost so far**: every `GET /pods/{id}` carries `credits_charged` (credits
spent on this rental to date), `price_per_hour_credits`, and `started_at`
— so per-pod spend and runtime (`now − started_at`) are visible at a glance,
including on the historical row after the pod is released.

`GET /pods/{id}` works even after the pod is gone (falls back to the
historical row), so you can always read why it ended:

| `end_reason` | meaning |
|---|---|
| `returned` | you called `DELETE` |
| `auto_idle` | idle timeout |
| `out_of_credits` | balance hit zero |
| `evicted_by_kubelet:OutOfephemeral-storage` | disk exceeded — bump it or rotate checkpoints |
| `evicted_by_kubelet:OOMKilled` | memory limit hit |
| `start_failed:<reason>` | never escaped Pending (bad image tag, etc.); auto-released ~5 min after creation, quota refunded |

---

## Windows / PowerShell

PowerShell's `curl` is an alias for `Invoke-WebRequest` and mangles JSON
bodies. Prefer **`Invoke-RestMethod`**:

```powershell
$T = $env:KGPU_API_TOKEN
$B = "https://api.kgpu.net/v1"
$H = @{ Authorization = "Bearer $T" }

$rent = Invoke-RestMethod -Method Post -Uri "$B/pods" -Headers $H `
    -ContentType "application/json" -Body (@{ gpu_model = "rtx4090" } | ConvertTo-Json)
$RENT = $rent.pod_id

do { Start-Sleep 10; $p = Invoke-RestMethod -Uri "$B/pods/$RENT" -Headers $H } until ($p.ready)

Invoke-RestMethod -Method Post -Uri "$B/pods/$RENT/exec" -Headers $H `
    -ContentType "application/json" -Body (@{ cmd = @("nvidia-smi") } | ConvertTo-Json)
# release
Invoke-RestMethod -Method Delete -Uri "$B/pods/$RENT" -Headers $H
```

Or call the real `curl.exe` with a single-quoted here-string to copy the
bash examples verbatim (`@'...'@`, closing `'@` at column 0). `jq` works
the same once installed (`winget install jqlang.jq`).

---

## Status codes & errors

`200` ok · `400` bad request · `401` invalid/revoked token or account not
approved · `402` insufficient credits · `403` forbidden scope (in-pod token
tried to rent/exec) · `404` not found · `409` not ready / quota
exceeded / duplicate · `422` validation failed · `500` server error.

Bodies carry a machine-readable `error` **code** + `message` (auth /
middleware errors at the top level — `{"error":"token_revoked","message":…}`;
endpoint errors nest the same under `detail`). Codes worth handling:
**`token_revoked`** — you reissued your token in the dashboard, so the old
one is dead **immediately, no grace period**; a stale cached token 401s with
this, so re-read `KGPU_API_TOKEN` from the dashboard / env on a 401 instead
of guessing. Others: `invalid_token`, `missing_token`, `forbidden_scope`,
`insufficient_credits`.

Every response carries `X-Request-ID` (8-byte hex); 5xx bodies echo it as
`trace_id`. Quote it when filing a bug.