← Back

Kestrel

A multi-tenant GPU compute control plane with runway-aware scheduling.

PythonFastAPIKubernetesKueuePostgreSQLPrometheus

Problem

Every GPU platform bills you after the fact. The scheduler places your job, the meter counts what it cost, and the two never speak, so a tenant who has burned through their budget keeps getting capacity until someone reads a report and turns them off. Billing is a receipt, not a control signal.

Approach

Architecture

Control plane

A FastAPI service backed by Postgres and Redis. Tenants submit jobs (run and finish) or endpoints (stay up behind a URL); the plane admits, places, meters and autoscales them.

Data plane

A real Kubernetes cluster on kind, with a genuine API server, etcd and scheduler, and Kueue admitting workloads against real ClusterQueue quota. GPU nodes are KWOK fakes advertising nvidia.com/gpu, while CPU work runs on a real kubelet.

Metering

An append-only usage ledger records GPU-seconds per tenant, exported to Prometheus and a Next.js operator console. Every figure derives from a ledger row; nothing is seeded.

Decisions

Runway feeds back into scheduling

The same ledger that produces a tenant's bill is read by the scheduler and autoscaler, so remaining budget influences where and whether work runs, not just what it costs afterwards. That feedback loop is the whole point of the project.

Two pools, deliberately

CPU work runs on a real kubelet and takes real time; GPU work lands on KWOK fakes and returns instantly. Keeping both visible makes it obvious which measurements are real, rather than hiding the simulation behind a uniform dashboard.

Allocation, not utilisation

With no DCGM telemetry, GPU-seconds means seconds reserved, not compute performed. Real clouds bill on allocation too, but the distinction is stated rather than glossed over.

Outcome

← All work