An open-source LLM inference and metering/billing platform built on Kubernetes. Deploy Qwen, DeepSeek, LLaMA and more on multiple heterogeneous accelerators, expose an OpenAI-compatible API, meter per token, and settle per API Key.
Everything an AI platform team needs to serve, meter, and bill LLM traffic.
Unified management of multiple heterogeneous accelerator cards (vendor-agnostic). Node labeling, Device Plugins, and driver deployment are handled by each vendor's GPU Operator, with a live accelerator inventory and a model × engine × card-type compatibility matrix.
Management APIs and inference APIs are served by separate gateways, so platform operations stay available while the inference path scales, upgrades, or degrades.
Deploy models from the console or API. The platform provisions inference workloads through a message queue and a Kubernetes controller — no hand-written YAML.
Register, version, and map engine images across the supported accelerator cards, with node-level pre-pulling to shorten cold starts.
/v1/chat/completions, /v1/completions, and /v1/embeddings served by an Envoy-based gateway. Auth, metering, and routing run in Wasm plugins — inference traffic never passes through business processes.
Token usage (prompt / completion / cached / reasoning) is metered at the gateway, recorded as tamper-proof vouchers, and settled asynchronously per API Key.
A model × card-type price matrix with separate input/output token rates, tiered pricing, and packages; balance (prepaid) and quota (postpaid) account modes.
Organization/project isolation with full API Key lifecycle management. Keys are stored only as salted hashes.
Pluggable identity providers — LDAP/LDAPS, OIDC/OAuth 2.0 (GitHub, Google, GitLab), SAML 2.0, and WeChat — with account binding, JIT provisioning, and attribute-to-role mapping.
Ordered routing policies over healthy deployments with bounded same-model failover and retryable-failure categories — no silent model or version substitution.
Inference autoscaling with scale-to-zero, driven by live load signals so idle models cost nothing and busy ones scale out.
Per-model latency / throughput / error dashboards, per-service CPU / memory / GPU metrics, and end-to-end request tracing with a TTFT / generation / total breakdown.
A versioned prompt library, an interactive playground with side-by-side model comparison, and deterministic prompt evaluation for regression testing.
Submit a JSONL batch of requests, process it asynchronously at a discounted rate, monitor progress, and download result / error files.
Per-key model scoping, request-rate limits, and org spend limits, so a leaked or over-broad key can never reach models it was not granted.
Audit logging and activity export, deployment history & audit, and GDPR-style data export of usage, billing, and request logs as JSON / CSV.
Choose the platform serving scheme — plain Kubernetes Deployments (default) or an AIBrix-managed distributed runtime — and inspect runtime component health and the models registered with the runtime.
Register third-party providers (DeepSeek / OpenAI / OpenAI-compatible), probe connectivity with five states (ok / unauthorized / unreachable / error / unknown), curate upstream models, and publish them into the catalog — consumed through the same metered gateway.
Routing groups expose virtual model names (e.g. auto-chat) over an ordered candidate list of real models, with weighted, cost-priority, and latency-priority strategies plus keyword intent rules — and a logged decision trail.
A deterministic, rule-based engine turns settled usage, price, and deployment data into ranked, quantified, dismissible savings recommendations, each linking straight to the page where the operator can act.
Per-organization quotas on concurrent requests, inference services, batch jobs per day, and max output tokens per request — with tier presets, fail-closed enforcement, and an audited change history.
Register and manage inference clusters, view cluster health and workload placement, and route model deployments across clusters.
HAMI-based vGPU slicing: label nodes into slicing pools, define slice templates, and co-schedule multiple small inference jobs onto a single physical GPU with binpack or spread placement.
Detect service, node, and accelerator failures, classify the root cause deterministically, and recover automatically (restart / reschedule) with operator-confirmed drain — every incident a timeline in the admin console.
Control plane and data plane, separated by audience and network.
One Helm chart deploys the whole stack on Kubernetes — control plane, consoles, and dependencies.
git clone https://github.com/go-taas/go-taas.git
cd go-taashelm install taas deploy/helm/go-taas \
--namespace taas --create-namespaceWith no overrides, the chart provisions in-cluster PostgreSQL, Redis, and RabbitMQ and wires their credentials to the control plane.
kubectl --namespace taas port-forward svc/taas-server 9091:9091End-user console: http://127.0.0.1:9091/ · Admin console: http://127.0.0.1:9091/admin
Each console has its own session. End-user APIs live under /api/v1/*; management APIs live under /api/v1/admin/*.
9091 — HTTP/JSON + console9090 — gRPC9092 — metrics / healthzmake helm-lint # lint the chart
make helm-template # render templates
helm uninstall taas # remove the releaseFor sizing, external PostgreSQL / Redis / RabbitMQ, and TLS, see the chart values: deploy/helm/go-taas/values.yaml.
Any OpenAI SDK works out of the box — just point it at your TaaS host.
from openai import OpenAI
client = OpenAI(
base_url="https://<your-taas-host>/v1",
api_key="sk-xxxxxxxx",
)
resp = client.chat.completions.create(
model="qwen2.5-7b",
messages=[{"role": "user", "content": "Hello!"}],
)
print(resp.choices[0].message.content)curl https://<your-taas-host>/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer sk-xxxxxxxx" \
-d '{
"model": "qwen2.5-7b",
"messages": [{"role": "user", "content": "Hello!"}]
}'Contributions of all kinds are welcome — code, documentation, bug reports, and feature ideas.
Built with Go and Kubernetes
Standing on the shoulders of the Kubernetes, AIBrix, Volcano, and HAMi communities.