Early development · Design finalized Apache-2.0 Go 1.25+

Token as a Service
for AI Agents & SDKs

An open-source LLM inference and metering/billing platform built on Kubernetes. Deploy Qwen, DeepSeek, LLaMA and more on heterogeneous accelerators, expose an OpenAI-compatible API, meter per token, and settle per API Key.

Get Started View on GitHub

Features

Everything an AI platform team needs to serve, meter, and bill LLM traffic.

Heterogeneous Compute

Unified support for NVIDIA GPU, Iluvatar CoreX, and MetaX. Node labeling, Device Plugins, and driver deployment are handled by each vendor's GPU Operator.

Control / Data Plane Split

Management APIs and inference APIs are served by separate gateways, so platform operations stay available while the inference path scales, upgrades, or degrades.

One-click Model Deployment

Deploy models from the console or API. The platform provisions inference workloads through a message queue and a Kubernetes controller — no hand-written YAML.

Inference Image Management

Register, version, and map engine images across NVIDIA / CoreX / MetaX, with node-level pre-pulling to shorten cold starts.

OpenAI-compatible API

/v1/chat/completions, /v1/completions, and /v1/embeddings served by an Envoy-based gateway. Auth, metering, and routing run in Wasm plugins — inference traffic never passes through business processes.

Token Metering & Billing

Token usage (prompt / completion / cached / reasoning) is metered at the gateway, recorded as tamper-proof vouchers, and settled asynchronously per API Key.

Flexible Pricing

A model × card-type price matrix with separate input/output token rates, tiered pricing, and packages; balance (prepaid) and quota (postpaid) account modes.

Multi-tenancy

Organization/project isolation with full API Key lifecycle management. Keys are stored only as salted hashes.

SSO Federation

Pluggable identity providers — LDAP/LDAPS, OIDC/OAuth 2.0 (GitHub, Google, GitLab), SAML 2.0, and WeChat — with account binding, JIT provisioning, and attribute-to-role mapping.

Architecture

Control plane and data plane, separated by audience and network.

Agents / SDK

Admin Console / CLI

Kubernetes Cluster

Control Plane · standard network

Control Gateway (HTTP)

gRPC Server — auth · model · image · infer · billing · metering

Message Queue (Kafka / NATS)

Controller (K8s reconciliation)

Data Plane · RDMA network

Inference Gateway — Envoy + Wasm

Inference Services — vLLM / SGLang / CoreX / MetaX

Storage & Stateful Components

Longhorn

PostgreSQL

Redis

MinIO

JuiceFS

Traffic is split by audience: administrators enter through the Control Gateway (HTTP, generated from the Protobuf contract by grpc-gateway), while agents and SDKs call the Inference Gateway (Envoy + Wasm).
The two gateways are deployed as separate workloads, so control plane APIs and inference APIs scale, upgrade, and fail independently.
All microservice gRPC servers run in a single Deployment and hand work to the Controller through a message queue (Kafka / NATS), decoupling API serving from Kubernetes reconciliation.
Authentication, metering, and routing run in Wasm plugins at the Inference Gateway, so inference traffic never passes through business processes.
The Inference Gateway verifies every API Key by calling the auth module over gRPC, backed by a two-level cache (Wasm-local TTL cache, then Redis); on timeout or auth unavailability it fails closed.
Inference services and the control plane share one Kubernetes cluster, isolated through Namespaces, node labels, and taints/tolerations.

Quick Start

A one-command experience environment with Docker Compose.

Clone the repository

git clone https://github.com/go-taas/go-taas.git
cd go-taas

Start the local stack

make compose-up

Builds the images (console included) and starts the control plane, PostgreSQL, Redis, and the message queue.

Open the console

http://localhost:9091/admin

Management APIs live under /api/v1/admin/*; user-facing APIs (login, signup) live under /api/v1/auth/*.

Ports

9091 — HTTP/JSON + console
9090 — gRPC
9092 — metrics / healthz

More commands

make compose-ps    # stack status
make compose-logs  # follow logs
make compose-down  # stop & remove

On restricted networks, point the build at local mirrors: make compose-up GOPROXY=… NPM_REGISTRY=…

Usage

Any OpenAI SDK works out of the box — just point it at your TaaS host.

Python · OpenAI SDK

from openai import OpenAI

client = OpenAI(
    base_url="https://<your-taas-host>/v1",
    api_key="sk-xxxxxxxx",
)

resp = client.chat.completions.create(
    model="qwen2.5-7b",
    messages=[{"role": "user", "content": "Hello!"}],
)
print(resp.choices[0].message.content)

cURL

curl https://<your-taas-host>/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer sk-xxxxxxxx" \
  -d '{
    "model": "qwen2.5-7b",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

Roadmap

From single-cluster MVP to production-grade heterogeneous platform.

Phase 1 — MVP

Single-cluster NVIDIA: model CRUD, API Keys, control/inference gateway split, vLLM deployment, gateway-side auth and metering.
Phase 2 — Billing & Multi-tenancy

Pricing, balance/quota, tiered pricing, asynchronous settlement, audit, operations dashboard.
Phase 3 — Heterogeneous Accelerators

GPU Operator integration, Iluvatar / MetaX engine and image adaptation, compatibility matrix.
Phase 4 — Productionization

Helm one-click install, autoscaling, RDMA networking, load testing, SDK, community operations.

Build the open token infrastructure with us

Contributions of all kinds are welcome — code, documentation, bug reports, and feature ideas.

Star on GitHub Join Discussions

Token as a Service for AI Agents & SDKs