What is an Inference Platform?
Inference is not just a model call. It is routing, caching, observability, evaluation, fine-tuning, and the runtime that executes the call. A complete guide.
The Shape of the Problem
The day you ship your first AI feature, you have an inference call. The day you have ten AI features in production, you have an inference system — and that system has at least eight moving parts whether you built them or not.
An inference platform is the consolidation of those parts behind one interface. This post is the long version.
The Eight Parts
1. Gateway
Every model provider has a slightly different API. A gateway speaks one normalized protocol — almost always OpenAI-compatible — and translates to whatever the destination provider expects. The win is that your application code stops caring whether a request lands on OpenAI, Anthropic, an open-weight model, or your own fine-tune.
2. Router
Given a request, the router decides which model and which provider serve it, subject to declared rules, health, cost, and latency budgets. Routing is where reliability lives.
3. Cache
There are at least three caches worth stacking. Exact-match for identical requests. Prefix for shared system prompts. Semantic for paraphrased questions. On real workloads, stacking all three cuts input tokens by 40 to 70%. The platform that owns the gateway is the only place a cache can correctly account for token cost across providers.
4. Observability
Every request needs to land in a replayable audit log with input, output, model, provider, latency, cost, and quality signals. Without this you cannot answer "what did the model say to user X on Tuesday" — which is the question you get asked in your first compliance review.
5. Evaluation
Production evaluation is the hard one. The naive answer is "LLM-as-judge with three judges and take the majority." The real answer is shadow evaluation against production traffic — run candidate models in parallel with the live one, score outputs with a stack of synthetic data, deterministic heuristics, and LLM judges, and only promote when the evidence is clean. This is how you safely swap a frontier model for a cheaper one.
6. Fine-tuning
When evaluation surfaces a workflow that a smaller model can handle, you fine-tune. LoRA adapters are the workhorse — they update less than 1% of parameters, fit on a single GPU, and serve cheaply. The platform that owns evaluation can spot candidate fine-tunes from traffic automatically.
7. Custom hosting
For open-weight models and your own fine-tunes, you need a place to put them. Custom hosting is just inference for weights you brought.
8. The runtime
Underneath everything is a kernel that actually executes the forward pass on a GPU. vLLM, SGLang, TensorRT-LLM, and Cumulus' Ion are runtimes. The runtime decides your tokens-per-second ceiling.
The Argument for Consolidation
You can buy these as seven vendors and one cloud GPU rental. Most teams do. The problem is that none of them know what the others know — the router does not know what the cache stored, the eval tool does not know what the router routed, the fine-tune service does not know which workflow generated the LoRA. Every seam between vendors is a place where context is lost.
A platform that does all eight in one place can do things the seven-vendor version cannot: evaluate routing decisions, route around degraded models that evaluation flagged, fine-tune based on what observability captured, and serve the result on its own inference engine. The whole point of consolidation is that the parts inform each other.
What Cumulus Is
Cumulus is one of these platforms. It is OpenAI-compatible at the edge, deterministic and traceable in the middle, and runs on its own NVIDIA Grace and Blackwell fleet at the bottom. The pitch is short: change one line, keep your code.