Drop-In OpenAI: Migrating in One Line
How to move an existing OpenAI-SDK application to Cumulus without changing your code — and what you get from the routing, caching, and observability layers once you do.
The Migration
If your application already uses the OpenAI Python SDK, the migration is one line.
from openai import OpenAI
client = OpenAI(
api_key=os.environ["CUMULUS_KEY"],
- base_url="https://api.openai.com/v1",
+ base_url="https://api.cumuluslabs.io/v1",
)
Same client, same method signatures, same response shapes. client.chat.completions.create(...) works identically. Tool calls, structured outputs, streaming, vision, and audio modalities pass through.
What Happens Behind the Line
The first request hits the Cumulus Gateway. The Router consults the declared rule for your workflow and picks a model and provider — by default this is the model name you asked for, on the cheapest healthy provider that serves it.
The Cache is checked before any provider call. If the request hits an exact, prefix, or semantic match, the response is returned in single-digit milliseconds.
The response, the request, the chosen model, the latency, and the token cost are written to Observability.
Working With Other SDKs
The base-URL trick works for the Anthropic SDK, LangChain, LlamaIndex, and the Vercel AI SDK. For each, you point the client's base URL at api.cumuluslabs.io/v1.
When You Want More Than Drop-In
The interesting work begins after the migration. With every request flowing through one place, you can:
- Define routing rules so a workflow uses a cheap model by default and falls back to a frontier model when a confidence signal is low.
- Turn on the semantic cache for FAQ-shaped traffic.
- Open the audit log and find the seven hardest requests to your model and add them to your eval set.
- Pull the top three workflows by spend into the fine-tune pipeline and serve them on Ion.
None of those require code changes — they are configuration on the routing rule.
Cost
Pay-per-token, no idle infrastructure cost. The free tier is generous and the volume tier is on request.