From If-Else to Declared Routing

The 3 AM Question

A production AI system goes down. The pager rings. The on-call engineer opens the logs and finds an upstream provider returning 5xx for an entire region. The codebase has a comment from six months ago that says # fallback to Anthropic if OpenAI down. The fallback path has never been exercised in production. It does not work.

This is the most common AI incident shape we see. It is also the most preventable.

Why Hand-Written Failover Fails

Three reasons.

1. It is untested. Failover code only runs during incidents. Anything that only runs during incidents is dead code that comes alive at the worst possible moment.
2. It is decentralized. Routing logic ends up in three different services, each with a slightly different idea of what "healthy" means.
3. It does not learn. Provider behavior shifts. A model gets deprecated. A new region opens. Hand-written rules do not notice.

Declared Routing

The Cumulus Router is the inverse. You declare, per workflow, what should happen — model, provider, fallback chain, latency budget — and the platform routes every request accordingly. The router is deterministic (the same request always picks the same path for the same rule version) and traceable (every dispatch decision is in the audit log).

A workflow rule looks like this:

workflow: inbox-triage.classify
primary:
  model: llama-3.2-1b
  provider: ion           # Cumulus' own engine
fallback:
  - model: gpt-5-mini
    provider: openai
  - model: claude-3.5-haiku
    provider: anthropic
budget:
  p95_latency_ms: 400
  max_cost_per_request_usd: 0.0002

The router health-checks every provider continuously. When p95 latency on the primary breaches budget, traffic shifts before the user sees it. When OpenAI drops a region, the dispatch graph re-resolves in milliseconds.

What This Replaces

For most teams, declared routing replaces:

• Bespoke retry-with-backoff loops in five different services.
• A homegrown circuit breaker that closes too aggressively.
• A Slack channel where on-call posts "GPT-5 looks slow, anyone else seeing it?"

It does not replace your code's understanding of business semantics — Cumulus does not decide whether a request is allowed to use a frontier model for cost reasons. It decides which model and provider, given the rules you wrote.

The Underrated Win: Latency

Declared routing usually starts as a reliability project and ends as a latency project. Once routing is centralized, the platform can short-circuit slow tails — if the primary breaches budget, the secondary fires speculatively, the first response wins. P99 drops in ways that hand-written failover does not touch.

Try Cumulus →