Cost-aware AI routing

The problem

An LLM on every message is a cost and latency problem dressed up as intelligence. Keyword matching alone is brittle. The useful design is a cascade: cheap, then similar, then expensive — with a way to see when the layers disagree.

What it is

Not every inbound message should hit a language model. I built a routing engine that spends almost nothing on the common case, escalates only when cheaper layers disagree or miss, and logs those disagreements as labelled data for later tuning.

What I built

Routing is a cascade. A normalized keyword layer handles the obvious cases. Misses go to pgvector HNSW similarity. Only then does an LLM classifier run, behind a provider-agnostic protocol so the rest of the product does not care which model is on duty.

When retrieval and the model disagree, that pair is logged as labelled tuning data instead of being discarded. The engine gets cheaper over time because the disagreements are the training set for the cheaper layers.

How it fails

A downed model provider does not take down routing: earlier layers still answer, and the protocol can swap the expensive step. Ambiguous matches escalate rather than invent a confident wrong route.

Focus

• Cheap-to-expensive cascade on the hot path
• pgvector HNSW similarity as the middle layer
• Provider-agnostic LLM step with disagreement logs

What changed

• Hot-path cost stays near zero for traffic the cheap layers can already handle.
• Model-vs-retrieval disagreements become a tuning corpus instead of silent mistakes.

Built with