AI prompt classification is the step that reads an incoming prompt, works out what kind of task it is, and labels it so a router can send it to the right model.
It looks like plumbing. It is actually the decision everything downstream hangs off, and it has to be almost free. Spend half a second working out the category and you have burned the entire latency budget deciding who should answer, before anyone has started answering.
ChatFuse runs this on every turn, in front of more than 130 models. Here is how it works, including the parts we got wrong first.
Why does a router need AI prompt classification at all?
A router needs a category because model quality is task specific, and you cannot pick a model until you know what the job is. Anthropic's Claude models, OpenAI's GPT family, Google's Gemini, and open models like Meta's Llama each pull ahead on different work. A request to refactor a function and a request to translate a paragraph want different models. All the router has to go on is the text someone typed.
So classification feeds routing. Get the category wrong and every sensible decision after it inherits the mistake, because the router then optimizes very confidently in the wrong direction.
We covered the layer sitting on top of this in AI model orchestration. This post is the layer underneath.
What categories does a prompt get sorted into?
ChatFuse sorts every prompt into one of 10 routing categories, and the category it lands in decides which model answers.
We settled on 10 deliberately. Too few and the buckets stop being useful, because coding and creative writing want genuinely different models. Too many and the boundaries blur, the classifier gets less sure which side of a line a prompt falls on, and more traffic drops through to the slow tiers. We stopped adding categories when the boundaries stopped being arguable.
How does ChatFuse classify a prompt without adding latency?
ChatFuse runs a cascade of 5 tiers, fastest first, and most prompts never reach the slow one.
Tier 0 is keyed on a hash of the message, the attachment types, the classifier model, and the category set. Send the same prompt twice and the second one comes back instantly.
On a miss we compute one embedding and reuse it everywhere below. It runs locally through sentence-transformers all-MiniLM-L6-v2 and only looks at the first 500 characters. Every tier below shares that one vector, so the expensive part happens once per prompt instead of once per tier.
Tier 1 is the tier that catches paraphrases. "Write me a function that sorts a list" and "can you write a sorting function" are different strings with identical intent, and Tier 0 has no way to see that.
Tier 2 is where most real traffic resolves. We precompute an embedding for each of the 10 category descriptions at startup, then compare the incoming prompt against all 10 and take the best match.
Tier 3 is left over from an earlier design and only fires on very high confidence matches, mostly image generation and coding. Tier 4 is an actual LLM call, with 3 retries and a circuit breaker so an OpenRouter outage fails fast instead of hanging. It is also more accurate than anything above it, which is the awkward part: the entire cascade exists to avoid calling the thing that works best.
How does the ChatFuse embedding classifier decide?
The ChatFuse classifier scores the prompt against all 10 category descriptions by cosine similarity, then applies two separate guards before committing to an answer.
First there is an absolute floor of 0.25. Nothing clears it, nothing matches, and the prompt is usually gibberish or a stray fragment. Then there is a confidence threshold of 0.45. Clearing the floor but missing the threshold means the prompt resembles something without resembling it enough to bet a model choice on.
Confidence itself is calibrated rather than raw. It blends the best score with the margin between the top two categories, because a prompt scoring 0.6 on one category and 0.58 on another is a much shakier call than one scoring 0.6 against a field of 0.2. The margin is what separates them.
What happens when the classifier is not sure?
ChatFuse refuses to answer and drops through to the next tier. That is the whole mechanism, and it took us a while to get comfortable with it.
The instinct when you build a classifier is to always return its best guess. Routing punishes that. A confident wrong answer costs more than an admission of uncertainty, because falling through does not produce an error, it produces a slower tier that gets the answer right. Falling through costs a few hundred milliseconds. A wrong category costs you a coding question answered by a model picked for translation.
How does ChatFuse test a classifier before trusting it?
You run it in shadow mode, where it classifies every prompt in parallel with the existing LLM classifier, returns nothing to the user, and logs every disagreement.
Then you read the disagreements. We held ours to a 93% agreement target against LLM ground truth and watched for 48 to 72 hours before promoting the embedding classifier to active. Shadow mode is unglamorous. It is also the only honest way to find out whether a classifier works on your actual traffic instead of on the examples you thought up while writing it.
The disagreement logs turned out to be more valuable than the agreement rate. They cluster. Each cluster is a category boundary you defined badly.
What does the confidence threshold trade off?
It trades latency against accuracy, and both ends are bad.
At 0.7 the classifier almost never feels sure enough, so nearly everything falls through to the 400 to 800ms tier and you have built an elaborate way to achieve nothing. At 0.3 it commits to weak matches, false positives climb, and prompts land on the wrong model. We run 0.45. It started as a conservative guess and we left it alone once the shadow data said it was holding.
Does caching actually help, or is it just theater?
It helps, though the reason has more to do with startup than with steady state.
A cold instance has an empty semantic cache, so the first user in every category pays full price. The service prewarms itself to avoid that. It waits 3 seconds for the Cloud Run readiness probe to pass, precomputes the 10 category embeddings, then pushes 7 to 12 representative queries per category through the embedding path and stores the results. By the time real traffic arrives, the cache already knows what a coding question looks like.
One guard is worth calling out. The semantic cache only stores a result when the category is one of the 10 real ones. Skip that check and a single hallucinated category from the LLM tier gets cached, then served to every similar prompt for the next 10 minutes.
What is still on the list?
A locally trained classifier to replace the cosine similarity step. The plan is a logistic regression model over the 384 dimensional embeddings, held to 95% accuracy under 5 fold cross validation before it replaces anything, running inference in about 5ms.
The generation script is written. The training pipeline is not, and it is blocked on having enough production traffic to validate against rather than on anything technical. We could bootstrap it from synthetic data tomorrow and report an accuracy number that means almost nothing.
Is prompt classification the same as model routing?
No. Classification decides what kind of task a prompt is, and routing decides which model handles that kind of task. They are separate steps and they fail in different ways, which is why they are separate services here. A classification failure looks like a coding question landing on a translation model. A routing failure looks like the right category going to a model that is down.
Does classification send my prompt anywhere?
In the common path, no. The embedding runs locally on our own infrastructure and the cache tiers never leave the process. Only the Tier 4 fallback involves an external model call, and it is capped at 200 prompt tokens. Our wider position on data handling is in zero trust AI data security.
Can I skip classification and pick the model myself?
Yes. Pinning a model bypasses the cascade entirely, because there is no decision left to make. Automatic routing is the default because it is right for most turns, and the override exists for when you want a specific model for a specific reason.
Does classification cost me credits?
No. Everything up to Tier 4 runs on our own infrastructure, and the Tier 4 fallback is capped at 200 prompt tokens on a small model. Routing is designed to lower what you spend rather than add to it, which we went into in energy efficient AI model routing and on the pricing page.
Where can I try ChatFuse prompt classification?
Every prompt you send to ChatFuse goes through this, and the point is that you never notice it happening. Start free and send something, or compare the plans on the pricing page first.
Comments
Loading comments…