AI Prompt Classification: How Routing Decides

AI prompt classification reads a user's question, figures out what kind of job it is, and tags it for routing to the best available model.

It seems like a simple plumbing job. But it's the most important call in the whole system, and it has to cost almost nothing. If you use even 500 milliseconds just to decide where a query should go, you've already spent all the time you had for an answer before the real work even starts. That time is the same precious resource that drives time to first token in LLM streaming.

ChatFuse handles this on every single request, before it ever hits one of our more than 100 models. Here's what that process looks like and where we messed it up at the start.

Why does a router need AI prompt classification at all?

A router can't pick a model until it figures out what the job is. That's because every model has its own strengths, and a task like translating text is nothing like refactoring a block of code. The only clue it gets is the prompt itself.

The model families we use, Google's Gemini, Anthropic's Claude, open models like Meta's Llama, and OpenAI's GPT family, each perform best on specific types of work.

That's why classification has to happen first. If you get the category wrong, the router makes all its next moves based on that mistake. It'll confidently pick the best model for a job that isn't actually there.

The system that handles routing has its own write up, AI model orchestration. For a look at the whole platform, there's the ChatFuse AI orchestration platform. What follows is the foundation it all sits on.

What categories does a prompt get sorted into?

ChatFuse puts every single request you make into one of 10 buckets. Which bucket it lands in tells the platform exactly which AI model to use for your answer. We've got categories for general chat, coding, making images, generating video, searching the web, looking at documents, creating documents, creative writing, analyzing data, and translating languages.

We picked 10 on purpose. If you only have a few, they don't mean much, since programming and creative writing need totally different AI models. But if you have too many, the lines between them get fuzzy. It makes the system less certain about where a question belongs, so more requests end up on the slow tiers. We quit adding new groups once it became impossible to argue about where one ended and the next began.

How does ChatFuse classify a prompt without adding latency?

ChatFuse puts every prompt through a cascade of 5 tiers, and it starts with the quickest one. The vast majority of requests never make it to the slowest level. Most responses come straight from a cache or a local embedding model, and that happens fast, in under 20ms. You only get the slower LLM call, which takes 400 to 800ms, when the faster tiers just aren't sure enough.

Tier 0 works by hashing the message itself, what sort of files are attached, the model doing the classifying, and the categories we're using. That means if you send the exact same prompt a second time, it comes back immediately.

When that hash doesn't find a match, we create one embedding for the prompt and use it for every tier after that. We generate it on our own servers using sentence-transformers all-MiniLM-L6-v2, and we only look at the first 500 characters of your request. The heavy lifting happens just this one time for the whole process, not over and over.

Tier 1 is what handles different ways of saying the same thing. For example, "sort this list for me in Python" and "I need code that orders a list" are 2 different sentences that mean the same thing, and Tier 0 can't see that connection.

Most requests get an answer from Tier 2. When the system starts, we turn each of the 10 category descriptions into a vector. Your prompt's vector gets a score against all 10 of those, and we pick the closest one.

Tier 3 is mostly legacy now and only runs on matches it's very sure about, like for making images or writing code. Tier 4 is a full LLM call through OpenRouter. It gets 3 tries and has a circuit breaker to stop it from hanging if the service has an outage. The tricky part is that this tier is actually the most accurate. The whole tiered setup is built to avoid using the method that gives the best results.

How does the ChatFuse embedding classifier decide?

The ChatFuse classifier compares your prompt to all 10 category descriptions using cosine similarity. It then runs 2 checks before it picks an answer: a 0.25 floor and a 0.45 confidence bar. If your prompt misses either one, it goes to the next tier.

The floor gets checked first. If nothing passes 0.25, we don't see a match. That usually means the input is nonsense or just a fragment. Even if it clears the floor, it still has to hit the 0.45 threshold. If it doesn't, the prompt looks a bit like a category but not enough for us to assign a model with any confidence.

We don't just use the raw score for confidence. It's calibrated. We mix the top score with the gap between the first and second place categories. A prompt that scores 0.6 on one and 0.58 on another is a much closer call than one that gets 0.6 against a bunch of 0.2s. That margin makes the difference.

What happens when the classifier is not sure?

ChatFuse doesn't guess. It just sends your prompt down to the next tier. That takes a few hundred extra milliseconds. Getting it wrong means a code question might get answered by a translation model, so saying "I'm not sure" is the cheaper move.

That's the entire system, and it took some getting used to. When you build a classifier, your first thought is to always give its best answer. But routing punishes that, because falling through doesn't create an error; it just makes a slower tier give you the right result.

How does ChatFuse test a classifier before trusting it?

You set it to run in shadow mode, so it tags every prompt alongside your current system but doesn't show the user. It just logs every single mismatch. At ChatFuse, our embedding classifier hit 93% agreement with the LLM's labels. We kept it that way for 48 to 72 hours, watching closely, and then we moved it to active.

Then you see the places it gets wrong. Running a classifier in shadow mode isn't exciting work. But it's the only real way to know if it handles your actual data and not just the examples you had in mind.

The logs of those disagreements ended up being more useful than the percent it got right. They group together. Each group is a category line you didn't draw well.

What does the confidence threshold trade off?

The confidence threshold decides how often we choose speed or accuracy, and neither extreme works well. A high value pushes almost everything to the slower LLM level. A low one makes the router pick weak signals and assign prompts incorrectly. We run it at 0.45 at ChatFuse, a middle ground.

The classifier set at 0.7 almost never decides anything. It just sends almost every single request to the slow 400 to 800ms tier. You basically built a complicated system that doesn't do much. But if you drop it to 0.3, you get way too many false positives. We picked 0.45 as our starting point, just a safe guess. And then we saw the shadow data was backing it up, so we just stuck with it.

Does caching actually help, or is it just theater?

Caching matters, but it's really about how things start up, not how they run once they're going. A brand new instance has nothing in its semantic cache, so the first person to ask about any category would get hit with full latency. We prewarm every instance at ChatFuse, so the cache already has common prompts sorted before any actual users show up.

Here's what that prewarm does. It hangs back for 3 seconds to let the Cloud Run readiness probe finish, calculates embeddings for all 10 categories, and then runs 7 to 12 example queries for each one through the embedding pipeline to save the outcomes. When real traffic finally comes, the cache already gets what a coding question should be.

There's one specific guard in place you should know about. The cache only keeps a result if the category is one of those 10 real ones. Miss that step and a single made up category from the LLM tier could get stored, then sent out to every matching prompt for the next 10 minutes.

What is next for ChatFuse prompt classification?

Next, we swap out the cosine piece with a classifier we train ourselves. It's a logistic regression model on those 384 dimensional embeddings. We won't put it in until it hits 95% accuracy with 5 fold cross validation. It also has to run inference in about 5ms.

The script that creates the data is done. The part that trains the model isn't, and it's stuck. We're waiting for more real traffic to test it on, not on some technical problem. We could fake it with synthetic data right now and get an accuracy number, but that number wouldn't mean a thing.

Is prompt classification the same as model routing?

No. Classification figures out what sort of job a prompt is asking for, and routing picks which model gets that type of job. They are 2 different pieces, and they break for their own reasons. That's why they run as separate systems here.

A classification mistake might send a request about code to a model built for translation. A routing error happens when the task type is right but it gets sent to a model that isn't working.

Does classification send my prompt anywhere?

In the common path, no. That embedding step happens right on our own hardware, and the cache layers stay entirely inside the process. The only time we ever call an outside model is for the Tier 4 fallback, and that's limited to 200 prompt tokens. You can see how we handle data over at zero trust AI data security.

Can I skip classification and pick the model myself?

Yes, you can. Pinning a model skips the cascade completely. The automatic routing is on by default since it works for most chats, and the override is there for when you need one specific model. You can also change AI models during a conversation and not lose your place.

Does classification cost me credits?

No, everything before Tier 4 runs on our own hardware. The Tier 4 fallback is limited to 200 prompt tokens on a smaller model. The routing system is built to cut your costs, not add to them. We cover the energy piece in energy efficient AI model routing. The billing part is in AI credits vs tokens. And you can see the plans on our pricing page.

Try ChatFuse prompt classification

Every time you type something into ChatFuse, this is the system that handles it. The goal is that you won't ever see it working. You can start free and try it for yourself, or take a look at the options on our pricing page before you do.