LLM Streaming Latency: Why Time to First Token Wins

LLM streaming latency is the wait from when you send a prompt until words actually show up. What decides if it feels fast isn't the total time to finish, it's the time to first token.

2 responses might both need 8 seconds. One begins writing in 400ms and then adds to it. The other just shows a loading spinner for 7 seconds before dumping everything at once. Users will call the first one quick and the second one broken, even though the clock says they took the same time. ChatFuse is built on that exact idea.

Getting that first case to show up isn't just about flipping the streaming switch.

What makes LLM streaming latency feel slow?

Streaming doesn't feel slow because it's long; it feels slow because it's lumpy. The language model sends text in bursts. If you draw every chunk the instant it arrives, you get about 30 words at once, then a pause. That's much harder to read than a smooth, steady crawl at the exact same overall speed.

Your eye can follow a smooth flow easily, but a sudden jump makes it stop. Because of that, you can't let the delivery speed set the rendering speed. That's the entire point of having a buffer.

How is the ChatFuse streaming pipeline structured?

ChatFuse runs text through 7 layers. Every one does just one thing, so they don't have to know about each other. First it goes from the SSE consumer into a smoothing hook, then a content context, and then the message list. One markdown renderer handles it all, whether the text is still coming in or it's done.

The speed here comes from one simple thing: we don't touch the markdown while the text is still streaming in. The raw text goes right to the renderer without any processing. All the parsing work waits until the entire message has come through. The canvas markup that gets stripped out by that process is part of our AI document editor, which lives next to the chat window.

Why does a smoothing buffer beat rendering every chunk?

The ChatFuse buffer separates the speed text is written from the speed it arrives over the network, since data comes in bursts. Each piece of data writes its full text to a reference, which doesn't cause a re-render. A separate process then decides exactly when to show that text to the user. Drawing every chunk immediately would put each burst on the screen all at once.

If fewer than 12 characters are waiting, we show them right away. This prevents a slow model from getting an extra delay on top of its own. But if more than 12 characters are waiting, a 16ms timer starts and releases text at a fixed rate.

The timer doesn't cut a word in half. It goes to the next space before it shows anything, so you always see complete words, not pieces of them.

Does the ChatFuse buffer fall behind on a fast model?

No, the release rate adjusts based on how much text is in the queue. A small amount of text goes out at 8 characters each interval. When a model produces tokens quickly, it can send up to 40 per interval. The interval stops itself once it catches up to the source, so it doesn't run idle. A fast model won't ever get ahead of the reader.

Why not use requestAnimationFrame instead?

We tried requestAnimationFrame, and it gave us the very issue we'd set out to avoid. A single frame callback renders every token queued since the last refresh. A burst of network activity dumps a huge block of text all at once, so the reader gets a blob of words instead of a steady stream.

Display refresh drives the frame rate, but reading pace should drive the text. The interval method feels clunkier, but it reads far more naturally, and that's what made the decision for us.

How do you stop the whole thread re-rendering on every token?

ChatFuse makes it so only the message that's actively streaming gets re-rendered. Everything else just sits there. Streaming content runs through its own separate context, and it completely ignores the memo boundary that protects the rest of the conversation. A thread with 60 messages ends up doing the exact same amount of work for each new token as one that only has 2.

If you didn't do it this way, every single token would cost more as the thread gets longer. That's the opposite of what you need for a big chat. We talk about how this gets laid out on a phone over in mobile AI chat architecture.

Does the message flicker when streaming ends?

It doesn't, and the solution is to stick with a single renderer for everything. It's tempting to use a simple text streamer first and then swap in the full markdown engine after it's done. But that switch makes the whole thing jump around just as the user is reading the final line, which is the absolute worst time to have that happen.

You want one system that does it all, with a simple toggle for turning animations on or off. That way, finishing is just a change in state, not a complete swap of parts, and the layout stays perfectly still.

What happens when someone stops a response mid stream?

The ChatFuse cursor goes back to zero and the smoothing process stops. It sounds straightforward, but this exact part failed on 2 separate occasions. When text is wiped out or swapped while the buffer keeps a location from an earlier stream, the final piece gets calculated using content that has since been removed.

Because of that, the handler now looks to see if the new text is less than what was already there. It treats that as a full reset instead of a step forward.

All the code for tidying up after a stream sits in a single spot for a similar reason. When you spread that cleanup across different paths, like an abort, an error, or a normal finish, the logic starts to drift. The final state you get depends entirely on which exit you took.

How does scrolling keep up with the text?

The ChatFuse message list automatically scrolls to show new messages as they stream in, but it stops doing that once the stream is finished. The trick is to only track the most recent message. If you try to track the entire list, a new message arriving anywhere in the thread will pull the screen away from whatever you were looking at.

When it's done right, the view sticks to the message that's still being written. It won't bother you if you've already scrolled back up to read something else.

What should you measure instead of total time?

Measure 3 things: how fast the first word shows up, the longest wait between chunks, and the full run time. The first 2 tell you what a person actually feels. The total time only matters when you're checking models against the same job, and it doesn't say much about how responsive your product seems.

The time to first token is how long it takes before the user sees any text appear. That number decides if your product feels fast or slow, and it mostly comes down to which model you pick and how much processing happens before the request even leaves your system.

Most teams never check the gaps between each new piece of text. The average gap might look okay, but it's the worst one that really matters. Users notice the longest pause, not the overall average.

Total completion time only helps when you're comparing models on identical tasks, like in our Claude Fable 5 vs GPT-5.6 Sol breakdown. It doesn't tell you anything about how responsive the product feels. A conversation that takes longer to finish can actually seem faster to the person using it.

Is time to first token the same as latency?

No. Latency is typically the full round trip time. Time to first token is that initial wait before any text shows up, and that's the part people actually feel as speed. If a model takes 5 seconds to start but then finishes 2 seconds faster overall, users will still call it the slower one.

This is exactly where routing helps. If you send a quick prompt to a giant model, you're paying a first token cost you didn't need to. We talked about that in our post on AI model orchestration.

Does streaming change what the model produces?

No. Streaming is just how we deliver the tokens, and they are exactly the same as the ones you get in a full response. All the details here are about when the text appears, not what it says. Turning streaming on or off doesn't change the content you get back.

How do I see ChatFuse streaming for myself?

Send a good chunk of text if you want to see it stream, like asking for a full explanation instead of a quick answer. You can start for free and just watch the words come in, or check out AI prompt classification for a look at how the model gets chosen.