smart_toy Artificial Intelligence › Article

Latency and Tokens per Second: Why It Matters for UX

LLM latency is measured by time to first token and tokens per second. What causes it, how users perceive the wait and how to design for it and measure it.

info Quick Definition

The latency of an LLM is the time the model takes to process a prompt and generate the response. In a chat interface it helps to split it into two measures: the time to first token (TTFT), the wait before the first word appears, and the generation speed, measured in tokens per second, which decides how fast the rest of the text fills in.

What do time to first token and tokens per second measure?

Anthropic’s documentation defines latency as the time the model takes to process a prompt and generate an output, and notes that it depends on the size of the model, the complexity of the prompt and the underlying infrastructure. To measure it, it separates TTFT, which runs from the moment the prompt is sent until the first token is generated, from total latency.

A token is the piece of text the model works with, and generation happens one token at a time. That gives a formula that works for estimating any response:

total time ≈ TTFT + (output tokens − 1) × time between tokens

The time between tokens (ITL, inter-token latency) is the inverse of tokens per second: at 50 tokens per second, each token takes 20 milliseconds. Careful: “TPOT” (time per output token) is sometimes used as a synonym for ITL and sometimes not, so check how each source defines its measurement before comparing figures.

What makes a response slow?

A request goes through two phases. In the first, called prefill, the model reads the whole prompt at once. In the second, decode, it generates the response one token after another. Each phase has a different bottleneck and therefore a different symptom:

What the person sees	Likely phase	What influences it	What you can do
A still screen before the first word	Prefill (plus queueing and network)	Prompt length, retrieved context, provider load	Shorten the prompt, reuse the prompt prefix, show immediate feedback
Text that advances with pauses	Decode	Model size, provider load	Smaller model, less load
A long total wait with a long answer	Decode	Number of output tokens	Ask for shorter outputs, split the task

Output weighs far more than input. OpenAI states in its optimization guide that cutting 50% of the output tokens may cut about 50% of the latency, while cutting 50% of the prompt may improve it by only 1 to 5%.
A very long prompt does show up in TTFT. NVIDIA measured Llama 3.1 70B on a GH200 NVL32 system with batch size 1: 472 ms to first token with 32,768 input tokens and about 2.2 seconds with 122,880. That is a lab measurement on specialized hardware: your numbers will differ, but the trend (more context, longer wait) holds. Anthropic says prompt caching generally improves TTFT on long documents.

A reasoning model adds another factor: it generates intermediate steps before answering, so the visible text takes longer to arrive (see reasoning models).

How does the user perceive the wait?

Jakob Nielsen set three response-time limits in his book Usability Engineering (1993), and Nielsen Norman Group keeps them published with a 2014 update:

0.1 second: the limit for the person to feel the system reacts instantly. No feedback is needed beyond showing the result.
1 second: the limit for the train of thought to stay uninterrupted. Between 0.1 and 1 s the delay is noticeable, but it needs no special notice.
10 seconds: the limit for keeping attention on the dialogue. Beyond that, NN/g recommends a percent-done indicator when progress can be measured, and a clearly signposted way to interrupt the operation.

Those figures predate LLMs, and almost no model response fits in 0.1 seconds. That is why the design relies on perception: something should move early, and the person should know what is happening.

What can design do?

To lower the perceived wait, without changing the real time:

Streaming. Showing text as it is generated lets people start reading before it finishes. OpenAI describes it as the most effective technique for reducing perceived waiting. Details are in response streaming.
Immediate feedback. On submit, the interface responds in under 0.1 s with a state that describes the real step. Which indicator to use for each duration is covered in AI loading states.
Visible steps. If the task has stages (read, search, write), showing them turns an empty wait into progress. See long task progress.
Cancellation. Letting people stop a long response avoids waiting for something they no longer need. See interrupting and steering.

To lower the real time, with the technical team:

Shorter output and prompt. Ask for brief answers and remove redundant context from the prompt. Anthropic advises limiting by paragraphs or sentences, because word-count limits work worse with tokens. max_tokens cuts off abruptly, even mid-sentence.
Faster model. Choosing a smaller model is one of the most direct routes, as long as the task allows it (see small vs. large models).
Fewer calls, or parallel calls. OpenAI recommends combining sequential steps into one request and running independent steps in parallel.

How do you measure it in practice?

Save a timestamp when the request is sent.
Save another when the first visible text fragment arrives. The difference is TTFT.
Save another when it ends. Divide the remaining time by the output tokens minus one to get the time between tokens.
Log the input and output tokens of each request, to see which phase weighs more.

Measure on real requests, not just short examples, and look at the high percentiles. Wikipedia explains that tail latency (the 95th, 99th or 99.9th percentiles) matters because averages and medians can mislead.

When should you NOT optimize only for speed?

When speed comes from cutting quality or context. A fast wrong answer is worse than a slow correct one, because the person spends time fixing it. Anthropic recommends first getting a prompt that works well without constraints and applying latency strategies afterwards, so you know what the best result looks like.

Example

Illustrative example: a writing assistant takes 6 seconds to show the first word because it first searches the brand’s documents, and then writes 300 tokens in about 6 more seconds. The team makes three changes: it shows the step “Searching the style guide” at once, turns on streaming so text appears from the first fragment, and adds a stop button. The sum of times barely changes, but the person sees activity within the first tenth of a second and starts reading at 6 seconds instead of 12.

Common mistakes

Measuring only total latency and forgetting time to first token.
Showing a generic spinner for 20 seconds.
Trusting the average and ignoring the slow cases.
Optimizing speed without measuring answer quality.
Not allowing people to cancel a long generation.

Sources

Frequently asked questions

What is LLM latency?

It is the time the model takes to process a prompt and generate the response. It splits into two measures: time to first token, and generation speed, either in tokens per second or as time between tokens.

What are tokens per second?

It is the speed at which the model generates the response once it has started. Its inverse, the time between one token and the next, is what the person perceives as smooth typing or as pauses.

How long will users wait?

Nielsen Norman Group records three limits: 0.1 seconds to feel an instant reaction, 1 second to keep the train of thought, and 10 seconds as the limit of attention. Beyond 10 seconds it recommends a progress indicator and a clear way to interrupt.

How can I reduce perceived latency without changing models?

Stream the response, confirm immediately that the request was sent, show the real steps of a long task and let users cancel. To reduce real latency, shorten the output and the prompt.