inference performance / TOKEN WIKI
The total system throughput of
TTFT / First Token Time
In one sentence
TTFT refers to the time from the start of the request to the receipt of the first output Token, and is often used to measure the waiting experience of streaming interactions.
Why it matters
does not mean that the entire segment is generated faster. It is also necessary to observe the subsequent Token interval and total time consumption.
Practical considerations
clearly measures start and end points, includes necessary network and queuing times, and records input length and concurrency conditions.
FAQ
Is TTFT the same as full answer time?
is different. The complete answer also includes the subsequent generation process.
