inference performance / TOKEN WIKI
The total system throughput of
inference delay
In one sentence
Inference latency describes the time elapsed from request initiation to output generation or completion. Queuing, first word, and total time consuming should be distinguished.
Why it matters
average may mask a long-tail experience, and interactive applications also need to observe high-quantile latency.
Practical considerations
records the first word time, generation speed, end-to-end time consumption, and p95 or p99 quantiles respectively. Does
FAQ
have a good experience if its average latency is low?
Not necessarily, long tail, failed retries and waiting for the first word may affect the user experience.
