inference performance / TOKEN WIKI
The total system throughput of
Throughput / Throughput
In one sentence
throughput is the number of requests or tokens processed within a given time. Statistical objects and test conditions need to be explained.
Why it matters
and the single user generation speed are not the same indicators. Higher aggregate throughput may be accompanied by longer queuing times.
Practical considerations
simultaneously reports concurrency, input and output length, success rate and latency quantiles to avoid out-of-condition comparisons. Is
FAQ
Token/s a single request or an indicator of the entire machine?
Both are possible, you need to confirm the test caliber first.
