inference performance / TOKEN WIKI
The total system throughput of
batch processing / Batching
In one sentence
batch processing combines multiple computing tasks to improve hardware utilization; the service system may also use continuous batch processing.
Why it matters
Batch size, queuing policy, and input differences affect throughput, memory, and request latency.
Practical considerations
tests batching strategies with real request distribution to balance resource utilization and service latency goals.
FAQ
Does batch processing necessarily make each request faster?
Not necessarily, it may improve the total throughput, but increase the waiting for individual requests. What is
