inference performance / TOKEN WIKI
The total system throughput of
KV Cache
In one sentence
KV Cache caches the keys and values of previous Tokens in attention mechanism calculations to reduce repeated calculations in autoregressive generation.
Why it matters
It will occupy video memory, and memory management needs to be paid attention to when the sequence length and concurrency increase.
Practical considerations
combines the framework to evaluate cache memory, paging management and concurrency capacity, and does not confuse KV Cache with commercial billing cache. Is
FAQ
KV Cache just the supplier’s cache discount?
is not the same concept, and project cache and product billing rules need to be understood separately.
