Computing Power Basics / TOKEN WIKI
There is a difference between the theoretical peak value of
Video Memory/VRAM
In one sentence
video memory is a dedicated memory used by the GPU to store model weights, intermediate states and other calculation data.
Why it matters
model weight is not the only one occupied. Long context, KV Cache and concurrency will also increase memory requirements.
Practical considerations
uses the actual deployment framework to test the peak memory, leaving system overhead and operating margin. Quantization reduces some weight memory. Can the
FAQ
model weight be installed and run stably?
Not necessarily, additional memory is required for runtime state and concurrent requests.
