Computing Power Basics / TOKEN WIKI
There is a difference between the theoretical peak value of
bandwidth and interconnection
In one sentence
bandwidth represents the amount of data that can be transmitted per unit time; model services involve bandwidth at different levels such as video memory, device interconnection, and network.
Why it matters
correspond to different optimization methods, and Internet bandwidth cannot solve the communication problem between GPUs.
Practical considerations
Identify whether the bottleneck occurs in memory, equipment, nodes or client networks, and then select the corresponding hardware or architecture optimization.
FAQ
The higher the bandwidth, the lower the first word delay?
Not necessarily, queuing, model calculation, routing and round-trip delay will also affect the first word time.
