Model and Inference / TOKEN WIKI
The capabilities of the
Inference / Inference
In one sentence
Inference is the process of using a trained model to calculate new inputs and generate outputs, which is different from training, which updates model parameters.
Why it matters
online inference focuses on request latency, throughput, stability and unit task cost.
Practical considerations
is stress tested according to the actual input and output length, concurrency and success criteria. Do not only use idealized short requests. What is the difference between
FAQ
model inference and model training?
training adjusts model parameters; inference typically uses existing parameters to process new inputs.
