Throughput / Throughput
throughput is the number of requests or tokens processed within a given time. Statistical objects and test conditions need to be explained.
Read article ↗TOKEN WIKI / Industry Knowledge Base
Understanding word elements starts with a concept.
has 52 introductory entries, starting from word units, computing power going overseas, and word units going overseas, covering models, reasoning, API billing and global delivery.
All termsword elements and billingcomputing power basismodel and inferenceinference performanceoverseas and architectureSearch and DiscoveryAI development toolsintelligent body and application
7 个词条
inference delay
Inference latency describes the time elapsed from request initiation to output generation or completion. Queuing, first word, and total time consuming should be distinguished.
Read article ↗TTFT / First Token Time
TTFT refers to the time from the start of the request to the receipt of the first output Token, and is often used to measure the waiting experience of streaming interactions.
Read article ↗KV Cache
KV Cache caches the keys and values of previous Tokens in attention mechanism calculations to reduce repeated calculations in autoregressive generation.
Read article ↗batch processing / Batching
batch processing combines multiple computing tasks to improve hardware utilization; the service system may also use continuous batch processing.
Read article ↗model quantification
model quantization uses lower-precision numerical values to represent part of the model data or calculations to reduce resource overhead.
Read article ↗Streaming output / Streaming
streaming output allows applications to receive content incrementally before the complete result is generated, often used for conversations and real-time interactions.
Read article ↗