Knowledge Maxxing← Back to topic library
AI MaxxingIntermediate60 sec
Inference latency
Inference latency is the time between sending a request to a model and receiving its response, a key factor in how responsive an AI product feels.
Prerequisite: Model serving
What matters
Larger models and longer outputs generally increase latency, so product teams often trade off model size or output length against how fast a response needs to feel.
Example
A model that takes eight seconds to respond may feel too slow for a live chat interface, even if its answer quality is excellent.
Common misconception
Lower inference latency does not automatically mean lower quality; latency and accuracy are related to model choices but are not the same thing.
Recall question
What trade-off often has to be made to reduce a model's inference latency?
Keywords and related concepts
inference latencyresponse timeperformancemodel serving