Knowledge Maxxing← Back to topic library
ML MaxxingAdvanced75 sec
Online inference
Online inference runs a trained model on a single request as it arrives, returning a prediction immediately for that one input.
Prerequisite: Batch inference
What matters
This mode is required whenever a live product needs a real-time answer, but it demands the model and serving infrastructure be fast enough to respond within an acceptable delay.
Example
A fraud-detection system evaluating a live transaction the instant a card is swiped uses online inference to make an immediate approve-or-block decision.
Common misconception
Online inference does not always require the largest or most accurate available model; teams often trade off some accuracy for the speed a live use case demands.
Recall question
Why does online inference put more pressure on serving speed than batch inference does?
Keywords and related concepts
online inferencereal-time predictionMLOpsbatch inference