AI MaxxingIntermediate75 sec

Model serving

Model serving is the infrastructure that receives a request, runs it through a trained model, and returns the result, making a model usable in a live application.

Mark as completed

What matters

Serving decisions, like batching requests together or choosing hardware, directly affect how quickly and cheaply an application can get results back from a model.

Example

A model-serving system might batch ten incoming requests together to run through the model at once, improving hardware efficiency compared to handling each individually.

Common misconception

Model serving is not the same as training; it only runs an already-trained model, and does not update the model's parameters.

Recall question

What is the difference between model serving and model training?

Keywords and related concepts