Knowledge Maxxing← Back to topic library
AI MaxxingIntermediate75 sec
Model gateway
A model gateway is a service layer that routes application requests to one or more underlying models, handling concerns like fallback, logging and rate limiting in one place.
Prerequisite: Model serving
What matters
Centralizing these concerns means an application can switch or add model providers, and enforce guardrails or logging consistently, without changing code throughout the whole application.
Example
A model gateway might route a request to a backup provider automatically if the primary model's service is down, without the calling application noticing.
Common misconception
A model gateway does not itself generate any responses; it only forwards requests to the models that do the actual generation.
Recall question
What problem does centralizing model requests through a gateway solve?
Keywords and related concepts
model gatewayroutingfallbackmodel serving