Inference Gateways
KServe LLMInferenceService integrates with Kubernetes Gateway API
implementations to expose models, route inference requests, and apply
gateway-specific traffic policies. Choose a reference integration based on the
gateway and LLM traffic-management features you want to use.
Reference integrations
| Gateway | Integration | Use this guide for |
|---|---|---|
| agentgateway | LLMInferenceService with agentgateway | Standard InferencePool routing with optional AgentgatewayBackend configuration for AI policies |
| Envoy AI Gateway | LLMInferenceService with Envoy AI Gateway | Routing to an InferencePool with an endpoint picker, OpenAI-compatible routing, token accounting, and usage-based rate limiting |
Both guides use Gateway API resources and KServe's configurable router templates. Their gateway-specific custom resources and supported policies are different, so follow the guide for the gateway installed in your cluster.
For production llm-d Router deployment patterns, see the llm-d well-lit paths and gateway guides. The llm-d documentation also provides provider-specific infrastructure guides for agentgateway and Envoy AI Gateway.