LLM Gateway Compared: LiteLLM vs Portkey vs OpenRouter
LLM Gateway Compared: LiteLLM vs Portkey vs OpenRouter vs Cloudflare vs Vercel
TL;DR: An LLM gateway is a proxy between your application and model providers that unifies the API, routes and retries requests, and tracks spend. LiteLLM and Portkey are open-source gateways you can self-host. OpenRouter, Cloudflare AI Gateway and Vercel AI Gateway are managed services. Self-host when data residency and key custody matter most. Use a managed gateway when you want failover without running a proxy.
Key Takeaways
- An LLM gateway solves four problems at one layer: one API format across providers, automatic fallback when a provider fails, central key custody, and per-team spend control.
- LiteLLM is the self-hosted default for Python-heavy platform teams: an OpenAI-format proxy for 100+ providers with virtual keys, budgets, and YAML-defined fallback chains.
- Portkey's MIT-licensed gateway is the lightest self-hosted option, a Node.js gateway with JSON routing configs, conditional fallbacks on status codes, and built-in guardrails.
- OpenRouter is a model marketplace: one key, hundreds of models, no markup on inference, a fee on credit purchases, and provider routing by price, latency or throughput.
- Cloudflare AI Gateway and Vercel AI Gateway are edge-hosted control planes. Cloudflare's core features are free with a 5% fee on unified-billing credits. Vercel adds zero token markup, including with BYOK.
- An LLM gateway is not an MCP tool gateway. AgentCore Gateway routes agent tool calls; LiteLLM and its peers route model calls. Production agent stacks usually need both.
Why do teams put a gateway in front of their LLM calls?
The first LLM feature in most codebases calls one provider SDK directly. That works until the second provider shows up. Maybe a cheaper model wins on a subtask, a region needs a different vendor, or an outage takes down the only model you depend on. At that point every call site has to learn a second API shape, a second retry policy, and a second way of counting tokens.
The timing matters in late 2026. Frontier labs now distribute their models through several clouds at once: OpenAI's open-weight models on AWS Bedrock and, per this week's announcement, OpenAI models on Google Cloud, plus Anthropic models on Bedrock and Vertex AI. That makes "which model" and "which provider serves it" two separate decisions. The same model can be reached through three or four endpoints with different prices, rate limits and data-handling terms. A gateway is the layer that makes that choice at request time instead of at code-review time.
Search data from our own readers points the same way. Queries like "pros and cons of model-agnostic llm gateway for small apps" and "agentcore gateway llm gateway" show people trying to work out which kind of gateway they actually need. This guide answers that with a side-by-side comparison of the five gateways most teams shortlist, plus a framework for when you need none of them.
What exactly does an LLM gateway do?
An LLM gateway is a reverse proxy that speaks an LLM API (almost always the OpenAI Chat Completions format, increasingly also the Anthropic Messages and OpenAI Responses formats) on the front and translates to each provider's native API on the back. Everything else is policy that runs on that request path:
┌──────────────────────────── LLM Gateway ────────────────────────────┐
App ───▶ │ auth (virtual key) → budget check → cache lookup → route selection │ ───▶ Provider A
(OpenAI │ ↓ ↓ │ ───▶ Provider B
format) │ guardrails (pre) retry / fallback / cooldown │ ───▶ Self-hosted vLLM
│ ↓ ↓ │
│ guardrails (post) ← response normalization ← usage + cost log │
└─────────────────────────────────────────────────────────────────────┘
The capabilities break down into five groups:
1. API unification. One request shape and one response shape, so changing models means changing a string, not a client library.
2. Reliability. Retries with backoff, fallback chains across providers and models, and cooldowns that stop routing to a deployment after repeated failures.
3. Key custody. Provider keys live in the gateway. Applications and developers get scoped virtual keys that can be rotated or revoked without touching provider accounts.
4. Cost control. Token and dollar accounting per key, team or project, with budgets that reject requests once a limit is hit.
5. Observability and safety. Request logs, latency and token metrics, optional caching, and pre- or post-call guardrails such as PII redaction.
None of these are hard to build for one provider. What makes a gateway worth having is getting all five right across every provider at once, including streaming, tool calls and the error-code quirks that differ between vendors.
How do LiteLLM, Portkey, OpenRouter, Cloudflare and Vercel compare at a glance?
| Dimension | LiteLLM | Portkey | OpenRouter | Cloudflare AI Gateway | Vercel AI Gateway |
|---|---|---|---|---|---|
| Deployment | Self-hosted proxy (Docker/K8s) or Python SDK; enterprise tier | Self-hosted OSS gateway or Portkey Cloud | Managed only | Managed (Cloudflare edge) | Managed (callable from any infra) |
| License (core) | Open source + commercial enterprise features | MIT | Proprietary service | Proprietary service | Proprietary service |
| Implementation | Python | Node.js / TypeScript | — | — | — |
| Provider coverage | 100+ providers | 45+ providers | Hundreds of models across many providers | Major providers + Workers AI | Major providers, multi-modality |
| Fallback config | YAML: fallbacks, context_window_fallbacks, content_policy_fallbacks | JSON config: strategy.mode: fallback, on_status_codes | Request body: models array + provider.order | Gateway-level retries and model fallbacks | Provider order + model fallbacks |
| Keys & budgets | Virtual keys, per-key/team budgets (Postgres) | Virtual keys, budgets in hosted tier | One account key, credit balance | Gateway auth, rate limits | Team/project/key budgets |
| Pricing model | Free to self-host; you pay infra | Free to self-host; hosted plans | No inference markup; fee on credit purchases; 5% BYOK fee above allowance | Core free; 5% fee on unified-billing credits | Zero markup on tokens, incl. BYOK |
| Data path | Your network only | Your network (self-hosted) | Through OpenRouter | Through Cloudflare | Through Vercel |
| Best fit | Platform teams wanting full control | Lightweight self-hosting, guardrails | Fast access to many models, prototyping | Apps already on Cloudflare | TypeScript/AI SDK stacks |
Two rows decide most evaluations: deployment (who operates the proxy) and data path (whose network your prompts cross). Everything else is a feature delta you can usually work around.
How does LiteLLM work, and when is it the right choice?
LiteLLM ships in two forms. The Python SDK gives you litellm.completion() with one call signature across providers. The proxy server is the gateway: a standalone OpenAI-compatible service, configured through a config.yaml, that any language can call.
The proxy's strength is how much routing behavior you can express in configuration. Model groups are named aliases. Repeating a model_name across entries load-balances across deployments, and fallback chains map one group to others:
model_list:
- model_name: chat-default # alias your app calls
litellm_params:
model: bedrock/anthropic.claude-sonnet-4-5
aws_region_name: us-east-1
- model_name: chat-default # same alias = load-balanced deployment
litellm_params:
model: vertex_ai/claude-sonnet-4-5
vertex_location: us-east5
- model_name: chat-backup
litellm_params:
model: openai/gpt-5-mini
api_key: os.environ/OPENAI_API_KEY
router_settings:
enable_pre_call_checks: true # required for context-window fallbacks
fallbacks: [{"chat-default": ["chat-backup"]}]
context_window_fallbacks: [{"chat-default": ["chat-backup"]}]
litellm_settings:
num_retries: 3 # per group, before falling back
request_timeout: 60
allowed_fails: 3 # failures/minute before cooldown
cooldown_time: 30
This example shows the pattern described above: the same model served from two clouds as one load-balanced group, with a different model as the fallback. Your application only ever sends model: "chat-default".
On top of routing, the proxy adds virtual keys, per-key and per-team spend tracking and budgets (backed by Postgres), callbacks to observability tools such as Langfuse, and an admin UI. LiteLLM publishes benchmark figures for proxy overhead (it cites 8 ms P95 at 1k RPS). Treat that as a vendor number to reproduce on your own hardware, not a guarantee.
Choose LiteLLM when you have a platform team, a Python-friendly ops stack, and requirements like "keys never leave our VPC" or "every team gets a budget". Watch out for the operational surface. You own Postgres, Redis (if you enable shared caching or rate limits), upgrades, and supply-chain hygiene. Pin exact versions and verify the signed Docker images LiteLLM publishes rather than pulling latest. A gateway holds every provider credential you own, so it is the highest-value target in your AI stack.
How does Portkey's gateway differ from LiteLLM?
Portkey's open-source AI Gateway (MIT licensed) runs on Node.js: npx @portkey-ai/gateway starts it locally, and it deploys to Docker, a plain Node server, or Cloudflare Workers. Routing is defined in JSON configs that can be attached to a request header or saved on the hosted platform:
{
"strategy": { "mode": "fallback", "on_status_codes": [429, 503] },
"targets": [
{ "override_params": { "model": "@openai-prod/gpt-5-mini" } },
{ "override_params": { "model": "@anthropic-prod/claude-sonnet-4-5" } }
]
}
By default, a fallback fires on any non-2xx response. on_status_codes narrows it, which matters more than it looks. You usually want to fall back on a 429 (rate limit) or 503 (overload), but a 400 (your request is malformed) will fail identically on every target. Retrying it just multiplies latency and cost. Targets can themselves be load-balancers or nested strategies, so you can build "load-balance across two keys, fall back to a second provider" as one tree.
Portkey's other differentiator is guardrails. The gateway has a large catalogue of pre-built checks (regex, JSON-schema, PII and others) that can run before or after a model call and can deny, flag or trigger a fallback. Semantic caching, the richer observability UI and team management live in the hosted and enterprise tiers.
Choose Portkey when you want a small self-hosted footprint, a TypeScript-native stack, or guardrails at the gateway rather than in each service. Choose LiteLLM instead when you need the deepest provider coverage and a self-hosted admin and budget system without a paid tier.
When does OpenRouter make more sense than a self-hosted gateway?
OpenRouter flips the model. You don't run a gateway; you buy credits and get one API key for a very large catalogue of models from many inference providers. It has two routing layers:
- Model fallback. A models array in the request lists alternatives in priority order.
- Provider routing. The provider object controls which upstream serves a given model: order, only, ignore, sort by price/latency/throughput, a max_price ceiling, and data-policy filters such as data_collection: "deny" and zdr: true for zero-data-retention endpoints.
{
"models": ["anthropic/claude-sonnet-4.5", "openai/gpt-5-mini"],
"messages": [{ "role": "user", "content": "Summarize this incident report." }],
"provider": {
"sort": "latency",
"data_collection": "deny",
"zdr": true,
"max_price": { "prompt": 3, "completion": 15 }
}
}
The economics are simple. OpenRouter states there is no markup on inference pricing. Its revenue is a fee on credit purchases, plus a 5% fee on bring-your-own-key usage above a monthly allowance. For open-weight models served by many competing providers, sort: "price" can land well below what one provider would charge.
Choose OpenRouter when you need breadth fast: prototyping across many models, evaluating open-weight models without signing up with each host, or a small app that wants failover with zero infrastructure. Avoid it as the only path when your contracts require prompts to stay inside your cloud account, or when you need enterprise discounts negotiated directly with a provider.
What do Cloudflare AI Gateway and Vercel AI Gateway add?
Both are managed control planes run by edge platforms. You keep calling a model API, but through their endpoint, and you get logging, caching and failover without deploying anything.
Cloudflare AI Gateway sits in front of Workers AI, OpenAI, Anthropic, Google Gemini and other providers. Its core features (analytics dashboard, caching and rate limiting) are free on all plans, and it supports request retries and model fallbacks. Logs are subject to per-plan storage limits. With unified billing, Cloudflare buys inference on your behalf at the provider's per-token rate and charges a 5% fee on purchased credits. Guardrails are billed as Workers AI inference. It fits best if your app already runs on Workers, because caching and rate limiting then sit on the same network as your code.
Vercel AI Gateway is callable from any infrastructure, not just Vercel deployments, through the AI SDK, OpenAI Chat Completions, OpenAI Responses, or Anthropic Messages formats. It routes across providers with configurable provider order and model fallbacks, records every routing attempt in request logs, and supports budgets per team, project, API key or member. Vercel states it adds zero markup to provider token prices, including BYOK traffic. One detail to read before relying on it: budgets cover spend billed through Vercel's system credentials, while BYOK spend is metered separately. If a BYOK request fails, the gateway can fall back to system credentials, so check that behavior if "only use our provider contract" is a hard requirement.
Choose an edge gateway when your app already lives on that platform and you want observability and failover in an afternoon. Look elsewhere when you need custom routing logic, on-prem models behind your firewall, or a single audit log spanning several clouds you control.
Is an LLM gateway the same as AgentCore Gateway or an MCP gateway?
No, and mixing them up is the most common mistake we see in architecture reviews. Both are called "gateways", but they sit on different arrows of an agent diagram:
| | LLM gateway | MCP / tool gateway |
|---|---|---|
| Routes | Model inference calls (prompt → completion) | Tool calls (agent → external API/function) |
| Protocol | OpenAI / Anthropic-style chat APIs | Model Context Protocol (MCP) |
| Examples | LiteLLM, Portkey, OpenRouter, Cloudflare, Vercel | AWS Bedrock AgentCore Gateway, self-hosted MCP routers |
| Core concern | Provider choice, failover, token spend | Tool discovery, auth to downstream systems, authorization |
[AgentCore Gateway](/articles/aws-bedrock-agentcore-gateway-evaluation-how-does-it-compare) turns Lambda functions and APIs into MCP tools that agents can discover and call with IAM-scoped credentials. It does not choose which model answers a prompt. A production agent typically needs both: an LLM gateway for the reasoning loop, and a tool gateway for actions, with [deterministic authorization](/articles/AI-Agent-Authorization-LLM-vs-Policy-Engine) on the tool side. Some vendors now market a combined "AI gateway" for both. Evaluate the two halves separately anyway, because their failure modes don't overlap.
What are the pros and cons of a model-agnostic gateway for a small app?
This is the question behind the most specific search query we see on the topic, and the honest answer is that a small app often doesn't need a self-hosted one.
Pros, even at small scale:
- Outage insurance. One fallback entry turns a provider incident from downtime into slightly different answers.
- Model agility. When a cheaper model matches your quality bar, switching is a config change, not a refactor.
- One bill and one dashboard. Managed gateways consolidate spend that would otherwise be scattered across provider consoles.
Cons:
- An extra hop and an extra dependency. Your availability is now the gateway's availability multiplied by the provider's. A self-hosted proxy that you forgot to scale is a worse single point of failure than any provider.
- Lowest-common-denominator APIs. Provider-specific features (prompt caching controls, extended-thinking parameters, server-side tools, citations) arrive in gateways late or are flattened by the translation layer. If your product depends on one of them, test it through the gateway before committing.
- Another party in the data path. With a managed gateway, prompts cross a third network and fall under a third set of retention terms.
The pragmatic path for a small app: start with a provider-agnostic SDK rather than a gateway. The Vercel AI SDK or LiteLLM's Python SDK gives you one call signature with no proxy to run; our [TypeScript LLM SDK guide](/articles/best-llm-sdk-for-typescript-apps) compares the options. Add a managed gateway when you need cross-provider failover or spend visibility, and self-host only when key custody or data residency force it.
How should you choose an LLM gateway?
Work through four questions in order. Each one eliminates options:
1. Must prompts and keys stay inside your network? If yes, self-host: LiteLLM or Portkey OSS. Managed gateways are out, whatever their features.
2. Who will operate it? With no platform team, a managed gateway (OpenRouter, Cloudflare, Vercel) almost always beats a self-hosted proxy nobody upgrades. With one, LiteLLM's budgets and admin UI pay for their operational cost quickly.
3. Where does your app already run? On Workers, start with Cloudflare. On Vercel or the AI SDK, start with Vercel AI Gateway. In a Python service mesh, start with LiteLLM.
4. How many models do you really need? Two or three providers on enterprise contracts favor a self-hosted router over your own keys. Dozens of models for evaluation or open-weight cost-shopping favor OpenRouter.
Whatever you pick, test it the same way before production:
- Kill a provider. Point one deployment at an invalid key or a blackholed endpoint, then confirm traffic shifts and the switch shows up in logs.
- Exceed a context window. Send an oversized prompt and confirm it falls back to a larger-context model (or fails fast) rather than retrying the same error.
- Exhaust a budget. Set a $1 limit on a test key and confirm the next request is rejected, not silently billed.
- Stream a tool call. Run a streaming response with function calling end-to-end, because streaming plus tools is where translation layers break most often.
- Measure overhead. Record P50/P95 latency with and without the gateway on your real payload sizes.
Feed the gateway's logs into the same tracing backend as the rest of your agent. Our [LangSmith vs Langfuse vs Phoenix comparison](/articles/LangSmith-vs-Langfuse-vs-Phoenix-LLM-Agent-Observability) covers how to pick one. A fallback that fires silently is a cost and quality change you want to see.
Frequently Asked Questions
What is an LLM gateway?
An LLM gateway is a proxy between applications and model providers that exposes one API, usually OpenAI-compatible, and applies routing, retries, fallbacks, key management, caching and spend tracking to every request. It lets you change models or providers through configuration instead of code.
Is LiteLLM or Portkey better for self-hosting?
LiteLLM is better when you want the widest provider coverage and built-in virtual keys, team budgets and an admin UI in the self-hosted product. Portkey's MIT gateway is better when you want a lighter Node.js deployment, JSON routing trees with status-code-specific fallbacks, and gateway-level guardrails.
Does OpenRouter charge more than calling providers directly?
OpenRouter states it adds no markup to inference pricing. It charges a fee when you buy credits, and a 5% fee on bring-your-own-key usage above a monthly allowance. Because it can route to the cheapest provider for open-weight models, total cost can come out lower than a single direct provider.
Do I need an LLM gateway if I already use AWS Bedrock?
Not always. Bedrock already gives you many models behind one AWS API, plus cross-region inference. A gateway becomes useful when you also call providers outside Bedrock, need failover to another cloud, or want per-team virtual keys and budgets that span providers. AgentCore Gateway handles MCP tool calls, not model routing.
Does an LLM gateway add noticeable latency?
A well-run gateway adds single-digit to low tens of milliseconds, which is small next to model generation time measured in seconds. The bigger latency risks are misconfigured retries that repeat a non-retryable error, and an undersized self-hosted proxy. Measure P95 with your own payloads before and after adding it.
Can an LLM gateway route to self-hosted models like vLLM?
Yes. LiteLLM, Portkey and Cloudflare AI Gateway can all target any OpenAI-compatible endpoint, which includes [vLLM](/articles/what-is-vllm) servers. A common pattern is to route routine traffic to a self-hosted open-weight model and fall back to a hosted frontier model on errors or context-length limits.
Sources
- LiteLLM GitHub Repository - OpenAI-format proxy and SDK for 100+ providers
- LiteLLM Proxy Reliability - Fallbacks, retries and cooldowns
- Portkey AI Gateway GitHub Repository - MIT-licensed open-source gateway
- Portkey Fallbacks Documentation - Strategy configs and status-code triggers
- OpenRouter Provider Routing - Order, sort, data policy and price ceilings
- OpenRouter FAQ - Pricing, credit fees and BYOK
- Cloudflare AI Gateway Documentation - Caching, rate limiting and fallbacks
- Cloudflare AI Gateway Pricing - Free features and unified billing fee
- Vercel AI Gateway Documentation - Routing, budgets, BYOK and zero markup
---
Originally published at fp8.co. Subscribe for weekly AI engineering analysis at fp8.co.
Mentions
社内のLLMリクエストを一か所に集約したAI Gateway構築記 ! この記事は2026年9月8日に執筆されたものです。 こんにちは、DevOpsエンジニアのPoka（ポカ）です。 Inflabでは、字幕翻訳、講座詳細ページの生成、学習エージェント、コミュニティの回答など、かなり多くの場所でLLM APIを使っています。 当初は各サービスが必要なプロバイダーでAPIキーを発行し、直接呼び出す方式でした。 しかし利用箇所が30を超えたあたりから、この方式では徐々に限界が見えてきました。 本記事では、社内LLMルーティングサービスであるAI Gatewayを作ることになった背景と、構築の過程を共有します。 背景 モデルの入れ替わりが速すぎる GPT、Claude、Gemini、Qwenといったプロバイダーが、ほぼ毎週のように、より安くて性能の良いモデルを出しています。 同じプロバイダーの中でも、せいぜい3か月ほどの周期で新バージョンが出て、古いモデルは提供終了となり、呼び出せなくなります。 ここで2つの問題が同時に起きます。 「今いちばんコスパの良いモデル」の寿命が非常に短い - 割引キャンペーンが絡むとさらに短くなります。 モデルの更新を見逃すとサービスが止まる - 提供終了日までに切り替えられなければ、そのまま障害になります。 モデルを切り替えるときは、既存モデルと新モデルの両方が満足のいく結果を出すかを検証する必要があります。 この検証を各サービスの担当者が短いサイクルで繰り返すと、相当な時間がかかります。 忙しければ切り替えはどんどん後回しになり、後回しが続くと上記の2つ目の問題につながります。 コストも無視できない水準まで膨らんでいました。 2026年6月時点で、InflabのLLM API費用はAWS費用の約23% に相当していました。 同じ性能をより安く使えるかどうかが、インフラ全体のコスト効率を左右するほどになっていたわけです。 キーと使用量が散らばっている 各サービスがプロバイダーを直接呼び出すと、サービスの数だけキーが増えます。 キーが増えると失効やローテーションが難しくなり、漏洩したときの影響範囲の把握も難しくなります。 使用量の確認も面倒でした。 どの機能がどれだけトークンを使っているかを見るには、プロバイダーのダッシュボードを一つずつ開く必要があり、それでも「どのサービスのどの機能が使ったのか」は区別できませんでした。 まとめると、モデル選択・キー管理・使用量の可観測性の3つを一か所に集約することが目標でした。 では、どのゲートウェイを使うべきか？ LiteLLM Proxy、Portkey AI Gateway、Envoy AI Gatewayを比較しました。 3つとも複数LLMへの統合ルーティングとフォールバックには対応しているので、決め手はそれ以外の項目になりました。 最初に比較したのは2025年11月で、本記事の執筆にあたり2026年9月時点の情報で再確認しています。 いずれもセルフホスティング + 無料プランで使えるかどうかを基準に記載しています。 3製品とも変化が速いので、導入を検討される場合は最新のドキュメントを改めて確認することをおすすめします。 機能 Envoy AI Gateway LiteLLM Proxy Portkey AI Gateway 予算/レートリミット（コスト/トークンベース） 可能（ポリシー設定） 有料のみ 有料のみ 可観測性（ログ/メトリクス/トレース） 可能（OpenTelemetry） ロギングのみ可能 メトリクスのエクスポートは有料 基本ダッシュボードのみ コストトラッキング 可能（設定が必要） 基本集計のみ可能 キー/チーム別は有料 有料のみ ガードレール 可能 有料のみ 可能 プロンプト管理 なし 可能 有料のみ 決め手はCNCFプロジェクトであることでした 表をもう一度見ていただくと、有料のみと書かれたセルが一つもないのは、Envoy AI Gatewayの列だけです。 プロンプト管理はそもそも存在しませんが、有料プランでなければ使えない機能は一つもありません。 Envoy AI GatewayはCNCFが管理するEnvoyプロジェクトの一部なので、機能が商用プラン限定になることがありません。 これは「今いくら払うか」という問題ではありませんでした。 ゲートウェイを置く目的そのものが、予算ポリシー、レートリミット、使用量の追跡、認証といった統制の仕組みを一か所に集めることです。 そして、まさにこの統制の仕組みこそが、たいてい商用プランのセールスポイントになっています。 利用箇所が増えてポリシーを強化すべきタイミングで、機能ごとにプランの交渉をやり直すような状況は避けたかったのです。 可観測性がまさにその例でした。 私たちはすでにメトリクスとトレースをPrometheusやOpenTelemetryベースのスタックに集約して見ていましたが、LiteLLM Proxyは使用量を独自UIの中でしか見られず、既存のダッシュボードに統合する方法がありませんでした。 社内ダッシュボード一か所で使用量を見ることが今回の主な目標だったので、そのまま不採用の理由になりました。 受け入れたトレードオフ もちろんタダで手に入ったわけではなく、付随するコストもありました。 Envoy Gatewayも一緒に導入する必要がある - Envoy AI GatewayはEnvoy Gatewayの上で動作します。当時はEnvoy Gatewayを使っていなかったので、AI Gatewayの導入に合わせて一緒に取り入れました。 プロンプト管理機能が弱い - プロンプトは各サービスのリポジトリで管理するほうが自然だと考え、許容しました。 その代わり、ルーティングと認証ポリシーをすべてKubernetesのカスタムリソースとして宣言できる点は、私たちの運用によく合っていました。 社内のインフラ設定はすべてGitOpsで管理しているので、新しいプロバイダーを追加する作業が、コンソールのクリックではなく、コードレビュー付きのPull Request 1件で完結するようになりました。 構成 まず使う側の話をすると、知っておくべきことはほとんどありません。 ゲートウェイがOpenAI APIの仕様でリクエストを受け取り、各プロバイダーの仕様に変換してくれるので、ほとんどの場合、OpenAIのライブラリ1つでClaudeもGeminiもQwenもすべて呼び出せます。 Inflabの構成では、1つのゲートウェイが3つのアクセス経路を受け持ち、その背後に5つのプロバイダーがつながっています。 使ううえで必要なのはここまでです。 ここから先はこのゲートウェイがどう構成されているかの話なので、興味のある方だけ読んでいただければ大丈夫です。 Envoy AI Gatewayは、設定を管理するControl Planeと、実際のリクエストが流れるData Planeに分かれています。 Envoy Gatewayの上に載る拡張なので、ルーティングと認証をKubernetesリソースとして宣言すると、コントローラーがそれをプロキシの設定に落とし込んでくれる仕組みです。 AIGatewayRoute - どのモデルへのリクエストをどのバックエンドに送るかを定義します AIServiceBackend - プロバイダーのエンドポイントとAPI仕様を定義します BackendSecurityPolicy - そのバックエンドで使う認証情報を定義します これらのリソースを適用すると、次のような流れで反映されます。 AIGatewayRouteを適用 -> AI Gateway ControllerがHTTPRouteと設定用Secretを生成 -> Envoy Gateway ControllerがxDS設定を生成 -> AI Gateway Extension ServerがxDSにLLM処理用フィルターを追加 -> Envoy Proxy + ExtProc sidecarがリクエストを処理 モデル名を消して「ティア」だけを残す モデルの切り替えを中央で処理するには、サービス側がモデル名を直接指定してはいけません。 サービスのコードに gemini-2.5-flash がハードコードされていると、そのモデルが提供終了になるときに結局サービスのコードを修正することになります。 それではゲートウェイを置いた意味がなくなってしまいます。 そこで、モデル名の代わりに low、medium、high の3つのティアだけを公開することにしました。 名前 入力100万トークンあたりの最大コスト 推奨用途 high $2 ハルシネーションの影響を非常に受けやすい場合や、複雑な推論が必要な場合に使用 medium $1 ユーザーがテキストを直接目にする場合や、後処理に使うデータの生成に使用 low $0.3 簡単なロジック、短い回答、分類、構造化データの生成に使用 各ティアに実際に紐づくモデルは、時間とともに変わります。 ただし、表に記載した最大コストは契約のように固定されます。 どのモデルが割り当てられるかは分からなくても上限は分かるので、月間の想定入力トークン数 × 上限コスト で入力トークンの費用を事前に見積もることができます。 例えば、月に入力トークンを1億使うと見込まれる機能で low を選んだ場合、100万トークン単位に換算すると100なので 100 × $0.3 = $30 です。出力トークンの費用はこれとは別にかかります。 実際に呼び出すときは、model にティア名を入れるだけです。 curl -s '<社内AI GatewayのURL>/v1/chat/completions' \ -H 'Content-Type: application/json' \ -H 'Authorization: Bearer <サービス名>' \ -H 'X-Service-Feature: onboarding-chat' \ -d '{"model":"low","messages":[{"role":"user","content":"hi"}]}' -> レスポンスのmodelフィールドには、実際にルーティングされたモデル名が入って返ってきます ルーティング構成 ゲートウェイの設定で、ティアを実際のモデルに置き換えます。 同じモデルを複数のプロバイダーから利用できる場合は、優先順位を付けてフォールバックを設定しています。 medium: 割引中のルートを先に叩き、レートリミットにかかったら定価ルートに切り替える - matches: - headers: - type: Exact name: x-ai-eg-model value: medium backendRefs: - name: envoy-ai-gateway-openrouter modelNameOverride: openai/gpt-5.6-luna priority: 0 - name: envoy-ai-gateway-openai modelNameOverride: gpt-5.6-luna priority: 1 これに加えて、429レスポンスと接続失敗に対するリトライポリシーを設定しました。 同じモデルをより安く提供するルートが出てきたらそちらを最優先にし、レートリミットにかかったときだけ定価ルートに切り替わります。 この判断がサービスのコードではなく、ゲートウェイの設定にあることがポイントです。 モデルを切り替えるときの手順 モデルが変わると、モデルの特性によってレスポンスの形が変わったり、これまでうまく動いていたパターンが動かなくなったりと、副作用が出てサービスに影響を与える可能性があります。 これを中央で極力抑えることもゲートウェイの役割だと考え、次の手順を定めました。 ティアごとに、最低限の品質基準を自動で検証できるテストセットを用意します。 上限コストを超えず、よりコスパが良さそうなモデルが出たら候補に挙げます。 既存モデルと候補モデルに対して同じテストセットを実行します。1つでも失敗したら不採用とします。 通過したら開発部門全体に切り替え予定日を告知し、関連機能のモニタリングをお願いします。 切り替え後、そのティアを使う機能が正常に動作しているかを重点的に確認し、新たに見つかった品質基準はテストセットに追加します。 でも、ちょっと窮屈じゃないですか？ はい、機能ごとにぴったりのモデルを選ぶ自由は減ります。 それでもこの方向を選んだのは、ある時点で完璧に最適化された選択をすることよりも、変化に素早く追従できる能力のほうが長く利益をもたらすと考えたからです。 あるモデルが大幅に値下げされたとき、30のサービスがそれぞれ対応する構造と、ゲートウェイが一度だけ対応する構造とでは、時間が経つほど差が開いていくと思います。 誰がどれだけ使ったかをどう把握するか？ ゲートウェイがキーを一元管理すると、プロバイダーのダッシュボード上ではすべてのリクエストがひとまとまりに見えます。 そのため、ゲートウェイ自身が「誰が呼び出したか」を記録する必要がありました。 最初は、サービスごとにゲートウェイ用のAPIキーを発行する案を検討しました。 しかし、なくそうとしていたキーがまた増えるのが引っかかり、発行ではなく命名規則と検証を使う方向に切り替えました。 APIキーの代わりに、デプロイしているアプリケーション名を入れてもらうと、ゲートウェイが許可された形式かを確認したうえで、サービス識別用のヘッダーに移し替えます。 さらに、先ほどの例にあった X-Service-Feature ヘッダーで機能名も受け取ります。 これにより、「どのサービスのどの機能が、どのモデルをどれだけ呼び出したか」を社内ダッシュボード一か所で確認できるようになりました。 成果 社内の30を超えるサービスが、ゲートウェイを経由してLLMを呼び出しています。サービスのコードからプロバイダーのキーがなくなりました。 新しいプロバイダーを追加する作業が、GitOpsのPull Request 1件に集約されました。 サービスと機能の2軸で、トークン使用量とコストを確認できるようになりました。 モデルの切り替えがゲートウェイの設定変更1回で完了します。大幅に値下げされたモデルが出たら、ティアの紐付けを変えるだけで全サービスに一度に反映されます。 では、コストはどれくらい下がったのか？ InflabのLLM費用が最も高かったのは6月でした。 ティアルーティングを本番環境に適用したのは8月初めなので、適用後初めて丸1か月運用した8月を現状の値として、ピーク時と比較しました。 LLM費用全体が、ピーク時と比べて約78% 減少しました。 これには2つの要因が同時に作用しました。 1つ目はティアルーティングです。 medium ティアの実際のブレンド単価は100万トークンあたり$0.18でした。 ティア表に記載した medium の上限$1の5分の1程度です。 キャッシュヒット率が80%を超えたことも、単価を押し下げるのに一役買いました。 2つ目は、使用量が見えるようになったことです。 それまではプロバイダーのダッシュボードにリクエストがひとまとまりで記録され、どの機能がいくら使っているのか分かりませんでした。 サービスと機能の2軸に分けてみると、想定よりはるかに多く使っていた機能や、わざわざ高いモデルを使う必要のなかった機能が浮かび上がってきました。 ルーティングを変えなくても、見えるようになっただけで改善される部分がありました。 おわりに 変化の速い領域では、最適な選択を維持し続けるよりも、選択を変えるコストを下げるほうが得策です。 モデル名をティアで隠すという決定によって、個々の機能の最適化を少しあきらめる代わりに、組織全体の反応速度を手に入れたと言えます。 管理対象を増やさずに解決する方法がないか、まず探してみるのがおすすめです。 サービスを識別するためにゲートウェイ用のキーを新たに発行しようとして、命名規則と検証に方向転換したのがその一例です。 そもそもキーを減らすために始めた作業だったので、キーを再び増やさない選択が正しかったと考えています。 今後は、ティアごとのテストセットをより細かく作り込み、モデル切り替えの判断を自動化することに注力していく予定です。 同じような悩みを抱えている方の参考になれば幸いです。最後まで読んでいただき、ありがとうございました！ 参考資料 Envoy AI Gateway Envoy AI Gateway Concepts Envoy Gateway LiteLLM Proxy Portkey AI Gateway
LLM Gateway Compared: LiteLLM vs Portkey vs OpenRouter LLM Gateway Compared: LiteLLM vs Portkey vs OpenRouter vs Cloudflare vs Vercel TL;DR: An LLM gateway is a proxy between your application and model providers that unifies the API, routes and retries requests, and tracks spend. LiteLLM and Portkey are open-source gateways you can self-host. OpenRouter, Cloudflare AI Gateway and Vercel AI Gateway are managed services. Self-host when data residency and key custody matter most. Use a managed gateway when you want failover without running a proxy. Key Takeaways - An LLM gateway solves four problems at one layer: one API format across providers, automatic fallback when a provider fails, central key custody, and per-team spend control. - LiteLLM is the self-hosted default for Python-heavy platform teams: an OpenAI-format proxy for 100+ providers with virtual keys, budgets, and YAML-defined fallback chains. - Portkey's MIT-licensed gateway is the lightest self-hosted option, a Node.js gateway with JSON routing configs, conditional fallbacks on status codes, and built-in guardrails. - OpenRouter is a model marketplace: one key, hundreds of models, no markup on inference, a fee on credit purchases, and provider routing by price, latency or throughput. - Cloudflare AI Gateway and Vercel AI Gateway are edge-hosted control planes. Cloudflare's core features are free with a 5% fee on unified-billing credits. Vercel adds zero token markup, including with BYOK. - An LLM gateway is not an MCP tool gateway. AgentCore Gateway routes agent tool calls; LiteLLM and its peers route model calls. Production agent stacks usually need both. Why do teams put a gateway in front of their LLM calls? The first LLM feature in most codebases calls one provider SDK directly. That works until the second provider shows up. Maybe a cheaper model wins on a subtask, a region needs a different vendor, or an outage takes down the only model you depend on. At that point every call site has to learn a second API shape, a second retry policy, and a second way of counting tokens. The timing matters in late 2026. Frontier labs now distribute their models through several clouds at once: OpenAI's open-weight models on AWS Bedrock and, per this week's announcement, OpenAI models on Google Cloud, plus Anthropic models on Bedrock and Vertex AI. That makes "which model" and "which provider serves it" two separate decisions. The same model can be reached through three or four endpoints with different prices, rate limits and data-handling terms. A gateway is the layer that makes that choice at request time instead of at code-review time. Search data from our own readers points the same way. Queries like "pros and cons of model-agnostic llm gateway for small apps" and "agentcore gateway llm gateway" show people trying to work out which kind of gateway they actually need. This guide answers that with a side-by-side comparison of the five gateways most teams shortlist, plus a framework for when you need none of them. What exactly does an LLM gateway do? An LLM gateway is a reverse proxy that speaks an LLM API (almost always the OpenAI Chat Completions format, increasingly also the Anthropic Messages and OpenAI Responses formats) on the front and translates to each provider's native API on the back. Everything else is policy that runs on that request path: ┌──────────────────────────── LLM Gateway ────────────────────────────┐ App ───▶ │ auth (virtual key) → budget check → cache lookup → route selection │ ───▶ Provider A (OpenAI │ ↓ ↓ │ ───▶ Provider B format) │ guardrails (pre) retry / fallback / cooldown │ ───▶ Self-hosted vLLM │ ↓ ↓ │ │ guardrails (post) ← response normalization ← usage + cost log │ └─────────────────────────────────────────────────────────────────────┘ The capabilities break down into five groups: 1. API unification. One request shape and one response shape, so changing models means changing a string, not a client library. 2. Reliability. Retries with backoff, fallback chains across providers and models, and cooldowns that stop routing to a deployment after repeated failures. 3. Key custody. Provider keys live in the gateway. Applications and developers get scoped virtual keys that can be rotated or revoked without touching provider accounts. 4. Cost control. Token and dollar accounting per key, team or project, with budgets that reject requests once a limit is hit. 5. Observability and safety. Request logs, latency and token metrics, optional caching, and pre- or post-call guardrails such as PII redaction. None of these are hard to build for one provider. What makes a gateway worth having is getting all five right across every provider at once, including streaming, tool calls and the error-code quirks that differ between vendors. How do LiteLLM, Portkey, OpenRouter, Cloudflare and Vercel compare at a glance? | Dimension | LiteLLM | Portkey | OpenRouter | Cloudflare AI Gateway | Vercel AI Gateway | |---|---|---|---|---|---| | Deployment | Self-hosted proxy (Docker/K8s) or Python SDK; enterprise tier | Self-hosted OSS gateway or Portkey Cloud | Managed only | Managed (Cloudflare edge) | Managed (callable from any infra) | | License (core) | Open source + commercial enterprise features | MIT | Proprietary service | Proprietary service | Proprietary service | | Implementation | Python | Node.js / TypeScript | — | — | — | | Provider coverage | 100+ providers | 45+ providers | Hundreds of models across many providers | Major providers + Workers AI | Major providers, multi-modality | | Fallback config | YAML: fallbacks, context_window_fallbacks, content_policy_fallbacks | JSON config: strategy.mode: fallback, on_status_codes | Request body: models array + provider.order | Gateway-level retries and model fallbacks | Provider order + model fallbacks | | Keys & budgets | Virtual keys, per-key/team budgets (Postgres) | Virtual keys, budgets in hosted tier | One account key, credit balance | Gateway auth, rate limits | Team/project/key budgets | | Pricing model | Free to self-host; you pay infra | Free to self-host; hosted plans | No inference markup; fee on credit purchases; 5% BYOK fee above allowance | Core free; 5% fee on unified-billing credits | Zero markup on tokens, incl. BYOK | | Data path | Your network only | Your network (self-hosted) | Through OpenRouter | Through Cloudflare | Through Vercel | | Best fit | Platform teams wanting full control | Lightweight self-hosting, guardrails | Fast access to many models, prototyping | Apps already on Cloudflare | TypeScript/AI SDK stacks | Two rows decide most evaluations: deployment (who operates the proxy) and data path (whose network your prompts cross). Everything else is a feature delta you can usually work around. How does LiteLLM work, and when is it the right choice? LiteLLM ships in two forms. The Python SDK gives you litellm.completion() with one call signature across providers. The proxy server is the gateway: a standalone OpenAI-compatible service, configured through a config.yaml, that any language can call. The proxy's strength is how much routing behavior you can express in configuration. Model groups are named aliases. Repeating a model_name across entries load-balances across deployments, and fallback chains map one group to others: model_list: - model_name: chat-default # alias your app calls litellm_params: model: bedrock/anthropic.claude-sonnet-4-5 aws_region_name: us-east-1 - model_name: chat-default # same alias = load-balanced deployment litellm_params: model: vertex_ai/claude-sonnet-4-5 vertex_location: us-east5 - model_name: chat-backup litellm_params: model: openai/gpt-5-mini api_key: os.environ/OPENAI_API_KEY router_settings: enable_pre_call_checks: true # required for context-window fallbacks fallbacks: [{"chat-default": ["chat-backup"]}] context_window_fallbacks: [{"chat-default": ["chat-backup"]}] litellm_settings: num_retries: 3 # per group, before falling back request_timeout: 60 allowed_fails: 3 # failures/minute before cooldown cooldown_time: 30 This example shows the pattern described above: the same model served from two clouds as one load-balanced group, with a different model as the fallback. Your application only ever sends model: "chat-default". On top of routing, the proxy adds virtual keys, per-key and per-team spend tracking and budgets (backed by Postgres), callbacks to observability tools such as Langfuse, and an admin UI. LiteLLM publishes benchmark figures for proxy overhead (it cites 8 ms P95 at 1k RPS). Treat that as a vendor number to reproduce on your own hardware, not a guarantee. Choose LiteLLM when you have a platform team, a Python-friendly ops stack, and requirements like "keys never leave our VPC" or "every team gets a budget". Watch out for the operational surface. You own Postgres, Redis (if you enable shared caching or rate limits), upgrades, and supply-chain hygiene. Pin exact versions and verify the signed Docker images LiteLLM publishes rather than pulling latest. A gateway holds every provider credential you own, so it is the highest-value target in your AI stack. How does Portkey's gateway differ from LiteLLM? Portkey's open-source AI Gateway (MIT licensed) runs on Node.js: npx @portkey-ai/gateway starts it locally, and it deploys to Docker, a plain Node server, or Cloudflare Workers. Routing is defined in JSON configs that can be attached to a request header or saved on the hosted platform: { "strategy": { "mode": "fallback", "on_status_codes": [429, 503] }, "targets": [ { "override_params": { "model": "@openai-prod/gpt-5-mini" } }, { "override_params": { "model": "@anthropic-prod/claude-sonnet-4-5" } } ] } By default, a fallback fires on any non-2xx response. on_status_codes narrows it, which matters more than it looks. You usually want to fall back on a 429 (rate limit) or 503 (overload), but a 400 (your request is malformed) will fail identically on every target. Retrying it just multiplies latency and cost. Targets can themselves be load-balancers or nested strategies, so you can build "load-balance across two keys, fall back to a second provider" as one tree. Portkey's other differentiator is guardrails. The gateway has a large catalogue of pre-built checks (regex, JSON-schema, PII and others) that can run before or after a model call and can deny, flag or trigger a fallback. Semantic caching, the richer observability UI and team management live in the hosted and enterprise tiers. Choose Portkey when you want a small self-hosted footprint, a TypeScript-native stack, or guardrails at the gateway rather than in each service. Choose LiteLLM instead when you need the deepest provider coverage and a self-hosted admin and budget system without a paid tier. When does OpenRouter make more sense than a self-hosted gateway? OpenRouter flips the model. You don't run a gateway; you buy credits and get one API key for a very large catalogue of models from many inference providers. It has two routing layers: - Model fallback. A models array in the request lists alternatives in priority order. - Provider routing. The provider object controls which upstream serves a given model: order, only, ignore, sort by price/latency/throughput, a max_price ceiling, and data-policy filters such as data_collection: "deny" and zdr: true for zero-data-retention endpoints. { "models": ["anthropic/claude-sonnet-4.5", "openai/gpt-5-mini"], "messages": [{ "role": "user", "content": "Summarize this incident report." }], "provider": { "sort": "latency", "data_collection": "deny", "zdr": true, "max_price": { "prompt": 3, "completion": 15 } } } The economics are simple. OpenRouter states there is no markup on inference pricing. Its revenue is a fee on credit purchases, plus a 5% fee on bring-your-own-key usage above a monthly allowance. For open-weight models served by many competing providers, sort: "price" can land well below what one provider would charge. Choose OpenRouter when you need breadth fast: prototyping across many models, evaluating open-weight models without signing up with each host, or a small app that wants failover with zero infrastructure. Avoid it as the only path when your contracts require prompts to stay inside your cloud account, or when you need enterprise discounts negotiated directly with a provider. What do Cloudflare AI Gateway and Vercel AI Gateway add? Both are managed control planes run by edge platforms. You keep calling a model API, but through their endpoint, and you get logging, caching and failover without deploying anything. Cloudflare AI Gateway sits in front of Workers AI, OpenAI, Anthropic, Google Gemini and other providers. Its core features (analytics dashboard, caching and rate limiting) are free on all plans, and it supports request retries and model fallbacks. Logs are subject to per-plan storage limits. With unified billing, Cloudflare buys inference on your behalf at the provider's per-token rate and charges a 5% fee on purchased credits. Guardrails are billed as Workers AI inference. It fits best if your app already runs on Workers, because caching and rate limiting then sit on the same network as your code. Vercel AI Gateway is callable from any infrastructure, not just Vercel deployments, through the AI SDK, OpenAI Chat Completions, OpenAI Responses, or Anthropic Messages formats. It routes across providers with configurable provider order and model fallbacks, records every routing attempt in request logs, and supports budgets per team, project, API key or member. Vercel states it adds zero markup to provider token prices, including BYOK traffic. One detail to read before relying on it: budgets cover spend billed through Vercel's system credentials, while BYOK spend is metered separately. If a BYOK request fails, the gateway can fall back to system credentials, so check that behavior if "only use our provider contract" is a hard requirement. Choose an edge gateway when your app already lives on that platform and you want observability and failover in an afternoon. Look elsewhere when you need custom routing logic, on-prem models behind your firewall, or a single audit log spanning several clouds you control. Is an LLM gateway the same as AgentCore Gateway or an MCP gateway? No, and mixing them up is the most common mistake we see in architecture reviews. Both are called "gateways", but they sit on different arrows of an agent diagram: | | LLM gateway | MCP / tool gateway | |---|---|---| | Routes | Model inference calls (prompt → completion) | Tool calls (agent → external API/function) | | Protocol | OpenAI / Anthropic-style chat APIs | Model Context Protocol (MCP) | | Examples | LiteLLM, Portkey, OpenRouter, Cloudflare, Vercel | AWS Bedrock AgentCore Gateway, self-hosted MCP routers | | Core concern | Provider choice, failover, token spend | Tool discovery, auth to downstream systems, authorization | [AgentCore Gateway](/articles/aws-bedrock-agentcore-gateway-evaluation-how-does-it-compare) turns Lambda functions and APIs into MCP tools that agents can discover and call with IAM-scoped credentials. It does not choose which model answers a prompt. A production agent typically needs both: an LLM gateway for the reasoning loop, and a tool gateway for actions, with [deterministic authorization](/articles/AI-Agent-Authorization-LLM-vs-Policy-Engine) on the tool side. Some vendors now market a combined "AI gateway" for both. Evaluate the two halves separately anyway, because their failure modes don't overlap. What are the pros and cons of a model-agnostic gateway for a small app? This is the question behind the most specific search query we see on the topic, and the honest answer is that a small app often doesn't need a self-hosted one. Pros, even at small scale: - Outage insurance. One fallback entry turns a provider incident from downtime into slightly different answers. - Model agility. When a cheaper model matches your quality bar, switching is a config change, not a refactor. - One bill and one dashboard. Managed gateways consolidate spend that would otherwise be scattered across provider consoles. Cons: - An extra hop and an extra dependency. Your availability is now the gateway's availability multiplied by the provider's. A self-hosted proxy that you forgot to scale is a worse single point of failure than any provider. - Lowest-common-denominator APIs. Provider-specific features (prompt caching controls, extended-thinking parameters, server-side tools, citations) arrive in gateways late or are flattened by the translation layer. If your product depends on one of them, test it through the gateway before committing. - Another party in the data path. With a managed gateway, prompts cross a third network and fall under a third set of retention terms. The pragmatic path for a small app: start with a provider-agnostic SDK rather than a gateway. The Vercel AI SDK or LiteLLM's Python SDK gives you one call signature with no proxy to run; our [TypeScript LLM SDK guide](/articles/best-llm-sdk-for-typescript-apps) compares the options. Add a managed gateway when you need cross-provider failover or spend visibility, and self-host only when key custody or data residency force it. How should you choose an LLM gateway? Work through four questions in order. Each one eliminates options: 1. Must prompts and keys stay inside your network? If yes, self-host: LiteLLM or Portkey OSS. Managed gateways are out, whatever their features. 2. Who will operate it? With no platform team, a managed gateway (OpenRouter, Cloudflare, Vercel) almost always beats a self-hosted proxy nobody upgrades. With one, LiteLLM's budgets and admin UI pay for their operational cost quickly. 3. Where does your app already run? On Workers, start with Cloudflare. On Vercel or the AI SDK, start with Vercel AI Gateway. In a Python service mesh, start with LiteLLM. 4. How many models do you really need? Two or three providers on enterprise contracts favor a self-hosted router over your own keys. Dozens of models for evaluation or open-weight cost-shopping favor OpenRouter. Whatever you pick, test it the same way before production: - Kill a provider. Point one deployment at an invalid key or a blackholed endpoint, then confirm traffic shifts and the switch shows up in logs. - Exceed a context window. Send an oversized prompt and confirm it falls back to a larger-context model (or fails fast) rather than retrying the same error. - Exhaust a budget. Set a $1 limit on a test key and confirm the next request is rejected, not silently billed. - Stream a tool call. Run a streaming response with function calling end-to-end, because streaming plus tools is where translation layers break most often. - Measure overhead. Record P50/P95 latency with and without the gateway on your real payload sizes. Feed the gateway's logs into the same tracing backend as the rest of your agent. Our [LangSmith vs Langfuse vs Phoenix comparison](/articles/LangSmith-vs-Langfuse-vs-Phoenix-LLM-Agent-Observability) covers how to pick one. A fallback that fires silently is a cost and quality change you want to see. Frequently Asked Questions What is an LLM gateway? An LLM gateway is a proxy between applications and model providers that exposes one API, usually OpenAI-compatible, and applies routing, retries, fallbacks, key management, caching and spend tracking to every request. It lets you change models or providers through configuration instead of code. Is LiteLLM or Portkey better for self-hosting? LiteLLM is better when you want the widest provider coverage and built-in virtual keys, team budgets and an admin UI in the self-hosted product. Portkey's MIT gateway is better when you want a lighter Node.js deployment, JSON routing trees with status-code-specific fallbacks, and gateway-level guardrails. Does OpenRouter charge more than calling providers directly? OpenRouter states it adds no markup to inference pricing. It charges a fee when you buy credits, and a 5% fee on bring-your-own-key usage above a monthly allowance. Because it can route to the cheapest provider for open-weight models, total cost can come out lower than a single direct provider. Do I need an LLM gateway if I already use AWS Bedrock? Not always. Bedrock already gives you many models behind one AWS API, plus cross-region inference. A gateway becomes useful when you also call providers outside Bedrock, need failover to another cloud, or want per-team virtual keys and budgets that span providers. AgentCore Gateway handles MCP tool calls, not model routing. Does an LLM gateway add noticeable latency? A well-run gateway adds single-digit to low tens of milliseconds, which is small next to model generation time measured in seconds. The bigger latency risks are misconfigured retries that repeat a non-retryable error, and an undersized self-hosted proxy. Measure P95 with your own payloads before and after adding it. Can an LLM gateway route to self-hosted models like vLLM? Yes. LiteLLM, Portkey and Cloudflare AI Gateway can all target any OpenAI-compatible endpoint, which includes [vLLM](/articles/what-is-vllm) servers. A common pattern is to route routine traffic to a self-hosted open-weight model and fall back to a hosted frontier model on errors or context-length limits. Sources - LiteLLM GitHub Repository - OpenAI-format proxy and SDK for 100+ providers - LiteLLM Proxy Reliability - Fallbacks, retries and cooldowns - Portkey AI Gateway GitHub Repository - MIT-licensed open-source gateway - Portkey Fallbacks Documentation - Strategy configs and status-code triggers - OpenRouter Provider Routing - Order, sort, data policy and price ceilings - OpenRouter FAQ - Pricing, credit fees and BYOK - Cloudflare AI Gateway Documentation - Caching, rate limiting and fallbacks - Cloudflare AI Gateway Pricing - Free features and unified billing fee - Vercel AI Gateway Documentation - Routing, budgets, BYOK and zero markup --- Originally published at fp8.co. Subscribe for weekly AI engineering analysis at fp8.co.
CỔNG ĐIỀU PHỐI AI TỐC ĐỘ CAO CHO DOANH NGHIỆP: PORTKEY GIẢI BÀI TOÁN FALLBACK, LOAD BALANCING VÀ GUARDRAILS CHO LLM Khi xây dựng ứng dụng AI phụ thuộc vào các API thương mại, sự cố nhà cung cấp ngừng hoạt động (OpenAI outage) hoặc tài khoản bị dính giới hạn tần suất (Rate Limit 429) sẽ khiến toàn bộ hệ thống của bạn sụp đổ ngay lập tức. Để đưa AI vào các nghiệp vụ sống còn, bạn cần một tầng điều phối thông minh có khả năng dự phòng tự động. Portkey AI Gateway là cổng điều phối AI mã nguồn mở siêu tốc (AI Gateway & Control Plane), hỗ trợ kết nối hơn 250 mô hình ngôn ngữ lớn với các tính năng định tuyến thông minh, bảo vệ dữ liệu và bộ nhớ đệm tự động. Tech Stack: Được phát triển bằng TypeScript trên nền tảng Cloudflare Workers / Node.js runtime, mang lại độ trễ chuyển tiếp cực thấp dưới 1 mili-giây với mức tiêu thụ tài nguyên siêu nhỏ. Tại sao nên dùng? - Cơ chế Fallback và Load Balancing tự động: Khi mô hình chính (ví dụ GPT-4o) gặp sự cố quá tải hoặc trả về lỗi 5xx, Portkey tự động chuyển hướng request sang mô hình dự phòng (Claude 3.5 Sonnet hoặc Gemini Pro) mà người dùng không hề hay biết. - Quản lý ngân sách và giới hạn chi phí theo nhóm: Thiết lập hạn mức chi tiêu token cho từng phòng ban, từng dự án hoặc từng người dùng cụ thể, tự động ngắt kết nối khi vượt ngân sách. - Tích hợp Guardrails bảo vệ dữ liệu nhạy cảm: Tự động phát hiện và che giấu các thông tin định danh cá nhân (PII) như số thẻ ngân hàng, email, mật khẩu trước khi gửi đến máy chủ LLM. Dự án đạt hơn 6.000 stars trên GitHub và là thành phần cốt lõi giúp các doanh nghiệp xây dựng hạ tầng AI có độ tin cậy đạt chuẩn 99.99%. Link tham khảo: Portkey-AI/gateway #OpenSourceVN #VNOSS #opensourcevn #vnoss #DevCommunity #Portkey #AIGateway #LLMOps #Reliability #TypeScript #AIWorkflow