Deploy this version​

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:v1.79.0.rc.1

pip install litellm==1.79.0

Major Changes​

• Cohere models will now be routed to Cohere v2 API by default - PR #15722

Key Highlights​

• Search APIs - Native /v1/search endpoint with support for Perplexity, Tavily, Parallel AI, Exa AI, DataforSEO, and Google PSE with cost tracking
• Vector Stores - Vertex AI Search API integration as vector store through LiteLLM with passthrough endpoint support
• Guardrails Expansion - Apply guardrails across Responses API, Image Gen, Text completions, Audio transcriptions, Audio Speech, Rerank, and Anthropic Messages API via unified apply_guardrails function
• New Guardrail Providers - Gray Swan, Dynamo AI, IBM Guardrails, Lasso Security v3, and Bedrock Guardrail apply_guardrail endpoint support
• Video Generation API - Native support for OpenAI Sora-2 and Azure Sora-2 (Pro, Pro-High-Res) with cost tracking and logging support
• Azure AI Speech (TTS) - Native Azure AI Speech integration with cost tracking for standard and HD voices

New Models / Updated Models​

New Model Support​

Features​

• Anthropic

  • Fix cache_control incorrectly applied to all content items instead of last item only - PR #15699
  • Forward anthropic-beta headers to Bedrock, VertexAI - PR #15700
  • Change max_tokens value to match max_output_tokens for claude sonnet - PR #15715

• Bedrock

  • Add AWS us-gov-west-1 Claude 3.7 Sonnet costs - PR #15775
  • Fix the date for sonnet 3.7 in govcloud - PR #15800
  • Use proper bedrock model name in health check - PR #15808
  • Support for embeddings_by_type Response Format in Bedrock Cohere Embed v1 - PR #15707
  • Add titan image generations with cost tracking - PR #15916

• Gemini

  • Add imageConfig parameter for gemini-2.5-flash-image - PR #15530
  • Replace deprecated gemini-1.5-pro-preview-0514 - PR #15852
  • Update vertex ai gemini costs - PR #15911

• Ollama

  • Set 'think' to False when reasoning effort is minimal/none/disable - PR #15763
  • Handle parsing ollama chunk error - PR #15717

• Vertex AI

  • Add mistral medium 3 and Codestral 2 on vertex - PR #15887

• Databricks

  • Allow prompt caching to be used for Anthropic Claude on Databricks - PR #15801

• Azure

  • Add Azure AVA TTS integration - PR #15749
  • Add Azure AVA (Speech AI) Cost Tracking - PR #15754
  • Azure AI Speech - Ensure voice is mapped from request body to SSML body, allow sending role and style - PR #15810
  • Add Azure support for video generation functionality (Sora-2) - PR #15901

• OpenAI

  • OpenAI videos refactoring - PR #15900

• General

  • Read from custom-llm-provider header - PR #15528

LLM API Endpoints​

Features​

• Responses API

  • Add gpt 4.1 pricing for response endpoint - PR #15593
  • Fix Incorrect status value in responses api with gemini - PR #15753
  • Simplify reasoning item handling for gpt-5-codex - PR #15815
  • ErrorEvent ValidationError when OpenAI Responses API returns nested error structure - PR #15804
  • Fix reasoning item ID auto-generation causing encrypted content verification errors - PR #15782
  • Support tags in metadata - PR #15867
  • Security: prevent User A from retrieving User B's response, if response.id is leaked - PR #15757

• Batch API

  • Add pre and post call for list batches - PR #15673
  • Add function responsible to call precall - PR #15636
  • Fix "User default_user_id does not have access to the object" when object not in db - PR #15873

• OCR API

  • Add Azure AI - OCR to docs - PR #15768
  • Add mode + Health check support for OCR models - PR #15767

• Search API

  • Add def search() APIs for Web Search - Perplexity API - PR #15769
  • Add Tavily Search API - PR #15770
  • Add Parallel AI - Search API - PR #15772
  • Add EXA AI Search API to LiteLLM - PR #15774
  • Add /search endpoint on LiteLLM Gateway - PR #15780
  • Add DataforSEO Search API - PR #15817
  • Add Google PSE Search Provider - PR #15816
  • Add cost tracking for Search API requests - Google PSE, Tavily, Parallel AI, Exa AI - PR #15821
  • Backend: Allow storing configured Search APIs in DB - PR #15862
  • Exa Search API - ensure request params are sent to Exa AI - PR #15855

• Vector Stores

  • Support Vertex AI Search API as vector store through LiteLLM - PR #15781
  • Azure AI - Search Vector Stores - PR #15873
  • VertexAI Search Vector Store - Passthrough endpoint support + Vector store search Cost tracking support - PR #15824
  • Don't raise error if managed object is not found - PR #15873
  • Show config.yaml vector stores on UI - PR #15873
  • Cost tracking for search spend - PR #15859

• Images API

  • Pass user-defined headers and extra_headers to image-edit calls - PR #15811

• Video Generation API

  • Add Azure support for video generation functionality (Sora-2, Sora-2-Pro, Sora-2-Pro-High-Res) - PR #15901
  • OpenAI video generation refactoring (Sora-2) - PR #15900

• Bedrock /invoke

  • Fix: Hooks broken on /bedrock passthrough due to missing metadata - PR #15849

• Realtime API

  • Fix: OpenAI Realtime API integration fails due to websockets.exceptions.PayloadTooBig error - PR #15751

Management Endpoints / UI​

Features​

• Passthrough

  • Set auth on passthrough endpoints, on the UI - PR #15778
  • Fix pass-through endpoint budget enforcement bug - PR #15805

• Organizations

  • Allow org admins to create teams on UI - PR #15924

• Search Tools

  • UI - Search Tools, allow adding search tools on UI + testing search - PR #15871
  • UI - Add logos for search providers - PR #15872

• General

  • Fix routing for custom server root path - PR #15701

Logging / Guardrail / Prompt Management Integrations​

Features​

• OpenTelemetry

  • Fix OpenTelemetry Logging functionality - PR #15645
  • Fix issue where headers were not being split correctly - PR #15916

• Sentry

  • Add SENTRY_ENVIRONMENT configuration for Sentry integration - PR #15760

• Helicone

  • Fix JSON serialization error in Helicone logging by removing OpenTelemetry span from metadata - PR #15728

• MLFlow

  • Fix MLFlow tags - split request_tags into (key, val) if request_tag has colon - PR #15914

• General

  • Rename configured_cold_storage_logger to cold_storage_custom_logger - PR #15798

Guardrails​

• Gray Swan

  • Add GraySwan Guardrails support - PR #15756
  • Rename GraySwan to Gray Swan - PR #15771

• Dynamo AI

  • New Guardrail - Dynamo AI Guardrail - PR #15920

• IBM Guardrails

  • IBM Guardrails integration - PR #15924

• Lasso Security

  • Add v3 API Support - PR #12452
  • Fixed lasso import config, redis cluster hash tags for test keys - PR #15917

• Bedrock Guardrails

  • Implement Bedrock Guardrail apply_guardrail endpoint support - PR #15892

• General

  • Guardrails - Responses API, Image Gen, Text completions, Audio transcriptions, Audio Speech, Rerank, Anthropic Messages API support via the unified apply_guardrails function - PR #15706

Spend Tracking, Budgets and Rate Limiting​

• Rate Limiting
  • Support absolute RPM/TPM in priority_reservation - PR #15813
  • Org level tpm/rpm limits + Team tpm/rpm validation when assigned to org - PR #15549

MCP Gateway​

• OAuth
  • Auth Header Fix for MCP Tool Call - PR #15736
  • Add response_type + PKCE parameters to OAuth authorization endpoint - PR #15720

Performance / Loadbalancing / Reliability improvements​

• Database

  • Minimize the occurrence of deadlocks - PR #15281

• Redis

  • Apply max_connections configuration to Redis async client - PR #15797

• Caching

  • Add documentation for enable_caching_on_provider_specific_optional_params setting - PR #15885

Documentation Updates​

• Provider Documentation
  • Update worker recommendation - PR #15702
  • Fix the wrong request body in json mode doc - PR #15729
  • Add details in docs - PR #15721
  • Add responses api on openai docs - PR #15866
  • Add OpenAI responses api - PR #15868

New Contributors​

• @tlecomte made their first contribution in PR #15528
• @tomhaynes made their first contribution in PR #15645
• @talalryz made their first contribution in PR #15720
• @1vinodsingh1 made their first contribution in PR #15736
• @nuernber made their first contribution in PR #15775
• @Thomas-Mildner made their first contribution in PR #15760
• @javiergarciapleo made their first contribution in PR #15721
• @lshgdut made their first contribution in PR #15717
• @kk-wangjifeng made their first contribution in PR #15530
• @anthonyivn2 made their first contribution in PR #15801
• @romanglo made their first contribution in PR #15707
• @mythral made their first contribution in PR #15859
• @mubashirosmani made their first contribution in PR #15866
• @CAFxX made their first contribution in PR #15281
• @reflection made their first contribution in PR #15914
• @shadielfares made their first contribution in PR #15917

PR Count Summary​

10/26/2025​

• New Models / Updated Models: 20
• LLM API Endpoints: 29
• Management Endpoints / UI: 5
• Logging / Guardrail / Prompt Management Integrations: 10
• Spend Tracking, Budgets and Rate Limiting: 2
• MCP Gateway: 2
• Performance / Loadbalancing / Reliability improvements: 3
• Documentation Updates: 5

Full Changelog​

View complete changelog on GitHub

Deploy this version​

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:v1.78.5-stable

pip install litellm==1.78.5

Key Highlights​

• Native OCR Endpoints - Native /v1/ocr endpoint support with cost tracking for Mistral OCR and Azure AI OCR
• Global Vendor Discounts - Specify global vendor discount percentages for accurate cost tracking and reporting
• Team Spending Reports - Team admins can now export detailed spending reports for their teams
• Claude Haiku 4.5 - Day 0 support for Claude Haiku 4.5 across Bedrock, Vertex AI, and OpenRouter with 200K context window
• GPT-5-Codex - Support for GPT-5-Codex via Responses API on OpenAI and Azure
• Performance Improvements - Major router optimizations: O(1) model lookups, 10-100x faster shallow copy, 30-40% faster timing calls, and O(n) to O(1) hash generation

New Models / Updated Models​

New Model Support​

Features​

• OpenAI

  • GPT-5 return reasoning content via /chat/completions + GPT-5-Codex working on Claude Code - PR #15441

• Anthropic

  • Reduce claude-4-sonnet max_output_tokens to 64k - PR #15409
  • Added claude-haiku-4.5 - PR #15579
  • Add support for thinking blocks and redacted thinking blocks in Anthropic v1/messages API - PR #15501

• Bedrock

  • Add anthropic.claude-haiku-4-5-20251001-v1:0 on Bedrock, VertexAI - PR #15581
  • Add Claude Haiku 4.5 support for Bedrock global and US regions - PR #15650
  • Add Claude Haiku 4.5 support for Bedrock Other regions - PR #15653
  • Add JP Cross-Region Inference jp.anthropic.claude-haiku-4-5-20251001 - PR #15598
  • Fix: bedrock-pricing-geo-inregion-cross-region / add Global Cross-Region Inference - PR #15685
  • Fix: Support us-gov prefix for AWS GovCloud Bedrock models - PR #15626
  • Fix GPT-OSS in Bedrock now supports streaming. Revert fake streaming - PR #15668

• Gemini

  • Feat(pricing): Add Gemini 2.5 Flash Image (Nano Banana) in GA - PR #15557
  • Fix: Gemini 2.5 Flash Image should not have supports_web_search=true - PR #15642
  • Remove penalty params as supported params for gemini preview model - PR #15503

• Ollama

  • Fix(ollama/chat): correctly map reasoning_effort to think in requests - PR #15465

• OpenRouter

  • Add anthropic/claude-sonnet-4.5 to OpenRouter cost map - PR #15472
  • Prompt caching for anthropic models with OpenRouter - PR #15535
  • Get completion cost directly from OpenRouter - PR #15448
  • Fix OpenRouter Claude Opus 4 model naming - PR #15495

• CometAPI

  • Fix(cometapi): improve CometAPI provider support (embeddings, image generation, docs) - PR #15591

• Lemonade

  • Adding new models to the lemonade provider - PR #15554

• Watson X

  • Fix (pricing): Fix pricing for watsonx model family for various models - PR #15670

• Vercel AI Gateway

  • Add glm-4.6 model to pricing configuration - PR #15679

• Vertex AI

  • Add Vertex AI Discovery Engine Rerank Support - PR #15532

Bug Fixes​

• Anthropic

  • Fix: Pricing for Claude Sonnet 4.5 in US regions is 10x too high - PR #15374

• OpenRouter

  • Change gpt-5-codex support in model_price json - PR #15540

• Bedrock

  • Fix filtering headers for signature calcs - PR #15590

• General

  • Add native reasoning and streaming support flag for gpt-5-codex - PR #15569

LLM API Endpoints​

Features​

• Responses API

  • Responses API - enable calling anthropic/gemini models in Responses API streaming in openai ruby sdk + DB - sanity check pending migrations before startup - PR #15432
  • Add support for responses mode in health check - PR #15658

• OCR API

  • Feat: Add native litellm.ocr() functions - PR #15567
  • Feat: Add /ocr route on LiteLLM AI Gateway - Adds support for native Mistral OCR calling - PR #15571
  • Feat: Add Azure AI Mistral OCR Integration - PR #15572
  • Feat: Native /ocr endpoint support - PR #15573
  • Feat: Add Cost Tracking for /ocr endpoints - PR #15678

• /generateContent

  • Fix: GEMINI - CLI - add google_routes to llm_api_routes - PR #15500
  • Fix Pydantic validation error for citationMetadata.citationSources in Google GenAI responses - PR #15592

• Images API

  • Fix: Dall-e-2 for Image Edits API - PR #15604

• Bedrock Passthrough

  • Feat: Allow calling /invoke, /converse routes through AI Gateway + models on config.yaml - PR #15618

Bugs​

• General
  • Fix: Convert object to a correct type - PR #15634
  • Bug Fix: Tags as metadata dicts were raising exceptions - PR #15625
  • Add type hint to function_to_dict and fix typo - PR #15580

Management Endpoints / UI​

Features​

• Virtual Keys

  • Docs: Key Rotations - PR #15455
  • Fix: UI - Key Max Budget Removal Error Fix - PR #15672
  • litellm_Key Settings Max Budget Removal Error Fix - PR #15669

• Teams

  • Feat: Allow Team Admins to export a report of the team spending - PR #15542

• Passthrough

  • Feat: Passthrough - allow admin to give access to specific passthrough endpoints - PR #15401

• SCIM v2

  • Feat(scim_v2.py): if group.id doesn't exist, use external id + Passthrough - ensure updates and deletions persist across instances - PR #15276

• SSO

  • Feat: UI SSO - Add PKCE for OKTA SSO - PR #15608
  • Fix: Separate OAuth M2M authentication from UI SSO + Handle Introspection endpoint for Oauth2 - PR #15667
  • Fix/entraid app roles jwt claim clean - PR #15583

Logging / Guardrail / Prompt Management Integrations​

Guardrails​

• General

  • Fix apply_guardrail endpoint returning raw string instead of ApplyGuardrailResponse - PR #15436
  • Fix: Ensure guardrail memory sync after database updates - PR #15633
  • Feat: add guardrail for image generation - PR #15619
  • Feat: Add Guardrails for /v1/messages and /v1/responses API - PR #15686

• Pillar Security

  • Feature: update pillar security integration to support no persistence mode in litellm proxy - PR #15599

Prompt Management​

• General
  • Small fix code snippet custom_prompt_management.md - PR #15544

Spend Tracking, Budgets and Rate Limiting​

• Cost Tracking

  • Feat: Cost Tracking - specify a global vendor discount for costs - PR #15546
  • Feat: UI - Allow setting Provider Discounts on UI - PR #15550

• Budgets

  • Fix: improve budget clarity - PR #15682

Performance / Loadbalancing / Reliability improvements​

• Router Optimizations

  • Perf(router): use shallow copy instead of deepcopy for model aliases - 10-100x faster than deepcopy on nested dict structures - PR #15576
  • Perf(router): optimize string concatenation in hash generation - Improves time complexity from O(n²) to O(n) - PR #15575
  • Perf(router): optimize model lookups with O(1) data structures - Replace O(n) scans with index map lookups - PR #15578
  • Perf(router): optimize model lookups with O(1) index maps - Use model_id_to_deployment_index_map and model_name_to_deployment_indices for instant lookups - PR #15574
  • Perf(router): optimize timing functions in completion hot path - Use time.perf_counter() for duration measurements and time.monotonic() for timeout calculations, providing 30-40% faster timing calls - PR #15617

• SSL/TLS Performance

  • Feat(ssl): add configurable ECDH curve for TLS performance - Configure via ssl_ecdh_curve setting to disable PQC on OpenSSL 3.x for better performance - PR #15617

• Token Counter

  • Fix(token-counter): extract model_info from deployment for custom_tokenizer - PR #15680

• Performance Metrics

  • Add: perf summary - PR #15458

• CI/CD

  • Fix: CI/CD - Missing env key & Linter type error - PR #15606

Documentation Updates​

• Provider Documentation

  • Litellm docs 10 11 2025 - PR #15457
  • Docs: add ecs deployment guide - PR #15468
  • Docs: Update benchmark results - PR #15461
  • Fix: add missing context to benchmark docs - PR #15688

• General

  • Fixed a few typos - PR #15267

New Contributors​

• @jlan-nl made their first contribution in PR #15374
• @ImadSaddik made their first contribution in PR #15267
• @huangyafei made their first contribution in PR #15472
• @mubashir1osmani made their first contribution in PR #15468
• @kowyo made their first contribution in PR #15465
• @dhruvyad made their first contribution in PR #15448
• @davizucon made their first contribution in PR #15544
• @FelipeRodriguesGare made their first contribution in PR #15540
• @ndrsfel made their first contribution in PR #15557
• @shinharaguchi made their first contribution in PR #15598
• @TensorNull made their first contribution in PR #15591
• @TeddyAmkie made their first contribution in PR #15583
• @aniketmaurya made their first contribution in PR #15580
• @eddierichter-amd made their first contribution in PR #15554
• @konekohana made their first contribution in PR #15535
• @Classic298 made their first contribution in PR #15495
• @afogel made their first contribution in PR #15599
• @orolega made their first contribution in PR #15633
• @LucasSugi made their first contribution in PR #15634
• @uc4w6c made their first contribution in PR #15619
• @Sameerlite made their first contribution in PR #15658
• @yuneng-jiang made their first contribution in PR #15672
• @Nikro made their first contribution in PR #15680

Full Changelog​

View complete changelog on GitHub

Deploy this version​

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:v1.78.0-stable

pip install litellm==1.78.0.post1

Key Highlights​

• MCP Gateway - Control Tool Access by Team, Key - Control MCP tool access by team/key.
• Performance Improvements - 70% Lower p99 Latency
• GPT-5 Pro & GPT-Image-1-Mini - Day 0 support for OpenAI's GPT-5 Pro (400K context) and gpt-image-1-mini image generation
• EnkryptAI Guardrails - New guardrail integration for content moderation
• Tag-Based Budgets - Support for setting budgets based on request tags

MCP Gateway - Control Tool Access by Team, Key​

Proxy admins can now control MCP tool access by team or key. This makes it easy to grant different teams selective access to tools from the same MCP server.

For example, you can now give your Engineering team access to list_repositories, create_issue, and search_code tools, while Sales only gets search_code and close_issue tools.

This makes it easier for Proxy Admins to govern MCP Tool Access.

Get Started

Performance - 70% Lower p99 Latency​

This release cuts p99 latency by 70% on LiteLLM AI Gateway, making it even better for low-latency use cases.

These gains come from two key enhancements:

Reliable Sessions

Added support for shared sessions with aiohttp. The shared_session parameter is now consistently used across all calls, enabling connection pooling.

Faster Routing

A new model_name_to_deployment_indices hash map replaces O(n) list scans in _get_all_deployments() with O(1) hash lookups, boosting routing performance and scalability.

As a result, performance improved across all latency percentiles:

• Median latency: 110 ms → 100 ms (−9.1%)
• p95 latency: 440 ms → 150 ms (−65.9%)
• p99 latency: 810 ms → 240 ms (−70.4%)
• Average latency: 310 ms → 111.73 ms (−64.0%)

Test Setup​

Locust

• Concurrent users: 1,000
• Ramp-up: 500

System Specs

• Database was used
• CPU: 4 vCPUs
• Memory: 8 GB RAM
• LiteLLM Workers: 4
• Instances: 4

Configuration (config.yaml)

View the complete configuration: gist.github.com/AlexsanderHamir/config.yaml

Load Script (no_cache_hits.py)

View the complete load testing script: gist.github.com/AlexsanderHamir/no_cache_hits.py

New Models / Updated Models​

New Model Support​

Features​

• OpenAI

  • Add GPT-5 Pro model configuration and documentation - PR #15258
  • Add stop parameter to non-supported params for GPT-5 - PR #15244
  • Day 0 Support, Add gpt-image-1-mini - PR #15259
  • Add gpt-realtime-mini support - PR #15283
  • Add gpt-5-pro-2025-10-06 to model costs - PR #15344
  • Minimal fix: gpt5 models should not go on cooldown when called with temperature!=1 - PR #15330

• Snowflake Cortex

  • Add function calling support for Snowflake Cortex REST API - PR #15221

• Gemini

  • Fix header forwarding for Gemini/Vertex AI providers in proxy mode - PR #15231

• Azure

  • Removed stop param from unsupported azure models - PR #15229
  • Fix(azure/responses): remove invalid status param from azure call - PR #15253
  • Add new Azure AI models with pricing details - PR #15387
  • AzureAD Default credentials - select credential type based on environment - PR #14470

• Bedrock

  • Add Global Cross-Region Inference - PR #15210
  • Add Cohere Embed v4 support for AWS Bedrock - PR #15298
  • Fix(bedrock): include cacheWriteInputTokens in prompt_tokens calculation - PR #15292
  • Add Bedrock AU Cross-Region Inference for Claude Sonnet 4.5 - PR #15402
  • Converse → /v1/messages streaming doesn't handle parallel tool calls with Claude models - PR #15315

• Vertex AI

  • Implement Context Caching for Vertex AI provider - PR #15226
  • Support for Vertex AI Gemma Models on Custom Endpoints - PR #15397
  • VertexAI - gemma model family support (custom endpoints) - PR #15419
  • VertexAI Gemma model family streaming support + Added MedGemma - PR #15427

• OCI

  • Add OCI Cohere support with tool calling and streaming capabilities - PR #15365

• Watson X

  • Add Watson X foundation model definitions to model_prices_and_context_window.json - PR #15219
  • Watsonx - Apply correct prompt templates for openai/gpt-oss model family - PR #15341

• OpenRouter

  • Fix - (openrouter): move cache_control to content blocks for claude/gemini - PR #15345
  • Fix - OpenRouter cache_control to only apply to last content block - PR #15395

• Together AI

  • Add new together models - PR #15383

Bug Fixes​

• General
  • Bug fix: gpt-5-chat-latest has incorrect max_input_tokens value - PR #15116
  • Fix reasoning response ID - PR #15265
  • Fix issue with parsing assistant messages - PR #15320
  • Fix litellm_param based costing - PR #15336
  • Fix lint errors - PR #15406

LLM API Endpoints​

Features​

• Responses API

  • Added streaming support for response api streaming image generation - PR #15269
  • Add native Responses API support for litellm_proxy provider - PR #15347
  • Temporarily relax ResponsesAPIResponse parsing to support custom backends (e.g., vLLM) - PR #15362

• Files API

  • Feat(files): add @client decorator to file operations - PR #15339

• /generateContent

  • Fix gemini cli by actually streaming the response - PR #15264

• Azure Passthrough

  • Azure - passthrough support with router models - PR #15240

Bugs​

• General
  • Fix x-litellm-cache-key header not being returned on cache hit - PR #15348

Management Endpoints / UI​

Features​

• Proxy CLI Auth

  • Proxy CLI - dont store existing key in the URL, store it in the state param - PR #15290

• Models + Endpoints

  • Make PATCH /model/{model_id}/update handle team_id consistently with POST /model/new - PR #15297
  • Feature: adds Infinity as a provider in the UI - PR #15285
  • Fix: model + endpoints page crash when config file contains router_settings.model_group_alias - PR #15308
  • Models & Endpoints Initial Refactor - PR #15435
  • Litellm UI API Reference page updates - PR #15438

• Teams

  • Teams page: new column "Your Role" on the teams table - PR #15384
  • LiteLLM Dashboard Teams UI refactor - PR #15418

• UI Infrastructure

  • Added prettier to autoformat frontend - PR #15215
  • Adds turbopack to the npm run dev command in UI to build faster during development - PR #15250
  • (perf) fix: Replaces bloated key list calls with lean key aliases endpoint - PR #15252
  • Potentially fixes a UI spasm issue with an expired cookie - PR #15309
  • LiteLLM UI Refactor Infrastructure - PR #15236
  • Enforces removal of unused imports from UI - PR #15416
  • Fix: usage page >> Model Activity >> spend per day graph: y-axis clipping on large spend values - PR #15389
  • Updates guardrail provider logos - PR #15421

• Admin Settings

  • Fix: Router settings do not update despite success message - PR #15249
  • Fix: Prevents DB from accidentally overriding config file values if they are empty in DB - PR #15340

• SSO

  • SSO - support EntraID app roles - PR #15351

Logging / Guardrail / Prompt Management Integrations​

Features​

• PostHog
  • Feat: posthog per request api key - PR #15379

Guardrails​

• EnkryptAI
  • Add EnkryptAI Guardrails on LiteLLM - PR #15390

Spend Tracking, Budgets and Rate Limiting​

• Tag Management

  • Tag Management - Add support for setting tag based budgets - PR #15433

• Dynamic Rate Limiter v3

  • QA/Fixes - Dynamic Rate Limiter v3 - final QA - PR #15311
  • Fix dynamic Rate limiter v3 - inserting litellm_model_saturation - PR #15394

• Shared Health Check

  • Implement Shared Health Check State Across Pods - PR #15380

MCP Gateway​

• Tool Control

  • MCP Gateway - UI - Select allowed tools for Key, Teams - PR #15241
  • MCP Gateway - Backend - Allow storing allowed tools by team/key - PR #15243
  • MCP Gateway - Fine-grained Database Object Storage Control - PR #15255
  • MCP Gateway - Litellm mcp fixes team control - PR #15304
  • MCP Gateway - QA/Fixes - Ensure Team/Key level enforcement works for MCPs - PR #15305
  • Feature: Include server_name in /v1/mcp/server/health endpoint response - PR #15431

• OpenAPI Integration

  • MCP - support converting OpenAPI specs to MCP servers - PR #15343
  • MCP - specify allowed params per tool - PR #15346

• Configuration

  • MCP - support setting CA_BUNDLE_PATH - PR #15253
  • Fix: Ensure MCP client stays open during tool call - PR #15391
  • Remove hardcoded "public" schema in migration.sql - PR #15363

Performance / Loadbalancing / Reliability improvements​

• Router Optimizations

  • Fix - Router: add model_name index for O(1) deployment lookups - PR #15113
  • Refactor Utils: extract inner function from client - PR #15234
  • Fix Networking: remove limitations - PR #15302

• Session Management

  • Fix - Sessions not being shared - PR #15388
  • Fix: remove panic from hot path - PR #15396
  • Fix - shared session parsing and usage issue - PR #15440
  • Fix: handle closed aiohttp sessions - PR #15442
  • Fix: prevent session leaks when recreating aiohttp sessions - PR #15443

• SSL/TLS Performance

  • Perf: optimize SSL/TLS handshake performance with prioritized cipher - PR #15398

• Dependencies

  • Upgrades tenacity version to 8.5.0 - PR #15303

• Data Masking

  • Fix - SensitiveDataMasker converts lists to string - PR #15420

General AI Gateway Improvements​

Security​

• General
  • Fix: redact AWS credentials when redact_user_api_key_info enabled - PR #15321

Documentation Updates​

• Provider Documentation

  • Update doc: perf update - PR #15211
  • Add W&B Inference documentation - PR #15278

• Deployment

  • Deletion of docker-compose buggy comment that cause config.yaml based startup fail - PR #15425

New Contributors​

• @Gal-bloch made their first contribution in PR #15219
• @lcfyi made their first contribution in PR #15315
• @ashengstd made their first contribution in PR #15362
• @vkolehmainen made their first contribution in PR #15363
• @jlan-nl made their first contribution in PR #15330
• @BCook98 made their first contribution in PR #15402
• @PabloGmz96 made their first contribution in PR #15425

Full Changelog​

Deploy this version​

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:v1.77.7.rc.1

pip install litellm==1.77.7.rc.1

Key Highlights​

• Dynamic Rate Limiter v3 - Automatically maximizes throughput when capacity is available (< 80% saturation) by allowing lower-priority requests to use unused capacity, then switches to fair priority-based allocation under high load (≥ 80%) to prevent blocking
• Major Performance Improvements - 2.9x lower median latency at 1,000 concurrent users.
• Claude Sonnet 4.5 - Support for Anthropic's new Claude Sonnet 4.5 model family with 200K+ context and tiered pricing
• MCP Gateway Enhancements - Fine-grained tool control, server permissions, and forwardable headers
• AMD Lemonade & Nvidia NIM - New provider support for AMD Lemonade and Nvidia NIM Rerank
• GitLab Prompt Management - GitLab-based prompt management integration

Performance - 2.9x Lower Median Latency​

This update removes LiteLLM router inefficiencies, reducing complexity from O(M×N) to O(1). Previously, it built a new array and ran repeated checks like data["model"] in llm_router.get_model_ids(). Now, a direct ID-to-deployment map eliminates redundant allocations and scans.

As a result, performance improved across all latency percentiles:

• Median latency: 320 ms → 110 ms (−65.6%)
• p95 latency: 850 ms → 440 ms (−48.2%)
• p99 latency: 1,400 ms → 810 ms (−42.1%)
• Average latency: 864 ms → 310 ms (−64%)

Test Setup​

Locust

• Concurrent users: 1,000
• Ramp-up: 500

System Specs

• CPU: 4 vCPUs
• Memory: 8 GB RAM
• LiteLLM Workers: 4
• Instances: 4

Configuration (config.yaml)

View the complete configuration: gist.github.com/AlexsanderHamir/config.yaml

Load Script (no_cache_hits.py)

View the complete load testing script: gist.github.com/AlexsanderHamir/no_cache_hits.py

MCP OAuth 2.0 Support​

This release adds support for OAuth 2.0 Client Credentials for MCP servers. This is great for Internal Dev Tools use-cases, as it enables your users to call MCP servers, with their own credentials. E.g. Allowing your developers to call the Github MCP, with their own credentials.

Set it up today on Claude Code

Scheduled Key Rotations​

This release brings support for scheduling virtual key rotations on LiteLLM AI Gateway.

From this release you can enforce Virtual Keys to rotate on a schedule of your choice e.g every 15 days/30 days/60 days etc.

This is great for Proxy Admins who need to enforce security policies for production workloads.

Get Started

New Models / Updated Models​

New Model Support​

Features​

• Anthropic
  • Add new claude-sonnet-4-5 model family with tiered pricing above 200K tokens - PR #15041
  • Add anthropic/claude-sonnet-4-5 to model price json with prompt caching support - PR #15049
  • Add 200K prices for Sonnet 4.5 - PR #15140
  • Add cost tracking for /v1/messages in streaming response - PR #15102
  • Add /v1/messages/count_tokens to Anthropic routes for non-admin user access - PR #15034
• Gemini
  • Ignore type param for gemini tools - PR #15022
• Vertex AI
  • Add LiteLLM Overhead metric for VertexAI - PR #15040
  • Support googlemap grounding in vertex ai - PR #15179
• Azure
  • Add azure_ai grok-4 model family - PR #15137
  • Use the extra_query parameter for GET requests in Azure Batch - PR #14997
  • Use extra_query for download results (Batch API) - PR #15025
  • Add support for Azure AD token-based authorization - PR #14813
• Ollama
  • Add ollama cloud models - PR #15008
• Groq
  • Add groq/moonshotai/kimi-k2-instruct-0905 - PR #15079
• OpenAI
  • Add support for GPT 5 codex models - PR #14841
• DeepInfra
  • Update DeepInfra model data refresh with latest pricing - PR #14939
• Bedrock
  • Add JP Cross-Region Inference - PR #15188
  • Add "eu.anthropic.claude-sonnet-4-5-20250929-v1:0" - PR #15181
  • Add twelvelabs bedrock Async Invoke Support - PR #14871
• Nvidia NIM
  • Add Nvidia NIM Rerank Support - PR #15152

Bug Fixes​

• VLLM
  • Fix response_format bug in hosted vllm audio_transcription - PR #15010
  • Fix passthrough of atranscription into kwargs going to upstream provider - PR #15005
• OCI
  • Fix OCI Generative AI Integration when using Proxy - PR #15072
• General
  • Fix: Authorization header to use correct "Bearer" capitalization - PR #14764
  • Bug fix: gpt-5-chat-latest has incorrect max_input_tokens value - PR #15116
  • Update request handling for original exceptions - PR #15013

New Provider Support​

• AMD Lemonade
  • Add AMD Lemonade provider support - PR #14840

LLM API Endpoints​

Features​

• Responses API

  • Return Cost for Responses API Streaming requests - PR #15053

• /generateContent

  • Add full support for native Gemini API translation - PR #15029

• Passthrough Gemini Routes

  • Add Gemini generateContent passthrough cost tracking - PR #15014
  • Add streamGenerateContent cost tracking in passthrough - PR #15199

• Passthrough Vertex AI Routes

  • Add cost tracking for Vertex AI Passthrough /predict endpoint - PR #15019
  • Add cost tracking for Vertex AI Live API WebSocket Passthrough - PR #14956

• General

  • Preserve Whitespace Characters in Model Response Streams - PR #15160
  • Add provider name to payload specification - PR #15130
  • Ensure query params are forwarded from origin url to downstream request - PR #15087

Management Endpoints / UI​

Features​

• Virtual Keys

  • Ensure LLM_API_KEYs can access pass through routes - PR #15115
  • Support 'guaranteed_throughput' when setting limits on keys belonging to a team - PR #15120

• Models + Endpoints

  • Ensure OCI secret fields not shared on /models and /v1/models endpoints - PR #15085
  • Add snowflake on UI - PR #15083
  • Make UI theme settings publicly accessible for custom branding - PR #15074

• Admin Settings

  • Ensure OTEL settings are saved in DB after set on UI - PR #15118
  • Top api key tags - PR #15151, PR #15156

• MCP

  • show health status of MCP servers - PR #15185
  • allow setting extra headers on the UI - PR #15185
  • allow editing allowed tools on the UI - PR #15185

Bug Fixes​

• Virtual Keys

  • (security) prevent user key from updating other user keys - PR #15201
  • (security) don't return all keys with blank key alias on /v2/key/info - PR #15201
  • Fix Session Token Cookie Infinite Logout Loop - PR #15146

• Models + Endpoints

  • Make UI theme settings publicly accessible for custom branding - PR #15074

• Teams

  • fix failed copy to clipboard for http ui - PR #15195

• Logs

  • fix logs page render logs on filter lookup - PR #15195
  • fix lookup list of end users (migrate to more efficient /customers/list lookup) - PR #15195

• Test key

  • update selected model on key change - PR #15197

• Dashboard

  • Fix LiteLLM model name fallback in dashboard overview - PR #14998

Logging / Guardrail / Prompt Management Integrations​

Features​

• OpenTelemetry
  • Use generation_name for span naming in logging method - PR #14799
• Langfuse
  • Handle non-serializable objects in Langfuse logging - PR #15148
  • Set usage_details.total in langfuse integration - PR #15015
• Prometheus
  • support custom metadata labels on key/team - PR #15094

Guardrails​

• Javelin
  • Add Javelin standalone guardrails integration for LiteLLM Proxy - PR #14983
  • Add logging for important status fields in guardrails - PR #15090
  • Don't run post_call guardrail if no text returned from Bedrock - PR #15106

Prompt Management​

• GitLab
  • GitLab based Prompt manager - PR #14988

Spend Tracking, Budgets and Rate Limiting​

• Cost Tracking
  • Proxy: end user cost tracking in the responses API - PR #15124
• Parallel Request Limiter v3
  • Use well known redis cluster hashing algorithm - PR #15052
  • Fixes to dynamic rate limiter v3 - add saturation detection - PR #15119
  • Dynamic Rate Limiter v3 - fixes for detecting saturation + fixes for post saturation behavior - PR #15192
• Teams
  • Add model specific tpm/rpm limits to teams on LiteLLM - PR #15044

MCP Gateway​

• Server Configuration
  • Specify forwardable headers, specify allowed/disallowed tools for MCP servers - PR #15002
  • Enforce server permissions on call tools - PR #15044
  • MCP Gateway Fine-grained Tools Addition - PR #15153
• Bug Fixes
  • Remove servername prefix mcp tools tests - PR #14986
  • Resolve regression with duplicate Mcp-Protocol-Version header - PR #15050
  • Fix test_mcp_server.py - PR #15183

Performance / Loadbalancing / Reliability improvements​

• Router Optimizations
  • +62.5% P99 Latency Improvement - Remove router inefficiencies (from O(M*N) to O(1)) - PR #15046
  • Remove hasattr checks in Router - PR #15082
  • Remove Double Lookups - PR #15084
  • Optimize _filter_cooldown_deployments from O(n×m + k×n) to O(n) - PR #15091
  • Optimize unhealthy deployment filtering in retry path (O(n*m) → O(n+m)) - PR #15110
• Cache Optimizations
  • Reduce complexity of InMemoryCache.evict_cache from O(n*log(n)) to O(log(n)) - PR #15000
  • Avoiding expensive operations when cache isn't available - PR #15182
• Worker Management
  • Add proxy CLI option to recycle workers after N requests - PR #15007
• Metrics & Monitoring
  • LiteLLM Overhead metric tracking - Add support for tracking litellm overhead on cache hits - PR #15045

Documentation Updates​

• Provider Documentation
  • Update litellm docs from latest release - PR #15004
  • Add missing api_key parameter - PR #15058
• General Documentation
  • Use docker compose instead of docker-compose - PR #15024
  • Add railtracks to projects that are using litellm - PR #15144
  • Perf: Last week improvement - PR #15193
  • Sync models GitHub documentation with Loom video and cross-reference - PR #15191

Security Fixes​

• JWT Token Security - Don't log JWT SSO token on .info() log - PR #15145

New Contributors​

• @herve-ves made their first contribution in PR #14998
• @wenxi-onyx made their first contribution in PR #15008
• @jpetrucciani made their first contribution in PR #15005
• @abhijitjavelin made their first contribution in PR #14983
• @ZeroClover made their first contribution in PR #15039
• @cedarm made their first contribution in PR #15043
• @Isydmr made their first contribution in PR #15025
• @serializer made their first contribution in PR #15013
• @eddierichter-amd made their first contribution in PR #14840
• @malags made their first contribution in PR #15000
• @henryhwang made their first contribution in PR #15029
• @plafleur made their first contribution in PR #15111
• @tyler-liner made their first contribution in PR #14799
• @Amir-R25 made their first contribution in PR #15144
• @georg-wolflein made their first contribution in PR #15124
• @niharm made their first contribution in PR #15140
• @anthony-liner made their first contribution in PR #15015
• @rishiganesh2002 made their first contribution in PR #15153
• @danielaskdd made their first contribution in PR #15160
• @JVenberg made their first contribution in PR #15146
• @speglich made their first contribution in PR #15072
• @daily-kim made their first contribution in PR #14764

Full Changelog​

Deploy this version​

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:v1.77.5-stable

pip install litellm==1.77.5

Key Highlights​

• MCP OAuth 2.0 Support - Enhanced authentication for Model Context Protocol integrations
• Scheduled Key Rotations - Automated key rotation capabilities for enhanced security
• New Gemini 2.5 Flash & Flash-lite Models - Latest September 2025 preview models with improved pricing and features
• Performance Improvements - 54% RPS improvement

Performance Improvements - 54% RPS Improvement​

This release brings a 54% RPS improvement (1,040 → 1,602 RPS, aggregated) per instance.

The improvement comes from fixing O(n²) inefficiencies in the LiteLLM Router, primarily caused by repeated use of in statements inside loops over large arrays.

Tests were run with a database-only setup (no cache hits).

Test Setup​

All benchmarks were executed using Locust with 1,000 concurrent users and a ramp-up of 500. The environment was configured to stress the routing layer and eliminate caching as a variable.

System Specs

• CPU: 8 vCPUs
• Memory: 32 GB RAM

Configuration (config.yaml)

View the complete configuration: gist.github.com/AlexsanderHamir/config.yaml

Load Script (no_cache_hits.py)

View the complete load testing script: gist.github.com/AlexsanderHamir/no_cache_hits.py

New Models / Updated Models​

New Model Support​

Features​

• Gemini
  • Added Gemini 2.5 Flash and Flash-lite preview models (September 2025 release) with improved pricing - PR #14948
  • Added new Anthropic web fetch tool support - PR #14951
• XAI
  • Add xai/grok-4-fast models - PR #14833
• Anthropic
  • Updated Claude Sonnet 4 configs to reflect million-token context window pricing - PR #14639
  • Added supported text field to anthropic citation response - PR #14164
• Bedrock
  • Added support for Qwen models family & Deepseek 3.1 to Amazon Bedrock - PR #14845
  • Support requestMetadata in Bedrock Converse API - PR #14570
• Vertex AI
  • Added vertex_ai/qwen models and azure/gpt-5-codex - PR #14844
  • Update vertex ai qwen model pricing - PR #14828
  • Vertex AI Context Caching: use Vertex ai API v1 instead of v1beta1 and accept 'cachedContent' param - PR #14831
• SambaNova
  • Add sambanova deepseek v3.1 and gpt-oss-120b - PR #14866
• OpenAI
  • Fix inconsistent token configs for gpt-5 models - PR #14942
  • GPT-3.5-Turbo price updated - PR #14858
• OpenRouter
  • Add gpt-5 and gpt-5-codex to OpenRouter cost map - PR #14879
• VLLM
  • Fix vllm passthrough - PR #14778
• Flux
  • Support flux image edit - PR #14790

Bug Fixes​

• Anthropic
  • Fix: Support claude code auth via subscription (anthropic) - PR #14821
  • Fix Anthropic streaming IDs - PR #14965
  • Revert incorrect changes to sonnet-4 max output tokens - PR #14933
• OpenAI
  • Fix a bug where openai image edit silently ignores multiple images - PR #14893
• VLLM
  • Fix: vLLM provider's rerank endpoint from /v1/rerank to /rerank - PR #14938

New Provider Support​

• W&B Inference
  • Add W&B Inference to LiteLLM - PR #14416

LLM API Endpoints​

Features​

• General
  • Add SDK support for additional headers - PR #14761
  • Add shared_session parameter for aiohttp ClientSession reuse - PR #14721

Bugs​

• General
  • Fix: Streaming tool call index assignment for multiple tool calls - PR #14587
  • Fix load credentials in token counter proxy - PR #14808

Management Endpoints / UI​

Features​

• Proxy CLI Auth
  • Allow re-using cli auth token - PR #14780
  • Create a python method to login using litellm proxy - PR #14782
  • Fixes for LiteLLM Proxy CLI to Auth to Gateway - PR #14836

Virtual Keys

• Initial support for scheduled key rotations - PR #14877
• Allow scheduling key rotations when creating virtual keys - PR #14960

Models + Endpoints

• Fix: added Oracle to provider's list - PR #14835

Bugs​

• SSO - Fix: SSO "Clear" button writes empty values instead of removing SSO config - PR #14826
• Admin Settings - Remove useful links from admin settings - PR #14918
• Management Routes - Add /user/list to management routes - PR #14868

Logging / Guardrail / Prompt Management Integrations​

Features​

• DataDog
  • Logging - datadog callback Log message content w/o sending to datadog - PR #14909
• Langfuse
  • Adding langfuse usage details for cached tokens - PR #10955
• Opik
  • Improve opik integration code - PR #14888
• SQS
  • Error logging support for SQS Logger - PR #14974

Guardrails​

• LakeraAI v2 Guardrail - Ensure exception is raised correctly - PR #14867
• Presidio Guardrail - Support custom entity types in Presidio guardrail with Union[PiiEntityType, str] - PR #14899
• Noma Guardrail - Add noma guardrail provider to ui - PR #14415

Prompt Management​

• BitBucket Integration - Add BitBucket Integration for Prompt Management - PR #14882

Spend Tracking, Budgets and Rate Limiting​

• Service Tier Pricing - Add service_tier based pricing support for openai (BOTH Service & Priority Support) - PR #14796
• Cost Tracking - Show input, output, tool call cost breakdown in StandardLoggingPayload - PR #14921
• Parallel Request Limiter v3
  • Ensure Lua scripts can execute on redis cluster - PR #14968
  • Fix: get metadata info from both metadata and litellm_metadata fields - PR #14783
• Priority Reservation - Fix: Priority Reservation: keys without priority metadata receive higher priority than keys with explicit priority configurations - PR #14832

MCP Gateway​

• MCP Configuration - Enable custom fields in mcp_info configuration - PR #14794
• MCP Tools - Remove server_name prefix from list_tools - PR #14720
• OAuth Flow - Initial commit for v2 oauth flow - PR #14964

Performance / Loadbalancing / Reliability improvements​

• Memory Leak Fix - Fix InMemoryCache unbounded growth when TTLs are set - PR #14869
• Cache Performance - Fix: cache root cause - PR #14827
• Concurrency Fix - Fix concurrency/scaling when many Python threads do streaming using sync completions - PR #14816
• Performance Optimization - Fix: reduce get_deployment cost to O(1) - PR #14967
• Performance Optimization - Fix: remove slow string operation - PR #14955
• DB Connection Management - Fix: DB connection state retries - PR #14925

Documentation Updates​

• Provider Documentation - Fix docs for provider_specific_params.md - PR #14787
• Model References - Update model references from gemini-pro to gemini-2.5-pro - PR #14775
• Letta Guide - Add Letta Guide documentation - PR #14798
• README - Make the README document clearer - PR #14860
• Session Management - Update docs for session management availability - PR #14914
• Cost Documentation - Add documentation for additional cost-related keys in custom pricing - PR #14949
• Azure Passthrough - Add azure passthrough documentation - PR #14958
• General Documentation - Doc updates sept 2025 - PR #14769
  • Clarified bridging between endpoints and mode in docs.
  • Added Vertex AI Gemini API configuration as an alternative in relevant guides. Linked AWS authentication info in the Bedrock guardrails documentation.
  • Added Cancel Response API usage with code snippets
  • Clarified that SSO (Single Sign-On) is free for up to 5 users:
  • Alphabetized sidebar, leaving quick start / intros at top of categories
  • Documented max_connections under cache_params.
  • Clarified IAM AssumeRole Policy requirements.
  • Added transform utilities example to Getting Started (showing request transformation).
  • Added references to models.litellm.ai as the full models list in various docs.
  • Added a code snippet for async_post_call_success_hook.
  • Removed broken links to callbacks management guide. - Reformatted and linked cookbooks + other relevant docs
• Documentation Corrections - Corrected docs updates sept 2025 - PR #14916

New Contributors​

• @uzaxirr made their first contribution in PR #14761
• @xprilion made their first contribution in PR #14416
• @CH-GAGANRAJ made their first contribution in PR #14779
• @otaviofbrito made their first contribution in PR #14778
• @danielmklein made their first contribution in PR #14639
• @Jetemple made their first contribution in PR #14826
• @akshoop made their first contribution in PR #14818
• @hazyone made their first contribution in PR #14821
• @leventov made their first contribution in PR #14816
• @fabriciojoc made their first contribution in PR #10955
• @onlylonly made their first contribution in PR #14845
• @Copilot made their first contribution in PR #14869
• @arsh72 made their first contribution in PR #14899
• @berri-teddy made their first contribution in PR #14914
• @vpbill made their first contribution in PR #14415
• @kgritesh made their first contribution in PR #14893
• @oytunkutrup1 made their first contribution in PR #14858
• @nherment made their first contribution in PR #14933
• @deepanshululla made their first contribution in PR #14974
• @TeddyAmkie made their first contribution in PR #14758
• @SmartManoj made their first contribution in PR #14775
• @uc4w6c made their first contribution in PR #14720
• @luizrennocosta made their first contribution in PR #14783
• @AlexsanderHamir made their first contribution in PR #14827
• @dharamendrak made their first contribution in PR #14721
• @TomeHirata made their first contribution in PR #14164
• @mrFranklin made their first contribution in PR #14860
• @luisfucros made their first contribution in PR #14866
• @huangyafei made their first contribution in PR #14879
• @thiswillbeyourgithub made their first contribution in PR #14949
• @Maximgitman made their first contribution in PR #14965
• @subnet-dev made their first contribution in PR #14938
• @22mSqRi made their first contribution in PR #14972

Full Changelog​

Deploy this version​

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:v1.77.3-stable

pip install litellm==1.77.3

Key Highlights​

• +550 RPS Performance Improvements - Optimizations in request handling and object initialization.
• Priority Quota Reservation - Proxy admins can now reserve TPM/RPM capacity for specific keys.

Priority Quota Reservation​

This release adds support for priority quota reservation. This allows Proxy Admins to reserve specific percentages of model capacity for different use cases.

This is great for use cases where you want to ensure your realtime use cases must always get priority responses and background development jobs can take longer.

This release adds support for priority quota reservation. This allows Proxy Admins to reserve TPM/RPM capacity for keys based on metadata priority levels, ensuring critical production workloads get guaranteed access regardless of development traffic volume.

Get started here

+550 RPS Performance Improvements​

This release delivers significant RPS improvements through targeted optimizations.

We've achieved a +500 RPS boost by fixing cache type inconsistencies that were causing frequent cache misses, plus an additional +50 RPS by removing unnecessary coroutine checks from the hot path.

New Models / Updated Models​

New Model Support​

Features​

• OVHCloud AI Endpoints
  • New provider support with comprehensive model catalog - PR #14494
• CompactifAI
  • New provider integration - PR #14532
• SambaNova
  • Added DeepSeek v3.1 and GPT-OSS-120B models - PR #14500
• Bedrock
  • Cross-region inference profile cost calculation - PR #14566
  • AWS external ID parameter support for authentication - PR #14582
  • CountTokens API implementation - PR #14557
  • Titan V2 encoding_format parameter support - PR #14687
  • Nova Canvas image generation inference profiles - PR #14578
  • Bedrock Batches API - batch processing support with file upload and request transformation - PR #14618
  • Bedrock Twelve Labs embedding provider support - PR #14697
• Vertex AI
  • Gemini labels field provider-aware filtering - PR #14563
  • Gemini Batch API support - PR #14733
• Volcengine
  • Fixed thinking parameters when disabled - PR #14569
• Cohere
  • Handle Generate API deprecation, default to chat endpoints - PR #14676
• TwelveLabs
  • Added Marengo Embed 2.7 embedding support - PR #14674

Bug Fixes​

• Bedrock
  • Empty arguments handling in tool call invocation - PR #14583
• Vertex AI
  • Avoid deepcopy crash with non-pickleables in Gemini/Vertex - PR #14418
• XAI
  • Fix unsupported stop parameter for grok-code models - PR #14565
• Gemini
  • Updated error message for Gemini API - PR #14589
  • Fixed 2.5 Flash Image Preview model routing - PR #14715
  • API key passing for token counting endpoints - PR #14744

New Provider Support​

• OVHCloud AI Endpoints
  • Complete provider integration with model catalog and authentication - PR #14494
• CompactifAI
  • New provider support with documentation - PR #14532

LLM API Endpoints​

Features​

• /responses
  • Added cancel endpoint support for non-admin users - PR #14594
  • Improved response session handling and cold storage configuration with s3 - PR #14534
  • Added OpenAI & Azure /responses/cancel endpoint support - PR #14561
• General
  • Enhanced rate limit error messages with details - PR #14736
  • Middle-truncation for spend log payloads - PR #14637

Bugs​

• /chat/completions
  • Fixed completion chat ID handling - PR #14548
  • Prevent AttributeError for _get_tags_from_request_kwargs - PR #14735
• /responses
  • Fixed cost calculation - PR #14675
• General
  • Rate limiter AttributeError fix - PR #14609

Spend Tracking, Budgets and Rate Limiting​

• Responses API Cost Calculation fix - PR #14675
• Anthropic Cache Token Pricing - Separate 1-hour vs 5-minute cache creation costs - PR #14620, PR #14652
• Indochina Time Timezone support for budget resets - PR #14666
• Soft Budget Alert Cache Issues - Resolved soft budget alert cache issues - PR #14491
• Dynamic Rate Limiter v3 - Priority routing improvements - PR #14734
• Enhanced Rate Limit Errors - More detailed error messages - PR #14736

Management Endpoints / UI​

Features​

• Team Member Service Account Keys - Allow team members to view keys they create - PR #14619
• Default Budget for JWT Teams - Auto-assign budgets to generated teams - PR #14514
• SSO Access Control Groups - Enhanced token info endpoint integration - PR #14738
• Health Test Connect Protection - Restrict access based on model creation permissions - PR #14650
• Amazon Bedrock Guardrail Info View - Enhanced logging visualization - PR #14696

Bug Fixes​

• SCIM v2 - Fix group PUSH and PUT operations for non-existent members - PR #14581
• Guardrail View/Edit/Delete behavior fixes - PR #14622
• In-Memory Guardrail update failures - PR #14653

Logging / Guardrail Integrations​

Features​

• DataDog
  • Enhanced spend tracking metrics - PR #14555
  • Stream support with is_streamed_request parameter - PR #14673
  • Fixed tool calls metadata passing - PR #14531
• Langfuse
  • Added logging support for Responses API - PR #14597
• Langsmith
  • Langsmith Sampling Rate - Key/Team-level tracing configuration - PR #14740
• Prometheus
  • Multi-worker support improvements - PR #14530
  • User email labels in monitoring - PR #14520
• Opik
  • Fixed timezone issue - PR #14708

Bug Fixes​

• S3
  • Fixed 404 error when using s3_endpoint_url - PR #14559

Guardrails​

• Tool Permission Guardrail - Fine-grained tool access control - PR #14519
• Bedrock Guardrails - Selective guarding support with runtime endpoint configuration - PR #14575, PR #14650
• Default Last Message in guardrails - PR #14640
• AWS exceptions handling despite 200 response - PR #14658

New Integration​

• PostHog - Complete observability integration for LiteLLM usage tracking and analytics - PR #14610

MCP Gateway​

• MCP Server Alias Parsing - Multi-part URL path support - PR #14558
• MCP Filter Recomputation - After server deletion - PR #14542
• MCP Gateway Tools List improvements - PR #14695

Performance / Loadbalancing / Reliability improvements​

• +500 RPS Performance Boost when sending the user field - PR #14616
• +50 RPS by removing iscoroutine from hot path - PR #14649
• 7% reduction in init overhead - PR #14689
• Generic Object Pool implementation for better resource management - PR #14702

General Proxy Improvements​

• Middle-Truncation for spend log payloads - PR #14637

Security​

• Security Update - Bump aiohttp==3.12.14, fix CVE-2025-53643 - PR #14638

New Contributors​

• @luisfucros made their first contribution in PR #14500
• @hanakannzashi made their first contribution in PR #14548
• @eliasto made their first contribution in PR #14494
• @Rasmusafj made their first contribution in PR #14491
• @LingXuanYin made their first contribution in PR #14569
• @ronaldpereira made their first contribution in PR #14613
• @hula-la made their first contribution in PR #14534
• @carlos-marchal-ph made their first contribution in PR #14610
• @akraines made their first contribution in PR #14637
• @mrFranklin made their first contribution in PR #14708
• @tcx4c70 made their first contribution in PR #14675
• @michaeltansg made their first contribution in PR #14666
• @tosi29 made their first contribution in PR #14725
• @gmdfalk made their first contribution in PR #14735
• @FelipeRodriguesGare made their first contribution in PR #14733
• @mritunjaysharma394 made their first contribution in PR #14678

Full Changelog​

Deploy this version​

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.77.2-stable

pip install litellm==1.77.2.post1

Key Highlights​

• Bedrock Batches API - Support for creating Batch Inference Jobs on Bedrock using LiteLLM's unified batch API (OpenAI compatible)
• Qwen API Tiered Pricing - Cost tracking support for Dashscope (Qwen) models with multiple pricing tiers

New Models / Updated Models​

New Model Support​

Features​

• Bedrock
  • Bedrock Batches API - batch processing support with file upload and request transformation - PR #14518, PR #14522
• VLLM
  • Added transcription endpoint support - PR #14523
• Ollama
  • ollama_chat/ - images, thinking, and content as list handling - PR #14523
• General
  • New debug flag for detailed request/response logging PR #14482

Bug Fixes​

• Azure OpenAI
  • Fixed extra_body injection causing payload rejection in image generation - PR #14475
• LM Studio
  • Resolved illegal Bearer header value issue - PR #14512

LLM API Endpoints​

Bug Fixes​

• /messages
  • Don't send content block after message w/ finish reason + usage block - PR #14477
• /generateContent
  • Gemini CLI Integration - Fixed token count errors - PR #14451, PR #14417

Spend Tracking, Budgets and Rate Limiting​

Features​

• Qwen API Tiered Pricing - Added comprehensive tiered cost tracking for Dashscope/Qwen models - PR #14471, PR #14479

Bug Fixes​

• Provider Budgets - Fixed provider budget calculations - PR #14459

Management Endpoints / UI​

Features​

• User Headers Mapping - New X-LiteLLM Users mapping feature for enhanced user tracking - PR #14485
• Key Unblocking - Support for hashed tokens in /key/unblock endpoint - PR #14477
• Model Group Header Forwarding - Enhanced wildcard model support with documentation - PR #14528

Bug Fixes​

• Log Tab Key Alias - Fixed filtering inaccuracies for failed logs - PR #14469, PR #14529

Logging / Guardrail Integrations​

Features​

• Noma Integration - Added non-blocking monitor mode with anonymize input support - PR #14401

Performance / Loadbalancing / Reliability improvements​

Performance​

• Removed dynamic creation of static values - PR #14538
• Using _PROXY_MaxParallelRequestsHandler_v3 by default for optimal throughput - PR #14450
• Improved execution context propagation into logging tasks - PR #14455

New Contributors​

• @Sameerlite made their first contribution in PR #14460
• @holzman made their first contribution in PR #14459
• @sashank5644 made their first contribution in PR #14469
• @TomAlon made their first contribution in PR #14401
• @AlexsanderHamir made their first contribution in PR #14538

Full Changelog​

Deploy this version​

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:v1.76.3

pip install litellm==1.76.3

Key Highlights​

• Major Performance Improvements +400 RPS when using correct amount of workers + CPU cores combination
• Video Generation Support - Added Google AI Studio and Vertex AI Veo Video Generation through LiteLLM Pass through routes
• CloudZero Integration - New cost tracking integration for exporting LiteLLM Usage and Spend data to CloudZero.

Major Changes​

• Performance Optimization: LiteLLM Proxy now achieves +400 RPS when using correct amount of CPU cores - PR #14153, PR #14242

  By default, LiteLLM will now use num_workers = os.cpu_count() to achieve optimal performance.

  Override Options:

  Set environment variable:

  DEFAULT_NUM_WORKERS_LITELLM_PROXY=1
  

  Or start LiteLLM Proxy with:

  litellm --num_workers 1
  

• Security Fix: Fixed memory_usage_in_mem_cache cache endpoint vulnerability - PR #14229

Performance Improvements​

This release includes significant performance optimizations. On our internal benchmarks we saw 1 instance get +400 RPS when using correct amount of workers + CPU cores combination.

• +400 RPS Performance Boost - LiteLLM Proxy now uses correct amount of CPU cores for optimal performance - PR #14153
• Default CPU Workers - Changed DEFAULT_NUM_WORKERS_LITELLM_PROXY default to number of CPUs - PR #14242

New Models / Updated Models​

New Model Support​

Features​

• Google Gemini
  • Added 'thoughtSignature' support via 'thinking_blocks' - PR #14122
  • Added support for reasoning_effort='minimal' for Gemini models - PR #14262
• OpenRouter
  • Added GPT-4.1 model family - PR #14101
• Groq
  • Added support for reasoning_effort parameter - PR #14207
• X.AI
  • Fixed XAI cost calculation - PR #14127
• Vertex AI
  • Added support for GPT-OSS models on Vertex AI - PR #14184
  • Added additionalProperties to Vertex AI Schema definition - PR #14252
• VLLM
  • Handle output parsing responses API output - PR #14121
• Ollama
  • Added unified 'thinking' param support via reasoning_content - PR #14121
• Anthropic
  • Added supported text field to anthropic citation response - PR #14126
• OCI Provider
  • Handle assistant messages with both content and tool_calls - PR #14171
• Bedrock
  • Fixed structure output - PR #14130
  • Added initial support for Bedrock Batches API - PR #14190
• Databricks
  • Added support for anthropic citation API in Databricks - PR #14077

Bug Fixes​

• Google Gemini (Google AI Studio + Vertex AI)
  • Fixed Gemini 2.5 Pro schema validation with OpenAI-style type arrays in tools - PR #14154
  • Fixed Gemini Tool Calling empty enum property - PR #14155

New Provider Support​

• Volcengine
  • Added Volcengine embedding module with handler and transformation logic - PR #14028

LLM API Endpoints​

Features​

• Images API
  • Added pass through image generation and image editing on OpenAI - PR #14292
  • Support extra_body parameter for image generation - PR #14211
• Responses API
  • Fixed response API for reasoning item in input for litellm proxy - PR #14200
  • Added structured output for SDK - PR #14206
• Bedrock Passthrough
  • Support AWS_BEDROCK_RUNTIME_ENDPOINT on bedrock passthrough - PR #14156
• Google AI Studio Passthrough
  • Allow using Veo Video Generation through LiteLLM Pass through routes - PR #14228
• General
  • Added support for safety_identifier parameter in chat.completions.create - PR #14174
  • Fixed misclassified 500 error on invalid image_url in /chat/completions request - PR #14149
  • Fixed token count error for Gemini CLI - PR #14133

Bugs​

• General
  • Remove "/" or ":" from model name when being used as h11 header name - PR #14191
  • Bug fix for openai.gpt-oss when using reasoning_effort parameter - PR #14300

Spend Tracking, Budgets and Rate Limiting​

Features​

• Added header support for spend_logs_metadata - PR #14186
• Litellm passthrough cost tracking for chat completion - PR #14256

Bug Fixes​

• Fixed TPM Rate Limit Bug - PR #14237
• Fixed Key Budget not resets at expectable times - PR #14241

Management Endpoints / UI​

Features​

• UI Improvements
  • Logs page screen size fixed - PR #14135
  • Create Organization Tooltip added on Success - PR #14132
  • Back to Keys should say Back to Logs - PR #14134
  • Add client side pagination on All Models table - PR #14136
  • Model Filters UI improvement - PR #14131
  • Remove table filter on user info page - PR #14169
  • Team name badge added on the User Details - PR #14003
  • Fix: Log page parameter passing error - PR #14193
• Authentication & Authorization
  • Support for ES256/ES384/ES512 and EdDSA JWT verification - PR #14118
  • Ensure team_id is a required field for generating service account keys - PR #14270

Bugs​

• General
  • Validate store model in db setting - PR #14269

Logging / Guardrail Integrations​

Features​

• Datadog
  • Ensure apm_id is set on DD LLM Observability traces - PR #14272
• Braintrust
  • Fix logging when OTEL is enabled - PR #14122
• OTEL
  • Optional Metrics and Logs following semantic conventions - PR #14179
• Slack Alerting
  • Added alert type to alert message to slack for easier handling - PR #14176

Guardrails​

• Added guardrail to the Anthropic API endpoint - PR #14107

New Integration​

• CloudZero
  • LiteLLM x CloudZero Integration for Cost Tracking - PR #14296

Performance / Loadbalancing / Reliability improvements​

Features​

• Performance
  • LiteLLM Proxy: +400 RPS when using correct amount of CPU cores - PR #14153
  • Allow using x-litellm-stream-timeout header for stream timeout in requests - PR #14147
  • Change DEFAULT_NUM_WORKERS_LITELLM_PROXY default to number CPUs - PR #14242
• Monitoring
  • Added Prometheus missing metrics - PR #14139
• Timeout
  • Stream Timeout Control - Allow using x-litellm-stream-timeout header for stream timeout in requests - PR #14147
• Routing
  • Fixed x-litellm-tags not routing with Responses API - PR #14289

Bugs​

• Security
  • Fixed memory_usage_in_mem_cache cache endpoint vulnerability - PR #14229

General Proxy Improvements​

Features​

• SCIM Support
  • Added better SCIM debugging - PR #14221
  • Bug fixes for handling SCIM Group Memberships - PR #14226
• Kubernetes
  • Added optional PodDisruptionBudget for litellm proxy - PR #14093
• Error Handling
  • Add model to azure error message - PR #14294

New Contributors​

• @iabhi4 made their first contribution in PR #14093
• @zainhas made their first contribution in PR #14087
• @LifeDJIK made their first contribution in PR #14146
• @retanoj made their first contribution in PR #14133
• @zhxlp made their first contribution in PR #14193
• @kayoch1n made their first contribution in PR #14191
• @kutsushitaneko made their first contribution in PR #14171
• @mjmendo made their first contribution in PR #14176
• @HarshavardhanK made their first contribution in PR #14213
• @eycjur made their first contribution in PR #14207
• @22mSqRi made their first contribution in PR #14241
• @onlylhf made their first contribution in PR #14028
• @btpemercier made their first contribution in PR #11319
• @tremlin made their first contribution in PR #14287
• @TobiMayr made their first contribution in PR #14262
• @Eitan1112 made their first contribution in PR #14252

Full Changelog​

Deploy this version​

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:v1.76.1

pip install litellm==1.76.1

Key Highlights​

• Major Performance Improvements - 6.5x faster LiteLLM Python SDK completion with fastuuid integration.
• New Model Support - Gemini 2.5 Flash Image Preview, Grok Code Fast, and GPT Realtime models
• Enhanced Provider Support - DeepSeek-v3.1 pricing on Fireworks AI, Vercel AI Gateway, and improved Anthropic/GitHub Copilot integration
• MCP Improvements - Better connection testing and SSE MCP tools bug fixes

Major Changes​

• Added support for using Gemini 2.5 Flash Image Preview with /chat/completions. 🚨 Warning If you were using gemini-2.0-flash-exp-image-generation please follow this migration guide. Gemini Image Generation Migration Guide

Performance Improvements​

This release includes significant performance optimizations:

• 6.5x faster LiteLLM Python SDK Completion - Major performance boost for completion operations - PR #13990
• fastuuid Integration - 2.1x faster UUID generation with +80 RPS improvement for /chat/completions and other LLM endpoints - PR #13992, PR #14016
• Optimized Request Logging - Don't print request params by default for +50 RPS improvement - PR #14015
• Cache Performance - 21% speedup in InMemoryCache.evict_cache and 45% speedup in _is_debugging_on function - PR #14012, PR #13988

New Models / Updated Models​

New Model Support​

Additional Models Added: Various other Vercel AI Gateway models were added too. See models.litellm.ai for the full list.

Features​

• Google Gemini
  • Added support for gemini-2.5-flash-image-preview with image return capability - PR #13979, PR #13983
  • Support for requests with only system prompt - PR #14010
  • Fixed invalid model name error for Gemini Imagen models - PR #13991
• X.AI
  • Added xai/grok-code-fast model family support - PR #14054
  • Fixed frequency_penalty parameter for grok-4 models - PR #14078
• OpenAI
  • Added support for gpt-realtime models - PR #14082
  • Support for reasoning and reasoning_effort parameters by default - PR #12865
• Fireworks AI
  • Added DeepSeek-v3.1 pricing - PR #13958
• DeepInfra
  • Fixed reasoning_effort setting for DeepSeek-V3.1 - PR #14053
• GitHub Copilot
  • Added support for thinking and reasoning_effort parameters - PR #13691
  • Added image headers support - PR #13955
• Anthropic
  • Support for custom Anthropic-compatible API endpoints - PR #13945
  • Fixed /messages fallback from Anthropic API to Bedrock API - PR #13946
• Nebius
  • Expanded provider models and normalized model IDs - PR #13965
• Vertex AI
  • Fixed Vertex Mistral streaming issues - PR #13952
  • Fixed anyOf corner cases for Gemini tool calls - PR #12797
• Bedrock
  • Fixed structure output issues - PR #14005
• OpenRouter
  • Added GPT-5 family models pricing - PR #13536

New Provider Support​

• Vercel AI Gateway
  • New provider support added - PR #13144
• DataRobot
  • Added provider documentation - PR #14038, PR #14074

LLM API Endpoints​

Features​

• Images API
  • Support for multiple images in OpenAI images/edits endpoint - PR #13916
  • Allow using dynamic api_key for image generation requests - PR #14007
• Responses API
  • Fixed /responses endpoint ignoring extra_headers in GitHub Copilot - PR #13775
  • Added support for new web_search tool - PR #14083
• Azure Passthrough
  • Fixed Azure Passthrough request with streaming - PR #13831

Bugs​

• General
  • Fixed handling of None metadata in batch requests - PR #13996
  • Fixed token_counter with special token input - PR #13374
  • Removed incorrect web search support for azure/gpt-4.1 family - PR #13566

MCP Gateway​

Features​

• SSE MCP Tools
  • Bug fix for adding SSE MCP tools - improved connection testing when adding MCPs - PR #14048

Read More

Management Endpoints / UI​

Features​

• Team Management
  • Allow setting Team Member RPM/TPM limits when creating a team - PR #13943
• UI Improvements
  • Fixed Next.js Security Vulnerabilities in UI Dashboard - PR #14084
  • Fixed collapsible navbar design - PR #14075

Bugs​

• Authentication
  • Fixed Virtual keys with llm_api type causing Internal Server Error for /anthropic/* and other LLM passthrough routes - PR #14046

Logging / Guardrail Integrations​

Features​

• Langfuse OTEL
  • Allow using LANGFUSE_OTEL_HOST for configuring host - PR #14013
• Braintrust
  • Added span name metadata feature - PR #13573
  • Fixed tests to reference moved attributes in braintrust_logging module - PR #13978
• OpenMeter
  • Set user from token user_id for OpenMeter integration - PR #13152

New Guardrail Support​

• Noma Security
  • Added Noma Security guardrail support - PR #13572
• Pangea
  • Updated Pangea Guardrail to support new AIDR endpoint - PR #13160

Performance / Loadbalancing / Reliability improvements​

Features​

• Caching
  • Verify if cache entry has expired prior to serving it to client - PR #13933
  • Fixed error saving latency as timedelta on Redis - PR #14040
• Router
  • Refactored router to choose weights by 'weight', 'rpm', 'tpm' in one loop for simple_shuffle - PR #13562
• Logging
  • Fixed LoggingWorker graceful shutdown to prevent CancelledError warnings - PR #14050
  • Enhanced logging for containers to log on files both with usual format and json format - PR #13394

Bugs​

• Dependencies
  • Bumped orjson version to "3.11.2" - PR #13969

General Proxy Improvements​

Features​

• AWS
  • Add support for AWS assume_role with a session token - PR #13919
• OCI Provider
  • Added oci_key_file as an optional_parameter - PR #14036
• Configuration
  • Allow configuration to set threshold before request entry in spend log gets truncated - PR #14042
  • Enhanced proxy_config configuration: add support for existing configmap in Helm charts - PR #14041
• Docker
  • Added back supervisor to non-root image - PR #13922

New Contributors​

• @ArthurRenault made their first contribution in PR #13922
• @stevenmanton made their first contribution in PR #13919
• @uc4w6c made their first contribution in PR #13914
• @nielsbosma made their first contribution in PR #13573
• @Yuki-Imajuku made their first contribution in PR #13567
• @codeflash-ai[bot] made their first contribution in PR #13988
• @ColeFrench made their first contribution in PR #13978
• @dttran-glo made their first contribution in PR #13969
• @manascb1344 made their first contribution in PR #13965
• @DorZion made their first contribution in PR #13572
• @edwardsamuel made their first contribution in PR #13536
• @blahgeek made their first contribution in PR #13374
• @Deviad made their first contribution in PR #13394
• @XSAM made their first contribution in PR #13775
• @KRRT7 made their first contribution in PR #14012
• @ikaadil made their first contribution in PR #13991
• @timelfrink made their first contribution in PR #13691
• @qidu made their first contribution in PR #13562
• @nagyv made their first contribution in PR #13243
• @xywei made their first contribution in PR #12885
• @ericgtkb made their first contribution in PR #12797
• @NoWall57 made their first contribution in PR #13945
• @lmwang9527 made their first contribution in PR #14050
• @WilsonSunBritten made their first contribution in PR #14042
• @Const-antine made their first contribution in PR #14041
• @dmvieira made their first contribution in PR #14040
• @gotsysdba made their first contribution in PR #14036
• @moshemorad made their first contribution in PR #14005
• @joshualipman123 made their first contribution in PR #13144

Full Changelog​

Deploy this version​

New Models / Updated Models​

Bugs​

• OpenAI
  • Gpt-5 chat: clarify does not support function calling PR #13612, s/o  @superpoussin22
• VertexAI
  • fix vertexai batch file format by @thiagosalvatore in PR #13576
• LiteLLM Proxy
  • Add support for calling image_edits + image_generations via SDK to Proxy - PR #13735
• OpenRouter
  • Fix max_output_tokens value for anthropic Claude 4 - PR #13526
• Gemini
  • Fix prompt caching cost calculation - PR #13742
• Azure
  • Support ../openai/v1/respones api base - PR #13526
  • Fix azure/gpt-5-chat max_input_tokens - PR #13660
• Groq
  • streaming ASCII encoding issue - PR #13675
• Baseten
  • Refactored integration to use new openai-compatible endpoints - PR #13783
• Bedrock
  • fix application inference profile for pass-through endpoints for bedrock - PR #13881
• DataRobot
  • Updated URL handling for DataRobot provider URL - PR #13880

Features​

• Together AI
  • Added Qwen3, Deepseek R1 0528 Throughput, GLM 4.5 and GPT-OSS models cost tracking - PR #13637, s/o  @Tasmay-Tibrewal
• Fireworks AI
  • add fireworks_ai/accounts/fireworks/models/deepseek-v3-0324 - PR #13821
• VertexAI
  • Add VertexAI qwen API Service - PR #13828
  • Add new VertexAI image models vertex_ai/imagen-4.0-generate-001, vertex_ai/imagen-4.0-ultra-generate-001, vertex_ai/imagen-4.0-fast-generate-001  - PR #13874
• Anthropic
  • Add long context support w/ cost tracking - PR #13759
• DeepInfra
  • Add rerank endpoint support for deepinfra - PR #13820
  • Add new models for cost tracking - PR #13883, s/o  @Toy-97
• Bedrock
  • Add tool prompt caching on async calls - PR #13803, s/o  @UlookEE
  • role chaining and session name with webauthentication for aws bedrock - PR #13753, s/o @RichardoC
• Ollama
  • Handle Ollama null response when using tool calling with non-tool trained models - PR #13902
• OpenRouter
  • Add deepseek/deepseek-chat-v3.1 support - PR #13897
• Mistral
  • Add support for calling mistral files via chat completions - PR #13866, s/o  @jinskjoy
  • Handle empty assistant content - PR #13671
  • Support new ‘thinking’ response block - PR #13671
• Databricks
  • remove deprecated dbrx models (dbrx-instruct, llama 3.1) - PR #13843
• AI/ML API
  • Image gen api support - PR #13893

LLM API Endpoints​

Bugs​

• Responses API
  • add default api version for openai responses api calls - PR #13526
  • support allowed_openai_params - PR #13671

MCP Gateway​

Bugs​

• fix StreamableHTTPSessionManager .run() error - PR #13666

Vector Stores​

Bugs​

• Bedrock
  • Using LiteLLM Managed Credentials for Query - PR #13787

Management Endpoints / UI​

Bugs​

• Passthrough
  • Fix query passthrough deletion - PR #13622

Features​

• Models
  • Add Search Functionality for Public Model Names in Model Dashboard - PR #13687
  • Auto-Add azure/ to deployment Name in UI - PR #13685
  • Models page row UI restructure - PR #13771
• Notifications
  • Add new notifications toast UI everywhere - PR #13813
• Keys
  • Fix key edit settings after regenerating a key - PR #13815
  • Require team_id when creating service account keys - PR #13873
  • Filter - show all options on filter option click - PR #13858
• Usage
  • Fix ‘Cannot read properties of undefined’ exception on user agent activity tab - PR #13892
• SSO
  • Free SSO usage for up to 5 users - PR #13843

Logging / Guardrail Integrations​

Bugs​

• Bedrock Guardrails
  • Add bedrock api key support - PR #13835

Features​

• Datadog LLM Observability
  • Add support for Failure Logging PR #13726
  • Add time to first token, litellm overhead, guardrail overhead latency metrics - PR #13734
  • Add support for tracing guardrail input/output - PR #13767
• Langfuse OTEL
  • Allow using Key/Team Based Logging - PR #13791
• AIM
  • Migrate to new firewall API - PR #13748
• OTEL
  • Add OTEL tracing for actual LLM API call - PR #13836
• MLFlow
  • Include predicted output in MLflow tracing - PR #13795, s/o @TomeHirata 

Performance / Loadbalancing / Reliability improvements​

Bugs​

• Cooldowns
  • don't return raw Azure Exceptions to client (can contain prompt leakage) - PR #13529
• Auto-router
  • Ensures the relevant dependencies for auto router existing on LiteLLM Docker - PR #13788
• Model Alias
  • Fix calling key with access to model alias - PR #13830

Features​

• S3 Caching
  • Use namespace as prefix for s3 cache - PR #13704
  • Async S3 Caching support (4x RPS improvement) - PR #13852, s/o @michal-otmianowski
• Model Group header forwarding
  • reuse same logic as global header forwarding - PR #13741
  • add support for hosted_vllm on UI - PR #13885
• Performance
  • Improve LiteLLM Python SDK RPS by +200 RPS (braintrust import + aiohttp transport fixes) - PR #13839
  • Use O(1) Set lookups for model routing - PR #13879
  • Reduce Significant CPU overhead from litellm_logging.py - PR #13895
  • Improvements for Async Success Handler (Logging Callbacks) - Approx +130 RPS - PR #13905

General Proxy Improvements​

Bugs​

• SDK
  • Fix litellm compatibility with newest release of openAI (>v1.100.0) - PR #13728
• Helm
  • Add possibility to configure resources for migrations-job - PR #13617
  • Ensure Helm chart auto generated master keys follow sk-xxxx format - PR #13871
  • Enhance database configuration: add support for optional endpointKey - PR #13763
• Rate Limits
  • fixing descriptor/response size mismatch on parallel_request_limiter_v3 - PR #13863, s/o  @luizrennocosta
• Non-root
  • fix permission access on prisma migrate in non-root image - PR #13848, s/o @Ithanil

Deploy this version​

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:v1.75.8-stable

pip install litellm==1.75.8

Key Highlights​

• Team Member Rate Limits - Individual rate limiting for team members with JWT authentication support.
• Performance Improvements - New experimental HTTP handler flag for 100+ RPS improvement on OpenAI calls.
• GPT-5 Model Family Support - Full support for OpenAI's GPT-5 models with reasoning_effort parameter and Azure OpenAI integration.
• Azure AI Flux Image Generation - Support for Azure AI's Flux image generation models.

Team Member Rate Limits​

LiteLLM MCP Architecture: Use MCP tools with all LiteLLM supported models

This release adds support for setting rate limits on individual members (including machine users) within a team. Teams can now give each agent its own rate limits—so that heavy-traffic agents don’t impact other agents or human users.

Agents can authenticate with LiteLLM using JWT and the same team role as human users, while still enforcing per-agent rate limits.

New Models / Updated Models​

New Model Support​

Features​

• OpenAI
  • Added reasoning_effort parameter support for GPT-5 model family - PR #13475, Get Started
  • Support for reasoning parameter in Responses API - PR #13475, Get Started
• Azure OpenAI
  • GPT-5 support with max_tokens and reasoning parameter - PR #13510, Get Started
• AWS Bedrock
  • Streaming support for bedrock gpt-oss model family - PR #13346, Get Started
  • /messages endpoint compatibility with bedrock/converse/<model> - PR #13627
  • Cache point support for assistant and tool messages - PR #13640
• Azure AI
  • New Azure AI Flux Image Generation provider - PR #13592, Get Started
  • Fixed Content-Type header for image generation - PR #13584
• CometAPI
  • New provider support with chat completions and streaming - PR #13458
• SambaNova
  • Added embedding model support - PR #13308, Get Started
• Vertex AI
  • Added /countTokens endpoint support for Gemini CLI integration - PR #13545
  • Token counter support for VertexAI models - PR #13558
• hosted_vllm
  • Added reasoning_effort parameter support - PR #13620, Get Started

Bugs​

• OCI
  • Fixed streaming issues - PR #13437
• Ollama
  • Fixed GPT-OSS streaming with 'thinking' field - PR #13375
• VolcEngine
  • Fixed thinking disabled parameter handling - PR #13598
• Streaming
  • Consistent 'finish_reason' chunk indexing - PR #13560

LLM API Endpoints​

Features​

• /messages
  • Tool use arguments properly returned for non-anthropic models - PR #13638

Bugs​

• Real-time API
  • Fixed endpoint for no intent scenarios - PR #13476
• Responses API
  • Fixed stream=True + background=True with Responses API - PR #13654

MCP Gateway​

Features​

• Access Control & Configuration
  • Enhanced MCPServerManager with access groups and description support - PR #13549

Bugs​

• Authentication
  • Fixed MCP gateway key authentication - PR #13630

Read More

Management Endpoints / UI​

Features​

• Team Management
  • Team Member Rate Limits implementation - PR #13601
  • JWT authentication support for team member rate limits - PR #13601
  • Show team member TPM/RPM limits in UI - PR #13662
  • Allow editing team member RPM/TPM limits - PR #13669
  • Allow unsetting TPM and RPM in Teams Settings - PR #13430
  • Team Member Permissions Page access column changes - PR #13145
• Key Management
  • Display errors from backend on the UI Keys page - PR #13435
  • Added confirmation modal before deleting keys - PR #13655
  • Support for user parameter in LiteLLM SDK to Proxy communication - PR #13555
• UI Improvements
  • Fixed internal users table overflow - PR #12736
  • Enhanced chart readability with short-form notation for large numbers - PR #12370
  • Fixed image overflow in LiteLLM model display - PR #13639
  • Removed ambiguous network response errors - PR #13582
• Credentials
  • Added CredentialDeleteModal component and integration with CredentialsPanel - PR #13550
• Admin & Permissions
  • Allow routes for admin viewer - PR #13588

Bugs​

• SCIM Integration
  • Fixed SCIM Team Memberships metadata handling - PR #13553
• Authentication
  • Fixed incorrect key info endpoint - PR #13633

Logging / Guardrail Integrations​

Features​

• Langfuse OTEL
  • Added key/team logging for Langfuse OTEL Logger - PR #13512
  • Fixed LangfuseOtelSpanAttributes constants to match expected values - PR #13659
• MLflow
  • Updated MLflow logger usage span attributes - PR #13561

Bugs​

• Security
  • Hide sensitive data in /model/info - azure entra client_secret - PR #13577
  • Fixed trivy/secrets false positives - PR #13631

Performance / Loadbalancing / Reliability improvements​

Features​

• HTTP Performance
  • New 'EXPERIMENTAL_OPENAI_BASE_LLM_HTTP_HANDLER' flag for +100 RPS improvement on OpenAI calls - PR #13625
• Database Monitoring
  • Added DB metrics to Prometheus - PR #13626
• Error Handling
  • Added safe divide by 0 protection to prevent crashes - PR #13624

Bugs​

• Dependencies
  • Updated boto3 to 1.36.0 and aioboto3 to 13.4.0 - PR #13665

General Proxy Improvements​

Features​

• Database
  • Removed redundant use_prisma_migrate flag - now default - PR #13555
• LLM Translation
  • Added model ID check - PR #13507
  • Refactored Anthropic configurations and added support for anthropic_beta headers - PR #13590

New Contributors​

• @TensorNull made their first contribution in PR #13458
• @MajorD00m made their first contribution in PR #13577
• @VerunicaM made their first contribution in PR #13584
• @huangyafei made their first contribution in PR #13607
• @TomeHirata made their first contribution in PR #13561
• @willfinnigan made their first contribution in PR #13659
• @dcbark01 made their first contribution in PR #13633
• @javacruft made their first contribution in PR #13631

Full Changelog​

Deploy this version​

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:v1.75.5-stable

pip install litellm==1.75.5.post2

Key Highlights​

• Redis - Latency Improvements - Reduces P99 latency by 50% with Redis enabled.
• Responses API Session Management - Support for managing responses API sessions with images.
• Oracle Cloud Infrastructure - New LLM provider for calling models on Oracle Cloud Infrastructure.
• Digital Ocean's Gradient AI - New LLM provider for calling models on Digital Ocean's Gradient AI platform.

Risk of Upgrade​

If you build the proxy from the pip package, you should hold off on upgrading. This version makes prisma migrate deploy our default for managing the DB. This is safer, as it doesn't reset the DB, but it requires a manual prisma generate step.

Users of our Docker image, are not affected by this change.

Redis Latency Improvements​

This release adds in-memory caching for Redis requests, enabling faster response times in high-traffic. Now, LiteLLM instances will check their in-memory cache for a cache hit, before checking Redis. This reduces caching-related latency from 100ms for LLM API calls to sub-1ms, on cache hits.

Responses API Session Management w/ Images​

LiteLLM now supports session management for Responses API requests with images. This is great for use-cases like chatbots, that are using the Responses API to track the state of a conversation. LiteLLM session management works across ALL LLM API's (including Anthropic, Bedrock, OpenAI, etc). LiteLLM session management works by storing the request and response content in an s3 bucket, you can specify.

New Models / Updated Models​

New Model Support​

Features​

• OCI
  • New LLM provider - PR #13206
• JinaAI
  • support multimodal embedding models - PR #13181
• GPT-5 (OpenAI/Azure)
  • Support drop_params for temperature - PR #13390
  • Map max_tokens to max_completion_tokens - PR #13390
• Anthropic
  • Add claude-opus-4-1 on model cost map - PR #13384
• OpenRouter
  • Add gpt-oss to model cost map - PR #13442
• Cerebras
  • Add gpt-oss to model cost map - PR #13442
• Azure
  • Support drop params for ‘temperature’ on o-series models - PR #13353
• GradientAI
  • New LLM Provider - PR #12169

Bugs​

• OpenAI
  • Add ‘service_tier’ and ‘safety_identifier’ as supported responses api params - PR #13258
  • Correct pricing for web search on 4o-mini - PR #13269
• Mistral
  • Handle $id and $schema fields when calling mistral - PR #13389

LLM API Endpoints​

Features​

• /responses
  • Responses API Session Handling w/ support for images - PR #13347
  • failed if input containing ResponseReasoningItem - PR #13465
  • Support custom tools - PR #13418

Bugs​

• /chat/completions
  • Fix completion_token_details usage object missing ‘text’ tokens - PR #13234
  • (SDK) handle tool being a pydantic object - PR #13274
  • include cost in streaming usage object - PR #13418
  • Exclude none fields on /chat/completion - allows usage with n8n - PR #13320
• /responses
  • Transform function call in response for non-openai models (gemini/anthropic) - PR #13260
  • Fix unsupported operand error with model groups - PR #13293
  • Responses api session management for streaming responses - PR #13396
• /v1/messages
  • Added litellm claude code count tokens - PR #13261
• /vector_stores
  • Fix create/search vector store errors - PR #13285

MCP Gateway​

Features​

• Add route check for internal users - PR #13350
• MCP Guardrails - docs - PR #13392

Bugs​

• Fix auth on UI for bearer token servers - PR #13312
• allow access group on mcp tool retrieval - PR #13425

Management Endpoints / UI​

Features​

• Teams
  • Add team deletion check for teams with keys - PR #12953
• Models
  • Add ability to set model alias per key/team - PR #13276
  • New button to reload model pricing from model cost map - PR #13464, PR #13470
• Keys
  • Make ‘team’ field required when creating service account keys - PR #13302
  • Gray out key-based logging settings for non-enterprise users - prevents confusion on if ‘logging’ all up is supported - PR #13431
• Navbar
  • Add logo customization for LiteLLM admin UI - PR #12958
• Logs
  • Add token breakdowns on logs + session page - PR #13357
• Usage
  • Ensure Usage Page loads after the DB has large entries - PR #13400
• Test Key Page
  • allow uploading images for /chat/completions and /responses - PR #13445
• MCP
  • Add auth tokens to local storage auth - PR #13473

Bugs​

• Custom Root Path
  • Fix login route when SSO is enabled - PR #13267
• Customers/End-users
  • Allow calling /v1/models when end user over budget - allows model listing to work on OpenWebUI when customer over budget - PR #13320
• Teams
  • Remove user - team membership, when user removed from team - PR #13433
• Errors
  • Bubble up network errors to user for Logging and Alerts page - PR #13427
• Model Hub
  • Show pricing for azure models, when base model is set - PR #13418

Logging / Guardrail Integrations​

Features​

• Bedrock Guardrails
  • Redacted sensitive information in bedrock guardrails error message - PR #13356
• Standard Logging Payload
  • Fix ‘can’t register atextexit’ bug - PR #13436

Bugs​

• Braintrust
  • Allow setting of braintrust callback base url - PR #13368
• OTEL
  • Track pre_call hook latency - PR #13362

Performance / Loadbalancing / Reliability improvements​

Features​

• Team-BYOK models
  • Add wildcard model support - PR #13278
• Caching
  • GCP IAM auth support for caching - PR #13275
• Latency
  • reduce p99 latency w/ redis enabled by 50% - only updates model usage if tpm/rpm limits set - PR #13362

General Proxy Improvements​

Features​

• Models
  • Support /v1/models/{model_id} retrieval - PR #13268
• Multi-instance
  • Ensure disable_llm_api_endpoints works - PR #13278
• Logs
  • Add apscheduler log suppress - PR #13299
• Helm
  • Add labels to migrations job template - PR #13343 s/o @unique-jakub

Bugs​

• Non-root image
  • Fix non-root image for migration - PR #13379
• Get Routes
  • Load get routes when using fastapi-offline - PR #13466
• Health checks
  • Generate unique trace IDs for Langfuse health checks - PR #13468
• Swagger
  • Allow using Swagger for /chat/completions - PR #13469
• Auth
  • Fix JWTs access not working with model access groups - PR #13474

New Contributors​

• @bbartels made their first contribution in https://github.com/BerriAI/litellm/pull/13244
• @breno-aumo made their first contribution in https://github.com/BerriAI/litellm/pull/13206
• @pascalwhoop made their first contribution in https://github.com/BerriAI/litellm/pull/13122
• @ZPerling made their first contribution in https://github.com/BerriAI/litellm/pull/13045
• @zjx20 made their first contribution in https://github.com/BerriAI/litellm/pull/13181
• @edwarddamato made their first contribution in https://github.com/BerriAI/litellm/pull/13368
• @msannan2 made their first contribution in https://github.com/BerriAI/litellm/pull/12169

Full Changelog​

Deploy this version​

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:v1.74.15-stable

pip install litellm==1.74.15.post2

Key Highlights​

• User Agent Activity Tracking - Track how much usage each coding tool gets.
• Prompt Management - Use Git-Ops style prompt management with prompt templates.
• MCP Gateway: Guardrails - Support for using Guardrails with MCP servers.
• Google AI Studio Imagen4 - Support for using Imagen4 models on Google AI Studio.

User Agent Activity Tracking​

This release brings support for tracking usage and costs for AI-powered coding tools like Claude Code, Roo Code, Gemini CLI through LiteLLM. You can now track LLM cost, total tokens used, and DAU/WAU/MAU for each coding tool.

This is great to central AI Platform teams looking to track how they are helping developer productivity.

Read More

Prompt Management​

Read More

New Models / Updated Models​

New Model Support​

Features​

• Google AI Studio
  • Added Google AI Studio Imagen4 model family support - PR #13065, Get Started
• Azure OpenAI
  • Azure api_version="preview" support - PR #13072, Get Started
  • Password protected certificate files support - PR #12995, Get Started
• AWS Bedrock
  • Cost tracking via Anthropic /v1/messages - PR #13072
  • Computer use support - PR #13150
• OpenRouter
  • Added Grok4 model support - PR #13018
• Anthropic
  • Auto Cache Control Injection - Improved cache_control_injection_points with negative index support - PR #13187, Get Started
  • Working mid-stream fallbacks with token usage tracking - PR #13149, PR #13170
• Perplexity
  • Citation annotations support - PR #13225

Bugs​

• Gemini
  • Fix merge_reasoning_content_in_choices parameter issue - PR #13066, Get Started
  • Added support for using GOOGLE_API_KEY environment variable for Google AI Studio - PR #12507
• vLLM/OpenAI-like
  • Fix missing extra_headers support for embeddings - PR #13198

LLM API Endpoints​

Bugs​

• /generateContent
  • Support for query_params in generateContent routes for API Key setting - PR #13100
  • Ensure "x-goog-api-key" is used for auth to google ai studio when using /generateContent on LiteLLM - PR #13098
  • Ensure tool calling works as expected on generateContent - PR #13189
• /vertex_ai (Passthrough)
  • Ensure multimodal embedding responses are logged properly - PR #13050

MCP Gateway​

Features​

• Health Check Improvements
  • Add health check endpoints for MCP servers - PR #13106
• Guardrails Integration
  • Add pre and during call hooks initialization - PR #13067
  • Move pre and during hooks to ProxyLogging - PR #13109
  • MCP pre and during guardrails implementation - PR #13188
• Protocol & Header Support
  • Add protocol headers support - PR #13062
• URL & Namespacing
  • Improve MCP server URL validation for internal/Kubernetes URLs - PR #13099

Bugs​

• UI
  • Fix scrolling issue with MCP tools - PR #13015
  • Fix MCP client list failure - PR #13114

Read More

Management Endpoints / UI​

Features​

• Usage Analytics

  • New tab for user agent activity tracking - PR #13146
  • Daily usage per user analytics - PR #13147
  • Default usage chart date range set to last 7 days - PR #12917
  • New advanced date range picker component - PR #13141, PR #13221
  • Show loader on usage cost charts after date selection - PR #13113

• Models

  • Added Voyage, Jinai, Deepinfra and VolcEngine providers on UI - PR #13131
  • Added Sagemaker on UI - PR #13117
  • Preserve model order in /v1/models and /model_group/info endpoints - PR #13178

• Key Management

  • Properly parse JSON options for key generation in UI - PR #12989

• Authentication

  • JWT Fields
    • Add dot notation support for all JWT fields - PR #13013

Bugs​

• Permissions
  • Fix object permission for organizations - PR #13142
  • Fix list team v2 security check - PR #13094
• Models
  • Fix model reload on model update - PR #13216
• Router Settings
  • Fix displaying models for fallbacks in UI - PR #13191
  • Fix wildcard model name handling with custom values - PR #13116
  • Fix fallback delete functionality - PR #12606

Logging / Guardrail Integrations​

Features​

• MLFlow
  • Allow adding tags for MLFlow logging requests - PR #13108
• Langfuse OTEL
  • Add comprehensive metadata support to Langfuse OpenTelemetry integration - PR #12956
• Datadog LLM Observability
  • Allow redacting message/response content for specific logging integrations - PR #13158

Bugs​

• API Key Logging
  • Fix API Key being logged inappropriately - PR #12978
• MCP Spend Tracking
  • Set default value for MCP namespace tool name in spend table - PR #12894

Performance / Loadbalancing / Reliability improvements​

Features​

• Background Health Checks
  • Allow disabling background health checks for specific deployments - PR #13186
• Database Connection Management
  • Ensure stale Prisma clients disconnect DB connections properly - PR #13140
• Jitter Improvements
  • Fix jitter calculation (should be added not multiplied) - PR #12901

Bugs​

• Anthropic Streaming
  • Always use choice index=0 for Anthropic streaming responses - PR #12666
• Custom Auth
  • Bubble up custom exceptions properly - PR #13093
• OTEL with Managed Files
  • Fix using managed files with OTEL integration - PR #13171

General Proxy Improvements​

Features​

• Database Migration
  • Move to use_prisma_migrate by default - PR #13117
  • Resolve team-only models on auth checks - PR #13117
• Infrastructure
  • Loosened MCP Python version restrictions - PR #13102
  • Migrate build_and_test to CI/CD Postgres DB - PR #13166
• Helm Charts
  • Allow Helm hooks for migration jobs - PR #13174
  • Fix Helm migration job schema updates - PR #12809

Bugs​

• Docker
  • Remove obsolete version attribute in docker-compose - PR #13172
  • Add openssl in runtime stage for non-root Dockerfile - PR #13168
• Database Configuration
  • Fix DB config through environment variables - PR #13111
• Logging
  • Suppress httpx logging - PR #13217
• Token Counting
  • Ignore unsupported keys like prefix in token counter - PR #11954

New Contributors​

• @5731la made their first contribution in https://github.com/BerriAI/litellm/pull/12989
• @restato made their first contribution in https://github.com/BerriAI/litellm/pull/12980
• @strickvl made their first contribution in https://github.com/BerriAI/litellm/pull/12956
• @Ne0-1 made their first contribution in https://github.com/BerriAI/litellm/pull/12995
• @maxrabin made their first contribution in https://github.com/BerriAI/litellm/pull/13079
• @lvuna made their first contribution in https://github.com/BerriAI/litellm/pull/12894
• @Maximgitman made their first contribution in https://github.com/BerriAI/litellm/pull/12666
• @pathikrit made their first contribution in https://github.com/BerriAI/litellm/pull/12901
• @huetterma made their first contribution in https://github.com/BerriAI/litellm/pull/12809
• @betterthanbreakfast made their first contribution in https://github.com/BerriAI/litellm/pull/13029
• @phosae made their first contribution in https://github.com/BerriAI/litellm/pull/12606
• @sahusiddharth made their first contribution in https://github.com/BerriAI/litellm/pull/12507
• @Amit-kr26 made their first contribution in https://github.com/BerriAI/litellm/pull/11954
• @kowyo made their first contribution in https://github.com/BerriAI/litellm/pull/13172
• @AnandKhinvasara made their first contribution in https://github.com/BerriAI/litellm/pull/13187
• @unique-jakub made their first contribution in https://github.com/BerriAI/litellm/pull/13174
• @tyumentsev4 made their first contribution in https://github.com/BerriAI/litellm/pull/13134
• @aayush-malviya-acquia made their first contribution in https://github.com/BerriAI/litellm/pull/12978
• @kankute-sameer made their first contribution in https://github.com/BerriAI/litellm/pull/13225
• @AlexanderYastrebov made their first contribution in https://github.com/BerriAI/litellm/pull/13178

Full Changelog​

Deploy this version​

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:v1.74.9-stable.patch.1

pip install litellm==1.74.9.post2

Key Highlights​

• Auto-Router - Automatically route requests to specific models based on request content.
• Model-level Guardrails - Only run guardrails when specific models are used.
• MCP Header Propagation - Propagate headers from client to backend MCP.
• New LLM Providers - Added Bedrock inpainting support and Recraft API image generation / image edits support.

Auto-Router​

This release introduces auto-routing to models based on request content. This means Proxy Admins can define a set of keywords that always routes to specific models when users opt in to using the auto-router.

This is great for internal use cases where you don't want users to think about which model to use - for example, use Claude models for coding vs GPT models for generating ad copy.

Read More

Model-level Guardrails​

This release brings model-level guardrails support to your config.yaml + UI. This is great for cases when you have an on-prem and hosted model, and just want to run prevent sending PII to the hosted model.

model_list:
  - model_name: claude-sonnet-4
    litellm_params:
      model: anthropic/claude-sonnet-4-20250514
      api_key: os.environ/ANTHROPIC_API_KEY
      api_base: https://api.anthropic.com/v1
      guardrails: ["azure-text-moderation"] # 👈 KEY CHANGE

guardrails:
  - guardrail_name: azure-text-moderation
    litellm_params:
      guardrail: azure/text_moderations
      mode: "post_call" 
      api_key: os.environ/AZURE_GUARDRAIL_API_KEY
      api_base: os.environ/AZURE_GUARDRAIL_API_BASE 

Read More

MCP Header Propagation​

v1.74.9-stable allows you to propagate MCP server specific authentication headers via LiteLLM

• Allowing users to specify which header_name is to be propagated to which mcp_server via headers
• Allows adding of different deployments of same MCP server type to use different authentication headers

Read More

New Models / Updated Models​

Pricing / Context Window Updates​

Features​

• Lambda AI
  • New LLM API provider - PR #12817
• Github Copilot
  • Dynamic endpoint support - PR #12827
• Morph
  • New LLM API provider - PR #12821
• Groq
  • Remove deprecated groq/qwen-qwq-32b - PR #12832
• Recraft
  • New image generation API - PR #12832
  • New image edits api - PR #12874
• Azure OpenAI
  • Support DefaultAzureCredential without hard-coded environment variables - PR #12841
• Hyperbolic
  • New LLM API provider - PR #12826
• OpenAI
  • /realtime API - pass through intent query param - PR #12838
• Bedrock
  • Add inpainting support for Amazon Nova Canvas - PR #12949 s/o @SantoshDhaladhuli

Bugs​

• Gemini (Google AI Studio + VertexAI)
  • Fix leaking file descriptor error on sync calls - PR #12824
• IBM Watsonx
  • use correct parameter name for tool choice - PR #9980
• Anthropic
  • Only show ‘reasoning_effort’ for supported models - PR #12847
  • Handle $id and $schema in tool call requests (Anthropic API stopped accepting them) - PR #12959
• Openrouter
  • filter out cache_control flag for non-anthropic models (allows usage with claude code) https://github.com/BerriAI/litellm/pull/12850
• Gemini
  • Shorten Gemini tool_call_id for Open AI compatibility - PR #12941 s/o @tonga54

LLM API Endpoints​

Features​

• Passthrough endpoints
  • Make key/user/team cost tracking OSS - PR #12847
• /v1/models
  • Return fallback models as part of api response - PR #12811 s/o @murad-khafizov
• /vector_stores
  • Make permission management OSS - PR #12990

Bugs​

1. /batches
   1. Skip invalid batch during cost tracking check (prev. Would stop all checks) - PR #12782
2. /chat/completions
   1. Fix async retryer on .acompletion() - PR #12886

MCP Gateway​

Features​

• Permission Management
  • Make permission management by key/team OSS - PR #12988
• MCP Alias
  • Support mcp server aliases (useful for calling long mcp server names on Cursor) - PR #12994
• Header Propagation
  • Support propagating headers from client to backend MCP (useful for sending personal access tokens to backend MCP) - PR #13003

Management Endpoints / UI​

Features​

• Usage
  • Support viewing usage by model group - PR #12890
• Virtual Keys
  • New key_type field on /key/generate - allows specifying if key can call LLM API vs. Management routes - PR #12909
• Models
  • Add ‘auto router’ on UI - PR #12960
  • Show global retry policy on UI - PR #12969
  • Add model-level guardrails on create + update - PR #13006

Bugs​

• SSO
  • Fix logout when SSO is enabled - PR #12703
  • Fix reset SSO when ui_access_mode is updated - PR #13011
• Guardrails
  • Show correct guardrails when editing a team - PR #12823
• Virtual Keys
  • Get updated token on regenerate key - PR #12788
  • Fix CVE with key injection - PR #12840

Logging / Guardrail Integrations​

Features​

• Google Cloud Model Armor
  • Document new guardrail - PR #12492
• Pillar Security
  • New LLM Guardrail - PR #12791
• CloudZero
  • Allow exporting spend to cloudzero - PR #12908
• Model-level Guardrails
  • Support model-level guardrails - PR #12968

Bugs​

• Prometheus
  • Fix [tag]=false when tag is set for tag-based metrics - PR #12916
• Guardrails AI
  • Use ‘validatedOutput’ to allow usage of “fix” guards - PR #12891 s/o @DmitriyAlergant

Performance / Loadbalancing / Reliability improvements​

Features​

• Auto-Router
  • New auto-router powered by semantic-router - PR #12955

Bugs​

• forward_clientside_headers
  • Filter out content-length from headers (caused backend requests to hang) - PR #12886
• Message Redaction
  • Fix cannot pickle coroutine object error - PR #13005

General Proxy Improvements​

Features​

• Benchmarks
  • Updated litellm proxy benchmarks (p50, p90, p99 overhead) - PR #12842
• Request Headers
  • Added new x-litellm-num-retries request header
• Swagger
  • Support local swagger on custom root paths - PR #12911
• Health
  • Track cost + add tags for health checks done by LiteLLM Proxy - PR #12880