Skip to content

fix(models): keep proxied file uploads inside the proxy - #6723

Open
vietnamesekid wants to merge 2 commits into
google:mainfrom
vietnamesekid:fix/litellm-proxy-upload-endpoint
Open

fix(models): keep proxied file uploads inside the proxy#6723
vietnamesekid wants to merge 2 commits into
google:mainfrom
vietnamesekid:fix/litellm-proxy-upload-endpoint

Conversation

@vietnamesekid

Copy link
Copy Markdown
Contributor

Link to Issue or Description of Change

Problem:

litellm.acreate_file resolves its endpoint independently of the completion
call. ADK passes only custom_llm_provider, so a proxied upload falls back to
the underlying provider's own credential resolution and the request splits in
two: the completion goes to the proxy, the upload goes elsewhere.

For azure that is a local failure. For openai it is not a failure at all.
get_openai_credentials defaults api_base to https://api.openai.com/v1, so
a stray OPENAI_API_KEY makes the upload succeed against the public OpenAI API
carrying proxied file content and the developer's own key.

Three configurations reach that fallback. Each row was run with the repro
script in the linked issue, changing only .env and the model string.

# Configuration Model Before
1 LITELLM_PROXY_API_BASE + LITELLM_PROXY_API_KEY litellm_proxy/openai/gpt-4o upload to api.openai.com
2 USE_LITELLM_PROXY=true openai/gpt-4o upload to api.openai.com
3 neither, only OPENAI_API_KEY litellm_proxy/openai/gpt-4o upload to api.openai.com

Case 1 exists because LiteLLM reads those two variables for the completion call
in litellm/llms/litellm_proxy/chat/transformation.py, while the upload path
does not. Case 2 exists because matching on the litellm_proxy/ prefix alone
misses USE_LITELLM_PROXY, which routes unprefixed models through the proxy.
LiteLLM's own docstring says that flag exists for Google ADK
(BerriAI/litellm#10559).

On 602e58d, the merge of #6578: the provider resolution part of that change is
correct and this PR keeps it. The same commit also set
custom_llm_provider="openai" for proxied uploads, and that is the line that
turns cases 1 and 3 from a local error into a request to api.openai.com,
since openai is the provider that defaults to the public endpoint. On main,
_get_content takes only parts, provider and model, and ADK never sets
litellm.api_base, so there is no configuration in which that override could
reach a proxy. This PR keeps the real provider, which is what preserves the
provider specific payload shape, and redirects the endpoint instead.

Solution:

  1. _get_upload_params falls back to LITELLM_PROXY_API_BASE and
    LITELLM_PROXY_API_KEY when the completion call carries no endpoint.
    Explicit completion arguments still win.
  2. _is_proxied_model also treats USE_LITELLM_PROXY as proxied, covering
    unprefixed models routed through the proxy.
  3. The upload raises when the model is proxied and no endpoint can be
    determined, rather than falling through to the provider default.

Direct models are untouched. They forward nothing and keep their existing
environment variable resolution, since their api_base already points at the
provider.

One behavior change worth deciding on rather than absorbing silently, item 3
above. Anyone who has OPENAI_API_KEY set and a proxied model has a working
upload today, to the wrong destination. After this they get a ValueError
naming the configuration to set. I think an explicit error beats a silent
misroute, but it is a trade off. If you would rather not take it, items 1 and 2
alone still fix every case where the proxy endpoint is knowable.

Testing Plan

Unit Tests:

  • I have added or updated unit tests for my change.
  • All unit tests pass locally.
$ pytest tests/unittests/models/test_litellm.py -q
398 passed in 2.68s

$ pytest tests/unittests/models/ -q
6 failed, 1068 passed, 34 warnings in 13.40s

The 6 failures are all in test_anthropic_llm.py and reproduce identically on
a pristine main worktree at 370027a, so they are pre-existing and unrelated
to this change.

New tests:

  • test_get_upload_params_falls_back_to_proxy_env, covering the env fallback
    and that explicit completion arguments take precedence over it.
  • test_is_proxied_model_honors_use_litellm_proxy_flag, covering the flag and
    that the explicit prefix still wins regardless of it.
  • test_get_content_proxied_upload_without_endpoint_raises, which asserts
    acreate_file is not called at all when the destination is unknown.

I also isolated test_get_upload_params from the ambient environment. It would
otherwise pick up LITELLM_PROXY_* from a developer or CI machine and pass or
fail depending on which shell it ran in.

Each part of the fix was reverted on its own to check the tests fail without it
instead of passing by construction. Reverting the env fallback, the flag check,
or the guard each fails exactly its own test and nothing else.

Manual End-to-End (E2E) Tests:

Driven through LiteLlm(...) rather than internal helpers, with configuration
loaded from .env by python-dotenv, a local HTTP server standing in for the
proxy, and every outbound request recorded and blocked if it left localhost.
Full script is in the linked issue.

.env:

LITELLM_PROXY_API_BASE=http://127.0.0.1:8877
LITELLM_PROXY_API_KEY=sk-proxy-secret
OPENAI_API_KEY=sk-personal-developer-key

Before, with model="litellm_proxy/openai/gpt-4o":

raised         : APIConnectionError: Connection error.
proxy received : []
left the proxy : ['https://api.openai.com/v1/files', ...]

After:

proxy received : ['/files', '/chat/completions']
left the proxy : []

Both halves of the request reach the proxy, and the proxy sees
Authorization: Bearer sk-proxy-secret, so it is the proxy credentials being
used and not the personal key.

Full matrix, same script throughout:

.env Model Before After
proxy only litellm_proxy/azure/my-deployment fails, proxy gets nothing reaches proxy
proxy + OPENAI_API_KEY litellm_proxy/openai/gpt-4o goes to api.openai.com reaches proxy
USE_LITELLM_PROXY=true openai/gpt-4o goes to api.openai.com reaches proxy
only OPENAI_API_KEY litellm_proxy/openai/gpt-4o goes to api.openai.com raises, sends nothing

Reproduced on litellm 1.85.7 and on 1.84.0, the floor of the litellm>=1.84
constraint in pyproject.toml, so this is not specific to one release.

I do not have an Azure backed proxy deployment, so none of this is confirmed
against a live endpoint. Everything above is the request ADK builds and where
it actually goes, measured against a local server.

Formatted with pyink 25.12.0 and isort 8.0.1, matching the pinned versions
in .pre-commit-config.yaml.

Checklist

  • I have read the CONTRIBUTING.md document.
  • I have performed a self-review of my own code.
  • I have commented my code, particularly in hard-to-understand areas.
  • I have added tests that prove my fix is effective or that my feature works.
  • New and existing unit tests pass locally with my changes.
  • I have manually tested my changes end-to-end.
  • Any dependent changes have been merged and published in downstream modules.

Additional context

Known limit, deliberately out of scope. This covers the proxy endpoint when it
comes from constructor arguments or from LITELLM_PROXY_API_BASE and
LITELLM_PROXY_API_KEY. A proxy configured by some other means that ADK cannot
see still lands in case 3 and raises. That is the safe outcome but it is still
a failure, and closing it would mean reimplementing more of LiteLLM's
configuration resolution inside ADK. That felt like your call rather than mine.

The two commits are separable if you would rather take them independently.
7ab75d24 forwards constructor supplied endpoints, which is the follow up I
raised on #6578, and d8643357 closes the three cases above.

`litellm.acreate_file` resolves its endpoint independently of the completion
call. ADK passed only `custom_llm_provider`, so the upload fell back to the
underlying provider's environment variables (`AZURE_API_BASE`,
`AZURE_API_KEY`).

For a proxied model that splits the request in two: the completion goes to the
proxy while the upload goes straight to Azure. Callers who configure only
`LiteLlm(model="litellm_proxy/azure/...", api_base=..., api_key=...)` and hold
no Azure credentials of their own get `OpenAIError: Missing credentials`, which
is the whole reason to front a provider with a proxy.

Forward `api_base`, `api_key`, and `api_version` from the completion arguments
to the upload when the model is proxied. Direct models forward nothing and keep
their existing environment-variable resolution, since their `api_base` already
points at the provider.

Verified against a local HTTP endpoint: with only proxy credentials set, the
upload now reaches the proxy and returns a real file_id, where before it raised
before sending anything.
`litellm.acreate_file` resolves its endpoint independently of the completion
call, so a proxied upload falls back to the underlying provider's own
credentials. That splits the request in two: the completion goes to the proxy
while the upload goes somewhere else.

For `azure` that fails locally with missing credentials. For `openai` it is
worse than a failure: `get_openai_credentials` defaults `api_base` to
`https://api.openai.com/v1`, so a stray `OPENAI_API_KEY` in the environment
silently sends proxied file content to the public OpenAI API.

Three paths reached that fallback:

- The proxy configured through `LITELLM_PROXY_API_BASE` /
  `LITELLM_PROXY_API_KEY` rather than constructor arguments. LiteLLM reads
  these for the completion call, so the upload has to follow the same
  resolution.
- `USE_LITELLM_PROXY=true`, which routes unprefixed models such as
  `openai/gpt-4o` through the proxy. Matching only on the `litellm_proxy/`
  prefix missed those entirely.
- No proxy endpoint determinable at all, where the upload previously fell
  through to the provider default.

Fall back to the proxy environment variables, treat the `USE_LITELLM_PROXY`
flag as proxied, and raise when the endpoint cannot be determined rather than
sending file content to the provider's default endpoint.

Note that the last case turns a silently misrouted upload into an explicit
error. Callers relying on a stray `OPENAI_API_KEY` to make proxied uploads
"work" will now see a failure that names the missing configuration.

Verified with dotenv-loaded .env files against a local proxy, driving
`LiteLlm(...)` end to end with all outbound requests recorded. Both the
upload and the completion reach the proxy, carrying the proxy credentials,
with no request to api.openai.com in any scenario.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Proxied file uploads bypass the LiteLLM Proxy and are sent to api.openai.com

1 participant