From 1289188f69210b5202b137be5a5ca68d7860862a Mon Sep 17 00:00:00 2001 From: Cameron G <156701171+camgrimsec@users.noreply.github.com> Date: Sat, 27 Jun 2026 17:22:44 -0400 Subject: [PATCH] docs(generators): document OpenAI token limits --- .../generators/azureopenaichatgenerator.mdx | 6 ++++++ .../pipeline-components/generators/azureopenaigenerator.mdx | 6 ++++++ .../pipeline-components/generators/openaichatgenerator.mdx | 6 ++++++ .../docs/pipeline-components/generators/openaigenerator.mdx | 6 ++++++ 4 files changed, 24 insertions(+) diff --git a/docs-website/docs/pipeline-components/generators/azureopenaichatgenerator.mdx b/docs-website/docs/pipeline-components/generators/azureopenaichatgenerator.mdx index d886817beb6..8a92db84c7e 100644 --- a/docs-website/docs/pipeline-components/generators/azureopenaichatgenerator.mdx +++ b/docs-website/docs/pipeline-components/generators/azureopenaichatgenerator.mdx @@ -59,6 +59,12 @@ Then, the component needs a list of `ChatMessage` objects to operate. `ChatMessa You can pass any chat completion parameters that are valid for the `openai.ChatCompletion.create` method directly to `AzureOpenAIChatGenerator` using the `generation_kwargs` parameter, both at initialization and to `run()` method. For more details on the supported parameters, refer to the [Azure documentation](https://learn.microsoft.com/en-us/azure/ai-services/openai/reference). +:::tip[Production cost control] + +For production workloads, set a completion token limit such as `generation_kwargs={"max_completion_tokens": 512}`. This bounds response length, latency, and API spend. If you use this generator inside an agent or retrying pipeline, the limit applies to each model call, so choose a value that fits your expected answer size. + +::: + You can also specify a model for this component through the `azure_deployment` init parameter. ### Structured Output diff --git a/docs-website/docs/pipeline-components/generators/azureopenaigenerator.mdx b/docs-website/docs/pipeline-components/generators/azureopenaigenerator.mdx index 70609181419..295595b7708 100644 --- a/docs-website/docs/pipeline-components/generators/azureopenaigenerator.mdx +++ b/docs-website/docs/pipeline-components/generators/azureopenaigenerator.mdx @@ -45,6 +45,12 @@ We recommend using environment variables instead of initialization parameters. Then, the component needs a prompt to operate, but you can pass any text generation parameters valid for the `openai.ChatCompletion.create` method directly to this component using the `generation_kwargs` parameter, both at initialization and to `run()` method. For more details on the supported parameters, refer to the [Azure documentation](https://learn.microsoft.com/en-us/azure/ai-services/openai/reference). +:::tip[Production cost control] + +For production workloads, set a completion token limit such as `generation_kwargs={"max_completion_tokens": 512}`. This bounds response length, latency, and API spend. If you use this generator inside a retrying pipeline, the limit applies to each model call, so choose a value that fits your expected answer size. + +::: + You can also specify a model for this component through the `azure_deployment` init parameter. ### Streaming diff --git a/docs-website/docs/pipeline-components/generators/openaichatgenerator.mdx b/docs-website/docs/pipeline-components/generators/openaichatgenerator.mdx index e4b7b47bd94..807509caddf 100644 --- a/docs-website/docs/pipeline-components/generators/openaichatgenerator.mdx +++ b/docs-website/docs/pipeline-components/generators/openaichatgenerator.mdx @@ -37,6 +37,12 @@ Then, the component needs a list of `ChatMessage` objects to operate. `ChatMessa You can pass any chat completion parameters valid for the `openai.ChatCompletion.create` method directly to `OpenAIChatGenerator` using the `generation_kwargs` parameter, both at initialization and to `run()` method. For more details on the parameters supported by the OpenAI API, refer to the [OpenAI documentation](https://platform.openai.com/docs/api-reference/chat). +:::tip[Production cost control] + +For production workloads, set a completion token limit such as `generation_kwargs={"max_completion_tokens": 512}`. This bounds response length, latency, and API spend. If you use this generator inside an agent or retrying pipeline, the limit applies to each model call, so choose a value that fits your expected answer size. + +::: + `OpenAIChatGenerator` can support custom deployments of your OpenAI models through the `api_base_url` init parameter. ### Structured Output diff --git a/docs-website/docs/pipeline-components/generators/openaigenerator.mdx b/docs-website/docs/pipeline-components/generators/openaigenerator.mdx index 0c6cf0e7d0d..86c793c0c8f 100644 --- a/docs-website/docs/pipeline-components/generators/openaigenerator.mdx +++ b/docs-website/docs/pipeline-components/generators/openaigenerator.mdx @@ -35,6 +35,12 @@ generator = OpenAIGenerator(api_key=Secret.from_token(""), model=" Then, the component needs a prompt to operate, but you can pass any text generation parameters valid for the `openai.ChatCompletion.create` method directly to this component using the `generation_kwargs` parameter, both at initialization and to `run()` method. For more details on the parameters supported by the OpenAI API, refer to the [OpenAI documentation](https://platform.openai.com/docs/api-reference/chat). +:::tip[Production cost control] + +For production workloads, set a completion token limit such as `generation_kwargs={"max_completion_tokens": 512}`. This bounds response length, latency, and API spend. If you use this generator inside a retrying pipeline, the limit applies to each model call, so choose a value that fits your expected answer size. + +::: + `OpenAIGenerator` supports custom deployments of your OpenAI models through the `api_base_url` init parameter. ### Streaming