You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@flink.apache.org by we...@apache.org on 2026/10/15 02:23:31 UTC
(flink-agents) branch main updated: [doc] Add a structured output user guide (#1252)
This is an automated email from the ASF dual-hosted git repository.
wenjin272 pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/flink-agents.git
The following commit(s) were added to refs/heads/main by this push:
new 7dcbe9f4 [doc] Add a structured output user guide (#1252)
7dcbe9f4 is described below
commit 7dcbe9f404704bf93ce8b9a13798ee6a1ac9e485
Author: Weiqing Yang <wi...@linkedin.com>
AuthorDate: Wed Oct 14 19:23:26 2026 -0700
[doc] Add a structured output user guide (#1252)
Generated-by: GitHub Copilot CLI 1.0.98 (Claude Sonnet 5.5)
---
.../calling-llms/chat-models/providers.md | 6 +-
docs/content/docs/development/calling-llms/chat.md | 4 +-
.../development/calling-llms/structured-output.md | 325 +++++++++++++++++++++
docs/content/docs/development/react_agent.md | 2 +
4 files changed, 334 insertions(+), 3 deletions(-)
diff --git a/docs/content/docs/development/calling-llms/chat-models/providers.md b/docs/content/docs/development/calling-llms/chat-models/providers.md
index 525a44bd..488b42a0 100644
--- a/docs/content/docs/development/calling-llms/chat-models/providers.md
+++ b/docs/content/docs/development/calling-llms/chat-models/providers.md
@@ -26,6 +26,8 @@ under the License.
## Built-in Providers
+For which providers and models enforce an `output_schema` natively, see [Structured Output]({{< ref "docs/development/calling-llms/structured-output#provider-support" >}}).
+
### Amazon Bedrock
Amazon Bedrock provides access to a wide range of foundation models from leading AI providers through a unified API. The Flink Agents Bedrock integration uses the [Converse API](https://docs.aws.amazon.com/bedrock/latest/userguide/conversation-inference.html), which provides a consistent interface across all supported models with native tool calling support. Authentication is handled via SigV4 using the AWS default credentials chain. No API keys are required.
@@ -337,7 +339,7 @@ The target is `OpenAICompletionsConnection` rather than `AzureOpenAIChatModelCon
|-----------|------|---------|-------------|
| `connection` | str | Required | Reference to connection method name |
| `model` | str | Required | Name of OpenAI model deployment on Azure |
-| `model_of_azure_deployment` | str | None | The underlying model name (e.g., 'gpt-4', 'gpt-35-turbo'). Used for token metrics tracking |
+| `model_of_azure_deployment` | str | None | The underlying model name (e.g., 'gpt-4', 'gpt-35-turbo'). Used for token metrics tracking, and to decide whether an `output_schema` is applied natively (see [Structured Output]({{< ref "docs/development/calling-llms/structured-output#provider-support" >}})) |
| `prompt` | Prompt \| str | None | Prompt template or reference to prompt resource |
| `tools` | List[str] | None | List of tool names available to the model |
| `temperature` | float | None | Sampling temperature (0.0 to 2.0). Not supported by reasoning models |
@@ -353,7 +355,7 @@ The target is `OpenAICompletionsConnection` rather than `AzureOpenAIChatModelCon
|-----------|------|---------|-------------|
| `connection` | String | Required | Reference to connection method name |
| `model` | String | Required | Azure deployment name (not the underlying OpenAI model name) |
-| `model_of_azure_deployment` | String | None | The underlying model name (e.g., 'gpt-4', 'gpt-4o'). Used solely for token metrics tracking |
+| `model_of_azure_deployment` | String | None | The underlying model name (e.g., 'gpt-4', 'gpt-4o'). Used for token metrics tracking, and to decide whether an `output_schema` is applied natively (see [Structured Output]({{< ref "docs/development/calling-llms/structured-output#provider-support" >}})) |
| `prompt` | Prompt \| String | None | Prompt template or reference to prompt resource |
| `tools` | List<String> | None | List of tool names available to the model |
| `temperature` | double | None | Sampling temperature (0.0 to 2.0). Not supported by reasoning models |
diff --git a/docs/content/docs/development/calling-llms/chat.md b/docs/content/docs/development/calling-llms/chat.md
index f747c473..05072ea8 100644
--- a/docs/content/docs/development/calling-llms/chat.md
+++ b/docs/content/docs/development/calling-llms/chat.md
@@ -183,7 +183,8 @@ In the request, `ollama_chat_model` (Python) or `ollamaChatModel` (Java) names t
response: accessing the response of a failed event raises an exception.
If you requested structured output, read the parsed result from
-`event.getStructuredOutput()` (Java) or `event.structured_output` (Python).
+`event.getStructuredOutput()` (Java) or `event.structured_output` (Python). See
+[Structured Output]({{< ref "docs/development/calling-llms/structured-output" >}}) for how to declare a schema and which providers enforce it natively.
Applications that need to recover from a failed chat request should handle the
failed event explicitly. The built-in ReAct agent propagates the failure.
@@ -315,5 +316,6 @@ The built-in `chat_model_action` and `tool_call_action` coordinate this cycle. F
- Use [Prompts]({{< ref "docs/development/calling-llms/prompts" >}}) to reuse message templates across requests.
- Add [Tool Use]({{< ref "docs/development/calling-llms/tool_use" >}}) or [Skills]({{< ref "docs/development/calling-llms/skills" >}}) when the model needs to perform operations beyond generating an answer.
+- Use [Structured Output]({{< ref "docs/development/calling-llms/structured-output" >}}) to get a typed object instead of free text.
- Use [Model Routing]({{< ref "docs/development/calling-llms/model_routing" >}}) to choose among registered models for each request.
- Consult [Built-in Chat Model Providers]({{< ref "docs/development/calling-llms/chat-models/providers" >}}) for provider-specific configuration and media support.
diff --git a/docs/content/docs/development/calling-llms/structured-output.md b/docs/content/docs/development/calling-llms/structured-output.md
new file mode 100644
index 00000000..bce09d9c
--- /dev/null
+++ b/docs/content/docs/development/calling-llms/structured-output.md
@@ -0,0 +1,325 @@
+---
+title: "Structured Output"
+weight: 8
+type: docs
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+## Overview
+
+Structured output makes a chat model return an object that matches a schema you define, instead of free text. A [`ChatResponseEvent`]({{< ref "docs/development/calling-llms/chat#handle-success-and-failure" >}}) carries the parsed object in `getStructuredOutput()` (Java) or `structured_output` (Python).
+
+There are two ways to get the model to follow the schema:
+
+- **Native:** the schema is sent in the provider's own structured-output parameter (for example OpenAI `response_format` or Anthropic `output_config`). The provider constrains the model's output to the schema.
+- **Prompt:** the model is asked in the prompt text to reply in JSON that matches the schema, and the framework parses the reply. This works with every model, but nothing enforces the shape.
+
+You choose a *strategy*, and each provider connection reports whether it can apply the schema natively. The framework combines the two for every request.
+
+## Declare a Schema
+
+A schema is one of:
+
+| Schema | Python | Java |
+|---|---|---|
+| Typed class | Pydantic `BaseModel` subclass | POJO `Class` |
+| Flink row type | `RowTypeInfo` | `RowTypeInfo` |
+
+Only a typed class can be applied natively. A `RowTypeInfo` always uses the prompt path.
+
+Pass the schema to a [ReAct Agent]({{< ref "docs/development/react_agent" >}}) with `output_schema`, or put it on a `ChatRequestEvent` when you [send chat requests yourself]({{< ref "docs/development/calling-llms/chat" >}}).
+
+On a `ChatRequestEvent`:
+
+- **Python:** wrap the schema in `OutputSchema`, as in the example below.
+- **Java:** pass a POJO class directly. Wrap a `RowTypeInfo` with `new OutputSchema(rowTypeInfo)`. Only the ReAct Agent wraps it for you. A bare `RowTypeInfo` on the event fails with `Unsupported output schema`.
+
+{{< tabs "Declare a Schema" >}}
+
+{{< tab "Python" >}}
+```python
+class Review(BaseModel):
+ summary: str
+ sentiment: Sentiment # an Enum, see "Enum Fields" below
+
+# ReAct Agent
+agent = ReActAgent(chat_model=chat_model_descriptor, output_schema=Review)
+
+# Custom action
+ctx.send_event(
+ ChatRequestEvent(
+ model="my_chat_model",
+ messages=[
+ ChatMessage.user(
+ "Review the text below. Reply in JSON with the fields"
+ f" summary and sentiment.\n\n{text}"
+ )
+ ],
+ output_schema=OutputSchema(output_schema=Review),
+ )
+)
+```
+{{< /tab >}}
+
+{{< tab "Java" >}}
+```java
+public class Review {
+ public String summary;
+ public Sentiment sentiment; // an enum, see "Enum Fields" below
+}
+
+// ReAct Agent
+ReActAgent agent = new ReActAgent(chatModelDescriptor, prompt, Review.class);
+
+// Custom action
+ctx.sendEvent(
+ new ChatRequestEvent(
+ "myChatModel",
+ List.of(
+ ChatMessage.user(
+ "Review the text below. Reply in JSON with the fields"
+ + " summary and sentiment.\n\n"
+ + text)),
+ Review.class));
+```
+{{< /tab >}}
+
+{{< /tabs >}}
+
+## Read the Result
+
+A ReAct Agent emits the parsed object as its output. In a custom action, read it from the successful `ChatResponseEvent`. For a typed class, Java returns an instance of the POJO. Python returns the model instance, or its fields as a `dict` once the event has been serialized, so validate it back into the model. For a `RowTypeInfo`, the result is a `Row`.
+
+{{< tabs "Read the Result" >}}
+
+{{< tab "Python" >}}
+```python
+@action(ChatResponseEvent.EVENT_TYPE)
+@staticmethod
+def process_response(event: Event, ctx: RunnerContext) -> None:
+ response = cast("ChatResponseEvent", event)
+ if response.is_failed:
+ raise RuntimeError(response.error)
+ review = Review.model_validate(response.structured_output)
+ ctx.send_event(OutputEvent(output=review.summary))
+```
+{{< /tab >}}
+
+{{< tab "Java" >}}
+```java
+@Action(EventType.ChatResponseEvent)
+public static void processResponse(Event event, RunnerContext ctx) throws Exception {
+ ChatResponseEvent response = ChatResponseEvent.fromEvent(event);
+ if (response.isFailed()) {
+ throw new IllegalStateException(response.getError());
+ }
+ Review review = (Review) response.getStructuredOutput();
+ ctx.sendEvent(new OutputEvent(review.summary));
+}
+```
+{{< /tab >}}
+
+{{< /tabs >}}
+
+## Choose a Strategy
+
+Set `structured_output_strategy` on the chat model **setup** (not the connection). The value is case-insensitive.
+
+| Strategy | Behavior |
+|---|---|
+| `AUTO` (default) | Native when the connection recommends native output for the model (see [Provider Support](#provider-support)). Otherwise the prompt path, silently. This includes a request that cannot carry the schema. |
+| `NATIVE` | Native whenever the request can carry the schema, even for a model that is not on the recommended list. If it cannot (for example a `RowTypeInfo` schema, or a blocker listed in [Provider Support](#provider-support)), it raises `IllegalArgumentException` (Java) or `ValueError` (Python) before any model call. |
+| `PROMPT` | Never native. |
+
+{{< tabs "Choose a Strategy" >}}
+
+{{< tab "Python" >}}
+```python
+ResourceDescriptor(
+ clazz=ResourceName.ChatModel.OPENAI_COMPLETIONS_SETUP,
+ connection="openai_connection",
+ model="gpt-4o-mini",
+ structured_output_strategy="native",
+)
+```
+{{< /tab >}}
+
+{{< tab "Java" >}}
+```java
+ResourceDescriptor.Builder.newBuilder(ResourceName.ChatModel.OPENAI_COMPLETIONS_SETUP)
+ .addInitialArgument("connection", "openaiConnection")
+ .addInitialArgument("model", "gpt-4o-mini")
+ .addInitialArgument("structured_output_strategy", "native")
+ .build();
+```
+{{< /tab >}}
+
+{{< /tabs >}}
+
+{{< hint info >}}
+`AUTO` falling back is silent. If a request you expect to be native is not, check the model name against [Provider Support](#provider-support), or set `NATIVE` to get an error that names the reason.
+{{< /hint >}}
+
+## How Native Output Behaves
+
+The events your actions send and handle are the same on both paths. These points differ:
+
+```text
+Prompt path: ChatRequestEvent → model call (tool loop)
+ → parse JSON answer → ChatResponseEvent
+
+Native path: ChatRequestEvent → model call (tool loop, no schema)
+ → one conversion call (schema, no tools)
+ → parse answer → ChatResponseEvent
+```
+
+- **The schema applies to the final answer only.** Tool-calling turns run without it, so tools work as usual.
+- **Native output adds one conversion call after the tool loop.** The model is asked to convert its answer into the schema's format without adding information. This adds latency and cost to each request. The conversion response is the `response` on the `ChatResponseEvent`.
+- **The prompt path has no conversion call.** The framework does not add the schema to your messages. The ReAct Agent adds its own instruction. In a custom action, ask for JSON that matches the schema in your own prompt. Do this even under `AUTO`, because the request falls back to the prompt path whenever native output does not apply.
+- **The ReAct Agent adjusts its prompt to the path.** On the native path it only asks the model to include the information for every required field, because the conversion call formats it. If it cannot tell whether the call will be native, it keeps the JSON instruction.
+- **A reply that cannot be parsed fails the request.** You get a failed `ChatResponseEvent` after the configured retries. See [Failures and Recovery]({{< ref "docs/development/calling-llms/chat#failures-and-recovery" >}}).
+
+## Provider Support
+
+"Recommended" is what `AUTO` needs to go native. Models outside it still go native under `NATIVE`, unless the last column lists a blocker. A blocked request uses the prompt path under `AUTO` and `PROMPT`, and fails under `NATIVE`.
+
+| Provider | Languages | Recommended models (`AUTO` goes native) | Blocked (besides a non-typed-class schema) |
+|---|---|---|---|
+| OpenAI | Java, Python | `gpt-4o-mini`, `gpt-4o-search-preview`, `gpt-4.1`, `gpt-5` and `o3`, `o4-mini` families (names that start with these), plus `gpt-4o`, `gpt-4o-2024-08-06`, `gpt-4o-2024-11-20`, `o1`, `o1-2024-12-17`. Audio, realtime, TTS and transcribe variants are excluded. | |
+| OpenAI Responses | Java | none | Always. Use the prompt path. |
+| Azure OpenAI | Java, Python | The model named by `model_of_azure_deployment` is one of `gpt-5.1`, `gpt-5.1-chat`, `gpt-5`, `gpt-5-mini`, `gpt-5-nano`, `o3`, `o3-mini`, `o4-mini`, `o1`, `gpt-4o-mini`, `gpt-4.1`, `gpt-4.1-mini`, `gpt-4.1-nano`. | `api_version` older than `2024-08-01`, or not a date |
+| vLLM | Java, Python | Any model | Same as OpenAI |
+| Anthropic | Java, Python | `claude-opus-4-6`, `-4-7`, `-4-8`, `-5`; `claude-sonnet-4-6`, `-5`; `claude-fable-5`; `claude-mythos-5`; `claude-mythos-preview`; and the `claude-opus-4-5`, `claude-sonnet-4-5`, `claude-haiku-4-5` aliases and their dated versions | You pass `output_config` in `additional_kwargs` |
+| Amazon Bedrock | Java | A fixed list of Converse models (a regional or global inference-profile prefix is ignored). ARNs are never recommended. | |
+| Gemini | Java | `gemini-*` text models. Image, TTS, audio, live, transcribe, embedding and omni variants are excluded. | The request binds tools |
+| Ollama | Java, Python | Any model | |
+| watsonx | Java, Python | Any model | |
+| DashScope | Python | `qwen3.7-max`, `qwen3.7-max-preview`, `qwen3.7-max-2026-05-17`, `qwen3.7-max-2026-05-20` | |
+
+Notes:
+
+- **Azure OpenAI** matches `model_of_azure_deployment`, not the deployment name, because a deployment name can be anything. If you do not set it, `AUTO` stays on the prompt path (`NATIVE` still goes native). Set it to get `AUTO` native on Azure, as in the example below.
+- **Gemini** blocks native output when the request binds tools. The conversion call is tool-free, so this only affects direct connection calls. A newer model that does not support the schema field fails with `400 INVALID_ARGUMENT`. Constraints such as `pattern` and `minLength` are ignored by the server.
+- **Ollama** needs a server at v0.5.0 or later. Ollama Cloud and the MLX runner accept the schema but do not enforce it.
+- **Bedrock** rejects recursive schemas, and map values are untyped.
+- **A native parameter you set yourself conflicts.** Do not put `response_format` (OpenAI-compatible providers) or `output_config` (Anthropic) in `additional_kwargs` or model parameters when you use an output schema. Azure OpenAI, watsonx and DashScope raise an error when the schema would overwrite your value. Anthropic treats the request as blocked.
+- **Cross-language setups never go native.** A chat model from the other language, reached through a [cross-language resource]({{< ref "docs/development/calling-llms/chat-models#using-cross-language-providers" >}}), cannot carry a schema across the bridge. `AUTO` and `PROMPT` use the prompt path, and `NATIVE` raises. To use a provider natively, declare the agent in that provider's language.
+
+For example, this Azure OpenAI setup goes native under `AUTO`, because the deployment's model is `gpt-4o-mini`. Without `model_of_azure_deployment`, the same setup would use the prompt path.
+
+{{< tabs "Azure OpenAI Example" >}}
+
+{{< tab "Python" >}}
+```python
+@chat_model_setup
+@staticmethod
+def azure_openai_chat_model() -> ResourceDescriptor:
+ return ResourceDescriptor(
+ clazz=ResourceName.ChatModel.AZURE_OPENAI_SETUP,
+ connection="azure_openai_connection",
+ model="my-deployment", # Your Azure deployment name
+ model_of_azure_deployment="gpt-4o-mini", # The model behind it
+ )
+```
+{{< /tab >}}
+
+{{< tab "Java" >}}
+```java
+@ChatModelSetup
+public static ResourceDescriptor azureOpenAIChatModel() {
+ return ResourceDescriptor.Builder.newBuilder(ResourceName.ChatModel.AZURE_OPENAI_SETUP)
+ .addInitialArgument("connection", "azureOpenAIConnection")
+ .addInitialArgument("model", "my-deployment") // Your Azure deployment name
+ .addInitialArgument("model_of_azure_deployment", "gpt-4o-mini") // The model behind it
+ .build();
+}
+```
+{{< /tab >}}
+
+{{< /tabs >}}
+
+The connection's `api_version` must be `2024-08-01` or later, for example `2024-10-21`.
+
+## Schema Rules
+
+### Enum Fields
+
+List the values the model should return, in the form the model reads back:
+
+- **Python:** a Pydantic `Enum` lists its values. A string enum gives `"type": "string"` and a numeric enum gives `"type": "integer"`.
+- **Java:** the schema lists the values Jackson deserializes. Put `@JsonProperty` on **every** constant, or use a `@JsonValue` method that returns a `String`.
+
+{{< tabs "Enum Fields" >}}
+
+{{< tab "Python" >}}
+```python
+class Sentiment(str, Enum):
+ POSITIVE = "positive"
+ NEGATIVE = "negative"
+```
+{{< /tab >}}
+
+{{< tab "Java" >}}
+```java
+public enum Sentiment {
+ @JsonProperty("positive") POSITIVE,
+ @JsonProperty("negative") NEGATIVE
+}
+
+// or
+public enum Sentiment {
+ POSITIVE("positive"),
+ NEGATIVE("negative");
+
+ private final String label;
+
+ Sentiment(String label) {
+ this.label = label;
+ }
+
+ @JsonValue
+ public String label() {
+ return label;
+ }
+}
+```
+{{< /tab >}}
+
+{{< /tabs >}}
+
+Two Java cases do not work as you might expect. They behave the same on every Java provider:
+
+- `@JsonProperty` on only **some** constants lists the Java names for **all** of them (`["POSITIVE","NEGATIVE"]`). The model may reply with a Java name where you annotated a different value, and parsing fails. Annotate every constant or none.
+- A `@JsonValue` that returns a number produces `{"type":"string","enum":[1,-1]}`, which contradicts itself. Use a string value instead.
+
+### Java POJOs
+
+- **Descriptions and required fields.** `@JsonPropertyDescription` and `@JsonClassDescription` are sent in the native schema, and every field is required (an `Optional` field is required but nullable on OpenAI, Azure OpenAI and Anthropic). The ReAct Agent's *prompt text* is rendered by a different generator, which has no descriptions, and lists a field as required only when it has `@JsonProperty(required = true)`. The prompt path therefore describes the schema less fully than the native path.
+- **`Map` fields.** On OpenAI, Azure OpenAI, Anthropic and vLLM, a `Map<String, V>` field becomes an empty object schema that allows no keys, so the model can only return `{}`. Avoid `Map` fields with these providers, or use a nested class. Gemini, Ollama and watsonx type the map values.
+- **Recursive types.** The ReAct Agent rejects a self-referential POJO at construction time. Bedrock also rejects a recursive schema.
+
+### Python Models
+
+The schema comes from Pydantic's `model_json_schema()`, so field descriptions and defaults are included. A model that Pydantic cannot express raises a `TypeError` naming the schema when it is rendered.
+
+## Next Steps
+
+- See [Built-in Chat Model Providers]({{< ref "docs/development/calling-llms/chat-models/providers" >}}) for how to declare each connection and setup.
+- Use the [ReAct Agent]({{< ref "docs/development/react_agent" >}}) to get structured output from an agent with tools.
diff --git a/docs/content/docs/development/react_agent.md b/docs/content/docs/development/react_agent.md
index 430ffc66..19dc49bc 100644
--- a/docs/content/docs/development/react_agent.md
+++ b/docs/content/docs/development/react_agent.md
@@ -257,6 +257,8 @@ User can set output schema to configure the ReAct Agent output type. If output s
The output schema should be a `BaseModel` subclass (Python) / a Pojo class (Java), or a `RowTypeInfo` (both).
+When the chat model's provider supports it, the schema is enforced natively by the provider rather than only described in the prompt. See [Structured Output]({{< ref "docs/development/calling-llms/structured-output" >}}) for the strategies, supported providers and schema rules.
+
{{< tabs "Output Schema" >}}
{{< tab "Python" >}}