You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Zhuoxi2000 (via GitHub)" <gi...@apache.org> on 2026/08/19 15:06:04 UTC

[D] [Discuss][API] Multimodal content parts in ChatMessage [flink-agents]

GitHub user Zhuoxi2000 created a discussion: [Discuss][API] Multimodal content parts in ChatMessage

Follow-up to the 0.4 planning discussion in #862. I volunteered for the “Support invoking multimodal models” Must item there, and @wenjin272 confirmed no one is working on it yet.

The proposal is to add first-class typed content parts to `ChatMessage` in both Java and Python, while keeping plain-string content fully backward compatible.

## Motivation

Today `ChatMessage.content` is a plain string, even though the chat-model SDKs we already integrate with support multimodal input natively.

There are also two concrete problems in the current code:

1. We already have provider-specific workarounds. For example, Python Anthropic carries native content blocks through `extra_args["anthropic_content_blocks"]`. This works, but it is untyped and provider-specific.

2. MCP tool results can already contain images, but `ChatModelAction` eventually stringifies the result before sending it back to the model. So an image currently becomes the string representation of a map containing base64 data rather than actual image content.

A first-class content-part API would give us one framework-level representation for these cases.

## Proposed API

Add a flat `ContentPart` type in both languages with three initial variants:

* `text`
* `image`
* `document`

Each part would carry only the fields it needs, e.g. `text`, `mime_type`, and either `data` or `url`.

I’d keep it flat rather than introducing a nested provider-style `source` object. Besides being simpler across providers, this also works better with the current Event Log depth limits.

`ChatMessage` would keep its existing `content: String` and gain `contentParts` / `content_parts`, defaulting to empty.

The compatibility rule would be:

* no parts → `content` behaves exactly as it does today
* parts present → parts are authoritative, while `content` remains a plain-text projection of the text parts for existing consumers

I considered making `content` itself a `String | List<ContentPart>` union, but that would be much more disruptive across Java callers, Jackson mapping, and the pemja bridge. An additive field seems safer for 0.4.

This also requires one small framework fix: image-only messages must not be dropped by the current “empty text content” filtering in the prompt/message merge logic.

## Cross-language / serialization

`content_parts` would be additive in the serialized Event Log format, with old records continuing to deserialize normally.

The pemja bridge needs explicit support because those conversions copy fields directly rather than going through JSON.

I’d leave multimodal YAML prompts out of scope for 0.4. Existing text YAML stays unchanged, and YAML `parts:` can be added separately once the API settles.

## Event Log and state

At STANDARD Event Log level, inline base64 is already bounded by the existing string truncation limit, so this does not make logs unbounded by default.

There is still a state-size concern for long agent/tool loops, since inline media may be carried through sensory memory. For 0.4, my preference would be to recommend URL-backed parts for large media and leave a first-class blob/reference type for follow-up work.

## Proposed 0.4 scope

**Phase 1 — framework**

* `ContentPart` + `ChatMessage` support in Java and Python
* Event Log / cross-language serialization
* pemja bridge
* image-only message filtering fix
* round-trip and snapshot tests

**Phase 2 — initial providers**

Start with OpenAI Chat Completions and Ollama.

OpenAI is a good first target because the shared conversion path also covers Azure OpenAI and vLLM. Ollama already has direct image support in both SDKs.

I’d start with USER-role multimodal input, which covers the primary invocation use case.

**Follow-ups**

Anthropic, Gemini, Bedrock, Responses API, Tongyi, MCP image tool results, YAML parts, and richer media-reference support can follow incrementally.

## Open questions

1. Should Event Log store a truncated base64 prefix, or replace media payloads with metadata such as `<image/png, 123 KB>`? In particular, should VERBOSE ever record full media bytes?
2. Is recommending URL-backed media enough for 0.4, or should we introduce a reference/blob type now?
3. For existing `extra_args` escape hatches, my inclination is to gradually subsume provider-specific ones like `anthropic_content_blocks`, while leaving generic `extra_args` available for other provider-specific options.
4. Naming: `content_parts` or simply `parts`?

If this direction looks reasonable, I’ll open a tracking issue with Phase 1 / Phase 2 subtasks and start with the framework work.


GitHub link: https://github.com/apache/flink-agents/discussions/1031

----
This is an automatically sent email for issues@flink.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@flink.apache.org


Re: [D] [Discuss][API] Multimodal content parts in ChatMessage [flink-agents]

Posted by "wenjin272 (via GitHub)" <gi...@apache.org>.
GitHub user wenjin272 added a comment to the discussion: [Discuss][API] Multimodal content parts in ChatMessage

Here are my thoughts on the four open questions.

### 1. Event Log representation

I would prefer recording media metadata rather than a truncated Base64 prefix. A Base64 prefix has little diagnostic value, but it can still expose part of the original content.

For example, the Event Log could contain:

```json
{
  "type": "image",
  "mime_type": "image/png",
  "source_type": "inline",
  "size_bytes": 126432,
  "sha256": "...",
  "data": "<omitted>"
}
```

For URL-backed content, we could record a sanitized URI or reference identifier, avoiding signed query parameters where appropriate.

I also do not think `VERBOSE` should automatically record complete media bytes. Binary content can be large and sensitive, and an Event Log should not implicitly become media storage. If preserving the complete payload is needed in the future, it would be better to introduce a separate, explicitly enabled artifact-capture mechanism.

### 2. URL-backed content versus a Blob/Reference type

For 0.4, I think supporting inline data and externally managed URLs is sufficient. I would not introduce a Flink Agents-managed Blob Storage service as part of this change, since that would also require decisions about storage backends, credentials, cleanup, lifetime, and recovery.

However, the data model should leave room for adding a managed reference later. For example, an image block could initially allow mutually exclusive `data` and `url` fields, together with `mime_type` and optional metadata such as size, checksum, and name. A `blob_id` or another reference form could then be added later without redesigning the content-block hierarchy.

The documentation should also make it clear that URL-backed content is externally managed. The user is responsible for its accessibility and lifetime. URLs may expire, may not be accessible to the model provider, and may no longer be valid after recovery from a checkpoint. Therefore, recommending URLs reduces state size, but it does not provide durable media storage.

### 3. Existing `extra_args` escape hatches

I agree with gradually replacing provider-specific content workarounds such as `anthropic_content_blocks` once the standard content-block API can represent the same information. Multimodal message content should have one canonical representation and should not be duplicated in both content blocks and `extra_args`.

Generic extensibility is still useful, so I would not remove `extra_args` entirely. However, its current name and responsibilities are somewhat ambiguous: it contains message metadata, framework protocol fields, provider response information, and provider-specific extensions. If its intended long-term role is auxiliary message metadata, `metadata` may be a clearer name, while provider-specific request fields could use a separate mechanism.

Some stable fields currently stored there, such as the tool-call ID and token usage, may also deserve dedicated typed fields. Usage may even belong to a response-level object rather than `ChatMessage`. This is orthogonal to the multimodal proposal and does not need to block it, but it may be worth tracking separately.

### 4. Naming

Since Flink Agents is still in Beta and backward compatibility is not required for this API, I would prefer replacing `content: String` rather than adding a second authoritative representation.

My preferred shape would be:

```text
ChatMessage
  - role
  - blocks: List<ContentBlock>
  - toolCalls
  - metadata
```

A text-only message would simply contain one `TextBlock`. Convenience factory methods can preserve the simple text-message experience without storing a separate string field.

Therefore, I would prefer `blocks` with `ContentBlock` over either `content_parts` or `parts`. If the existing string field must be retained, then `content_parts` is clearer than the more generic `parts`, but I think a single canonical `blocks` field would produce a cleaner API.

GitHub link: https://github.com/apache/flink-agents/discussions/1031#discussioncomment-18156757

----
This is an automatically sent email for issues@flink.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@flink.apache.org


Re: [D] [Discuss][API] Multimodal content parts in ChatMessage [flink-agents]

Posted by "Zhuoxi2000 (via GitHub)" <gi...@apache.org>.
GitHub user Zhuoxi2000 added a comment to the discussion: [Discuss][API] Multimodal content parts in ChatMessage

Thanks for the review! Opened #1059  to track the work with per-phase subtasks,starting with Phase 1.


GitHub link: https://github.com/apache/flink-agents/discussions/1031#discussioncomment-18183571

----
This is an automatically sent email for issues@flink.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@flink.apache.org


Re: [D] [Discuss][API] Multimodal content parts in ChatMessage [flink-agents]

Posted by "Zhuoxi2000 (via GitHub)" <gi...@apache.org>.
GitHub user Zhuoxi2000 added a comment to the discussion: [Discuss][API] Multimodal content parts in ChatMessage

Agreed on all four. I’ve folded these into the proposal above.

1. Event Log: I’ll use the metadata only shape you suggested for every log level, including VERBOSE. Media payloads won’t be written to the Event Log. URL query strings will also be stripped so signed parameters don’t end up in logs.

2. URL vs managed references: for 0.4, I’ll keep this to inline base64 and externally managed URLs. No managed blob storage yet. The model will leave room for something like `blob_id` later without changing the overall design. The docs will also make it clear that URLs are external references, not durable storage.

3. `extra_args`: agreed that blocks should become the canonical representation of message content. `anthropic_content_blocks` can be deprecated once Anthropic moves to the new block model. I’ll leave `extra_args` itself unchanged here and keep the broader metadata cleanup in #1056.

4. Naming: agreed on `blocks: List<ContentBlock>` as the single source of truth, with no separate string content field.


GitHub link: https://github.com/apache/flink-agents/discussions/1031#discussioncomment-18179004

----
This is an automatically sent email for issues@flink.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@flink.apache.org


Re: [D] [Discuss][API] Multimodal content parts in ChatMessage [flink-agents]

Posted by "wenjin272 (via GitHub)" <gi...@apache.org>.
GitHub user wenjin272 added a comment to the discussion: [Discuss][API] Multimodal content parts in ChatMessage

Thanks for proposing this. I agree that supporting multimodal messages is necessary for Flink Agents.

At the moment, however, I feel the proposal is still too high-level to evaluate as an API change. It would be helpful to extend it with two concrete parts:

**1. The public data model**

Please provide the proposed Java and Python structures, including their serialized representation. For example, one possible shape could be:

```text
ChatMessage
  - role
  - blocks: List<ContentBlock>

ContentBlock
  - TextBlock
  - ImageBlock
  - AudioBlock
  - VideoBlock
  - DocumentBlock / FileBlock
```

The proposal should also define:

- How binary content is represented, such as URL, URI, bytes, or Base64.
- How block types are identified during serialization and cross-language communication.
- Whether mixed blocks preserve their original order.
- Whether tool results can contain the same content block types.

Since Flink Agents is currently in Beta and we do not need to preserve this API for compatibility, I would prefer replacing `content: String` with the block list instead of keeping both. Having two representations would introduce two sources of truth. Convenience APIs can still be provided for creating text-only messages or extracting their text content.

**2. Integration with model providers**

The proposal should explain how these blocks flow into a model provider, for example:

```text
ChatMessage blocks
    -> ChatModelConnection/provider adapter
    -> provider-native request content
```

Each provider uses a different representation for multimodal input. Therefore, it would be useful to clarify:

- Whether each `ChatModelConnection` implementation is responsible for converting `ContentBlock`s into its provider-native format.
- How unsupported block types are handled: explicit validation/error, capability declaration, or fallback conversion.
- Whether provider responses and tool results can also return multimodal blocks.
- How this works consistently across the Java and Python implementations.

Frameworks such as LlamaIndex and AgentScope demonstrate different valid approaches: LlamaIndex exposes modality-specific block types, while AgentScope uses a more generic `DataBlock` routed by MIME type. I do not think Flink Agents must copy either design, but the proposal should make an explicit choice and provide at least one end-to-end example, such as a text-and-image user message being converted into an OpenAI, Anthropic, or Gemini request.

GitHub link: https://github.com/apache/flink-agents/discussions/1031#discussioncomment-18156558

----
This is an automatically sent email for issues@flink.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@flink.apache.org


Re: [D] [Discuss][API] Multimodal content parts in ChatMessage [flink-agents]

Posted by "wenjin272 (via GitHub)" <gi...@apache.org>.
GitHub user wenjin272 added a comment to the discussion: [Discuss][API] Multimodal content parts in ChatMessage

Thanks for the detailed follow-up. This addresses my concerns. The proposed block model, provider conversion boundary, Event Log handling, and sequencing with #1056 all look reasonable to me.

I don’t have any further comments on the design at this point.

GitHub link: https://github.com/apache/flink-agents/discussions/1031#discussioncomment-18183159

----
This is an automatically sent email for issues@flink.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@flink.apache.org


Re: [D] [Discuss][API] Multimodal content parts in ChatMessage [flink-agents]

Posted by "Zhuoxi2000 (via GitHub)" <gi...@apache.org>.
GitHub user Zhuoxi2000 added a comment to the discussion: [Discuss][API] Multimodal content parts in ChatMessage

Thanks for the detailed review. Agreed on both points. I also think replacing `content: String` directly is cleaner than keeping two representations. Since the API is still Beta, this feels like the right time to make that change.

## 1. Public data model

I’d go with modality specific blocks as the public API:

`TextBlock`, `ImageBlock`, `AudioBlock`, `VideoBlock`, and `DocumentBlock`.

Media blocks would share the same basic fields: `mime_type`, either `data` or `url`, plus optional metadata such as `name`, `size_bytes`, and `sha256`.

`ChatMessage` would then use an ordered `List<ContentBlock>` / `List[ContentBlock]` as the source of truth.

Existing helpers like `user(String)` would keep working by creating a single `TextBlock`, and `getText()` / `.text` would provide the concatenated text projection for existing text consumers.

Java and Python would use the same serialized shape with a `type` discriminator:

```json
{
  "role": "user",
  "blocks": [
    {"type": "text", "text": "What's in this picture?"},
    {"type": "image", "mime_type": "image/png", "data": "<base64>"}
  ]
}
```

A few choices I’d make explicit:

* Media uses base64 `data` or `url`, exactly one, with `mime_type` required.
* Block order is preserved through serialization and provider conversion.
* TOOL messages use the same block model. This also gives us a clean path to fix MCP image results, which are currently stringified before reaching the model.
* No dual representation. `content` goes away and `blocks` becomes the source of truth.

The cross language snapshots, pemja bridge, and existing call sites would all be migrated together.

## 2. Provider integration

The flow would stay simple:

`ChatMessage.blocks -> provider converter -> provider native content`

Each `ChatModelConnection` handles the conversion in its existing message conversion path.

For example, OpenAI maps image blocks to `image_url`, Anthropic maps them to image source blocks, and Ollama maps them to its `images` field.

If a provider does not support a block type, I’d fail fast with a clear validation error instead of silently dropping or converting it. A capability API can come later if we want preflight checks.

The same model also works for responses and tool results, so we do not need another representation later when providers start returning multimodal content.

Java and Python would follow the same block model and wire contract, with mirrored converter tests for providers supported in both runtimes.

GitHub link: https://github.com/apache/flink-agents/discussions/1031#discussioncomment-18178994

----
This is an automatically sent email for issues@flink.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@flink.apache.org


Re: [D] [Discuss][API] Multimodal content parts in ChatMessage [flink-agents]

Posted by "Zhuoxi2000 (via GitHub)" <gi...@apache.org>.
GitHub user Zhuoxi2000 added a comment to the discussion: [Discuss][API] Multimodal content parts in ChatMessage

The sequencing makes sense to me. Multimodal needs to land for 0.4 first, and that should leave #1056 with a cleaner surface to work from.

A few boundaries I’d keep clear:

1. The block model lands once with multimodal. #1056 can then handle typed `tool_calls`, the `extra_args` → `metadata` rename, and moving response-level fields like usage/model/id without reshaping `blocks`. I’ll leave `tool_calls` and `extra_args` untouched in the multimodal change.

2. Blocks will use one serialization contract everywhere: Jackson/Pydantic, event attributes, and pemja, pinned by the cross-language snapshots. No new side path.

3. The existing Java/Python differences around `externalId` and `extra_args` stay in #1056. The multimodal converters won’t depend on those behaviors.

Once multimodal lands, I’d also be happy to pick up part of #1056, especially typed `tool_calls` or the serialization cleanup.


GitHub link: https://github.com/apache/flink-agents/discussions/1031#discussioncomment-18179011

----
This is an automatically sent email for issues@flink.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@flink.apache.org


Re: [D] [Discuss][API] Multimodal content parts in ChatMessage [flink-agents]

Posted by "wenjin272 (via GitHub)" <gi...@apache.org>.
GitHub user wenjin272 added a comment to the discussion: [Discuss][API] Multimodal content parts in ChatMessage

As part of the 0.4 API refinement, we have also decided to review and refactor `ChatMessage` and its related end-to-end paths, tracked in #1056.

This overlaps with the multimodal work in areas such as the message data model, Java/Python serialization, pemja, Event Log, and provider conversion. To keep the scope manageable, we plan to complete the multimodal support first and then address the remaining broader concerns in #1056.

What do you think about this sequencing? Are there any parts of the broader refactoring that you think should be addressed together with the multimodal work to avoid rework?

GitHub link: https://github.com/apache/flink-agents/discussions/1031#discussioncomment-18173543

----
This is an automatically sent email for issues@flink.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@flink.apache.org


Re: [D] [Discuss][API] Multimodal content parts in ChatMessage [flink-agents]

Posted by "wenjin272 (via GitHub)" <gi...@apache.org>.
GitHub user wenjin272 added a comment to the discussion: [Discuss][API] Multimodal content parts in ChatMessage

Thanks for bringing this up. I’m tied up with other things right now, but I’ll try to check it out this week.

GitHub link: https://github.com/apache/flink-agents/discussions/1031#discussioncomment-18142459

----
This is an automatically sent email for issues@flink.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@flink.apache.org