You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Gavin McDonald (Jira)" <ji...@apache.org> on 2026/09/05 16:10:00 UTC
[jira] [Comment Edited] (FLINK-40120) Persist RowData field names in RowDataSerializerSnapshot
[ https://issues.apache.org/jira/browse/FLINK-40120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18095938#comment-18095938 ]
Gavin McDonald edited comment on FLINK-40120 at 9/5/26 4:09 PM:
----------------------------------------------------------------
Thank you [~weiqing] , looks good. We can expose the getters as the first reliant feature goes in.
was (Author: gyfora):
Thank you [~WeiqingYang] , looks good. We can expose the getters as the first reliant feature goes in.
> Persist RowData field names in RowDataSerializerSnapshot
> --------------------------------------------------------
>
> Key: FLINK-40120
> URL: https://issues.apache.org/jira/browse/FLINK-40120
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / Runtime
> Reporter: Weiqing Yang
> Assignee: Weiqing Yang
> Priority: Major
> Labels: pull-request-available
> Fix For: 2.4.0
>
>
> `RowDataSerializerSnapshot` currently persists only the field *types* of a `RowData` state serializer, not the field *names*. Multiple efforts need the names persisted:
> - FLIP-599 (State Catalog) needs the field names to infer and expose a readable schema for keyed state.
> - FLIP-527 (State Schema Evolution for RowData) needs them for name-based compatibility and migration.
> This issue covers the small, self-contained, shared building block: **persist the RowData field names in the serializer snapshot, written unconditionally and decoupled from any schema-evolution enable/disable logic.**
> Scope (in):
> - Add `String[] fieldNames` to `RowDataSerializer` (name-based metadata; a lightweight alternative to carrying the full `RowType`).
> - Bump `RowDataSerializerSnapshot` from V3 to V4, appending the field names after the existing field-types section. Names are written **unconditionally** (not gated on any feature flag).
> - V4 readers remain able to read V3 snapshots (field names absent -> `null`), so existing savepoints continue to restore.
> - Only the top-level `RowType` field names are persisted (a flat `String[]`). Nested `ROW` field names already survive today via the persisted field logical types, so no recursion is needed.
> Scope (out — intentionally not here):
> - No change to `resolveSchemaCompatibility` behavior: with only this change, a field-type-array difference stays `incompatible()` exactly as today. This issue adds no migration/evolution behavior; it only makes the names available.
> - The schema-evolution opt-in and the compatibility/migration logic that consumes the names are owned by FLIP-527 and are out of scope here. Decoupling the "names persisted" metadata from the "evolution permitted" signal is precisely why this is a standalone change.
> - Scoped to `RowDataSerializer`, since its snapshot is the one that persists field types without names. Field names for other serializer types, if the State Catalog needs them, are separate follow-ups (each serializer owns its own snapshot format).
> Compatibility:
> - Backward compatible. Snapshot format moves V3 -> V4 by appending names; V4 reads V3 with names absent. Runtime behavior is unchanged for all existing jobs (names are persisted but not yet consumed by any decision).
> Motivation for splitting this out: both FLIP-527 and FLIP-599 independently proposed the same V3 -> V4 name-storage bump; landing it once as shared infrastructure lets whichever FLIP lands first establish a single format that serves the catalog's schema-inference needs and schema evolution's needs, and avoids two conflicting snapshot bumps. Discussed on the FLIP-599 [DISCUSS] thread.
> Links:
> - FLIP-599: State Catalog — https://lists.apache.org/thread/xhojhcr0pkg7j8dblnt2crrdkkjz76vo
> - FLIP-527: State Schema Evolution for RowData — https://cwiki.apache.org/confluence/spaces/FLINK/pages/353601981/FLIP-527+State+Schema+Evolution+for+RowData
--
This message was sent by Atlassian Jira
(v8.20.10#820010)