[jira] [Commented] (FLINK-40120) Persist RowData field names in RowDataSerializerSnapshot

["Weiqing Yang (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-40120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18095389#comment-18095389 ] 

Weiqing Yang commented on FLINK-40120:
--------------------------------------

Filed to decouple persisting the RowData field names from FLIP-527's schema-evolution opt-in, per the FLIP-599 [DISCUSS] thread. The intent is a small, standalone change that both the State Catalog and schema evolution can build on: persist the field names in {{RowDataSerializerSnapshot}} (V3 -> V4), written unconditionally, with no compatibility/migration behavior change.

[~gyfora] Before I put up a first cut, a couple of scope questions for the State Catalog side:
 - *Serializer scope:* does the catalog need field names only from {{RowData}} keyed state, or a more general mechanism across other structured serializers too? Names for other serializer types would each be a separate change (each serializer owns its own snapshot format), and most non-RowData structured serializers already carry names (e.g. POJO), so {{RowDataSerializer}} looks like the concrete gap — but I'd like to confirm that matches the catalog's need.
 - *What to persist:* names only, or names plus anything else? Note V4 already carries the field *types* (from V3) alongside the new names, so the snapshot would expose both name and type per field, including nested {{ROW}} recursively. Is that sufficient for the catalog's schema inference?

I can take a first cut on the RowData piece once the above is confirmed, and open a draft PR linked here. FLIP-527's in-flight PR will then rebase onto this so there's a single V3 -> V4 format.

> Persist RowData field names in RowDataSerializerSnapshot
> --------------------------------------------------------
>
>                 Key: FLINK-40120
>                 URL: https://issues.apache.org/jira/browse/FLINK-40120
>             Project: Flink
>          Issue Type: Improvement
>          Components: Table SQL / Runtime
>            Reporter: Weiqing Yang
>            Priority: Major
>
> {\{RowDataSerializerSnapshot}} currently persists only the field *types* of a \{{RowData}} state serializer, not the field *names*. Multiple efforts need the names persisted:
> - FLIP-599 (State Catalog) needs the field names to infer and expose a readable schema for keyed state.
> - FLIP-527 (State Schema Evolution for RowData) needs them for name-based compatibility and migration.
> This issue covers the small, self-contained, shared building block: *persist the RowData field names in the serializer snapshot, written unconditionally and decoupled from any schema-evolution enable/disable logic.*
> *Scope (in):*
> - Add \{{String[] fieldNames}} to \{{RowDataSerializer}} (name-based metadata; a lightweight alternative to carrying the full \{{RowType}}).
> - Bump \{{RowDataSerializerSnapshot}} from V3 to V4, appending the field names after the existing field-types section. Names are written *unconditionally* (not gated on any feature flag).
> - V4 readers remain able to read V3 snapshots (field names absent -> null), so existing savepoints continue to restore.
> - Field names cover nested ROW types recursively.
> *Scope (out - intentionally not here):*
> - No change to \{{resolveSchemaCompatibility}} behavior: with only this change, a field-type-array difference stays \{{incompatible()}} exactly as today. This issue adds no migration/evolution behavior; it only makes the names available.
> - The schema-evolution opt-in and the compatibility/migration logic that consumes the names are owned by FLIP-527 and are out of scope here. Decoupling the "names persisted" metadata from the "evolution permitted" signal is precisely why this is a standalone change.
> - Scoped to \{{RowDataSerializer}}, since its snapshot is the one that persists field types without names. Field names for other serializer types, if the State Catalog needs them, are separate follow-ups (each serializer owns its own snapshot format).
> *Compatibility:*
> - Backward compatible. Snapshot format moves V3 -> V4 by appending names; V4 reads V3 with names absent. Runtime behavior is unchanged for all existing jobs (names are persisted but not yet consumed by any decision).
> Both FLIP-527 and FLIP-599 independently proposed the same V3 -> V4 name-storage bump; landing it once as shared infrastructure lets whichever FLIP lands first establish a single format that serves both, and avoids two conflicting snapshot bumps. Discussed on the FLIP-599 [DISCUSS] thread.
> Links:
> - FLIP-599: State Catalog - https://lists.apache.org/thread/xhojhcr0pkg7j8dblnt2crrdkkjz76vo
> - FLIP-527: State Schema Evolution for RowData - https://cwiki.apache.org/confluence/spaces/FLINK/pages/353601981/FLIP-527+State+Schema+Evolution+for+RowData



--
This message was sent by Atlassian Jira
(v8.20.10#820010)