You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by "Weiqing Yang (Jira)" <ji...@apache.org> on 2026/08/03 00:18:00 UTC

[jira] [Created] (FLINK-40298) Opt-in name-based state schema evolution for RowData

Weiqing Yang created FLINK-40298:
------------------------------------

             Summary: Opt-in name-based state schema evolution for RowData
                 Key: FLINK-40298
                 URL: https://issues.apache.org/jira/browse/FLINK-40298
             Project: Flink
          Issue Type: Sub-task
          Components: API / Type Serialization System, Table SQL / Runtime
            Reporter: Weiqing Yang


The core of FLIP-527. Adds:

  * A new option table.exec.state.schema-evolution.enabled (Boolean, default false).
  * Name-based compatibility in RowDataSerializer.RowDataSerializerSnapshot.resolveSchemaCompatibility: when the field layouts differ but the difference is a supported backward-compatible change, it returns compatibleAfterMigration() instead of incompatible(). Fields are matched by the names persisted by FLINK-40120. Added fields must be nullable, removed fields and changed leaf types are rejected, and nested ROW fields are validated by recursion.
  * An override of migrate that remaps a row into the new layout: fields placed by name, added fields null-filled, row kind preserved, nested ROW values remapped recursively.

The option is carried as an in-memory, non-persisted flag on the RowData serializer rather than as anything written into the snapshot. The flag is set only on a serializer that is a state's own value serializer, at the point the state descriptor initializes it. Consequently a RowData serializer that sits below a composite serializer (List or Tuple, as in interval and outer join buffers), or one an operator pre-builds itself, never carries the flag and is rejected on restore rather than migrated. This matters for correctness: migration is not propagated into nested serializers by composite snapshots, so those shapes must fail closed. A test asserts this rejection explicitly.

Snapshots written before field names were persisted have no names and are likewise rejected.

Depends on FLINK-40296.

FLIP-527: https://cwiki.apache.org/confluence/spaces/FLINK/pages/353601981/FLIP-527+State+Schema+Evolution+for+RowData



--
This message was sent by Atlassian Jira
(v8.20.10#820010)