You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Leonard Xu (Jira)" <ji...@apache.org> on 2026/09/24 06:55:00 UTC
[jira] [Updated] (FLINK-40349) Add Fluss sink schema compatibility option to avoid silent data loss
[ https://issues.apache.org/jira/browse/FLINK-40349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Leonard Xu updated FLINK-40349:
-------------------------------
Fix Version/s: cdc-3.7.0
(was: cdc-3.6.0)
> Add Fluss sink schema compatibility option to avoid silent data loss
> --------------------------------------------------------------------
>
> Key: FLINK-40349
> URL: https://issues.apache.org/jira/browse/FLINK-40349
> Project: Flink
> Issue Type: Improvement
> Components: Flink CDC
> Affects Versions: cdc-3.5.0
> Reporter: Hongshun Wang
> Assignee: Hongshun Wang
> Priority: Major
> Fix For: cdc-3.7.0
>
>
>
> h2. {{Motivation}}
> {{Currently, the Fluss sink supports schema mapping between the input schema and the target Fluss table schema. This behavior is useful because Fluss only supports limited schema evolution today, so the upstream and downstream schemas may not always be exactly the same.}}
> {{}}
> {{However, the current default behavior may silently drop data when the upstream input schema contains columns that do not exist in the downstream Fluss table schema. For example, if the upstream schema is wider than the target Fluss table schema, the extra input columns can be ignored during sink-side schema mapping, and users may not notice that data is being lost.
> We should provide an explicit sink option to control how strictly the sink validates schema compatibility.}}
> {{}}
> h2.
> h2. {{Proposal}}
>
> {{Introduce a new Flink sink option, tentatively named: schema-validation-mode.}}
> {{}}
> The option controls the allowed relationship between the input schema and the target Fluss table schema.
> Supported values:
> * {{permissive: }}
> Default value. Keeps the current behavior for backward compatibility.
> The sink may perform schema mapping, projection, and padding based on the target Fluss table schema. This allows existing jobs to continue working as before.
> *
> {{target-superset:}}
> The target Fluss table schema must contain all fields from the input schema by field name.
> This mode allows the target schema to be wider than the input schema, which is useful for schema evolution such as adding nullable columns to the Fluss table. However, it rejects the case where the input schema contains columns that do not exist in the target Fluss table schema, preventing silent data loss.
> *
> {{strict: }}
> The input schema and target Fluss table schema must be strictly identical.
> The validation should check field count, field names, field order, and field types. If any mismatch is found, the sink should fail during initialization with a clear validation error.
> {{{}{}}}{{{}{}}}
> {{{}{}}}{{{}{}}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)