[DISCUSS][STIP-35] Coordinate schema evolution across parallel sink writers

[Doyeon Kim <ek...@gmail.com>]

Hi community,

I would like to start a discussion on STIP-35: *Coordinate schema evolution
across parallel sink writers*.

GitHub issue: https://github.com/apache/seatunnel/issues/11785

The proposal comes from a correctness issue found in the current schema
evolution path when sink parallelism is greater than one.

Currently, SupportSchemaEvolutionSinkWriter.applySchemaChange(...) is
responsible for both applying the external DDL and refreshing the writer's
local schema state. With multiple sink writers, sending the schema change
to only one writer can leave the other writers with stale schemas, while
sending it to every writer may execute the same external DDL multiple times.

STIP-35 proposes separating these two responsibilities:

   - apply or reconcile the external schema change once through a
   designated SchemaChangeApplier;
   - refresh the local schema-dependent state of every related sink writer
   using the complete evolved schema;
   - block new-schema records until all related writers have refreshed;
   - provide an engine-independent SPI that can be coordinated by both Zeta
   and Flink;
   - keep the existing schema-evolution path backward compatible for
   connectors that do not adopt the new capabilities.

The initial implementation is intended to validate the design with JDBC and
BigQuery.

I would appreciate feedback on the proposed direction, especially the
separation between external schema application and writer-local schema
refresh, as well as the coordination model for parallel sink writers.

Best regards,

Doyeon

Re: [DISCUSS][STIP-35] Coordinate schema evolution across parallel sink writers

[David Zollo <da...@gmail.com>]

Good STIP.
It would be even better if there's a detailed design.



Best Regards,
David
Linkedin: https://www.linkedin.com/in/davidzollo



On Wed, Aug 12, 2026 at 1:36 PM Doyeon Kim <ek...@gmail.com> wrote:

> Hi community,
>
> I would like to start a discussion on STIP-35: *Coordinate schema evolution
> across parallel sink writers*.
>
> GitHub issue: https://github.com/apache/seatunnel/issues/11785
>
> The proposal comes from a correctness issue found in the current schema
> evolution path when sink parallelism is greater than one.
>
> Currently, SupportSchemaEvolutionSinkWriter.applySchemaChange(...) is
> responsible for both applying the external DDL and refreshing the writer's
> local schema state. With multiple sink writers, sending the schema change
> to only one writer can leave the other writers with stale schemas, while
> sending it to every writer may execute the same external DDL multiple
> times.
>
> STIP-35 proposes separating these two responsibilities:
>
>    - apply or reconcile the external schema change once through a
>    designated SchemaChangeApplier;
>    - refresh the local schema-dependent state of every related sink writer
>    using the complete evolved schema;
>    - block new-schema records until all related writers have refreshed;
>    - provide an engine-independent SPI that can be coordinated by both Zeta
>    and Flink;
>    - keep the existing schema-evolution path backward compatible for
>    connectors that do not adopt the new capabilities.
>
> The initial implementation is intended to validate the design with JDBC and
> BigQuery.
>
> I would appreciate feedback on the proposed direction, especially the
> separation between external schema application and writer-local schema
> refresh, as well as the coordination model for parallel sink writers.
>
> Best regards,
>
> Doyeon
>