You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "dybyte (via GitHub)" <gi...@apache.org> on 2026/08/12 16:56:29 UTC
[I] [STIP-35][Feature][API] Coordinate schema evolution across parallel sink writers [seatunnel]
dybyte opened a new issue, #11785:
URL: https://github.com/apache/seatunnel/issues/11785
### Search before asking
- [x] I had searched in the [feature](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22Feature%22) and found no similar feature requirement.
### Description
## Background
SeaTunnel currently uses `SupportSchemaEvolutionSinkWriter.applySchemaChange(...)` for both:
- applying DDL to the external system
- refreshing the sink writer’s local schema state
With parallel sink writers, sending a schema event to only one writer leaves the others with stale schemas. Sending it to every writer may execute the same external DDL multiple times.
## Motivation
Schema evolution should work correctly regardless of sink parallelism.
SeaTunnel needs to apply external schema changes once while ensuring that every related sink writer refreshes its local schema before processing records with the new schema.
## Goals
- Separate external schema application from writer-local refresh.
- Maintain an authoritative evolved schema for each sink table.
- Apply external DDL through one designated applier.
- Refresh every related writer with the complete evolved schema.
- Support both Zeta and Flink through an engine-independent SPI.
- Preserve backward compatibility for existing connectors.
## Non-Goals
- Redesign source-side CDC schema capture.
- Add new schema change event types.
- Require every connector to migrate immediately.
- Guarantee transactional exactly-once DDL.
- Restrict sink parallelism to one.
## Design
Introduce two experimental capabilities:
- `SchemaChangeApplier.apply(SchemaChangeEvent event)` applies or reconciles the external schema change.
- `SupportSchemaRefreshSinkWriter.refreshSchema(CatalogTable evolvedSchema)` rebuilds the writer’s local schema-dependent state.
The engine coordinates the following sequence:
1. Receive a `SchemaChangeEvent`.
2. Drain records using the previous schema.
3. Compute the authoritative evolved `CatalogTable`.
4. Execute the external schema change through one applier.
5. Send the complete evolved schema to every related writer.
6. Wait for all writer refresh acknowledgements.
7. Resume processing records with the new schema.
The complete evolved schema is passed instead of only the latest event. This allows writers with different local states to converge without relying on their schema event history.
External DDL and engine checkpoints are not one transaction. Therefore, the applier must treat an already-applied compatible change as success during recovery.
The initial implementations will cover JDBC and BigQuery.
## Compatibility, Deprecation, and Migration Plan
This proposal is additive.
Existing connectors using `SupportSchemaEvolutionSinkWriter.applySchemaChange(...)` continue using the current path. A connector enters the coordinated path only when it explicitly implements the new applier and writer-refresh capabilities.
No existing configuration, default value, or parallelism behavior changes. The new SPI remains `@Experimental` until it is validated through JDBC and BigQuery on both Zeta and Flink.
## Test Plan
- Verify one external apply and multiple writer refreshes.
- Verify all writers converge to the authoritative schema.
- Verify duplicate apply and refresh operations are safe.
- Verify new-schema records remain blocked until all writers refresh.
- Verify recovery after external apply and partial writer refresh.
- Add parallel sink writer tests for Zeta and Flink.
- Add JDBC and BigQuery integration tests.
## Alternatives Considered
### Send the existing method to every writer
Rejected because every writer may execute the same external DDL.
### Send the schema event to only one writer
Rejected because the remaining writers retain stale local schemas.
### Refresh using only the schema event
Rejected because it assumes every writer received all previous events. Passing the complete authoritative schema provides deterministic convergence.
### Restrict sink parallelism to one
Rejected because it reduces throughput and restricts existing usage.
### Usage Scenario
_No response_
### Related issues
#11752
### Are you willing to submit a PR?
- [x] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [STIP-35][Feature][API] Coordinate schema evolution across parallel sink writers [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on issue #11785:
URL: https://github.com/apache/seatunnel/issues/11785#issuecomment-5433183176
+1 for selecting the #11912 semantic model: preserve parallel sink writers for one physical sink table. We should not make a table-level single writer owner the default.
The distinction matters. #11960 routes data with `keyBy(tableId)` and emits the sink schema command only from the owning gate. For a job writing one table, all of that tables rows therefore reach one sink writer; a higher sink parallelism helps only when there are multiple tables. That is a material throughput/semantics change, and conflicts with STIP-35s explicit non-goal of restricting sink parallelism to one.
The agreed invariant should instead be:
1. Exactly one owner applies/reconciles external DDL for each physical sink/schema coordination key.
2. Every parallel writer that can receive rows for that physical table refreshes its writer-local state from the same complete target schema before it receives a dependent row.
3. The protocol is checkpointed and replayable: monotonic change IDs, pending controls/data, authoritative target schema, and recovery/rescale reconciliation must be durable. DDL remains at-least-once across the external-system/checkpoint boundary and must converge when it is replayed.
So I recommend using #11912s DDL-vs-local-refresh separation as the architectural contract, while carrying over #11960s data-plane dependency tracking and managed-state recovery design. A single DDL owner is necessary; a single data/writer owner for the table is not.
This is a design decision, not an approval of either current PR. Before merging an implementation, require an end-to-end proof for one physical table with sink parallelism > 1 covering: external DDL exactly once per attempt, every writer refresh, control/data reordering, failure after DDL but before the confirming checkpoint, and rescaling. #11912s current split data/control topology must specifically prove that no dependent row can overtake its writers refresh marker.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [STIP-35][Feature][API] Coordinate schema evolution across parallel sink writers [seatunnel]
Posted by "dybyte (via GitHub)" <gi...@apache.org>.
dybyte commented on issue #11785:
URL: https://github.com/apache/seatunnel/issues/11785#issuecomment-5292573723
> Hello, I've noticed the issue you mentioned. How exactly do you plan to ensure compatibility with Flink's current coordination process?
Thanks for pointing this out. I plan to preserve the existing Flink checkpoint-based schema evolution flow rather than replace it. The coordinated path will be opt-in, so existing sinks will continue to use the current behavior.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [STIP-35][Feature][API] Coordinate schema evolution across parallel sink writers [seatunnel]
Posted by "CloverDew (via GitHub)" <gi...@apache.org>.
CloverDew commented on issue #11785:
URL: https://github.com/apache/seatunnel/issues/11785#issuecomment-5275599780
Hello, I've noticed the issue you mentioned. How exactly do you plan to ensure compatibility with Flink's current coordination process?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org