You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "dybyte (via GitHub)" <gi...@apache.org> on 2026/07/22 19:18:17 UTC
[PR] [Fix][API] Honor multi-table failure policy during timer flush [seatunnel]
dybyte opened a new pull request, #11537:
URL: https://github.com/apache/seatunnel/pull/11537
### Purpose of this pull request
Honor `multi_table.failure_policy` when an aggregated timer flush fails for a specific table.
Previously, `MultiTableSinkWriter.aggregatedFlush()` propagated the first sub-writer flush exception immediately. Under `CONTINUE_OTHER_TABLES`, this could fail the sink task and prevent remaining healthy tables from being flushed.
This patch applies the existing table retry and isolation handling to timer flushes. A failed table is quarantined after its retries are exhausted, while healthy tables continue to be flushed. The existing `FAIL_FAST` behavior remains unchanged.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
- Added unit tests verifying:
- retry and table isolation under `CONTINUE_OTHER_TABLES`
- healthy tables continue to be flushed
- quarantined tables are skipped by subsequent timer flushes
- flush exceptions are immediately propagated under `FAIL_FAST`
### Check list
* [ ] If any new Jar binary package adding in your PR, please add License Notice according
[New License Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/developer/new-license.md)
* [ ] If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs
* [ ] If necessary, please update `incompatible-changes.md` to describe the incompatibility caused by this PR.
* [ ] If you are contributing the connector code, please check that the following files are updated:
1. Update [plugin-mapping.properties](https://github.com/apache/seatunnel/blob/dev/plugin-mapping.properties) and add new connector information in it
2. Update the pom file of [seatunnel-dist](https://github.com/apache/seatunnel/blob/dev/seatunnel-dist/pom.xml)
3. Add ci label in [label-scope-conf](https://github.com/apache/seatunnel/blob/dev/.github/workflows/labeler/label-scope-conf.yml)
4. Add e2e testcase in [seatunnel-e2e](https://github.com/apache/seatunnel/tree/dev/seatunnel-e2e/seatunnel-connector-v2-e2e/)
5. Update connector [plugin_config](https://github.com/apache/seatunnel/blob/dev/config/plugin_config)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][API] Honor multi-table failure policy during timer flush [seatunnel]
Posted by "davidzollo (via GitHub)" <gi...@apache.org>.
davidzollo merged PR #11537:
URL: https://github.com/apache/seatunnel/pull/11537
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Fix][API] Honor multi-table failure policy during timer flush [seatunnel]
Posted by "nzw921rx (via GitHub)" <gi...@apache.org>.
nzw921rx commented on code in PR #11537:
URL: https://github.com/apache/seatunnel/pull/11537#discussion_r3635184907
##########
seatunnel-api/src/main/java/org/apache/seatunnel/api/sink/multitablesink/MultiTableSinkWriter.java:
##########
@@ -725,7 +725,33 @@ void aggregatedFlush(Map<SinkIdentifier, SinkContextProxy> proxyContexts) throws
for (SinkIdentifier id : sinkWritersWithIndex.get(i).keySet()) {
SinkContextProxy proxy = proxyContexts.get(id);
if (proxy != null && proxy.getFlushAction() != null) {
- proxy.getFlushAction().run();
+ try {
+ executeWithTableRetry(
+ id.getTableIdentifier(),
+ MultiTableFailurePhase.RUNTIME_WRITE,
Review Comment:
Thank you for your contribution.🚀
1. A new failure type should be introduced here; otherwise, it will be impossible to distinguish a `write` failure from a `flush` failure.
2. I also have one concern. A flush operation handles a batch of records. On the JDBC side, the following logic is used to ensure that records are not inserted repeatedly during retries:
```java
public void addToBatch(SeaTunnelRow record) throws SQLException {
boolean exist = existRow(record);
if (exist) {
if (preExistFlag != null && !preExistFlag) {
insertStatement.executeBatch();
insertStatement.clearBatch();
}
rowConverter.toExternal(
valueTableSchema, databaseTableSchema, record, updateStatement);
updateStatement.addBatch();
} else {
if (preExistFlag != null && preExistFlag) {
updateStatement.executeBatch();
updateStatement.clearBatch();
}
rowConverter.toExternal(
valueTableSchema, databaseTableSchema, record, insertStatement);
insertStatement.addBatch();
}
preExistFlag = exist;
submitted = false;
}
@Override
public void executeBatch() throws SQLException {
if (preExistFlag != null) {
if (preExistFlag) {
updateStatement.executeBatch();
updateStatement.clearBatch();
} else {
insertStatement.executeBatch();
insertStatement.clearBatch();
}
}
submitted = true;
}
```
Do other connectors provide a similar mechanism for handling flush retries? If not, retrying a failed flush may cause unpredictable data issues, such as duplicate writes or inconsistent data.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org