You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "heguanhui (via GitHub)" <gi...@apache.org> on 2025/11/27 06:52:29 UTC
[PR] [Fix](cdc) Fix BE data loss caused by invalid checkpoint recovery during Flink CDC Stream Load with BE unavailable scenario [doris-flink-connector]
heguanhui opened a new pull request, #624:
URL: https://github.com/apache/doris-flink-connector/pull/624
[Fix](cdc) Fix BE data loss caused by invalid checkpoint recovery during Flink CDC Stream Load with BE unavailable scenario
## Problem Summary:
Fix data loss on BE after Flink job recovery: When performing Stream Load to BE via Flink CDC direct connection, Flink still succeeds in checkpointing even if BE is unavailable. After reaching the maximum retry count due to BE connection exceptions, the Flink job restarts and recovers from the previous checkpoint (which is actually invalid), leading to data loss on BE.
## Checklist(Required)
1. Does it affect the original behavior: (No)
2. Has unit tests been added: (No)
3. Has document been added or modified: (No)
4. Does it need to update dependencies: (No)
5. Are there any changes that cannot be rolled back: (No)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](cdc) Fix BE data loss caused by invalid checkpoint recovery during Flink CDC Stream Load with BE unavailable scenario [doris-flink-connector]
Posted by "JNSimba (via GitHub)" <gi...@apache.org>.
JNSimba commented on code in PR #624:
URL: https://github.com/apache/doris-flink-connector/pull/624#discussion_r2570440297
##########
flink-doris-connector/src/main/java/org/apache/doris/flink/sink/batch/DorisBatchStreamLoad.java:
##########
@@ -334,6 +334,7 @@ private void waitAsyncLoadFinish() {
BatchRecordBuffer empty = new BatchRecordBuffer();
putRecordToFlushQueue(empty);
}
+ checkFlushException();
Review Comment:
Is any additional checking needed here? It's already checked in `putRecordToFlushQueue`, and also before the `prepareCommit` and `snapshotState` methods.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org