You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "ruanhang1993 (via GitHub)" <gi...@apache.org> on 2023/03/16 08:02:01 UTC
[GitHub] [flink] ruanhang1993 commented on a diff in pull request #21589: [FLINK-25509][connector-base/connector-kafka] Add RecordEvaluator to dynamically stop source based on de-serialized records
ruanhang1993 commented on code in PR #21589:
URL: https://github.com/apache/flink/pull/21589#discussion_r1138257030
##########
flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/connector/kafka/source/reader/KafkaPartitionSplitReader.java:
##########
@@ -153,25 +157,55 @@ public RecordsWithSplitIds<ConsumerRecord<byte[], byte[]>> fetch() throws IOExce
return recordsBySplits;
}
- private void markEmptySplitsAsFinished(KafkaPartitionSplitRecords recordsBySplits) {
+ private void markSplitsAsFinished(KafkaPartitionSplitRecords recordsBySplits) {
+ // Some splits are discovered as empty when handling split additions. These splits should be
+ // added to finished splits to clean up states in split fetcher and source reader.
+ markSplitsAsFinished(emptySplits, recordsBySplits);
+
+ // Some splits are removed when handling split changes. These splits should be
+ // added to finished splits to clean up states in split fetcher and source reader.
+ markSplitsAsFinished(removedSplits, recordsBySplits);
+ }
+
+ private void markSplitsAsFinished(
+ Set<String> splits, KafkaPartitionSplitRecords recordsBySplits) {
// Some splits are discovered as empty when handling split additions. These splits should be
// added to finished splits to clean up states in split fetcher and source reader.
- if (!emptySplits.isEmpty()) {
- recordsBySplits.finishedSplits.addAll(emptySplits);
- emptySplits.clear();
+ if (!splits.isEmpty()) {
+ recordsBySplits.finishedSplits.addAll(splits);
+ splits.clear();
}
}
@Override
public void handleSplitsChanges(SplitsChange<KafkaPartitionSplit> splitsChange) {
- // Get all the partition assignments and stopping offsets.
- if (!(splitsChange instanceof SplitsAddition)) {
+ if (splitsChange instanceof SplitsAddition) {
+ // Get all the partition assignments and stopping offsets.
+ handleSplitsAddition(splitsChange);
+ } else if (splitsChange instanceof SplitsRemoval) {
+ handleSplitsRemoval(splitsChange);
+ } else {
throw new UnsupportedOperationException(
String.format(
"The SplitChange type of %s is not supported.",
splitsChange.getClass()));
}
+ }
+
+ private void handleSplitsRemoval(SplitsChange<KafkaPartitionSplit> splitsRemoval) {
Review Comment:
Judgments have been added to ensure that it is an instance of SplitsRemoval. So I think it is ok.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org