You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "xiaochen-zhou (via GitHub)" <gi...@apache.org> on 2025/07/21 05:31:42 UTC

[PR] Kafka partition stop consumer [seatunnel]

xiaochen-zhou opened a new pull request, #9597:
URL: https://github.com/apache/seatunnel/pull/9597

   ### Purpose of this pull request
   
   Fix indefinite blocking by adjusting stopping condition in split reader
   refer: [https://github.com/apache/flink-connector-kafka/pull/100](https://github.com/apache/flink-connector-kafka/pull/100)
   
   ### Does this PR introduce _any_ user-facing change?
   
   no
   
   ### How was this patch tested?
   
   Add tests KafkaTransactionalStoppingConditionTest#testStoppingConditionWithControlMessages()
    
   Consider partition 0, where offsets 13 and 14 are valid records and 15 is a control record. If stoppingOffset is set to 15 for partition 0and properties.max.poll.records is configured to 1, checking only partitions that return records would miss offset 15. By consistently reviewing all assigned partitions, the consumer’s position jumps control record in the subsequent poll, allowing the system to escape.
   
   
   
   
   ### Check list
   
   * [ ] If any new Jar binary package adding in your PR, please add License Notice according
     [New License Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/contribution/new-license.md)
   * [ ] If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs
   * [ ] If you are contributing the connector code, please check that the following files are updated:
     1. Update [plugin-mapping.properties](https://github.com/apache/seatunnel/blob/dev/plugin-mapping.properties) and add new connector information in it
     2. Update the pom file of [seatunnel-dist](https://github.com/apache/seatunnel/blob/dev/seatunnel-dist/pom.xml)
     3. Add ci label in [label-scope-conf](https://github.com/apache/seatunnel/blob/dev/.github/workflows/labeler/label-scope-conf.yml)
     4. Add e2e testcase in [seatunnel-e2e](https://github.com/apache/seatunnel/tree/dev/seatunnel-e2e/seatunnel-connector-v2-e2e/)
     5. Update connector [plugin_config](https://github.com/apache/seatunnel/blob/dev/config/plugin_config)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix][Connector-V2] Fix indefinite blocking by adjusting stopping condition in kafka split reader [seatunnel]

Posted by "xiaochen-zhou (via GitHub)" <gi...@apache.org>.
xiaochen-zhou commented on code in PR #9597:
URL: https://github.com/apache/seatunnel/pull/9597#discussion_r2221208518


##########
seatunnel-connectors-v2/connector-kafka/src/test/java/org/apache/seatunnel/connectors/seatunnel/kafka/source/KafkaTransactionalStoppingConditionTest.java:
##########
@@ -0,0 +1,146 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.seatunnel.connectors.seatunnel.kafka.source;
+
+import org.apache.seatunnel.api.source.SourceReader;
+import org.apache.seatunnel.connectors.seatunnel.kafka.config.StartMode;
+
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.clients.consumer.ConsumerRecords;
+import org.apache.kafka.clients.consumer.MockConsumer;
+import org.apache.kafka.clients.consumer.OffsetResetStrategy;
+import org.apache.kafka.common.TopicPartition;
+
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.DisplayName;
+import org.junit.jupiter.api.Test;
+import org.mockito.Mock;
+import org.mockito.MockitoAnnotations;
+
+import java.time.Duration;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Properties;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+import static org.mockito.Mockito.when;
+
+/** Test class demonstrating the fix for transactional message stopping condition. */
+public class KafkaTransactionalStoppingConditionTest {
+
+    private static final String TOPIC = "test-topic";
+    private static final int PARTITION = 0;
+    private static final TopicPartition TOPIC_PARTITION = new TopicPartition(TOPIC, PARTITION);
+
+    private MockConsumer<byte[], byte[]> mockConsumer;
+
+    @Mock private SourceReader.Context readerContext;
+
+    @Mock private KafkaSourceConfig kafkaSourceConfig;
+
+    @BeforeEach
+    void setUp() {
+        MockitoAnnotations.openMocks(this);
+
+        mockConsumer = new MockConsumer<>(OffsetResetStrategy.EARLIEST);
+        // Setup mock config
+        Properties properties = new Properties();
+        properties.setProperty(ConsumerConfig.GROUP_ID_CONFIG, "test-group");
+        properties.setProperty(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
+
+        when(kafkaSourceConfig.getProperties()).thenReturn(properties);
+        when(kafkaSourceConfig.getConsumerGroup()).thenReturn("test-group");
+        when(kafkaSourceConfig.getBootstrap()).thenReturn("localhost:9092");
+        when(kafkaSourceConfig.isCommitOnCheckpoint()).thenReturn(false);
+        when(kafkaSourceConfig.getPollTimeout()).thenReturn(1000L);
+
+        Map<String, ConsumerMetadata> mapMetadata = new HashMap<>();
+        ConsumerMetadata metadata = new ConsumerMetadata();
+        metadata.setTopic(TOPIC);
+        metadata.setStartMode(StartMode.EARLIEST);
+        mapMetadata.put(TOPIC, metadata);
+    }
+
+    /**
+     * Test the core issue: control messages can cause infinite blocking when using last record
+     * offset as stopping condition.
+     *
+     * <p>Note: MockConsumer cannot directly simulate control messages since they are invisible to
+     * consumers. Instead, we simulate the EFFECT of control messages: consumer position advancing
+     * beyond the last visible record.
+     */
+    @Test
+    @DisplayName("Test stopping condition with control messages - demonstrates the fix")
+    void testStoppingConditionWithControlMessages() {

Review Comment:
   
   
   
   > This test only verifies the consumer behavior of Kafka, but does not ensure that our connector behaves normally through the test case. Please modify the test case to verify that KafkaPartitionSplitReader::fetch will end normally.
   
   I tried it out, and I think it might be need to write an E2E test for this. Could you give me some suggestions? @Hisoka-X @Carl-Zhou-CN 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix][Connector-V2] Fix indefinite blocking by adjusting stopping condition in kafka split reader [seatunnel]

Posted by "xiaochen-zhou (via GitHub)" <gi...@apache.org>.
xiaochen-zhou commented on PR #9597:
URL: https://github.com/apache/seatunnel/pull/9597#issuecomment-3096450753

   Friendly ping, do you have time to take a look @Hisoka-X @Carl-Zhou-CN 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix][Connector-V2] Fix indefinite blocking by adjusting stopping condition in kafka split reader [seatunnel]

Posted by "xiaochen-zhou (via GitHub)" <gi...@apache.org>.
xiaochen-zhou commented on code in PR #9597:
URL: https://github.com/apache/seatunnel/pull/9597#discussion_r2220899785


##########
seatunnel-connectors-v2/connector-kafka/src/test/java/org/apache/seatunnel/connectors/seatunnel/kafka/source/KafkaTransactionalStoppingConditionTest.java:
##########
@@ -0,0 +1,146 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.seatunnel.connectors.seatunnel.kafka.source;
+
+import org.apache.seatunnel.api.source.SourceReader;
+import org.apache.seatunnel.connectors.seatunnel.kafka.config.StartMode;
+
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.clients.consumer.ConsumerRecords;
+import org.apache.kafka.clients.consumer.MockConsumer;
+import org.apache.kafka.clients.consumer.OffsetResetStrategy;
+import org.apache.kafka.common.TopicPartition;
+
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.DisplayName;
+import org.junit.jupiter.api.Test;
+import org.mockito.Mock;
+import org.mockito.MockitoAnnotations;
+
+import java.time.Duration;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Properties;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+import static org.mockito.Mockito.when;
+
+/** Test class demonstrating the fix for transactional message stopping condition. */
+public class KafkaTransactionalStoppingConditionTest {
+
+    private static final String TOPIC = "test-topic";
+    private static final int PARTITION = 0;
+    private static final TopicPartition TOPIC_PARTITION = new TopicPartition(TOPIC, PARTITION);
+
+    private MockConsumer<byte[], byte[]> mockConsumer;
+
+    @Mock private SourceReader.Context readerContext;
+
+    @Mock private KafkaSourceConfig kafkaSourceConfig;
+
+    @BeforeEach
+    void setUp() {
+        MockitoAnnotations.openMocks(this);
+
+        mockConsumer = new MockConsumer<>(OffsetResetStrategy.EARLIEST);
+        // Setup mock config
+        Properties properties = new Properties();
+        properties.setProperty(ConsumerConfig.GROUP_ID_CONFIG, "test-group");
+        properties.setProperty(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
+
+        when(kafkaSourceConfig.getProperties()).thenReturn(properties);
+        when(kafkaSourceConfig.getConsumerGroup()).thenReturn("test-group");
+        when(kafkaSourceConfig.getBootstrap()).thenReturn("localhost:9092");
+        when(kafkaSourceConfig.isCommitOnCheckpoint()).thenReturn(false);
+        when(kafkaSourceConfig.getPollTimeout()).thenReturn(1000L);
+
+        Map<String, ConsumerMetadata> mapMetadata = new HashMap<>();
+        ConsumerMetadata metadata = new ConsumerMetadata();
+        metadata.setTopic(TOPIC);
+        metadata.setStartMode(StartMode.EARLIEST);
+        mapMetadata.put(TOPIC, metadata);
+    }
+
+    /**
+     * Test the core issue: control messages can cause infinite blocking when using last record
+     * offset as stopping condition.
+     *
+     * <p>Note: MockConsumer cannot directly simulate control messages since they are invisible to
+     * consumers. Instead, we simulate the EFFECT of control messages: consumer position advancing
+     * beyond the last visible record.
+     */
+    @Test
+    @DisplayName("Test stopping condition with control messages - demonstrates the fix")
+    void testStoppingConditionWithControlMessages() {

Review Comment:
   > This test only verifies the consumer behavior of Kafka, but does not ensure that our connector behaves normally through the test case. Please modify the test case to verify that KafkaPartitionSplitReader::fetch will end normally.
   
   OK.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix][Connector-V2] Fix indefinite blocking by adjusting stopping condition in kafka split reader [seatunnel]

Posted by "xiaochen-zhou (via GitHub)" <gi...@apache.org>.
xiaochen-zhou commented on PR #9597:
URL: https://github.com/apache/seatunnel/pull/9597#issuecomment-3172700096

   Done. @Hisoka-X @Carl-Zhou-CN 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix][Connector-V2] Fix indefinite blocking by adjusting stopping condition in kafka split reader [seatunnel]

Posted by "Hisoka-X (via GitHub)" <gi...@apache.org>.
Hisoka-X commented on code in PR #9597:
URL: https://github.com/apache/seatunnel/pull/9597#discussion_r2265584110


##########
seatunnel-connectors-v2/connector-kafka/src/main/java/org/apache/seatunnel/connectors/seatunnel/kafka/source/KafkaPartitionSplitReader.java:
##########
@@ -85,6 +85,16 @@ public KafkaPartitionSplitReader(
         this.pollTimeout = kafkaSourceConfig.getPollTimeout();
     }
 
+    public KafkaPartitionSplitReader(

Review Comment:
   ```suggestion
       @VisibleForTesting
       public KafkaPartitionSplitReader(
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix][Connector-V2] Fix indefinite blocking by adjusting stopping condition in kafka split reader [seatunnel]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #9597: [Fix][Connector-V2] Fix indefinite blocking by adjusting stopping condition in kafka split reader
URL: https://github.com/apache/seatunnel/pull/9597


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix][Connector-V2] Fix indefinite blocking by adjusting stopping condition in kafka split reader [seatunnel]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #9597:
URL: https://github.com/apache/seatunnel/pull/9597#issuecomment-3672880466

   This pull request has been closed because it has not had recent activity. You could reopen it if you try to continue your work, and anyone who are interested in it are encouraged to continue work on this pull request.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix][Connector-V2] Fix indefinite blocking by adjusting stopping condition in kafka split reader [seatunnel]

Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #9597:
URL: https://github.com/apache/seatunnel/pull/9597#discussion_r2218407836


##########
seatunnel-connectors-v2/connector-kafka/src/main/java/org/apache/seatunnel/connectors/seatunnel/kafka/source/KafkaPartitionSplitReader.java:
##########
@@ -103,27 +103,21 @@ public RecordsWithSplitIds<ConsumerRecord<byte[], byte[]>> fetch() throws IOExce
         KafkaPartitionSplitRecords recordsBySplits =
                 new KafkaPartitionSplitRecords(consumerRecords);
         List<TopicPartition> finishedPartitions = new ArrayList<>();
-        for (TopicPartition tp : consumerRecords.partitions()) {
+        for (TopicPartition tp : consumer.assignment()) {
             long stoppingOffset = getStoppingOffset(tp);
-            final List<ConsumerRecord<byte[], byte[]>> recordsFromPartition =
-                    consumerRecords.records(tp);
-
-            if (recordsFromPartition.size() > 0) {
-                final ConsumerRecord<byte[], byte[]> lastRecord =
-                        recordsFromPartition.get(recordsFromPartition.size() - 1);
-
-                // After processing a record with offset of "stoppingOffset - 1", the split reader
-                // should not continue fetching because the record with stoppingOffset may not
-                // exist. Keep polling will just block forever.
-                if (lastRecord.offset() >= stoppingOffset - 1) {
-                    recordsBySplits.setPartitionStoppingOffset(tp, stoppingOffset);
-                    finishSplitAtRecord(
-                            tp,
-                            stoppingOffset,
-                            lastRecord.offset(),
-                            finishedPartitions,
-                            recordsBySplits);
-                }
+            long consumerPosition = consumer.position(tp);
+            // Stop fetching when the consumer's position reaches the stoppingOffset.
+            // Control messages may follow the last record; therefore, using the last record's
+            // offset as a stopping condition could result in indefinite blocking.
+            if (consumerPosition >= stoppingOffset) {
+                LOG.debug(
+                        "Position of {}: {}, has reached stopping offset: {}",
+                        tp,
+                        consumerPosition,
+                        stoppingOffset);
+                recordsBySplits.setPartitionStoppingOffset(tp, stoppingOffset);
+                finishSplitAtRecord(
+                        tp, stoppingOffset, consumerPosition, finishedPartitions, recordsBySplits);

Review Comment:
   The finishSplitAtRecord method is being called with consumerPosition instead of the actual last record offset. This changes the semantic meaning of the third parameter, which could break the method's contract if it expects the actual record offset for processing logic.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix][Connector-V2] Fix indefinite blocking by adjusting stopping condition in kafka split reader [seatunnel]

Posted by "Hisoka-X (via GitHub)" <gi...@apache.org>.
Hisoka-X commented on code in PR #9597:
URL: https://github.com/apache/seatunnel/pull/9597#discussion_r2222278180


##########
seatunnel-connectors-v2/connector-kafka/src/test/java/org/apache/seatunnel/connectors/seatunnel/kafka/source/KafkaTransactionalStoppingConditionTest.java:
##########
@@ -0,0 +1,146 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.seatunnel.connectors.seatunnel.kafka.source;
+
+import org.apache.seatunnel.api.source.SourceReader;
+import org.apache.seatunnel.connectors.seatunnel.kafka.config.StartMode;
+
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.clients.consumer.ConsumerRecords;
+import org.apache.kafka.clients.consumer.MockConsumer;
+import org.apache.kafka.clients.consumer.OffsetResetStrategy;
+import org.apache.kafka.common.TopicPartition;
+
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.DisplayName;
+import org.junit.jupiter.api.Test;
+import org.mockito.Mock;
+import org.mockito.MockitoAnnotations;
+
+import java.time.Duration;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Properties;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+import static org.mockito.Mockito.when;
+
+/** Test class demonstrating the fix for transactional message stopping condition. */
+public class KafkaTransactionalStoppingConditionTest {
+
+    private static final String TOPIC = "test-topic";
+    private static final int PARTITION = 0;
+    private static final TopicPartition TOPIC_PARTITION = new TopicPartition(TOPIC, PARTITION);
+
+    private MockConsumer<byte[], byte[]> mockConsumer;
+
+    @Mock private SourceReader.Context readerContext;
+
+    @Mock private KafkaSourceConfig kafkaSourceConfig;
+
+    @BeforeEach
+    void setUp() {
+        MockitoAnnotations.openMocks(this);
+
+        mockConsumer = new MockConsumer<>(OffsetResetStrategy.EARLIEST);
+        // Setup mock config
+        Properties properties = new Properties();
+        properties.setProperty(ConsumerConfig.GROUP_ID_CONFIG, "test-group");
+        properties.setProperty(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
+
+        when(kafkaSourceConfig.getProperties()).thenReturn(properties);
+        when(kafkaSourceConfig.getConsumerGroup()).thenReturn("test-group");
+        when(kafkaSourceConfig.getBootstrap()).thenReturn("localhost:9092");
+        when(kafkaSourceConfig.isCommitOnCheckpoint()).thenReturn(false);
+        when(kafkaSourceConfig.getPollTimeout()).thenReturn(1000L);
+
+        Map<String, ConsumerMetadata> mapMetadata = new HashMap<>();
+        ConsumerMetadata metadata = new ConsumerMetadata();
+        metadata.setTopic(TOPIC);
+        metadata.setStartMode(StartMode.EARLIEST);
+        mapMetadata.put(TOPIC, metadata);
+    }
+
+    /**
+     * Test the core issue: control messages can cause infinite blocking when using last record
+     * offset as stopping condition.
+     *
+     * <p>Note: MockConsumer cannot directly simulate control messages since they are invisible to
+     * consumers. Instead, we simulate the EFFECT of control messages: consumer position advancing
+     * beyond the last visible record.
+     */
+    @Test
+    @DisplayName("Test stopping condition with control messages - demonstrates the fix")
+    void testStoppingConditionWithControlMessages() {

Review Comment:
   Maybe you can mock a `KafkaConsumer` into `KafkaPartitionSplitReader `.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix][Connector-V2] Fix indefinite blocking by adjusting stopping condition in kafka split reader [seatunnel]

Posted by "Hisoka-X (via GitHub)" <gi...@apache.org>.
Hisoka-X commented on code in PR #9597:
URL: https://github.com/apache/seatunnel/pull/9597#discussion_r2219476682


##########
seatunnel-connectors-v2/connector-kafka/src/test/java/org/apache/seatunnel/connectors/seatunnel/kafka/source/KafkaTransactionalStoppingConditionTest.java:
##########
@@ -0,0 +1,146 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.seatunnel.connectors.seatunnel.kafka.source;
+
+import org.apache.seatunnel.api.source.SourceReader;
+import org.apache.seatunnel.connectors.seatunnel.kafka.config.StartMode;
+
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.clients.consumer.ConsumerRecords;
+import org.apache.kafka.clients.consumer.MockConsumer;
+import org.apache.kafka.clients.consumer.OffsetResetStrategy;
+import org.apache.kafka.common.TopicPartition;
+
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.DisplayName;
+import org.junit.jupiter.api.Test;
+import org.mockito.Mock;
+import org.mockito.MockitoAnnotations;
+
+import java.time.Duration;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Properties;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+import static org.mockito.Mockito.when;
+
+/** Test class demonstrating the fix for transactional message stopping condition. */
+public class KafkaTransactionalStoppingConditionTest {
+
+    private static final String TOPIC = "test-topic";
+    private static final int PARTITION = 0;
+    private static final TopicPartition TOPIC_PARTITION = new TopicPartition(TOPIC, PARTITION);
+
+    private MockConsumer<byte[], byte[]> mockConsumer;
+
+    @Mock private SourceReader.Context readerContext;

Review Comment:
   useless?



##########
seatunnel-connectors-v2/connector-kafka/src/test/java/org/apache/seatunnel/connectors/seatunnel/kafka/source/KafkaTransactionalStoppingConditionTest.java:
##########
@@ -0,0 +1,146 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.seatunnel.connectors.seatunnel.kafka.source;
+
+import org.apache.seatunnel.api.source.SourceReader;
+import org.apache.seatunnel.connectors.seatunnel.kafka.config.StartMode;
+
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.clients.consumer.ConsumerRecords;
+import org.apache.kafka.clients.consumer.MockConsumer;
+import org.apache.kafka.clients.consumer.OffsetResetStrategy;
+import org.apache.kafka.common.TopicPartition;
+
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.DisplayName;
+import org.junit.jupiter.api.Test;
+import org.mockito.Mock;
+import org.mockito.MockitoAnnotations;
+
+import java.time.Duration;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Properties;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+import static org.mockito.Mockito.when;
+
+/** Test class demonstrating the fix for transactional message stopping condition. */
+public class KafkaTransactionalStoppingConditionTest {
+
+    private static final String TOPIC = "test-topic";
+    private static final int PARTITION = 0;
+    private static final TopicPartition TOPIC_PARTITION = new TopicPartition(TOPIC, PARTITION);
+
+    private MockConsumer<byte[], byte[]> mockConsumer;
+
+    @Mock private SourceReader.Context readerContext;
+
+    @Mock private KafkaSourceConfig kafkaSourceConfig;
+
+    @BeforeEach
+    void setUp() {
+        MockitoAnnotations.openMocks(this);
+
+        mockConsumer = new MockConsumer<>(OffsetResetStrategy.EARLIEST);
+        // Setup mock config
+        Properties properties = new Properties();
+        properties.setProperty(ConsumerConfig.GROUP_ID_CONFIG, "test-group");
+        properties.setProperty(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
+
+        when(kafkaSourceConfig.getProperties()).thenReturn(properties);
+        when(kafkaSourceConfig.getConsumerGroup()).thenReturn("test-group");
+        when(kafkaSourceConfig.getBootstrap()).thenReturn("localhost:9092");
+        when(kafkaSourceConfig.isCommitOnCheckpoint()).thenReturn(false);
+        when(kafkaSourceConfig.getPollTimeout()).thenReturn(1000L);
+
+        Map<String, ConsumerMetadata> mapMetadata = new HashMap<>();
+        ConsumerMetadata metadata = new ConsumerMetadata();
+        metadata.setTopic(TOPIC);
+        metadata.setStartMode(StartMode.EARLIEST);
+        mapMetadata.put(TOPIC, metadata);
+    }
+
+    /**
+     * Test the core issue: control messages can cause infinite blocking when using last record
+     * offset as stopping condition.
+     *
+     * <p>Note: MockConsumer cannot directly simulate control messages since they are invisible to
+     * consumers. Instead, we simulate the EFFECT of control messages: consumer position advancing
+     * beyond the last visible record.
+     */
+    @Test
+    @DisplayName("Test stopping condition with control messages - demonstrates the fix")
+    void testStoppingConditionWithControlMessages() {

Review Comment:
   This test only verifies the consumer behavior of Kafka, but does not ensure that our connector behaves normally through the test case. Please modify the test case to verify that KafkaPartitionSplitReader::fetch will end normally.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix][Connector-V2] Fix indefinite blocking by adjusting stopping condition in kafka split reader [seatunnel]

Posted by "Carl-Zhou-CN (via GitHub)" <gi...@apache.org>.
Carl-Zhou-CN commented on code in PR #9597:
URL: https://github.com/apache/seatunnel/pull/9597#discussion_r2232825701


##########
seatunnel-connectors-v2/connector-kafka/src/test/java/org/apache/seatunnel/connectors/seatunnel/kafka/source/KafkaTransactionalStoppingConditionTest.java:
##########
@@ -0,0 +1,146 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.seatunnel.connectors.seatunnel.kafka.source;
+
+import org.apache.seatunnel.api.source.SourceReader;
+import org.apache.seatunnel.connectors.seatunnel.kafka.config.StartMode;
+
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.clients.consumer.ConsumerRecords;
+import org.apache.kafka.clients.consumer.MockConsumer;
+import org.apache.kafka.clients.consumer.OffsetResetStrategy;
+import org.apache.kafka.common.TopicPartition;
+
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.DisplayName;
+import org.junit.jupiter.api.Test;
+import org.mockito.Mock;
+import org.mockito.MockitoAnnotations;
+
+import java.time.Duration;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Properties;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+import static org.mockito.Mockito.when;
+
+/** Test class demonstrating the fix for transactional message stopping condition. */
+public class KafkaTransactionalStoppingConditionTest {
+
+    private static final String TOPIC = "test-topic";
+    private static final int PARTITION = 0;
+    private static final TopicPartition TOPIC_PARTITION = new TopicPartition(TOPIC, PARTITION);
+
+    private MockConsumer<byte[], byte[]> mockConsumer;
+
+    @Mock private SourceReader.Context readerContext;
+
+    @Mock private KafkaSourceConfig kafkaSourceConfig;
+
+    @BeforeEach
+    void setUp() {
+        MockitoAnnotations.openMocks(this);
+
+        mockConsumer = new MockConsumer<>(OffsetResetStrategy.EARLIEST);
+        // Setup mock config
+        Properties properties = new Properties();
+        properties.setProperty(ConsumerConfig.GROUP_ID_CONFIG, "test-group");
+        properties.setProperty(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
+
+        when(kafkaSourceConfig.getProperties()).thenReturn(properties);
+        when(kafkaSourceConfig.getConsumerGroup()).thenReturn("test-group");
+        when(kafkaSourceConfig.getBootstrap()).thenReturn("localhost:9092");
+        when(kafkaSourceConfig.isCommitOnCheckpoint()).thenReturn(false);
+        when(kafkaSourceConfig.getPollTimeout()).thenReturn(1000L);
+
+        Map<String, ConsumerMetadata> mapMetadata = new HashMap<>();
+        ConsumerMetadata metadata = new ConsumerMetadata();
+        metadata.setTopic(TOPIC);
+        metadata.setStartMode(StartMode.EARLIEST);
+        mapMetadata.put(TOPIC, metadata);
+    }
+
+    /**
+     * Test the core issue: control messages can cause infinite blocking when using last record
+     * offset as stopping condition.
+     *
+     * <p>Note: MockConsumer cannot directly simulate control messages since they are invisible to
+     * consumers. Instead, we simulate the EFFECT of control messages: consumer position advancing
+     * beyond the last visible record.
+     */
+    @Test
+    @DisplayName("Test stopping condition with control messages - demonstrates the fix")
+    void testStoppingConditionWithControlMessages() {

Review Comment:
   > Maybe you can mock a `KafkaConsumer` into `KafkaPartitionSplitReader `.
   
   good suggestion



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix][Connector-V2] Fix indefinite blocking by adjusting stopping condition in kafka split reader [seatunnel]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #9597:
URL: https://github.com/apache/seatunnel/pull/9597#issuecomment-3639524199

   This pull request has been automatically marked as stale because it has not had recent activity for 120 days. It will be closed in 7 days if no further activity occurs.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org