You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Mason Chen (Jira)" <ji...@apache.org> on 2024/01/17 06:01:00 UTC
[jira] [Commented] (FLINK-34127) Kafka connector repo runs a duplicate of `IntegrationTests` framework tests
[ https://issues.apache.org/jira/browse/FLINK-34127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17807559#comment-17807559 ]
Mason Chen commented on FLINK-34127:
------------------------------------
I found this is an instance of a 50 minute, timed out CI run. This causes network instability which causes a KafkaConsumer to infinitely reset the offset:
```
2024-01-17T00:56:08.7826779Z 00:56:08,659 [Source Data Fetcher for Source: Tested Source (3/5)#0] INFO org.apache.kafka.clients.consumer.internals.SubscriptionState [] - [Consumer clientId=DynamicKafkaSourceExternalContext-cluster0-2, groupId=DynamicKafkaSourceExternalContext] Seeking to earliest offset of partition topic1-3301243828175368455-0
```
This log is repeated over 10 times through the log and the thread dump confirms that this test's thread was stuck waiting for the test to complete.
> Kafka connector repo runs a duplicate of `IntegrationTests` framework tests
> ---------------------------------------------------------------------------
>
> Key: FLINK-34127
> URL: https://issues.apache.org/jira/browse/FLINK-34127
> Project: Flink
> Issue Type: Improvement
> Components: Build System / CI, Connectors / Kafka
> Affects Versions: kafka-3.0.2
> Reporter: Mason Chen
> Priority: Major
>
> I found out this behavior when troubleshooting CI flakiness. These integration tests make heavy use of the CI since they require Kafka, Zookeeper, and Docker containers. We can further stablize CI by not redundantly running these set of tests.
> `grep -E ".*testIdleReader\[TestEnvironment.*" 14_Compile\ and\ test.txt` returns:
> ```
> 2024-01-17T00:51:05.2943150Z Test org.apache.flink.connector.kafka.dynamic.source.DynamicKafkaSourceITTest.IntegrationTests.testIdleReader[TestEnvironment: [MiniCluster], ExternalContext: [org.apache.flink.connector.kafka.testutils.DynamicKafkaSourceExternalContext@43e9a8a2], Semantic: [EXACTLY_ONCE]] is running.
> 2024-01-17T00:51:07.6922535Z Test org.apache.flink.connector.kafka.dynamic.source.DynamicKafkaSourceITTest.IntegrationTests.testIdleReader[TestEnvironment: [MiniCluster], ExternalContext: [org.apache.flink.connector.kafka.testutils.DynamicKafkaSourceExternalContext@43e9a8a2], Semantic: [EXACTLY_ONCE]] successfully run.
> 2024-01-17T00:56:27.1326332Z Test org.apache.flink.connector.kafka.dynamic.source.DynamicKafkaSourceITTest.IntegrationTests.testIdleReader[TestEnvironment: [MiniCluster], ExternalContext: [org.apache.flink.connector.kafka.testutils.DynamicKafkaSourceExternalContext@2db4a84a], Semantic: [EXACTLY_ONCE]] is running.
> 2024-01-17T00:56:28.4000830Z Test org.apache.flink.connector.kafka.dynamic.source.DynamicKafkaSourceITTest.IntegrationTests.testIdleReader[TestEnvironment: [MiniCluster], ExternalContext: [org.apache.flink.connector.kafka.testutils.DynamicKafkaSourceExternalContext@2db4a84a], Semantic: [EXACTLY_ONCE]] successfully run.
> 2024-01-17T00:56:58.7830792Z Test org.apache.flink.connector.kafka.source.KafkaSourceITCase.IntegrationTests.testIdleReader[TestEnvironment: [MiniCluster], ExternalContext: [KafkaSource-PARTITION], Semantic: [EXACTLY_ONCE]] is running.
> 2024-01-17T00:56:59.0544092Z Test org.apache.flink.connector.kafka.source.KafkaSourceITCase.IntegrationTests.testIdleReader[TestEnvironment: [MiniCluster], ExternalContext: [KafkaSource-PARTITION], Semantic: [EXACTLY_ONCE]] successfully run.
> 2024-01-17T00:56:59.3910987Z Test org.apache.flink.connector.kafka.source.KafkaSourceITCase.IntegrationTests.testIdleReader[TestEnvironment: [MiniCluster], ExternalContext: [KafkaSource-TOPIC], Semantic: [EXACTLY_ONCE]] is running.
> 2024-01-17T00:56:59.6025298Z Test org.apache.flink.connector.kafka.source.KafkaSourceITCase.IntegrationTests.testIdleReader[TestEnvironment: [MiniCluster], ExternalContext: [KafkaSource-TOPIC], Semantic: [EXACTLY_ONCE]] successfully run.
> 2024-01-17T00:57:37.8378640Z Test org.apache.flink.connector.kafka.source.KafkaSourceITCase.IntegrationTests.testIdleReader[TestEnvironment: [MiniCluster], ExternalContext: [KafkaSource-PARTITION], Semantic: [EXACTLY_ONCE]] is running.
> 2024-01-17T00:57:38.0144732Z Test org.apache.flink.connector.kafka.source.KafkaSourceITCase.IntegrationTests.testIdleReader[TestEnvironment: [MiniCluster], ExternalContext: [KafkaSource-PARTITION], Semantic: [EXACTLY_ONCE]] successfully run.
> 2024-01-17T00:57:38.2004796Z Test org.apache.flink.connector.kafka.source.KafkaSourceITCase.IntegrationTests.testIdleReader[TestEnvironment: [MiniCluster], ExternalContext: [KafkaSource-TOPIC], Semantic: [EXACTLY_ONCE]] is running.
> 2024-01-17T00:57:38.4072815Z Test org.apache.flink.connector.kafka.source.KafkaSourceITCase.IntegrationTests.testIdleReader[TestEnvironment: [MiniCluster], ExternalContext: [KafkaSource-TOPIC], Semantic: [EXACTLY_ONCE]] successfully run.
> 2024-01-17T01:06:11.2933375Z Test org.apache.flink.tests.util.kafka.KafkaSourceE2ECase.testIdleReader[TestEnvironment: [FlinkContainers], ExternalContext: [KafkaSource-PARTITION], Semantic: [EXACTLY_ONCE]] is running.
> 2024-01-17T01:06:12.1790031Z Test org.apache.flink.tests.util.kafka.KafkaSourceE2ECase.testIdleReader[TestEnvironment: [FlinkContainers], ExternalContext: [KafkaSource-PARTITION], Semantic: [EXACTLY_ONCE]] successfully run.
> 2024-01-17T01:06:12.5703927Z Test org.apache.flink.tests.util.kafka.KafkaSourceE2ECase.testIdleReader[TestEnvironment: [FlinkContainers], ExternalContext: [KafkaSource-TOPIC], Semantic: [EXACTLY_ONCE]] is running.
> 2024-01-17T01:06:13.3369574Z Test org.apache.flink.tests.util.kafka.KafkaSourceE2ECase.testIdleReader[TestEnvironment: [FlinkContainers], ExternalContext: [KafkaSource-TOPIC], Semantic: [EXACTLY_ONCE]] successfully run.
> ```
--
This message was sent by Atlassian Jira
(v8.20.10#820010)