You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Nicolaus Weidner (Jira)" <ji...@apache.org> on 2021/09/06 16:10:00 UTC
[jira] [Commented] (FLINK-23932)
KafkaTableITKafkaTableITCase.testKafkaSourceSinkWithKeyAndPartialValue
hangs on AzureCase
[ https://issues.apache.org/jira/browse/FLINK-23932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17410708#comment-17410708 ]
Nicolaus Weidner commented on FLINK-23932:
------------------------------------------
This looks like the same root cause as FLINK-22198 and FLINK-23778. This is from the logs, looks like log compaction deleted entries before they were read and then the test waited forever:
{code:java}
16:34:53,147 [docker-java-stream-1151525689] INFO org.apache.flink.streaming.connectors.kafka.table.KafkaTableTestBase [] - STDOUT: [2021-08-23 16:34:53,146] INFO [Log partition=key_partial_value_topic_json-0, dir=/var/lib/kafka/data] Rolled new log segment at offset 3 in 26 ms. (kafka.log.Log)
16:34:53,148 [docker-java-stream-1151525689] INFO org.apache.flink.streaming.connectors.kafka.table.KafkaTableTestBase [] - STDOUT: [2021-08-23 16:34:53,147] INFO [Log partition=key_partial_value_topic_json-0, dir=/var/lib/kafka/data] Scheduling segments for deletion List(LogSegment(baseOffset=0, size=389, lastModifiedTime=1629736493000, largestTime=1583845931123)) (kafka.log.Log)
16:34:53,148 [Source Data Fetcher for Source: KafkaSource-default_catalog.default_database.kafka -> Calc(select=[k_user_id, name, CAST(timestamp) AS timestamp, k_event_id, user_id, payload]) -> Sink: Collect table sink (1/1)#0] INFO org.apache.kafka.clients.consumer.KafkaConsumer [] - [Consumer clientId=flink-tests-0, groupId=flink-tests] Subscribed to partition(s): key_partial_value_topic_json-0
16:34:53,149 [docker-java-stream-1151525689] INFO org.apache.flink.streaming.connectors.kafka.table.KafkaTableTestBase [] - STDOUT: [2021-08-23 16:34:53,148] INFO [Log partition=key_partial_value_topic_json-0, dir=/var/lib/kafka/data] Incrementing log start offset to 3 (kafka.log.Log)
16:34:53,149 [Source Data Fetcher for Source: KafkaSource-default_catalog.default_database.kafka -> Calc(select=[k_user_id, name, CAST(timestamp) AS timestamp, k_event_id, user_id, payload]) -> Sink: Collect table sink (1/1)#0] INFO org.apache.kafka.clients.consumer.internals.SubscriptionState [] - [Consumer clientId=flink-tests-0, groupId=flink-tests] Seeking to EARLIEST offset of partition key_partial_value_topic_json-0
16:34:53,158 [Source Data Fetcher for Source: KafkaSource-default_catalog.default_database.kafka -> Calc(select=[k_user_id, name, CAST(timestamp) AS timestamp, k_event_id, user_id, payload]) -> Sink: Collect table sink (1/1)#0] INFO org.apache.kafka.clients.consumer.internals.SubscriptionState [] - [Consumer clientId=flink-tests-0, groupId=flink-tests] Resetting offset for partition key_partial_value_topic_json-0 to offset 3.
16:35:21,819 [ Debug Logging Timer] INFO org.apache.flink.streaming.connectors.kafka.table.KafkaTableTestBase [] - TopicPartition "key_partial_value_topic_json-0": starting offset: 3, stopping offset: 3
{code}
> KafkaTableITKafkaTableITCase.testKafkaSourceSinkWithKeyAndPartialValue hangs on AzureCase
> -----------------------------------------------------------------------------------------
>
> Key: FLINK-23932
> URL: https://issues.apache.org/jira/browse/FLINK-23932
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka, Table SQL / Ecosystem
> Affects Versions: 1.14.0
> Reporter: Till Rohrmann
> Priority: Critical
> Labels: test-stability
> Fix For: 1.14.0
>
>
> The test {{KafkaTableITCase.testKafkaSourceSinkWithKeyAndPartialValue}} hangs on Azure. Interestingly, the test case seems to spawn 400 {{kafka-admin-client-thread | adminclient}} threads. I think there is something wrong with the test case setup.
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=22674&view=logs&j=c5f0071e-1851-543e-9a45-9ac140befc32&t=15a22db7-8faa-5b34-3920-d33c9f0ca23c&l=22875
--
This message was sent by Atlassian Jira
(v8.3.4#803005)