You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@drill.apache.org by "Maksym Rymar (Jira)" <ji...@apache.org> on 2022/02/02 14:10:00 UTC
[jira] [Updated] (DRILL-8122) Change kafka metadata obtaining due to KAFKA-5697
[ https://issues.apache.org/jira/browse/DRILL-8122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Maksym Rymar updated DRILL-8122:
--------------------------------
Description:
Starting from kafka 2.0 was changed behavior of the [KafkaConsumer#poll|https://kafka.apache.org/11/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html#poll-long-] . In Drill was used one hack, where poll(0) is used only to update consumer metadata. But after the next change: [https://github.com/apache/kafka/pull/4855] this hack is not working now as poll() is not blocked more until metadata update yet more.
Unfortunately, Kafka doesn't have public method to only update assignment metadata in consumer and it's on the discussion now whether to add it or not: [KIP-505|https://cwiki.apache.org/confluence/display/KAFKA/KIP-505%3A+Add+new+public+method+to+only+update+assignment+metadata+in+consumer] . Once it will be introduced we can use it but until that it needs to implement a workaround.
Code to change:
[https://github.com/apache/drill/blob/15b2f52260e4f0026f2dfafa23c5d32e0fb66502/contrib/storage-kafka/src/main/java/org/apache/drill/exec/store/kafka/KafkaGroupScan.java#L185]
was:
Starting from kafka 2.0 was changed behavior of the [KafkaConsumer#poll|https://kafka.apache.org/11/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html#poll-long-] . In Drill was used one hack, where poll() is used only to update consumer metadata. But after the next change: [https://github.com/apache/kafka/pull/4855] this hack is not working now as poll() is not blocked more until metadata update yet more.
Unfortunately, Kafka doesn't have public method to only update assignment metadata in consumer and it's on the discussion now whether to add it or not: [KIP-505|https://cwiki.apache.org/confluence/display/KAFKA/KIP-505%3A+Add+new+public+method+to+only+update+assignment+metadata+in+consumer] . Once it will be introduced we can use it but until that it needs to implement a workaround.
Code to change:
https://github.com/apache/drill/blob/15b2f52260e4f0026f2dfafa23c5d32e0fb66502/contrib/storage-kafka/src/main/java/org/apache/drill/exec/store/kafka/KafkaGroupScan.java#L185
> Change kafka metadata obtaining due to KAFKA-5697
> -------------------------------------------------
>
> Key: DRILL-8122
> URL: https://issues.apache.org/jira/browse/DRILL-8122
> Project: Apache Drill
> Issue Type: Bug
> Affects Versions: 1.17.0, 1.18.0, 1.19.0
> Reporter: Maksym Rymar
> Assignee: Maksym Rymar
> Priority: Major
> Fix For: 1.20.0
>
>
> Starting from kafka 2.0 was changed behavior of the [KafkaConsumer#poll|https://kafka.apache.org/11/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html#poll-long-] . In Drill was used one hack, where poll(0) is used only to update consumer metadata. But after the next change: [https://github.com/apache/kafka/pull/4855] this hack is not working now as poll() is not blocked more until metadata update yet more.
> Unfortunately, Kafka doesn't have public method to only update assignment metadata in consumer and it's on the discussion now whether to add it or not: [KIP-505|https://cwiki.apache.org/confluence/display/KAFKA/KIP-505%3A+Add+new+public+method+to+only+update+assignment+metadata+in+consumer] . Once it will be introduced we can use it but until that it needs to implement a workaround.
> Code to change:
> [https://github.com/apache/drill/blob/15b2f52260e4f0026f2dfafa23c5d32e0fb66502/contrib/storage-kafka/src/main/java/org/apache/drill/exec/store/kafka/KafkaGroupScan.java#L185]
--
This message was sent by Atlassian Jira
(v8.20.1#820001)