[jira] [Closed] (DRILL-7364) Timeout reading from Kafka topic using Kafka plugin

["Arina Ielchiieva (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/DRILL-7364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Arina Ielchiieva closed DRILL-7364.
-----------------------------------
    Resolution: Done

> Timeout reading from Kafka topic using Kafka plugin
> ---------------------------------------------------
>
>                 Key: DRILL-7364
>                 URL: https://issues.apache.org/jira/browse/DRILL-7364
>             Project: Apache Drill
>          Issue Type: Bug
>          Components: Storage - Kafka
>    Affects Versions: 1.15.0, 1.16.0
>            Reporter: Aditya Allamraju
>            Priority: Major
>
> When we try to query Mapr-streams(similar to Apache Kafka) topic using Kafka plugin, we see the below timeout being thrown.
> {code:java}
> 0: jdbc:drill:drillbit=10.10.75.158:31010> 
> 0: jdbc:drill:drillbit=10.10.75.158:31010> select count(*) from `/sample-stream:fast-messages` where k<100;
> Error: DATA_READ ERROR: Failed to fetch messages within 200 milliseconds. Consider increasing the value of the property : store.kafka.poll.timeout
> DATA_READ ERROR: Failed to fetch messages within 200 milliseconds. Consider increasing the value of the property : store.kafka.poll.timeout
> [Error Id: 27112f7b-afd8-43cb-9376-32f4c63ad2d8 ]
> Fragment 0:0
> [Error Id: 27112f7b-afd8-43cb-9376-32f4c63ad2d8 on vm75-158.support.mapr.com:31010] (state=,code=0)
> 0: jdbc:drill:drillbit=10.10.75.158:31010> ALTER SYSTEM SET `store.kafka.poll.timeout` = 50000;
> +-------+------------------------------------+
> |  ok   |              summary               |
> +-------+------------------------------------+
> | true  | store.kafka.poll.timeout updated.  |
> +-------+------------------------------------+
> 1 row selected (0.148 seconds)
> 0: jdbc:drill:drillbit=10.10.75.158:31010>
> {code}
> The other interesting behavior is that:
> 1) Even after increasing the timeout value to 50secs, the drill query failed after the execution time crossed 50 secs(~51 secs).
> This pattern continued to whatever value we increased. For ex, after increasing the timeout to 100secs, query failed with above error after 101 secs of execution time.
> The user is using Drill 1.15.
> I tried to reproduce this on my test cluster. But this was not consistently reproducing in the test cluster. Whereas in the client's cluster, we were able to reproduce this behavior consistently. We collected the logs. But they have very little info on what's happening.
> I believe it is now essential to know how(and why) the timeout parameter "store.kafka.poll.timeout" is related to the Query execution time to understand this bug.
> I also have few more questions where i couldn't find much documentation.
> _1) Is there a one-to-one mapping between number of partitions of a topic to minor fragments of a query? For ex, If a given topic(t) has say 20 partitions, then will the query most likely have 20 minor fragments, other parameters being fairly sized._
> _2) Does each drillbit in the cluster equivalent to a Kafka-Consumer and all such drillbits of a cluster treated as part of a consumer group?_
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)