You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@drill.apache.org by Chetan Kothari <ch...@oracle.com> on 2017/08/29 17:14:14 UTC
Querying Streaming Data using Drill
Is there any support for querying streaming data using Drill?
Presto provides out-of-box Kafka and Amazon Kinesis Connectors for querying streaming data.
Regards
Chetan
RE: Querying Streaming Data using Drill
Posted by Chetan Kothari <ch...@oracle.com>.
Thanks Anil
Can you please share more details of the approach here for querying streaming data?
Are we going to create table mapped to Kafka Topic here and pull data from topic at regular intervals?
Are we going to extract event from message and insert as row to table.
Data pushed to Kafka topic may be unstructured.
Are we planning to provide some mechanism through which user can define meta-data of message pushed to Kafka Topic?
Regards
Chetan
-----Original Message-----
From: AnilKumar B [mailto:akumarb2010@gmail.com]
Sent: Wednesday, August 30, 2017 12:19 AM
To: user@drill.apache.org
Subject: Re: Querying Streaming Data using Drill
Hi Padma & Chetan,
Just wanted to update on https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_DRILL-2D4779&d=DwIBaQ&c=RoP1YumCXCgaWHvlZYR8PQcxBKCX5YTpkKY057SbK10&r=XdafK486-x96ZJoTcDF35EpcYc2j9nO0sbpt27_VGCg&m=_sbH_Ztnc2gGMatSIcqTEJ5Oy1A4Q6mDnQPvVOCqwhg&s=RhoBrti0LOnAXP0jWjDD4C1VH1keATistN8CMoAq8cQ&e=
We have developed and tested kafka integration and currently it's working for JSON messages. And currently we are working on test cases and Avro support.
We are planning to present this on Sept 18th Drill Developer's day.
Due to multiple reasons, this feature delayed from long time, but we are almost there.
Repo:
https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_akumarb2010_incubator-2Ddrill_tree_master_contrib_storage-2Dkafka&d=DwIBaQ&c=RoP1YumCXCgaWHvlZYR8PQcxBKCX5YTpkKY057SbK10&r=XdafK486-x96ZJoTcDF35EpcYc2j9nO0sbpt27_VGCg&m=_sbH_Ztnc2gGMatSIcqTEJ5Oy1A4Q6mDnQPvVOCqwhg&s=13f8RiuuC2F7dMIIuzBERWl6jIjmZuMu-X9SnCVxNP4&e=
Thanks & Regards,
B Anil Kumar.
On Tue, Aug 29, 2017 at 10:26 AM, Chetan Kothari <HYPERLINK "mailto:chetan.kothari@oracle.com"chetan.kothari@oracle.com>
wrote:
> Thanks Padma for quick response.
>
>
>
> This will be very critical feature to support in Drill as user will
> look for
>
> single SQL Engine which supports querying both batch and streaming data.
>
>
>
> Any inputs on when support for querying streaming data will be supported?
>
>
>
> Regards
>
> Chetan
>
>
>
> -----Original Message-----
> From: Padma Penumarthy [mailto:ppenumarthy@mapr.com]
> Sent: Tuesday, August 29, 2017 10:53 PM
> To: HYPERLINK "mailto:user@drill.apache.org"user@drill.apache.org
> Subject: Re: Querying Streaming Data using Drill
>
>
>
> Currently, we do not have support for these storage plugins.
>
> I see an open JIRA for Kafka, not sure how much progress was made (as
> last update was a while back).
>
>
>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.
> apache.org_jira_browse_DRILL-2D4779&d=DwIFAg&c=
> RoP1YumCXCgaWHvlZYR8PQcxBKCX5YTpkKY057SbK10&r=XdafK486-
> x96ZJoTcDF35EpcYc2j9nO0sbpt27_VGCg&m=bHLm9AOBdeCey8wjxCKXHxHTwlpzXu
> JF3dhrqs-qX-0&s=UCGsbhOa9QLDAmUEfJYiq_cOfGg5dUi1RdcPB7YTMMQ&e=
>
>
>
> Thanks,
>
> Padma
>
>
>
>
>
> On Aug 29, 2017, at 10:14 AM, Chetan Kothari <HYPERLINK "mailto:
> HYPERLINK "mailto:chetan.kothari@oracle.com%3cmailto:chetan.kothari@oracle.com"chetan.kothari@oracle.com%3cmailto:chetan.kothari@oracle.com"
> HYPERLINK "mailto:chetan.kothari@oracle.com%3cmailto:chetan.kothari@oracle.com"chetan.kothari@oracle.com<ma...@oracle.com>> wrote:
>
>
>
> Is there any support for querying streaming data using Drill?
>
>
>
> Presto provides out-of-box Kafka and Amazon Kinesis Connectors for
> querying streaming data.
>
>
>
>
>
>
>
> Regards
>
>
>
> Chetan
>
>
>
>
>
Re: Querying Streaming Data using Drill
Posted by Padma Penumarthy <pp...@mapr.com>.
That’s great. Thanks for the update. Looking forward for the presentation.
Thanks,
Padma
> On Aug 29, 2017, at 11:48 AM, AnilKumar B <ak...@gmail.com> wrote:
>
> Hi Padma & Chetan,
>
> Just wanted to update on https://issues.apache.org/jira/browse/DRILL-4779
>
> We have developed and tested kafka integration and currently it's working
> for JSON messages. And currently we are working on test cases and Avro
> support.
>
> We are planning to present this on Sept 18th Drill Developer's day.
>
> Due to multiple reasons, this feature delayed from long time, but we are
> almost there.
>
> Repo:
> https://github.com/akumarb2010/incubator-drill/tree/master/contrib/storage-kafka
>
>
>
>
> Thanks & Regards,
> B Anil Kumar.
>
> On Tue, Aug 29, 2017 at 10:26 AM, Chetan Kothari <ch...@oracle.com>
> wrote:
>
>> Thanks Padma for quick response.
>>
>>
>>
>> This will be very critical feature to support in Drill as user will look
>> for
>>
>> single SQL Engine which supports querying both batch and streaming data.
>>
>>
>>
>> Any inputs on when support for querying streaming data will be supported?
>>
>>
>>
>> Regards
>>
>> Chetan
>>
>>
>>
>> -----Original Message-----
>> From: Padma Penumarthy [mailto:ppenumarthy@mapr.com]
>> Sent: Tuesday, August 29, 2017 10:53 PM
>> To: user@drill.apache.org
>> Subject: Re: Querying Streaming Data using Drill
>>
>>
>>
>> Currently, we do not have support for these storage plugins.
>>
>> I see an open JIRA for Kafka, not sure how much progress was made (as last
>> update was a while back).
>>
>>
>>
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.
>> apache.org_jira_browse_DRILL-2D4779&d=DwIFAg&c=
>> RoP1YumCXCgaWHvlZYR8PQcxBKCX5YTpkKY057SbK10&r=XdafK486-
>> x96ZJoTcDF35EpcYc2j9nO0sbpt27_VGCg&m=bHLm9AOBdeCey8wjxCKXHxHTwlpzXu
>> JF3dhrqs-qX-0&s=UCGsbhOa9QLDAmUEfJYiq_cOfGg5dUi1RdcPB7YTMMQ&e=
>>
>>
>>
>> Thanks,
>>
>> Padma
>>
>>
>>
>>
>>
>> On Aug 29, 2017, at 10:14 AM, Chetan Kothari <HYPERLINK "mailto:
>> chetan.kothari@oracle.com%3cmailto:chetan.kothari@oracle.com"
>> chetan.kothari@oracle.com<ma...@oracle.com>> wrote:
>>
>>
>>
>> Is there any support for querying streaming data using Drill?
>>
>>
>>
>> Presto provides out-of-box Kafka and Amazon Kinesis Connectors for
>> querying streaming data.
>>
>>
>>
>>
>>
>>
>>
>> Regards
>>
>>
>>
>> Chetan
>>
>>
>>
>>
>>
Re: Querying Streaming Data using Drill
Posted by AnilKumar B <ak...@gmail.com>.
Hi Padma & Chetan,
Just wanted to update on https://issues.apache.org/jira/browse/DRILL-4779
We have developed and tested kafka integration and currently it's working
for JSON messages. And currently we are working on test cases and Avro
support.
We are planning to present this on Sept 18th Drill Developer's day.
Due to multiple reasons, this feature delayed from long time, but we are
almost there.
Repo:
https://github.com/akumarb2010/incubator-drill/tree/master/contrib/storage-kafka
Thanks & Regards,
B Anil Kumar.
On Tue, Aug 29, 2017 at 10:26 AM, Chetan Kothari <ch...@oracle.com>
wrote:
> Thanks Padma for quick response.
>
>
>
> This will be very critical feature to support in Drill as user will look
> for
>
> single SQL Engine which supports querying both batch and streaming data.
>
>
>
> Any inputs on when support for querying streaming data will be supported?
>
>
>
> Regards
>
> Chetan
>
>
>
> -----Original Message-----
> From: Padma Penumarthy [mailto:ppenumarthy@mapr.com]
> Sent: Tuesday, August 29, 2017 10:53 PM
> To: user@drill.apache.org
> Subject: Re: Querying Streaming Data using Drill
>
>
>
> Currently, we do not have support for these storage plugins.
>
> I see an open JIRA for Kafka, not sure how much progress was made (as last
> update was a while back).
>
>
>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.
> apache.org_jira_browse_DRILL-2D4779&d=DwIFAg&c=
> RoP1YumCXCgaWHvlZYR8PQcxBKCX5YTpkKY057SbK10&r=XdafK486-
> x96ZJoTcDF35EpcYc2j9nO0sbpt27_VGCg&m=bHLm9AOBdeCey8wjxCKXHxHTwlpzXu
> JF3dhrqs-qX-0&s=UCGsbhOa9QLDAmUEfJYiq_cOfGg5dUi1RdcPB7YTMMQ&e=
>
>
>
> Thanks,
>
> Padma
>
>
>
>
>
> On Aug 29, 2017, at 10:14 AM, Chetan Kothari <HYPERLINK "mailto:
> chetan.kothari@oracle.com%3cmailto:chetan.kothari@oracle.com"
> chetan.kothari@oracle.com<ma...@oracle.com>> wrote:
>
>
>
> Is there any support for querying streaming data using Drill?
>
>
>
> Presto provides out-of-box Kafka and Amazon Kinesis Connectors for
> querying streaming data.
>
>
>
>
>
>
>
> Regards
>
>
>
> Chetan
>
>
>
>
>
RE: Querying Streaming Data using Drill
Posted by Chetan Kothari <ch...@oracle.com>.
Thanks Padma for quick response.
This will be very critical feature to support in Drill as user will look for
single SQL Engine which supports querying both batch and streaming data.
Any inputs on when support for querying streaming data will be supported?
Regards
Chetan
-----Original Message-----
From: Padma Penumarthy [mailto:ppenumarthy@mapr.com]
Sent: Tuesday, August 29, 2017 10:53 PM
To: user@drill.apache.org
Subject: Re: Querying Streaming Data using Drill
Currently, we do not have support for these storage plugins.
I see an open JIRA for Kafka, not sure how much progress was made (as last update was a while back).
https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_DRILL-2D4779&d=DwIFAg&c=RoP1YumCXCgaWHvlZYR8PQcxBKCX5YTpkKY057SbK10&r=XdafK486-x96ZJoTcDF35EpcYc2j9nO0sbpt27_VGCg&m=bHLm9AOBdeCey8wjxCKXHxHTwlpzXuJF3dhrqs-qX-0&s=UCGsbhOa9QLDAmUEfJYiq_cOfGg5dUi1RdcPB7YTMMQ&e=
Thanks,
Padma
On Aug 29, 2017, at 10:14 AM, Chetan Kothari <HYPERLINK "mailto:chetan.kothari@oracle.com%3cmailto:chetan.kothari@oracle.com"chetan.kothari@oracle.com<ma...@oracle.com>> wrote:
Is there any support for querying streaming data using Drill?
Presto provides out-of-box Kafka and Amazon Kinesis Connectors for querying streaming data.
Regards
Chetan
Re: Querying Streaming Data using Drill
Posted by Padma Penumarthy <pp...@mapr.com>.
Currently, we do not have support for these storage plugins.
I see an open JIRA for Kafka, not sure how much progress was made (as last update was a while back).
https://issues.apache.org/jira/browse/DRILL-4779
Thanks,
Padma
On Aug 29, 2017, at 10:14 AM, Chetan Kothari <ch...@oracle.com>> wrote:
Is there any support for querying streaming data using Drill?
Presto provides out-of-box Kafka and Amazon Kinesis Connectors for querying streaming data.
Regards
Chetan