You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@drill.apache.org by Jacques Nadeau <ja...@apache.org> on 2013/11/02 18:02:24 UTC
Re: HBase Query Support
Yes. This is very much what I was thinking as well. If you're in the bay
area, you should come to the Drill meetup on Monday. We're hoping to try
to gather together interested parties to talk more. If not, no problem as
we'll report back any thoughts etc here.
thanks,
Jacques
On Tue, Oct 29, 2013 at 5:16 AM, Dhruv <yo...@gmail.com> wrote:
> Hi Aditya/James,
>
> As per my understanding Phoenix (having great features while
> storing[e.g. secondary index] + Schema) can be directly used for data
> insertion (Drill does not come in to picture in this flow).
>
> And for query retrieval; first level we can use Phoenix end user API
> in local Drillbit to query data.
>
> DrillUsers -> DrillBit+EmbaddedPhoenixJDBCClient (local mode, single
> instance) -> {HBase cluster..n}
>
> In the mean time James can draft a design where DrillBit can use more
> lower level API to benefit from Drill's Distributed operators,Joins and
> ValueVector memory model, and using Phoenix's Hbase side coprocessors and
> custom filters (as operator pushdown)
>
> DrillUsers -> {DrillBit+EmbaddedPhoenixClient (Distributed mode) ..n}
> -> -> {HBase cluster..n}
>
> Also we should be able to use HBase Table Schema generated by Phoenix.
>
> Your thoughts?
>
> -Dhruv
>
>
> On 10/29/2013 03:52 AM, James Taylor wrote:
>
>> Sure, I'd be happy to.
>>
>> Thanks,
>> James
>>
>>
>> On Mon, Oct 28, 2013 at 3:16 PM, Aditya <ad...@gmail.com> wrote:
>>
>> James,
>>>
>>> Jacques and I discussed this recently and we concur that it would be a
>>> beneficial to build such an integration between the two projects.
>>>
>>> Would you have some time to have a meeting/hangout with Drill team to
>>> talk
>>> this further?
>>>
>>> Regards,
>>> Aditya...
>>>
>>> Regards,
>>> Aditya...
>>>
>>>
>>> On Mon, Oct 28, 2013 at 1:31 PM, James Taylor <jtaylor@salesforce.com
>>>
>>>> wrote:
>>>> Hi,
>>>> The idea for the Phoenix integration was to come up with the right API
>>>> to
>>>> be able to plug Phoenix into Drill as the HBase query engine/store.
>>>>
>>> Jacque
>>>
>>>> and I got busy on our respective releases, but now that those are both
>>>>
>>> out,
>>>
>>>> I think we should get back to it. If we did it right, then both products
>>>> would be able to continue to evolve independently, but Drill would have
>>>>
>>> top
>>>
>>>> notch query performance for HBase data and Phoenix would have a new
>>>> customer.
>>>>
>>>> Thoughts/ideas?
>>>>
>>>> Thanks,
>>>> James
>>>>
>>>>
>>>> On Mon, Oct 28, 2013 at 5:34 AM, Dhruv <yo...@gmail.com>
>>>>
>>> wrote:
>>>
>>>> Hi,
>>>>>
>>>>> I am interested in contributing towards providing HBase Query support
>>>>>
>>>> in
>>>
>>>> DRILL.
>>>>>
>>>>> A) Can anyone summarize existing plan+Priority for it? (Or we can
>>>>>
>>>> discuss
>>>>
>>>>> in this thread and I will create WIKI page)
>>>>>
>>>>> B) I see already following JIRA issues, Anybody working on this
>>>>>
>>>> locally?
>>>
>>>> B-1) https://issues.apache.org/**jira/browse/DRILL-15<
>>>>>
>>>> https://issues.apache.org/jira/browse/DRILL-15> last updated 13-Aug-13.
>>>>
>>>>> B-2) https://issues.apache.org/**jira/browse/DRILL-88<
>>>>>
>>>> https://issues.apache.org/jira/browse/DRILL-88>Unassigned
>>>>
>>>>> B-3) https://issues.apache.org/**jira/browse/DRILL-63<
>>>>>
>>>> https://issues.apache.org/jira/browse/DRILL-63>James Taylor seems
>>>> interested to work
>>>>
>>>>> C) As per my understanding implimenting
>>>>>
>>>> org.apache.drill.exec.store.**hbase
>>>>
>>>>> package with following for local mode drill bit should suffice for POC?
>>>>> HBASERecordReader.java
>>>>> HBASEGroupScan
>>>>> HBASEScanBatchCreator
>>>>> HbaseSchemaProvider
>>>>> HBASEStorageEngine
>>>>> HBASEStorageEngineConfig
>>>>> HBASESubScan
>>>>>
>>>>> D) What part of https://github.com/**forcedotcom/phoenix<
>>>>>
>>>> https://github.com/forcedotcom/phoenix>can we reuse? Any phonix
>>>> contributors around?
>>>>
>>>>> -Dhruv
>>>>>
>>>>>
>
Re: HBase Query Support
Posted by Dhruv <yo...@gmail.com>.
Thanks Jacques,
Sorry I am little far from Bay area ;-) @India , so won't be able
to discuss Face2Face.
But surely waiting for continue the discussion on mailing list
towards some concrete approach.
Also would you mind answering originally posted quote?
> A) Can anyone summarize existing plan+Priority for it? *Hbase support*
-Dhruv
On 11/02/2013 10:32 PM, Jacques Nadeau wrote:
> Yes. This is very much what I was thinking as well. If you're in the bay
> area, you should come to the Drill meetup on Monday. We're hoping to try
> to gather together interested parties to talk more. If not, no problem as
> we'll report back any thoughts etc here.
>
> thanks,
> Jacques
>
>
> On Tue, Oct 29, 2013 at 5:16 AM, Dhruv <yo...@gmail.com> wrote:
>
>> Hi Aditya/James,
>>
>> As per my understanding Phoenix (having great features while
>> storing[e.g. secondary index] + Schema) can be directly used for data
>> insertion (Drill does not come in to picture in this flow).
>>
>> And for query retrieval; first level we can use Phoenix end user API
>> in local Drillbit to query data.
>>
>> DrillUsers -> DrillBit+EmbaddedPhoenixJDBCClient (local mode, single
>> instance) -> {HBase cluster..n}
>>
>> In the mean time James can draft a design where DrillBit can use more
>> lower level API to benefit from Drill's Distributed operators,Joins and
>> ValueVector memory model, and using Phoenix's Hbase side coprocessors and
>> custom filters (as operator pushdown)
>>
>> DrillUsers -> {DrillBit+EmbaddedPhoenixClient (Distributed mode) ..n}
>> -> -> {HBase cluster..n}
>>
>> Also we should be able to use HBase Table Schema generated by Phoenix.
>>
>> Your thoughts?
>>
>> -Dhruv
>>
>>
>> On 10/29/2013 03:52 AM, James Taylor wrote:
>>
>>> Sure, I'd be happy to.
>>>
>>> Thanks,
>>> James
>>>
>>>
>>> On Mon, Oct 28, 2013 at 3:16 PM, Aditya <ad...@gmail.com> wrote:
>>>
>>> James,
>>>> Jacques and I discussed this recently and we concur that it would be a
>>>> beneficial to build such an integration between the two projects.
>>>>
>>>> Would you have some time to have a meeting/hangout with Drill team to
>>>> talk
>>>> this further?
>>>>
>>>> Regards,
>>>> Aditya...
>>>>
>>>> Regards,
>>>> Aditya...
>>>>
>>>>
>>>> On Mon, Oct 28, 2013 at 1:31 PM, James Taylor <jtaylor@salesforce.com
>>>>
>>>>> wrote:
>>>>> Hi,
>>>>> The idea for the Phoenix integration was to come up with the right API
>>>>> to
>>>>> be able to plug Phoenix into Drill as the HBase query engine/store.
>>>>>
>>>> Jacque
>>>>
>>>>> and I got busy on our respective releases, but now that those are both
>>>>>
>>>> out,
>>>>
>>>>> I think we should get back to it. If we did it right, then both products
>>>>> would be able to continue to evolve independently, but Drill would have
>>>>>
>>>> top
>>>>
>>>>> notch query performance for HBase data and Phoenix would have a new
>>>>> customer.
>>>>>
>>>>> Thoughts/ideas?
>>>>>
>>>>> Thanks,
>>>>> James
>>>>>
>>>>>
>>>>> On Mon, Oct 28, 2013 at 5:34 AM, Dhruv <yo...@gmail.com>
>>>>>
>>>> wrote:
>>>>
>>>>> Hi,
>>>>>> I am interested in contributing towards providing HBase Query support
>>>>>>
>>>>> in
>>>>> DRILL.
>>>>>> A) Can anyone summarize existing plan+Priority for it? (Or we can
>>>>>>
>>>>> discuss
>>>>>
>>>>>> in this thread and I will create WIKI page)
>>>>>>
>>>>>> B) I see already following JIRA issues, Anybody working on this
>>>>>>
>>>>> locally?
>>>>> B-1) https://issues.apache.org/**jira/browse/DRILL-15<
>>>>> https://issues.apache.org/jira/browse/DRILL-15> last updated 13-Aug-13.
>>>>>
>>>>>> B-2) https://issues.apache.org/**jira/browse/DRILL-88<
>>>>>>
>>>>> https://issues.apache.org/jira/browse/DRILL-88>Unassigned
>>>>>
>>>>>> B-3) https://issues.apache.org/**jira/browse/DRILL-63<
>>>>>>
>>>>> https://issues.apache.org/jira/browse/DRILL-63>James Taylor seems
>>>>> interested to work
>>>>>
>>>>>> C) As per my understanding implimenting
>>>>>>
>>>>> org.apache.drill.exec.store.**hbase
>>>>>
>>>>>> package with following for local mode drill bit should suffice for POC?
>>>>>> HBASERecordReader.java
>>>>>> HBASEGroupScan
>>>>>> HBASEScanBatchCreator
>>>>>> HbaseSchemaProvider
>>>>>> HBASEStorageEngine
>>>>>> HBASEStorageEngineConfig
>>>>>> HBASESubScan
>>>>>>
>>>>>> D) What part of https://github.com/**forcedotcom/phoenix<
>>>>>>
>>>>> https://github.com/forcedotcom/phoenix>can we reuse? Any phonix
>>>>> contributors around?
>>>>>
>>>>>> -Dhruv
>>>>>>
>>>>>>