You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@hive.apache.org by Ray Duong <ra...@gmail.com> on 2010/07/01 19:36:01 UTC

Hive-Hbase Key lookup w/o full scan

Is there away to do a hbase key lookup using the Hive-Hbase integration
without doing a full scan?

Since I'm specifying the key='foo' in the where condition, shouldn't it be a
fast lookup?

thanks,
-ray

Re: Hive-Hbase Key lookup w/o full scan

Posted by John Sichi <js...@facebook.com>.

Thanks Ray!

A number of people are interested in this; I will send out a breakdown of the work required, as well as some pointers for getting started on it.

JVS

On Jul 1, 2010, at 2:57 PM, Ray Duong wrote:

Thanks John,

Can you provide me with some pointers?.  My team can try to work on it.

Our workaround right now is to call the Thrift API from within Hive using a UDF.

Thanks,
-ray

On Thu, Jul 1, 2010 at 1:19 PM, John Sichi <js...@facebook.com>> wrote:
On Jul 1, 2010, at 10:36 AM, Ray Duong wrote:

> Is there away to do a hbase key lookup using the Hive-Hbase integration without doing a full scan?
>
> Since I'm specifying the key='foo' in the where condition, shouldn't it be a fast lookup?

Hi Ray,

Pushing down filters to HBase is one of our roadmap items.

https://issues.apache.org/jira/browse/HIVE-1226

If you'd like to work on it, let me know and I'll give you some pointers.

JVS

Re: Hive-Hbase Key lookup w/o full scan

Posted by John Sichi <js...@facebook.com>.

Hi Ray,

Apologies for my very slow response.

Here is a draft of a doc which explains how I think we can tackle this:

http://wiki.apache.org/hadoop/Hive/FilterPushdownDev

Maybe you can work on translation from ExprNodeDesc -> HBase scan object?  If you can get that working in isolation in unit tests, I can help with the rest of the parts for plumbing the filter through from Hive's optimizer.

JVS

On Jul 1, 2010, at 2:57 PM, Ray Duong wrote:

Thanks John,

Can you provide me with some pointers?.  My team can try to work on it.

Our workaround right now is to call the Thrift API from within Hive using a UDF.

Thanks,
-ray

On Thu, Jul 1, 2010 at 1:19 PM, John Sichi <js...@facebook.com>> wrote:
On Jul 1, 2010, at 10:36 AM, Ray Duong wrote:

> Is there away to do a hbase key lookup using the Hive-Hbase integration without doing a full scan?
>
> Since I'm specifying the key='foo' in the where condition, shouldn't it be a fast lookup?

Hi Ray,

Pushing down filters to HBase is one of our roadmap items.

https://issues.apache.org/jira/browse/HIVE-1226

If you'd like to work on it, let me know and I'll give you some pointers.

JVS

Re: Hive-Hbase Key lookup w/o full scan

Posted by Ray Duong <ra...@gmail.com>.

Thanks John,

Can you provide me with some pointers?.  My team can try to work on it.

Our workaround right now is to call the Thrift API from within Hive using a
UDF.

Thanks,
-ray

On Thu, Jul 1, 2010 at 1:19 PM, John Sichi <js...@facebook.com> wrote:

> On Jul 1, 2010, at 10:36 AM, Ray Duong wrote:
>
> > Is there away to do a hbase key lookup using the Hive-Hbase integration
> without doing a full scan?
> >
> > Since I'm specifying the key='foo' in the where condition, shouldn't it
> be a fast lookup?
>
> Hi Ray,
>
> Pushing down filters to HBase is one of our roadmap items.
>
> https://issues.apache.org/jira/browse/HIVE-1226
>
> If you'd like to work on it, let me know and I'll give you some pointers.
>
> JVS
>
>

Re: Hive-Hbase Key lookup w/o full scan

Posted by John Sichi <js...@facebook.com>.

On Jul 1, 2010, at 10:36 AM, Ray Duong wrote:

> Is there away to do a hbase key lookup using the Hive-Hbase integration without doing a full scan?
> 
> Since I'm specifying the key='foo' in the where condition, shouldn't it be a fast lookup?

Hi Ray,

Pushing down filters to HBase is one of our roadmap items.

https://issues.apache.org/jira/browse/HIVE-1226

If you'd like to work on it, let me know and I'll give you some pointers.

JVS