You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by "Sterk, Paul (Contractor)" <Pa...@searshc.com> on 2011/05/13 17:52:00 UTC

Web-based Hbase Query Tool?

Hi,

 

I am looking for a web based query tool that is able to query Hbase
tables and columns.  Can someone point me to any?  I want to be able to
create secondary indexes on Hbase tables then issue queries against
similar to Solr/Lucene. For example: 'find all transactions with
transaction number 9040' with a query such as
'TransactionMaster.transaction_nbr:9040'.  

 

I have found this tool so far:
http://sourceforge.net/projects/hbaseexplorer/

 

Please let me know if there are any others.

 

Thanks in advance,

Paul

This message, including any attachments, is the property of Sears Holdings Corporation and/or one of its subsidiaries. It is confidential and may contain proprietary or legally privileged information. If you are not the intended recipient, please delete it without reading the contents. Thank you.

RE: Web-based Hbase Query Tool?

Posted by "Sterk, Paul (Contractor)" <Pa...@searshc.com>.
Jason,

Thanks for the quick response. What I want to do is use a query tool that leverages some kind of Hbase secondary index to support row filtering based on columnfamily values (not rowid values).  This will enable queries similar to SELECT col1 FROM table1 WHERE col1 LIKE '%test%'. Currently, this is not supported by hbase, AFAIK.

Now, it is possible to create Hbase secondary indexes using the classes in org.apache.hadoop.hbase.client.tableindexed (see SingleColumnValueFilter).  This approach works and I can retrieve filter results using a basic java client.  The problem is that there is nothing layered on top of the client.  I need something like a JavaEE app with a web front end where a user could enter a query text string and leverage the above java client code to retrieve filtered rows from bhase.

Now, I have implemented a solution using Solr/Lucene as the indexing and query server[1].  See below for details.  The downside is my implementation involves two queries: one to the Solr index store and a second one to hbase.  I would prefer just one query directly to hbase.  

I hope this makes sense. Please let me know if your implementation seems to fit my requirements. I'll continue to read about what you have done.

Paul

[1] What I am looking for is close to this:

https://issues.apache.org/jira/browse/SOLR-1301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

This enables the creation of Hbase secondary indexes via Solr/Lucene.  The Solr indexes are stored in Solr's native storage format. I have used this module to index Hbase rows with the row id mapped to the Solr Document id.  Similarly, Hbase column families are mapped to Solr fields. This enables someone to lookup an Hbase column using a term query such as TransactionInfo.transaction_nbr:9043.

I have also created an implementation of a org.apache.solr.response.QueryResponseWriter. This class receives the results of the Solr query, looks up the Solr Documents, retrieves the all ids which map to the Hbase row ids, sorts them, and then executes a range scan on the Hbase table using startRow and stopRow.  This works reasonably well but it involves two queries: one to Solr and one to Hbase.

-----Original Message-----
From: Jason Rutherglen [mailto:jason.rutherglen@gmail.com] 
Sent: Friday, May 13, 2011 12:00 PM
To: user@hbase.apache.org
Subject: Re: Web-based Hbase Query Tool?

Paul,

I have a patch to implement Lucene in HBase - HBASE-3529.  It's a work
in progress limited primarily by time.

For a web UI I'll probably eventually integrate Solr at some point,
which would provide a basic UI, though it'd need to be modified to
account for column families.  Is this what you're looking for?

Jason

On Fri, May 13, 2011 at 8:52 AM, Sterk, Paul (Contractor)
<Pa...@searshc.com> wrote:
> Hi,
>
>
>
> I am looking for a web based query tool that is able to query Hbase
> tables and columns.  Can someone point me to any?  I want to be able to
> create secondary indexes on Hbase tables then issue queries against
> similar to Solr/Lucene. For example: 'find all transactions with
> transaction number 9040' with a query such as
> 'TransactionMaster.transaction_nbr:9040'.
>
>
>
> I have found this tool so far:
> http://sourceforge.net/projects/hbaseexplorer/
>
>
>
> Please let me know if there are any others.
>
>
>
> Thanks in advance,
>
> Paul
>
> This message, including any attachments, is the property of Sears Holdings Corporation and/or one of its subsidiaries. It is confidential and may contain proprietary or legally privileged information. If you are not the intended recipient, please delete it without reading the contents. Thank you.
>

This message, including any attachments, is the property of Sears Holdings Corporation and/or one of its subsidiaries. It is confidential and may contain proprietary or legally privileged information. If you are not the intended recipient, please delete it without reading the contents. Thank you.

Re: Web-based Hbase Query Tool?

Posted by Jason Rutherglen <ja...@gmail.com>.
I probably need to delete the patch that's up there, and I'm not sure
it needs to be updated regularly given the other dependencies.

On Fri, May 13, 2011 at 9:35 AM, highpointe <hi...@gmail.com> wrote:
> Jason,
>
> Where could one find that patch. Also your implementation of Solr to Hbase is definitely something I'd love to hear about.
>
> HP
> Sent from my iPhone
>
> On May 13, 2011, at 10:00 AM, Jason Rutherglen <ja...@gmail.com> wrote:
>
>> Paul,
>>
>> I have a patch to implement Lucene in HBase - HBASE-3529.  It's a work
>> in progress limited primarily by time.
>>
>> For a web UI I'll probably eventually integrate Solr at some point,
>> which would provide a basic UI, though it'd need to be modified to
>> account for column families.  Is this what you're looking for?
>>
>> Jason
>>
>> On Fri, May 13, 2011 at 8:52 AM, Sterk, Paul (Contractor)
>> <Pa...@searshc.com> wrote:
>>> Hi,
>>>
>>>
>>>
>>> I am looking for a web based query tool that is able to query Hbase
>>> tables and columns.  Can someone point me to any?  I want to be able to
>>> create secondary indexes on Hbase tables then issue queries against
>>> similar to Solr/Lucene. For example: 'find all transactions with
>>> transaction number 9040' with a query such as
>>> 'TransactionMaster.transaction_nbr:9040'.
>>>
>>>
>>>
>>> I have found this tool so far:
>>> http://sourceforge.net/projects/hbaseexplorer/
>>>
>>>
>>>
>>> Please let me know if there are any others.
>>>
>>>
>>>
>>> Thanks in advance,
>>>
>>> Paul
>>>
>>> This message, including any attachments, is the property of Sears Holdings Corporation and/or one of its subsidiaries. It is confidential and may contain proprietary or legally privileged information. If you are not the intended recipient, please delete it without reading the contents. Thank you.
>>>
>

Re: Web-based Hbase Query Tool?

Posted by Ted Yu <yu...@gmail.com>.
See Jason's comment in HBASE-3529 @ 18/Apr/11 02:33

On Fri, May 13, 2011 at 9:35 AM, highpointe <hi...@gmail.com> wrote:

> Jason,
>
> Where could one find that patch. Also your implementation of Solr to Hbase
> is definitely something I'd love to hear about.
>
> HP
> Sent from my iPhone
>
> On May 13, 2011, at 10:00 AM, Jason Rutherglen <ja...@gmail.com>
> wrote:
>
> > Paul,
> >
> > I have a patch to implement Lucene in HBase - HBASE-3529.  It's a work
> > in progress limited primarily by time.
> >
> > For a web UI I'll probably eventually integrate Solr at some point,
> > which would provide a basic UI, though it'd need to be modified to
> > account for column families.  Is this what you're looking for?
> >
> > Jason
> >
> > On Fri, May 13, 2011 at 8:52 AM, Sterk, Paul (Contractor)
> > <Pa...@searshc.com> wrote:
> >> Hi,
> >>
> >>
> >>
> >> I am looking for a web based query tool that is able to query Hbase
> >> tables and columns.  Can someone point me to any?  I want to be able to
> >> create secondary indexes on Hbase tables then issue queries against
> >> similar to Solr/Lucene. For example: 'find all transactions with
> >> transaction number 9040' with a query such as
> >> 'TransactionMaster.transaction_nbr:9040'.
> >>
> >>
> >>
> >> I have found this tool so far:
> >> http://sourceforge.net/projects/hbaseexplorer/
> >>
> >>
> >>
> >> Please let me know if there are any others.
> >>
> >>
> >>
> >> Thanks in advance,
> >>
> >> Paul
> >>
> >> This message, including any attachments, is the property of Sears
> Holdings Corporation and/or one of its subsidiaries. It is confidential and
> may contain proprietary or legally privileged information. If you are not
> the intended recipient, please delete it without reading the contents. Thank
> you.
> >>
>

Re: Web-based Hbase Query Tool?

Posted by Jason Rutherglen <ja...@gmail.com>.
I think Solr is completely external?
http://www.lilyproject.org/lily/about.html  Also there's the use of a
message queue it looks like.  I generally think that it's adding
additional complexity?

2011/5/13 Frédéric Fondement <fr...@uha.fr>:
> From what I understood, they use HBase as the datastore for Solr, so the
> grid should be robust.
> But I'm just a doc reader so far...
>
> Le 13/05/11 19:07, Jason Rutherglen a écrit :
>>
>> I think Lily implements search using Solr distributed search (eg,
>> external to HBase), so it's possible for a server to fail and then all
>> queries will fail?  Also then there's overhead from reading the WAL
>> from an external process?
>>
>> 2011/5/13 Frédéric Fondement<fr...@uha.fr>:
>>>
>>> Hello,
>>>
>>> I think that it's the purpose of lily (http://www.lilyproject.org/),
>>> though
>>> you have to use lily's API instead of HBase client API. I mean it's not
>>> intended to be plugged to existing HBase data. Am I wrong ?
>>>
>>> Le 13/05/11 18:35, highpointe a écrit :
>>>>
>>>> Jason,
>>>>
>>>> Where could one find that patch. Also your implementation of Solr to
>>>> Hbase
>>>> is definitely something I'd love to hear about.
>>>>
>>>> HP
>>>> Sent from my iPhone
>>>>
>>>> On May 13, 2011, at 10:00 AM, Jason
>>>> Rutherglen<ja...@gmail.com>
>>>>  wrote:
>>>>
>>>>> Paul,
>>>>>
>>>>> I have a patch to implement Lucene in HBase - HBASE-3529.  It's a work
>>>>> in progress limited primarily by time.
>>>>>
>>>>> For a web UI I'll probably eventually integrate Solr at some point,
>>>>> which would provide a basic UI, though it'd need to be modified to
>>>>> account for column families.  Is this what you're looking for?
>>>>>
>>>>> Jason
>>>>>
>>>>> On Fri, May 13, 2011 at 8:52 AM, Sterk, Paul (Contractor)
>>>>> <Pa...@searshc.com>    wrote:
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>>
>>>>>>
>>>>>> I am looking for a web based query tool that is able to query Hbase
>>>>>> tables and columns.  Can someone point me to any?  I want to be able
>>>>>> to
>>>>>> create secondary indexes on Hbase tables then issue queries against
>>>>>> similar to Solr/Lucene. For example: 'find all transactions with
>>>>>> transaction number 9040' with a query such as
>>>>>> 'TransactionMaster.transaction_nbr:9040'.
>>>>>>
>>>>>>
>>>>>>
>>>>>> I have found this tool so far:
>>>>>> http://sourceforge.net/projects/hbaseexplorer/
>>>>>>
>>>>>>
>>>>>>
>>>>>> Please let me know if there are any others.
>>>>>>
>>>>>>
>>>>>>
>>>>>> Thanks in advance,
>>>>>>
>>>>>> Paul
>>>>>>
>>>>>> This message, including any attachments, is the property of Sears
>>>>>> Holdings Corporation and/or one of its subsidiaries. It is
>>>>>> confidential and
>>>>>> may contain proprietary or legally privileged information. If you are
>>>>>> not
>>>>>> the intended recipient, please delete it without reading the contents.
>>>>>> Thank
>>>>>> you.
>>>>>>
>>>
>
>

Re: Web-based Hbase Query Tool?

Posted by Frédéric Fondement <fr...@uha.fr>.
 From what I understood, they use HBase as the datastore for Solr, so 
the grid should be robust.
But I'm just a doc reader so far...

Le 13/05/11 19:07, Jason Rutherglen a écrit :
> I think Lily implements search using Solr distributed search (eg,
> external to HBase), so it's possible for a server to fail and then all
> queries will fail?  Also then there's overhead from reading the WAL
> from an external process?
>
> 2011/5/13 Frédéric Fondement<fr...@uha.fr>:
>> Hello,
>>
>> I think that it's the purpose of lily (http://www.lilyproject.org/), though
>> you have to use lily's API instead of HBase client API. I mean it's not
>> intended to be plugged to existing HBase data. Am I wrong ?
>>
>> Le 13/05/11 18:35, highpointe a écrit :
>>> Jason,
>>>
>>> Where could one find that patch. Also your implementation of Solr to Hbase
>>> is definitely something I'd love to hear about.
>>>
>>> HP
>>> Sent from my iPhone
>>>
>>> On May 13, 2011, at 10:00 AM, Jason Rutherglen<ja...@gmail.com>
>>>   wrote:
>>>
>>>> Paul,
>>>>
>>>> I have a patch to implement Lucene in HBase - HBASE-3529.  It's a work
>>>> in progress limited primarily by time.
>>>>
>>>> For a web UI I'll probably eventually integrate Solr at some point,
>>>> which would provide a basic UI, though it'd need to be modified to
>>>> account for column families.  Is this what you're looking for?
>>>>
>>>> Jason
>>>>
>>>> On Fri, May 13, 2011 at 8:52 AM, Sterk, Paul (Contractor)
>>>> <Pa...@searshc.com>    wrote:
>>>>> Hi,
>>>>>
>>>>>
>>>>>
>>>>> I am looking for a web based query tool that is able to query Hbase
>>>>> tables and columns.  Can someone point me to any?  I want to be able to
>>>>> create secondary indexes on Hbase tables then issue queries against
>>>>> similar to Solr/Lucene. For example: 'find all transactions with
>>>>> transaction number 9040' with a query such as
>>>>> 'TransactionMaster.transaction_nbr:9040'.
>>>>>
>>>>>
>>>>>
>>>>> I have found this tool so far:
>>>>> http://sourceforge.net/projects/hbaseexplorer/
>>>>>
>>>>>
>>>>>
>>>>> Please let me know if there are any others.
>>>>>
>>>>>
>>>>>
>>>>> Thanks in advance,
>>>>>
>>>>> Paul
>>>>>
>>>>> This message, including any attachments, is the property of Sears
>>>>> Holdings Corporation and/or one of its subsidiaries. It is confidential and
>>>>> may contain proprietary or legally privileged information. If you are not
>>>>> the intended recipient, please delete it without reading the contents. Thank
>>>>> you.
>>>>>
>>


Re: Web-based Hbase Query Tool?

Posted by Jason Rutherglen <ja...@gmail.com>.
I think Lily implements search using Solr distributed search (eg,
external to HBase), so it's possible for a server to fail and then all
queries will fail?  Also then there's overhead from reading the WAL
from an external process?

2011/5/13 Frédéric Fondement <fr...@uha.fr>:
> Hello,
>
> I think that it's the purpose of lily (http://www.lilyproject.org/), though
> you have to use lily's API instead of HBase client API. I mean it's not
> intended to be plugged to existing HBase data. Am I wrong ?
>
> Le 13/05/11 18:35, highpointe a écrit :
>>
>> Jason,
>>
>> Where could one find that patch. Also your implementation of Solr to Hbase
>> is definitely something I'd love to hear about.
>>
>> HP
>> Sent from my iPhone
>>
>> On May 13, 2011, at 10:00 AM, Jason Rutherglen<ja...@gmail.com>
>>  wrote:
>>
>>> Paul,
>>>
>>> I have a patch to implement Lucene in HBase - HBASE-3529.  It's a work
>>> in progress limited primarily by time.
>>>
>>> For a web UI I'll probably eventually integrate Solr at some point,
>>> which would provide a basic UI, though it'd need to be modified to
>>> account for column families.  Is this what you're looking for?
>>>
>>> Jason
>>>
>>> On Fri, May 13, 2011 at 8:52 AM, Sterk, Paul (Contractor)
>>> <Pa...@searshc.com>  wrote:
>>>>
>>>> Hi,
>>>>
>>>>
>>>>
>>>> I am looking for a web based query tool that is able to query Hbase
>>>> tables and columns.  Can someone point me to any?  I want to be able to
>>>> create secondary indexes on Hbase tables then issue queries against
>>>> similar to Solr/Lucene. For example: 'find all transactions with
>>>> transaction number 9040' with a query such as
>>>> 'TransactionMaster.transaction_nbr:9040'.
>>>>
>>>>
>>>>
>>>> I have found this tool so far:
>>>> http://sourceforge.net/projects/hbaseexplorer/
>>>>
>>>>
>>>>
>>>> Please let me know if there are any others.
>>>>
>>>>
>>>>
>>>> Thanks in advance,
>>>>
>>>> Paul
>>>>
>>>> This message, including any attachments, is the property of Sears
>>>> Holdings Corporation and/or one of its subsidiaries. It is confidential and
>>>> may contain proprietary or legally privileged information. If you are not
>>>> the intended recipient, please delete it without reading the contents. Thank
>>>> you.
>>>>
>
>

Re: Web-based Hbase Query Tool?

Posted by Frédéric Fondement <fr...@uha.fr>.
Hello,

I think that it's the purpose of lily (http://www.lilyproject.org/), 
though you have to use lily's API instead of HBase client API. I mean 
it's not intended to be plugged to existing HBase data. Am I wrong ?

Le 13/05/11 18:35, highpointe a écrit :
> Jason,
>
> Where could one find that patch. Also your implementation of Solr to Hbase is definitely something I'd love to hear about.
>
> HP
> Sent from my iPhone
>
> On May 13, 2011, at 10:00 AM, Jason Rutherglen<ja...@gmail.com>  wrote:
>
>> Paul,
>>
>> I have a patch to implement Lucene in HBase - HBASE-3529.  It's a work
>> in progress limited primarily by time.
>>
>> For a web UI I'll probably eventually integrate Solr at some point,
>> which would provide a basic UI, though it'd need to be modified to
>> account for column families.  Is this what you're looking for?
>>
>> Jason
>>
>> On Fri, May 13, 2011 at 8:52 AM, Sterk, Paul (Contractor)
>> <Pa...@searshc.com>  wrote:
>>> Hi,
>>>
>>>
>>>
>>> I am looking for a web based query tool that is able to query Hbase
>>> tables and columns.  Can someone point me to any?  I want to be able to
>>> create secondary indexes on Hbase tables then issue queries against
>>> similar to Solr/Lucene. For example: 'find all transactions with
>>> transaction number 9040' with a query such as
>>> 'TransactionMaster.transaction_nbr:9040'.
>>>
>>>
>>>
>>> I have found this tool so far:
>>> http://sourceforge.net/projects/hbaseexplorer/
>>>
>>>
>>>
>>> Please let me know if there are any others.
>>>
>>>
>>>
>>> Thanks in advance,
>>>
>>> Paul
>>>
>>> This message, including any attachments, is the property of Sears Holdings Corporation and/or one of its subsidiaries. It is confidential and may contain proprietary or legally privileged information. If you are not the intended recipient, please delete it without reading the contents. Thank you.
>>>


Re: Web-based Hbase Query Tool?

Posted by highpointe <hi...@gmail.com>.
Jason,

Where could one find that patch. Also your implementation of Solr to Hbase is definitely something I'd love to hear about. 

HP
Sent from my iPhone

On May 13, 2011, at 10:00 AM, Jason Rutherglen <ja...@gmail.com> wrote:

> Paul,
> 
> I have a patch to implement Lucene in HBase - HBASE-3529.  It's a work
> in progress limited primarily by time.
> 
> For a web UI I'll probably eventually integrate Solr at some point,
> which would provide a basic UI, though it'd need to be modified to
> account for column families.  Is this what you're looking for?
> 
> Jason
> 
> On Fri, May 13, 2011 at 8:52 AM, Sterk, Paul (Contractor)
> <Pa...@searshc.com> wrote:
>> Hi,
>> 
>> 
>> 
>> I am looking for a web based query tool that is able to query Hbase
>> tables and columns.  Can someone point me to any?  I want to be able to
>> create secondary indexes on Hbase tables then issue queries against
>> similar to Solr/Lucene. For example: 'find all transactions with
>> transaction number 9040' with a query such as
>> 'TransactionMaster.transaction_nbr:9040'.
>> 
>> 
>> 
>> I have found this tool so far:
>> http://sourceforge.net/projects/hbaseexplorer/
>> 
>> 
>> 
>> Please let me know if there are any others.
>> 
>> 
>> 
>> Thanks in advance,
>> 
>> Paul
>> 
>> This message, including any attachments, is the property of Sears Holdings Corporation and/or one of its subsidiaries. It is confidential and may contain proprietary or legally privileged information. If you are not the intended recipient, please delete it without reading the contents. Thank you.
>> 

Re: Web-based Hbase Query Tool?

Posted by Jason Rutherglen <ja...@gmail.com>.
Paul,

I have a patch to implement Lucene in HBase - HBASE-3529.  It's a work
in progress limited primarily by time.

For a web UI I'll probably eventually integrate Solr at some point,
which would provide a basic UI, though it'd need to be modified to
account for column families.  Is this what you're looking for?

Jason

On Fri, May 13, 2011 at 8:52 AM, Sterk, Paul (Contractor)
<Pa...@searshc.com> wrote:
> Hi,
>
>
>
> I am looking for a web based query tool that is able to query Hbase
> tables and columns.  Can someone point me to any?  I want to be able to
> create secondary indexes on Hbase tables then issue queries against
> similar to Solr/Lucene. For example: 'find all transactions with
> transaction number 9040' with a query such as
> 'TransactionMaster.transaction_nbr:9040'.
>
>
>
> I have found this tool so far:
> http://sourceforge.net/projects/hbaseexplorer/
>
>
>
> Please let me know if there are any others.
>
>
>
> Thanks in advance,
>
> Paul
>
> This message, including any attachments, is the property of Sears Holdings Corporation and/or one of its subsidiaries. It is confidential and may contain proprietary or legally privileged information. If you are not the intended recipient, please delete it without reading the contents. Thank you.
>