why not introduce reverse index into hbase for multi-dimensional query

[张景鹏 <zh...@jd.com>]

why not introduce reverse index into hbase for multi-dimensional query ?  looks like sorl + hbase

Re: why not introduce reverse index into hbase for multi-dimensional query

[Michael Segel <mi...@hotmail.com>]

See:
http://www.infoq.com/articles/LuceneHbase

Note: We did this early in 2011 but couldn’t talk about it for a while. 

SOLR / Elastic Search would follow the same pattern. 

Note that depending on what you’re indexing, the size of the index(s)  could be larger than the base table by a couple of orders in magnitude. 

If you wanted to tie SOLR to HBase for an in memory index you have a decision to make. 
Do you update the index data in hbase and have an eventual consistency model where it will take some time x (variable and measured in minutes to hours) before the data is available to the index, or do you want to update the data in memory and then persist to hbase. 

We built the index and updated the index in HBase because we didn’t care about the eventual consistency. So we had to modify the flow of information. 

If you are writing to SOLR directly, then SOLR has to persist in to HBase, and then you will have to deal with the issue of if SOLR isn’t available what do you do with the data? 
(Assume that you could on error write to HBase.) 

Its definitely an option but you would also have to write the co-processor code to handle the index writes as you update the base table. 

HTH

-Mike

On Jul 16, 2014, at 5:51 AM, 张景鹏 <zh...@jd.com> wrote:

> Sorry ,yes it's Solr
> When I combin hbase with solr ，I find it have a good performance . I store the colums which should be query as condition and hbase rowkey in solr and all colums stored in hbase . so I think hbase can add the solr's(lucene) some index function as hbase two secondary index.
> I recommend you to look at this article which will have a help.  
> http://static.druid.io/docs/druid.pdf 
> 
> 
> 
> -----邮件原件-----
> 发件人: Ted Yu [mailto:yuzhihong@gmail.com] 
> 发送时间: 2014年7月16日 10:58
> 收件人: user@hbase.apache.org
> 主题: Re: why not introduce reverse index into hbase for multi-dimensional query
> 
> Can you describe reverse index in bit more detail ?
> 
> BTW sorl should be solr, right ?
> 
> 
> On Tue, Jul 15, 2014 at 7:42 PM, 张景鹏 <zh...@jd.com> wrote:
> 
>> why not introduce reverse index into hbase for multi-dimensional query ?
>> looks like sorl + hbase
>> 

Re: 答复: why not introduce reverse index into hbase for multi-dimensional query

[Rayees <mr...@gmail.com>]

Perhaps, this can be of interest for this:
http://www.lilyproject.org/lily/index.html


On Tue, Jul 15, 2014 at 9:51 PM, 张景鹏 <zh...@jd.com> wrote:

> Sorry ,yes it's Solr
> When I combin hbase with solr ，I find it have a good performance . I store
> the colums which should be query as condition and hbase rowkey in solr and
> all colums stored in hbase . so I think hbase can add the solr's(lucene)
> some index function as hbase two secondary index.
> I recommend you to look at this article which will have a help.
> http://static.druid.io/docs/druid.pdf
>
>
>
> -----邮件原件-----
> 发件人: Ted Yu [mailto:yuzhihong@gmail.com]
> 发送时间: 2014年7月16日 10:58
> 收件人: user@hbase.apache.org
> 主题: Re: why not introduce reverse index into hbase for multi-dimensional
> query
>
> Can you describe reverse index in bit more detail ?
>
> BTW sorl should be solr, right ?
>
>
> On Tue, Jul 15, 2014 at 7:42 PM, 张景鹏 <zh...@jd.com> wrote:
>
> > why not introduce reverse index into hbase for multi-dimensional query ?
> >  looks like sorl + hbase
> >
>

答复: why not introduce reverse index into hbase for multi-dimensional query

[张景鹏 <zh...@jd.com>]

Sorry ,yes it's Solr
When I combin hbase with solr ，I find it have a good performance . I store the colums which should be query as condition and hbase rowkey in solr and all colums stored in hbase . so I think hbase can add the solr's(lucene) some index function as hbase two secondary index.
I recommend you to look at this article which will have a help.  
http://static.druid.io/docs/druid.pdf 



-----邮件原件-----
发件人: Ted Yu [mailto:yuzhihong@gmail.com] 
发送时间: 2014年7月16日 10:58
收件人: user@hbase.apache.org
主题: Re: why not introduce reverse index into hbase for multi-dimensional query

Can you describe reverse index in bit more detail ?

BTW sorl should be solr, right ?


On Tue, Jul 15, 2014 at 7:42 PM, 张景鹏 <zh...@jd.com> wrote:

> why not introduce reverse index into hbase for multi-dimensional query ?
>  looks like sorl + hbase
>

Re: why not introduce reverse index into hbase for multi-dimensional query

[Ted Yu <yu...@gmail.com>]

Can you describe reverse index in bit more detail ?

BTW sorl should be solr, right ?


On Tue, Jul 15, 2014 at 7:42 PM, 张景鹏 <zh...@jd.com> wrote:

> why not introduce reverse index into hbase for multi-dimensional query ?
>  looks like sorl + hbase
>