You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Anoop Sam John <an...@huawei.com> on 2012/12/04 09:10:32 UTC
HBase - Secondary Index
Hi All
Last week I got a chance to present the secondary indexing solution what we have done in Huawei at the China Hadoop Conference. You can see the presentation from http://hbtc2012.hadooper.cn/subject/track4Anoop%20Sam%20John2.pdf
I would like to hear what others think on this. :)
-Anoop-
RE: HBase - Secondary Index
Posted by Anoop Sam John <an...@huawei.com>.
David
Not using any existing library like Lucene. The index data of a table will be written in another HBase table.
-Anoop-
________________________________________
From: David Arthur [mumrah@gmail.com]
Sent: Thursday, December 20, 2012 8:17 AM
To: user@hbase.apache.org
Subject: Re: HBase - Secondary Index
Very cool design. Just curious, for the index did you write something
custom or using an existing library like Lucene?
-David
On 12/4/12 3:10 AM, Anoop Sam John wrote:
> Hi All
>
> Last week I got a chance to present the secondary indexing solution what we have done in Huawei at the China Hadoop Conference. You can see the presentation from http://hbtc2012.hadooper.cn/subject/track4Anoop%20Sam%20John2.pdf
>
>
>
> I would like to hear what others think on this. :)
>
>
>
> -Anoop-
>
Re: HBase - Secondary Index
Posted by David Arthur <mu...@gmail.com>.
Very cool design. Just curious, for the index did you write something
custom or using an existing library like Lucene?
-David
On 12/4/12 3:10 AM, Anoop Sam John wrote:
> Hi All
>
> Last week I got a chance to present the secondary indexing solution what we have done in Huawei at the China Hadoop Conference. You can see the presentation from http://hbtc2012.hadooper.cn/subject/track4Anoop%20Sam%20John2.pdf
>
>
>
> I would like to hear what others think on this. :)
>
>
>
> -Anoop-
>
Re: HBase - Secondary Index
Posted by Farah Karim <10...@seecs.edu.pk>.
Hi i am student of MS and i am doing my thesis work on Hadoop Mapreduce and
HBase. I have done major implementation of my work. Now I have to use
indexing to reduce the response time of query on hbase. I saw your
presentation and it seems great, I want you to make this as open source as
soon as possible. I welcome any other suggestions for indexing on HBase.
Thanks :)
On Thu, Dec 20, 2012 at 8:33 AM, Anoop Sam John <an...@huawei.com> wrote:
> Hi Nick, Andrew
> I am discussing this with the management here. Can tell more
> details after the new year.. Some people are not available due to Christmas
> & New year holidays :)
>
> -Anoop-
> ________________________________________
> From: Andrew Purtell [apurtell@apache.org]
> Sent: Wednesday, December 19, 2012 6:21 AM
> To: user@hbase.apache.org
> Cc: dev@hbase.apache.org
> Subject: Re: HBase - Secondary Index
>
> Hi Anoop,
>
> What Nick asked. I've also heard people wonder this out loud in a few
> places.
>
>
> On Tue, Dec 18, 2012 at 9:48 AM, Nick Dimiduk <nd...@gmail.com> wrote:
>
> > Hi Anoop,
> >
> > Your presentation has garnered quite a bit of community interest. Have
> you
> > considered providing your implementation to the community, perhaps in an
> > HBase-contrib module?
> >
> > Thanks,
> > Nick
> >
> > On Tue, Dec 4, 2012 at 12:10 AM, Anoop Sam John <an...@huawei.com>
> > wrote:
> >
> > > Hi All
> > >
> > > Last week I got a chance to present the secondary indexing
> > > solution what we have done in Huawei at the China Hadoop Conference.
> You
> > > can see the presentation from
> > > http://hbtc2012.hadooper.cn/subject/track4Anoop%20Sam%20John2.pdf
> > >
> > >
> > >
> > > I would like to hear what others think on this. :)
> > >
> > >
> > >
> > > -Anoop-
> > >
> >
>
>
>
> --
> Best regards,
>
> - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
>
Re: HBase - Secondary Index
Posted by Farah Karim <10...@seecs.edu.pk>.
Hi i am student of MS and i am doing my thesis work on Hadoop Mapreduce and
HBase. I have done major implementation of my work. Now I have to use
indexing to reduce the response time of query on hbase. I saw your
presentation and it seems great, I want you to make this as open source as
soon as possible. I welcome any other suggestions for indexing on HBase.
Thanks :)
On Thu, Dec 20, 2012 at 8:33 AM, Anoop Sam John <an...@huawei.com> wrote:
> Hi Nick, Andrew
> I am discussing this with the management here. Can tell more
> details after the new year.. Some people are not available due to Christmas
> & New year holidays :)
>
> -Anoop-
> ________________________________________
> From: Andrew Purtell [apurtell@apache.org]
> Sent: Wednesday, December 19, 2012 6:21 AM
> To: user@hbase.apache.org
> Cc: dev@hbase.apache.org
> Subject: Re: HBase - Secondary Index
>
> Hi Anoop,
>
> What Nick asked. I've also heard people wonder this out loud in a few
> places.
>
>
> On Tue, Dec 18, 2012 at 9:48 AM, Nick Dimiduk <nd...@gmail.com> wrote:
>
> > Hi Anoop,
> >
> > Your presentation has garnered quite a bit of community interest. Have
> you
> > considered providing your implementation to the community, perhaps in an
> > HBase-contrib module?
> >
> > Thanks,
> > Nick
> >
> > On Tue, Dec 4, 2012 at 12:10 AM, Anoop Sam John <an...@huawei.com>
> > wrote:
> >
> > > Hi All
> > >
> > > Last week I got a chance to present the secondary indexing
> > > solution what we have done in Huawei at the China Hadoop Conference.
> You
> > > can see the presentation from
> > > http://hbtc2012.hadooper.cn/subject/track4Anoop%20Sam%20John2.pdf
> > >
> > >
> > >
> > > I would like to hear what others think on this. :)
> > >
> > >
> > >
> > > -Anoop-
> > >
> >
>
>
>
> --
> Best regards,
>
> - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
>
Re: HBase - Secondary Index
Posted by Farah Karim <10...@seecs.edu.pk>.
Hi i am student of MS and i am doing my thesis work on Hadoop Mapreduce and
HBase. I have done major implementation of my work. Now I have to use
indexing to reduce the response time of query on hbase. I saw your
presentation and it seems great, I want you to make this as open source as
soon as possible. I welcome any other suggestions for indexing on HBase.
Thanks :)
On Thu, Dec 20, 2012 at 8:33 AM, Anoop Sam John <an...@huawei.com> wrote:
> Hi Nick, Andrew
> I am discussing this with the management here. Can tell more
> details after the new year.. Some people are not available due to Christmas
> & New year holidays :)
>
> -Anoop-
> ________________________________________
> From: Andrew Purtell [apurtell@apache.org]
> Sent: Wednesday, December 19, 2012 6:21 AM
> To: user@hbase.apache.org
> Cc: dev@hbase.apache.org
> Subject: Re: HBase - Secondary Index
>
> Hi Anoop,
>
> What Nick asked. I've also heard people wonder this out loud in a few
> places.
>
>
> On Tue, Dec 18, 2012 at 9:48 AM, Nick Dimiduk <nd...@gmail.com> wrote:
>
> > Hi Anoop,
> >
> > Your presentation has garnered quite a bit of community interest. Have
> you
> > considered providing your implementation to the community, perhaps in an
> > HBase-contrib module?
> >
> > Thanks,
> > Nick
> >
> > On Tue, Dec 4, 2012 at 12:10 AM, Anoop Sam John <an...@huawei.com>
> > wrote:
> >
> > > Hi All
> > >
> > > Last week I got a chance to present the secondary indexing
> > > solution what we have done in Huawei at the China Hadoop Conference.
> You
> > > can see the presentation from
> > > http://hbtc2012.hadooper.cn/subject/track4Anoop%20Sam%20John2.pdf
> > >
> > >
> > >
> > > I would like to hear what others think on this. :)
> > >
> > >
> > >
> > > -Anoop-
> > >
> >
>
>
>
> --
> Best regards,
>
> - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
>
RE: HBase - Secondary Index
Posted by Anoop Sam John <an...@huawei.com>.
Hi Nick, Andrew
I am discussing this with the management here. Can tell more details after the new year.. Some people are not available due to Christmas & New year holidays :)
-Anoop-
________________________________________
From: Andrew Purtell [apurtell@apache.org]
Sent: Wednesday, December 19, 2012 6:21 AM
To: user@hbase.apache.org
Cc: dev@hbase.apache.org
Subject: Re: HBase - Secondary Index
Hi Anoop,
What Nick asked. I've also heard people wonder this out loud in a few
places.
On Tue, Dec 18, 2012 at 9:48 AM, Nick Dimiduk <nd...@gmail.com> wrote:
> Hi Anoop,
>
> Your presentation has garnered quite a bit of community interest. Have you
> considered providing your implementation to the community, perhaps in an
> HBase-contrib module?
>
> Thanks,
> Nick
>
> On Tue, Dec 4, 2012 at 12:10 AM, Anoop Sam John <an...@huawei.com>
> wrote:
>
> > Hi All
> >
> > Last week I got a chance to present the secondary indexing
> > solution what we have done in Huawei at the China Hadoop Conference. You
> > can see the presentation from
> > http://hbtc2012.hadooper.cn/subject/track4Anoop%20Sam%20John2.pdf
> >
> >
> >
> > I would like to hear what others think on this. :)
> >
> >
> >
> > -Anoop-
> >
>
--
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)
RE: HBase - Secondary Index
Posted by Anoop Sam John <an...@huawei.com>.
Hi Nick, Andrew
I am discussing this with the management here. Can tell more details after the new year.. Some people are not available due to Christmas & New year holidays :)
-Anoop-
________________________________________
From: Andrew Purtell [apurtell@apache.org]
Sent: Wednesday, December 19, 2012 6:21 AM
To: user@hbase.apache.org
Cc: dev@hbase.apache.org
Subject: Re: HBase - Secondary Index
Hi Anoop,
What Nick asked. I've also heard people wonder this out loud in a few
places.
On Tue, Dec 18, 2012 at 9:48 AM, Nick Dimiduk <nd...@gmail.com> wrote:
> Hi Anoop,
>
> Your presentation has garnered quite a bit of community interest. Have you
> considered providing your implementation to the community, perhaps in an
> HBase-contrib module?
>
> Thanks,
> Nick
>
> On Tue, Dec 4, 2012 at 12:10 AM, Anoop Sam John <an...@huawei.com>
> wrote:
>
> > Hi All
> >
> > Last week I got a chance to present the secondary indexing
> > solution what we have done in Huawei at the China Hadoop Conference. You
> > can see the presentation from
> > http://hbtc2012.hadooper.cn/subject/track4Anoop%20Sam%20John2.pdf
> >
> >
> >
> > I would like to hear what others think on this. :)
> >
> >
> >
> > -Anoop-
> >
>
--
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)
Re: HBase - Secondary Index
Posted by ramkrishna vasudevan <ra...@gmail.com>.
Hi Anoop
Its great to see people accepting this design. Hope it comes out to the
contrib.
Very happy to see positive comments.
Regards
Ram
On Wed, Dec 19, 2012 at 6:21 AM, Andrew Purtell <ap...@apache.org> wrote:
> Hi Anoop,
>
> What Nick asked. I've also heard people wonder this out loud in a few
> places.
>
>
> On Tue, Dec 18, 2012 at 9:48 AM, Nick Dimiduk <nd...@gmail.com> wrote:
>
> > Hi Anoop,
> >
> > Your presentation has garnered quite a bit of community interest. Have
> you
> > considered providing your implementation to the community, perhaps in an
> > HBase-contrib module?
> >
> > Thanks,
> > Nick
> >
> > On Tue, Dec 4, 2012 at 12:10 AM, Anoop Sam John <an...@huawei.com>
> > wrote:
> >
> > > Hi All
> > >
> > > Last week I got a chance to present the secondary indexing
> > > solution what we have done in Huawei at the China Hadoop Conference.
> You
> > > can see the presentation from
> > > http://hbtc2012.hadooper.cn/subject/track4Anoop%20Sam%20John2.pdf
> > >
> > >
> > >
> > > I would like to hear what others think on this. :)
> > >
> > >
> > >
> > > -Anoop-
> > >
> >
>
>
>
> --
> Best regards,
>
> - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
>
Re: HBase - Secondary Index
Posted by Andrew Purtell <ap...@apache.org>.
Hi Anoop,
What Nick asked. I've also heard people wonder this out loud in a few
places.
On Tue, Dec 18, 2012 at 9:48 AM, Nick Dimiduk <nd...@gmail.com> wrote:
> Hi Anoop,
>
> Your presentation has garnered quite a bit of community interest. Have you
> considered providing your implementation to the community, perhaps in an
> HBase-contrib module?
>
> Thanks,
> Nick
>
> On Tue, Dec 4, 2012 at 12:10 AM, Anoop Sam John <an...@huawei.com>
> wrote:
>
> > Hi All
> >
> > Last week I got a chance to present the secondary indexing
> > solution what we have done in Huawei at the China Hadoop Conference. You
> > can see the presentation from
> > http://hbtc2012.hadooper.cn/subject/track4Anoop%20Sam%20John2.pdf
> >
> >
> >
> > I would like to hear what others think on this. :)
> >
> >
> >
> > -Anoop-
> >
>
--
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)
Re: HBase - Secondary Index
Posted by Andrew Purtell <ap...@apache.org>.
Hi Anoop,
What Nick asked. I've also heard people wonder this out loud in a few
places.
On Tue, Dec 18, 2012 at 9:48 AM, Nick Dimiduk <nd...@gmail.com> wrote:
> Hi Anoop,
>
> Your presentation has garnered quite a bit of community interest. Have you
> considered providing your implementation to the community, perhaps in an
> HBase-contrib module?
>
> Thanks,
> Nick
>
> On Tue, Dec 4, 2012 at 12:10 AM, Anoop Sam John <an...@huawei.com>
> wrote:
>
> > Hi All
> >
> > Last week I got a chance to present the secondary indexing
> > solution what we have done in Huawei at the China Hadoop Conference. You
> > can see the presentation from
> > http://hbtc2012.hadooper.cn/subject/track4Anoop%20Sam%20John2.pdf
> >
> >
> >
> > I would like to hear what others think on this. :)
> >
> >
> >
> > -Anoop-
> >
>
--
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)
Re: HBase - Secondary Index
Posted by Nick Dimiduk <nd...@gmail.com>.
Hi Anoop,
Your presentation has garnered quite a bit of community interest. Have you
considered providing your implementation to the community, perhaps in an
HBase-contrib module?
Thanks,
Nick
On Tue, Dec 4, 2012 at 12:10 AM, Anoop Sam John <an...@huawei.com> wrote:
> Hi All
>
> Last week I got a chance to present the secondary indexing
> solution what we have done in Huawei at the China Hadoop Conference. You
> can see the presentation from
> http://hbtc2012.hadooper.cn/subject/track4Anoop%20Sam%20John2.pdf
>
>
>
> I would like to hear what others think on this. :)
>
>
>
> -Anoop-
>
Re: HBase - Secondary Index
Posted by Anoop John <an...@gmail.com>.
Hi All
Thanks for the look and your concerns and comments..
@Jon Yes the consistency is maintained now. As the put to both the tables
in the same RS this was comparably easy job..More granular details on how
this is being done , I can share in later mails.
Yes as per Jon when the cluster is really big with 100s of RSs, for the
scan it need to visit every region in RSs. But may be from some regions no
data with the condition present and the scan will end immediately.. We
were also thinking about ways to add some custom blooms so that a special
bloom on the index region can tell whether a column value is present in
that region or not. Didn't try with this and the perf comparison.. Now we
are in the process of profiling and fine tuning.. What we felt is that
the per region indexing would be still better than the full table scan..
Also one main reason for not doing the client based approach where a global
ordered indexing will be there, is the put performance.. In the past when
we have done a fully client based solution ( something like Lily) the put
performance was not meeting our goal as one put was potentially making many
puts.. In our customer case some tables having upto 5 indices...
@Ted, at production level it is yet to get used.. The apps using the
indexing is under the way now...
Good to hear from all so that we can also get more scenarios and concerns
and opinions from experts..:)
-Anoop-
On Thu, Dec 6, 2012 at 3:28 AM, Doug Meil <do...@explorysmedical.com>wrote:
>
> re: "It seems everybody wants secondary indexes in HBase. The problem is
> that most folks don't agree what that actually means."
>
> Agreed. And often times when people say "secondary indexing" they may be
> talking about an access path on something that isn't represented in the
> key. So you'd not just need a "row key schema" (which is what we were
> talking about in HBASE-7221), but a schema for the columns too.
>
>
>
>
>
>
> On 12/5/12 4:03 PM, "lars hofhansl" <lh...@yahoo.com> wrote:
>
> >I'd say "it depends".
> >It seems everybody wants secondary indexes in HBase. The problem is that
> >most folks don't agree what that actually means.
> >
> >The most interesting problem and discussion point (IMHO) is that HBase
> >would need some form of schema description.
> >
> >See also HBASE-7221. Assuming we had something like that, it could be a
> >building block for a simple built-in secondary index solution.
> >
> >In the end you're probably right and we cannot prescribe a single
> >secondary index solution that will fit all use cases.
> >
> >-- Lars
> >
> >
> >----- Original Message -----
> >From: Jonathan Hsieh <jo...@cloudera.com>
> >To: dev@hbase.apache.org
> >Cc:
> >Sent: Wednesday, December 5, 2012 11:23 AM
> >Subject: Re: HBase - Secondary Index
> >
> >I personally feel that we should only add the primitive apis necessary to
> >make this work (and if non are required, all the better!), and to try to
> >keep secondary indexing work it as a separate project on top of hbase
> >because there are many possible valid architectures and implementations.
> >
> >I have two question areas -- one touched upon in previous messages -- what
> >logging and how do we get consistency guarantee's do we get bewteen the
> >index and primary?
> >
> >The other has to do with scalability. I'm not sure I interpreted the
> >slides
> >correctly, but from the slides 8 and 13, is the architecture such that
> >each
> >primary table region has a corresponding index table region?
> >
> >Is slide 14 a comparison of a full table scan vs the indexed lookups on 4
> >rs's? What happens if we go up to 20, or 100 rs's? If I'm right about
> >the
> >per index region per table region, I have a feeling this isn't going to
> >scale well with a large number of regions (since it would potentially have
> >to talk to each region and essentially every region server).
> >
> >Jon.
> >
> >On Wed, Dec 5, 2012 at 9:54 AM, Anoop John <an...@gmail.com> wrote:
> >
> >> I mean HBase devs to work on having sec indexing available with the
> >>HBase
> >> distribution... Now I guess many users of HBase implement different
> >>kinds
> >> of sec indexing ..:)
> >>
> >> We @Huawei would be happy to provide our support in it as per the
> >>interest
> >> from the community.. :)
> >>
> >> -Anoop-
> >>
> >> On Wed, Dec 5, 2012 at 9:29 PM, Andrey Stepachev <oc...@gmail.com>
> >>wrote:
> >>
> >> > Can you explain, what you mean under 'HBase community version with sec
> >> > indexing in it'. You will wait, until someone implements the same
> >> algorithm
> >> > in trunk hbase, or?
> >> >
> >> >
> >> > On Wed, Dec 5, 2012 at 4:55 PM, Anoop Sam John <an...@huawei.com>
> >> wrote:
> >> >
> >> > > No this is not open sourced yet.. As per the interest from the
> >>HBase
> >> > > community we can think of contributing..
> >> > > It is time to see HBase community version with sec indexing in it
> >> (IMHO)
> >> > > :)
> >> > >
> >> > > -Anoop-
> >> > >
> >> > > ________________________________________
> >> > > From: Andrey Stepachev [octo47@gmail.com]
> >> > > Sent: Wednesday, December 05, 2012 5:22 PM
> >> > > To: dev@hbase.apache.org
> >> > > Subject: Re: HBase - Secondary Index
> >> > >
> >> > > Hi.
> >> > >
> >> > > Indexing solution looks tempting.
> >> > > Are there any plans to open source your solution (or it already open
> >> and
> >> > I
> >> > > can't find it?).
> >> > >
> >> > >
> >> > >
> >> > >
> >> > > On Tue, Dec 4, 2012 at 12:10 PM, Anoop Sam John <anoopsj@huawei.com
> >
> >> > > wrote:
> >> > >
> >> > > > Hi All
> >> > > >
> >> > > > Last week I got a chance to present the secondary
> >> indexing
> >> > > > solution what we have done in Huawei at the China Hadoop
> >>Conference.
> >> > You
> >> > > > can see the presentation from
> >> > > > http://hbtc2012.hadooper.cn/subject/track4Anoop%20Sam%20John2.pdf
> >> > > >
> >> > > >
> >> > > >
> >> > > > I would like to hear what others think on this. :)
> >> > > >
> >> > > >
> >> > > >
> >> > > > -Anoop-
> >> > > >
> >> > >
> >> > >
> >> > >
> >> > > --
> >> > > Andrey.
> >> > >
> >> >
> >> >
> >> >
> >> > --
> >> > Andrey.
> >> >
> >>
> >
> >
> >
> >--
> >// Jonathan Hsieh (shay)
> >// Software Engineer, Cloudera
> >// jon@cloudera.com
> >
> >
>
>
>
Re: HBase - Secondary Index
Posted by ramkrishna vasudevan <ra...@gmail.com>.
Yes there is a one to one mapping, but even if 5 indices are there still we
will have only all the indices data in that same region.
May be more perf testing with more regions is needed.
Anoop, need to tell something on the range scans?
Regards
Ram
On Thu, Dec 6, 2012 at 10:27 AM, Anoop Sam John <an...@huawei.com> wrote:
> >I'd guess that would look like a coprocessor based solution, so
> there's no cost to those who don't want to use it, and minimal changes
> to core code.
>
> Yes Andrew, This was the main consideration from our side when we were
> doing the design of the secondary indexing solution here. Plus the minimum
> possible degradation for the Put as our customer was very much on to it :)
>
> >Everyone asks for it. All the time.
> Yes this is what we also been hearing from our customers. :)
>
> -Anoop-
> ________________________________________
> From: Andrew Purtell [apurtell@apache.org]
> Sent: Thursday, December 06, 2012 10:11 AM
> To: dev@hbase.apache.org
> Subject: Re: HBase - Secondary Index
>
> We won't find a secondary indexing scheme for HBase that will fit all
> use cases.
>
> However I am becoming convinced we should distribute as part of the
> distribution something good enough for some subset of simple/common
> use cases, so users have *something* to try, and can play around it.
> Everyone asks for it. All the time.
>
> I'd guess that would look like a coprocessor based solution, so
> there's no cost to those who don't want to use it, and minimal changes
> to core code.
>
> Maybe it would suit them, or if not at least they will have enough
> experience having played around with it to decide why they might need
> to do something else and what that might look like for their use case.
>
> On 12/6/12, lars hofhansl <lh...@yahoo.com> wrote:
> > I'd say "it depends".
> > It seems everybody wants secondary indexes in HBase. The problem is that
> > most folks don't agree what that actually means.
> >
> > The most interesting problem and discussion point (IMHO) is that HBase
> would
> > need some form of schema description.
> >
> > See also HBASE-7221. Assuming we had something like that, it could be a
> > building block for a simple built-in secondary index solution.
> >
> > In the end you're probably right and we cannot prescribe a single
> secondary
> > index solution that will fit all use cases.
> >
> > -- Lars
> >
> >
> > ----- Original Message -----
> > From: Jonathan Hsieh <jo...@cloudera.com>
> > To: dev@hbase.apache.org
> > Cc:
> > Sent: Wednesday, December 5, 2012 11:23 AM
> > Subject: Re: HBase - Secondary Index
> >
> > I personally feel that we should only add the primitive apis necessary to
> > make this work (and if non are required, all the better!), and to try to
> > keep secondary indexing work it as a separate project on top of hbase
> > because there are many possible valid architectures and implementations.
> >
> > I have two question areas -- one touched upon in previous messages --
> what
> > logging and how do we get consistency guarantee's do we get bewteen the
> > index and primary?
> >
> > The other has to do with scalability. I'm not sure I interpreted the
> slides
> > correctly, but from the slides 8 and 13, is the architecture such that
> each
> > primary table region has a corresponding index table region?
> >
> > Is slide 14 a comparison of a full table scan vs the indexed lookups on 4
> > rs's? What happens if we go up to 20, or 100 rs's? If I'm right about
> the
> > per index region per table region, I have a feeling this isn't going to
> > scale well with a large number of regions (since it would potentially
> have
> > to talk to each region and essentially every region server).
> >
> > Jon.
> >
> > On Wed, Dec 5, 2012 at 9:54 AM, Anoop John <an...@gmail.com>
> wrote:
> >
> >> I mean HBase devs to work on having sec indexing available with the
> HBase
> >> distribution... Now I guess many users of HBase implement different
> kinds
> >> of sec indexing ..:)
> >>
> >> We @Huawei would be happy to provide our support in it as per the
> interest
> >> from the community.. :)
> >>
> >> -Anoop-
> >>
> >> On Wed, Dec 5, 2012 at 9:29 PM, Andrey Stepachev <oc...@gmail.com>
> wrote:
> >>
> >> > Can you explain, what you mean under 'HBase community version with sec
> >> > indexing in it'. You will wait, until someone implements the same
> >> algorithm
> >> > in trunk hbase, or?
> >> >
> >> >
> >> > On Wed, Dec 5, 2012 at 4:55 PM, Anoop Sam John <an...@huawei.com>
> >> wrote:
> >> >
> >> > > No this is not open sourced yet.. As per the interest from the
> HBase
> >> > > community we can think of contributing..
> >> > > It is time to see HBase community version with sec indexing in it
> >> (IMHO)
> >> > > :)
> >> > >
> >> > > -Anoop-
> >> > >
> >> > > ________________________________________
> >> > > From: Andrey Stepachev [octo47@gmail.com]
> >> > > Sent: Wednesday, December 05, 2012 5:22 PM
> >> > > To: dev@hbase.apache.org
> >> > > Subject: Re: HBase - Secondary Index
> >> > >
> >> > > Hi.
> >> > >
> >> > > Indexing solution looks tempting.
> >> > > Are there any plans to open source your solution (or it already open
> >> and
> >> > I
> >> > > can't find it?).
> >> > >
> >> > >
> >> > >
> >> > >
> >> > > On Tue, Dec 4, 2012 at 12:10 PM, Anoop Sam John <anoopsj@huawei.com
> >
> >> > > wrote:
> >> > >
> >> > > > Hi All
> >> > > >
> >> > > > Last week I got a chance to present the secondary
> >> indexing
> >> > > > solution what we have done in Huawei at the China Hadoop
> Conference.
> >> > You
> >> > > > can see the presentation from
> >> > > > http://hbtc2012.hadooper.cn/subject/track4Anoop%20Sam%20John2.pdf
> >> > > >
> >> > > >
> >> > > >
> >> > > > I would like to hear what others think on this. :)
> >> > > >
> >> > > >
> >> > > >
> >> > > > -Anoop-
> >> > > >
> >> > >
> >> > >
> >> > >
> >> > > --
> >> > > Andrey.
> >> > >
> >> >
> >> >
> >> >
> >> > --
> >> > Andrey.
> >> >
> >>
> >
> >
> >
> > --
> > // Jonathan Hsieh (shay)
> > // Software Engineer, Cloudera
> > // jon@cloudera.com
> >
> >
>
RE: HBase - Secondary Index
Posted by Anoop Sam John <an...@huawei.com>.
>I'd guess that would look like a coprocessor based solution, so
there's no cost to those who don't want to use it, and minimal changes
to core code.
Yes Andrew, This was the main consideration from our side when we were doing the design of the secondary indexing solution here. Plus the minimum possible degradation for the Put as our customer was very much on to it :)
>Everyone asks for it. All the time.
Yes this is what we also been hearing from our customers. :)
-Anoop-
________________________________________
From: Andrew Purtell [apurtell@apache.org]
Sent: Thursday, December 06, 2012 10:11 AM
To: dev@hbase.apache.org
Subject: Re: HBase - Secondary Index
We won't find a secondary indexing scheme for HBase that will fit all
use cases.
However I am becoming convinced we should distribute as part of the
distribution something good enough for some subset of simple/common
use cases, so users have *something* to try, and can play around it.
Everyone asks for it. All the time.
I'd guess that would look like a coprocessor based solution, so
there's no cost to those who don't want to use it, and minimal changes
to core code.
Maybe it would suit them, or if not at least they will have enough
experience having played around with it to decide why they might need
to do something else and what that might look like for their use case.
On 12/6/12, lars hofhansl <lh...@yahoo.com> wrote:
> I'd say "it depends".
> It seems everybody wants secondary indexes in HBase. The problem is that
> most folks don't agree what that actually means.
>
> The most interesting problem and discussion point (IMHO) is that HBase would
> need some form of schema description.
>
> See also HBASE-7221. Assuming we had something like that, it could be a
> building block for a simple built-in secondary index solution.
>
> In the end you're probably right and we cannot prescribe a single secondary
> index solution that will fit all use cases.
>
> -- Lars
>
>
> ----- Original Message -----
> From: Jonathan Hsieh <jo...@cloudera.com>
> To: dev@hbase.apache.org
> Cc:
> Sent: Wednesday, December 5, 2012 11:23 AM
> Subject: Re: HBase - Secondary Index
>
> I personally feel that we should only add the primitive apis necessary to
> make this work (and if non are required, all the better!), and to try to
> keep secondary indexing work it as a separate project on top of hbase
> because there are many possible valid architectures and implementations.
>
> I have two question areas -- one touched upon in previous messages -- what
> logging and how do we get consistency guarantee's do we get bewteen the
> index and primary?
>
> The other has to do with scalability. I'm not sure I interpreted the slides
> correctly, but from the slides 8 and 13, is the architecture such that each
> primary table region has a corresponding index table region?
>
> Is slide 14 a comparison of a full table scan vs the indexed lookups on 4
> rs's? What happens if we go up to 20, or 100 rs's? If I'm right about the
> per index region per table region, I have a feeling this isn't going to
> scale well with a large number of regions (since it would potentially have
> to talk to each region and essentially every region server).
>
> Jon.
>
> On Wed, Dec 5, 2012 at 9:54 AM, Anoop John <an...@gmail.com> wrote:
>
>> I mean HBase devs to work on having sec indexing available with the HBase
>> distribution... Now I guess many users of HBase implement different kinds
>> of sec indexing ..:)
>>
>> We @Huawei would be happy to provide our support in it as per the interest
>> from the community.. :)
>>
>> -Anoop-
>>
>> On Wed, Dec 5, 2012 at 9:29 PM, Andrey Stepachev <oc...@gmail.com> wrote:
>>
>> > Can you explain, what you mean under 'HBase community version with sec
>> > indexing in it'. You will wait, until someone implements the same
>> algorithm
>> > in trunk hbase, or?
>> >
>> >
>> > On Wed, Dec 5, 2012 at 4:55 PM, Anoop Sam John <an...@huawei.com>
>> wrote:
>> >
>> > > No this is not open sourced yet.. As per the interest from the HBase
>> > > community we can think of contributing..
>> > > It is time to see HBase community version with sec indexing in it
>> (IMHO)
>> > > :)
>> > >
>> > > -Anoop-
>> > >
>> > > ________________________________________
>> > > From: Andrey Stepachev [octo47@gmail.com]
>> > > Sent: Wednesday, December 05, 2012 5:22 PM
>> > > To: dev@hbase.apache.org
>> > > Subject: Re: HBase - Secondary Index
>> > >
>> > > Hi.
>> > >
>> > > Indexing solution looks tempting.
>> > > Are there any plans to open source your solution (or it already open
>> and
>> > I
>> > > can't find it?).
>> > >
>> > >
>> > >
>> > >
>> > > On Tue, Dec 4, 2012 at 12:10 PM, Anoop Sam John <an...@huawei.com>
>> > > wrote:
>> > >
>> > > > Hi All
>> > > >
>> > > > Last week I got a chance to present the secondary
>> indexing
>> > > > solution what we have done in Huawei at the China Hadoop Conference.
>> > You
>> > > > can see the presentation from
>> > > > http://hbtc2012.hadooper.cn/subject/track4Anoop%20Sam%20John2.pdf
>> > > >
>> > > >
>> > > >
>> > > > I would like to hear what others think on this. :)
>> > > >
>> > > >
>> > > >
>> > > > -Anoop-
>> > > >
>> > >
>> > >
>> > >
>> > > --
>> > > Andrey.
>> > >
>> >
>> >
>> >
>> > --
>> > Andrey.
>> >
>>
>
>
>
> --
> // Jonathan Hsieh (shay)
> // Software Engineer, Cloudera
> // jon@cloudera.com
>
>
Re: HBase - Secondary Index
Posted by Andrew Purtell <ap...@apache.org>.
We won't find a secondary indexing scheme for HBase that will fit all
use cases.
However I am becoming convinced we should distribute as part of the
distribution something good enough for some subset of simple/common
use cases, so users have *something* to try, and can play around it.
Everyone asks for it. All the time.
I'd guess that would look like a coprocessor based solution, so
there's no cost to those who don't want to use it, and minimal changes
to core code.
Maybe it would suit them, or if not at least they will have enough
experience having played around with it to decide why they might need
to do something else and what that might look like for their use case.
On 12/6/12, lars hofhansl <lh...@yahoo.com> wrote:
> I'd say "it depends".
> It seems everybody wants secondary indexes in HBase. The problem is that
> most folks don't agree what that actually means.
>
> The most interesting problem and discussion point (IMHO) is that HBase would
> need some form of schema description.
>
> See also HBASE-7221. Assuming we had something like that, it could be a
> building block for a simple built-in secondary index solution.
>
> In the end you're probably right and we cannot prescribe a single secondary
> index solution that will fit all use cases.
>
> -- Lars
>
>
> ----- Original Message -----
> From: Jonathan Hsieh <jo...@cloudera.com>
> To: dev@hbase.apache.org
> Cc:
> Sent: Wednesday, December 5, 2012 11:23 AM
> Subject: Re: HBase - Secondary Index
>
> I personally feel that we should only add the primitive apis necessary to
> make this work (and if non are required, all the better!), and to try to
> keep secondary indexing work it as a separate project on top of hbase
> because there are many possible valid architectures and implementations.
>
> I have two question areas -- one touched upon in previous messages -- what
> logging and how do we get consistency guarantee's do we get bewteen the
> index and primary?
>
> The other has to do with scalability. I'm not sure I interpreted the slides
> correctly, but from the slides 8 and 13, is the architecture such that each
> primary table region has a corresponding index table region?
>
> Is slide 14 a comparison of a full table scan vs the indexed lookups on 4
> rs's? What happens if we go up to 20, or 100 rs's? If I'm right about the
> per index region per table region, I have a feeling this isn't going to
> scale well with a large number of regions (since it would potentially have
> to talk to each region and essentially every region server).
>
> Jon.
>
> On Wed, Dec 5, 2012 at 9:54 AM, Anoop John <an...@gmail.com> wrote:
>
>> I mean HBase devs to work on having sec indexing available with the HBase
>> distribution... Now I guess many users of HBase implement different kinds
>> of sec indexing ..:)
>>
>> We @Huawei would be happy to provide our support in it as per the interest
>> from the community.. :)
>>
>> -Anoop-
>>
>> On Wed, Dec 5, 2012 at 9:29 PM, Andrey Stepachev <oc...@gmail.com> wrote:
>>
>> > Can you explain, what you mean under 'HBase community version with sec
>> > indexing in it'. You will wait, until someone implements the same
>> algorithm
>> > in trunk hbase, or?
>> >
>> >
>> > On Wed, Dec 5, 2012 at 4:55 PM, Anoop Sam John <an...@huawei.com>
>> wrote:
>> >
>> > > No this is not open sourced yet.. As per the interest from the HBase
>> > > community we can think of contributing..
>> > > It is time to see HBase community version with sec indexing in it
>> (IMHO)
>> > > :)
>> > >
>> > > -Anoop-
>> > >
>> > > ________________________________________
>> > > From: Andrey Stepachev [octo47@gmail.com]
>> > > Sent: Wednesday, December 05, 2012 5:22 PM
>> > > To: dev@hbase.apache.org
>> > > Subject: Re: HBase - Secondary Index
>> > >
>> > > Hi.
>> > >
>> > > Indexing solution looks tempting.
>> > > Are there any plans to open source your solution (or it already open
>> and
>> > I
>> > > can't find it?).
>> > >
>> > >
>> > >
>> > >
>> > > On Tue, Dec 4, 2012 at 12:10 PM, Anoop Sam John <an...@huawei.com>
>> > > wrote:
>> > >
>> > > > Hi All
>> > > >
>> > > > Last week I got a chance to present the secondary
>> indexing
>> > > > solution what we have done in Huawei at the China Hadoop Conference.
>> > You
>> > > > can see the presentation from
>> > > > http://hbtc2012.hadooper.cn/subject/track4Anoop%20Sam%20John2.pdf
>> > > >
>> > > >
>> > > >
>> > > > I would like to hear what others think on this. :)
>> > > >
>> > > >
>> > > >
>> > > > -Anoop-
>> > > >
>> > >
>> > >
>> > >
>> > > --
>> > > Andrey.
>> > >
>> >
>> >
>> >
>> > --
>> > Andrey.
>> >
>>
>
>
>
> --
> // Jonathan Hsieh (shay)
> // Software Engineer, Cloudera
> // jon@cloudera.com
>
>
Re: HBase - Secondary Index
Posted by Doug Meil <do...@explorysmedical.com>.
re: "It seems everybody wants secondary indexes in HBase. The problem is
that most folks don't agree what that actually means."
Agreed. And often times when people say "secondary indexing" they may be
talking about an access path on something that isn't represented in the
key. So you'd not just need a "row key schema" (which is what we were
talking about in HBASE-7221), but a schema for the columns too.
On 12/5/12 4:03 PM, "lars hofhansl" <lh...@yahoo.com> wrote:
>I'd say "it depends".
>It seems everybody wants secondary indexes in HBase. The problem is that
>most folks don't agree what that actually means.
>
>The most interesting problem and discussion point (IMHO) is that HBase
>would need some form of schema description.
>
>See also HBASE-7221. Assuming we had something like that, it could be a
>building block for a simple built-in secondary index solution.
>
>In the end you're probably right and we cannot prescribe a single
>secondary index solution that will fit all use cases.
>
>-- Lars
>
>
>----- Original Message -----
>From: Jonathan Hsieh <jo...@cloudera.com>
>To: dev@hbase.apache.org
>Cc:
>Sent: Wednesday, December 5, 2012 11:23 AM
>Subject: Re: HBase - Secondary Index
>
>I personally feel that we should only add the primitive apis necessary to
>make this work (and if non are required, all the better!), and to try to
>keep secondary indexing work it as a separate project on top of hbase
>because there are many possible valid architectures and implementations.
>
>I have two question areas -- one touched upon in previous messages -- what
>logging and how do we get consistency guarantee's do we get bewteen the
>index and primary?
>
>The other has to do with scalability. I'm not sure I interpreted the
>slides
>correctly, but from the slides 8 and 13, is the architecture such that
>each
>primary table region has a corresponding index table region?
>
>Is slide 14 a comparison of a full table scan vs the indexed lookups on 4
>rs's? What happens if we go up to 20, or 100 rs's? If I'm right about
>the
>per index region per table region, I have a feeling this isn't going to
>scale well with a large number of regions (since it would potentially have
>to talk to each region and essentially every region server).
>
>Jon.
>
>On Wed, Dec 5, 2012 at 9:54 AM, Anoop John <an...@gmail.com> wrote:
>
>> I mean HBase devs to work on having sec indexing available with the
>>HBase
>> distribution... Now I guess many users of HBase implement different
>>kinds
>> of sec indexing ..:)
>>
>> We @Huawei would be happy to provide our support in it as per the
>>interest
>> from the community.. :)
>>
>> -Anoop-
>>
>> On Wed, Dec 5, 2012 at 9:29 PM, Andrey Stepachev <oc...@gmail.com>
>>wrote:
>>
>> > Can you explain, what you mean under 'HBase community version with sec
>> > indexing in it'. You will wait, until someone implements the same
>> algorithm
>> > in trunk hbase, or?
>> >
>> >
>> > On Wed, Dec 5, 2012 at 4:55 PM, Anoop Sam John <an...@huawei.com>
>> wrote:
>> >
>> > > No this is not open sourced yet.. As per the interest from the
>>HBase
>> > > community we can think of contributing..
>> > > It is time to see HBase community version with sec indexing in it
>> (IMHO)
>> > > :)
>> > >
>> > > -Anoop-
>> > >
>> > > ________________________________________
>> > > From: Andrey Stepachev [octo47@gmail.com]
>> > > Sent: Wednesday, December 05, 2012 5:22 PM
>> > > To: dev@hbase.apache.org
>> > > Subject: Re: HBase - Secondary Index
>> > >
>> > > Hi.
>> > >
>> > > Indexing solution looks tempting.
>> > > Are there any plans to open source your solution (or it already open
>> and
>> > I
>> > > can't find it?).
>> > >
>> > >
>> > >
>> > >
>> > > On Tue, Dec 4, 2012 at 12:10 PM, Anoop Sam John <an...@huawei.com>
>> > > wrote:
>> > >
>> > > > Hi All
>> > > >
>> > > > Last week I got a chance to present the secondary
>> indexing
>> > > > solution what we have done in Huawei at the China Hadoop
>>Conference.
>> > You
>> > > > can see the presentation from
>> > > > http://hbtc2012.hadooper.cn/subject/track4Anoop%20Sam%20John2.pdf
>> > > >
>> > > >
>> > > >
>> > > > I would like to hear what others think on this. :)
>> > > >
>> > > >
>> > > >
>> > > > -Anoop-
>> > > >
>> > >
>> > >
>> > >
>> > > --
>> > > Andrey.
>> > >
>> >
>> >
>> >
>> > --
>> > Andrey.
>> >
>>
>
>
>
>--
>// Jonathan Hsieh (shay)
>// Software Engineer, Cloudera
>// jon@cloudera.com
>
>
Re: HBase - Secondary Index
Posted by lars hofhansl <lh...@yahoo.com>.
I'd say "it depends".
It seems everybody wants secondary indexes in HBase. The problem is that most folks don't agree what that actually means.
The most interesting problem and discussion point (IMHO) is that HBase would need some form of schema description.
See also HBASE-7221. Assuming we had something like that, it could be a building block for a simple built-in secondary index solution.
In the end you're probably right and we cannot prescribe a single secondary index solution that will fit all use cases.
-- Lars
----- Original Message -----
From: Jonathan Hsieh <jo...@cloudera.com>
To: dev@hbase.apache.org
Cc:
Sent: Wednesday, December 5, 2012 11:23 AM
Subject: Re: HBase - Secondary Index
I personally feel that we should only add the primitive apis necessary to
make this work (and if non are required, all the better!), and to try to
keep secondary indexing work it as a separate project on top of hbase
because there are many possible valid architectures and implementations.
I have two question areas -- one touched upon in previous messages -- what
logging and how do we get consistency guarantee's do we get bewteen the
index and primary?
The other has to do with scalability. I'm not sure I interpreted the slides
correctly, but from the slides 8 and 13, is the architecture such that each
primary table region has a corresponding index table region?
Is slide 14 a comparison of a full table scan vs the indexed lookups on 4
rs's? What happens if we go up to 20, or 100 rs's? If I'm right about the
per index region per table region, I have a feeling this isn't going to
scale well with a large number of regions (since it would potentially have
to talk to each region and essentially every region server).
Jon.
On Wed, Dec 5, 2012 at 9:54 AM, Anoop John <an...@gmail.com> wrote:
> I mean HBase devs to work on having sec indexing available with the HBase
> distribution... Now I guess many users of HBase implement different kinds
> of sec indexing ..:)
>
> We @Huawei would be happy to provide our support in it as per the interest
> from the community.. :)
>
> -Anoop-
>
> On Wed, Dec 5, 2012 at 9:29 PM, Andrey Stepachev <oc...@gmail.com> wrote:
>
> > Can you explain, what you mean under 'HBase community version with sec
> > indexing in it'. You will wait, until someone implements the same
> algorithm
> > in trunk hbase, or?
> >
> >
> > On Wed, Dec 5, 2012 at 4:55 PM, Anoop Sam John <an...@huawei.com>
> wrote:
> >
> > > No this is not open sourced yet.. As per the interest from the HBase
> > > community we can think of contributing..
> > > It is time to see HBase community version with sec indexing in it
> (IMHO)
> > > :)
> > >
> > > -Anoop-
> > >
> > > ________________________________________
> > > From: Andrey Stepachev [octo47@gmail.com]
> > > Sent: Wednesday, December 05, 2012 5:22 PM
> > > To: dev@hbase.apache.org
> > > Subject: Re: HBase - Secondary Index
> > >
> > > Hi.
> > >
> > > Indexing solution looks tempting.
> > > Are there any plans to open source your solution (or it already open
> and
> > I
> > > can't find it?).
> > >
> > >
> > >
> > >
> > > On Tue, Dec 4, 2012 at 12:10 PM, Anoop Sam John <an...@huawei.com>
> > > wrote:
> > >
> > > > Hi All
> > > >
> > > > Last week I got a chance to present the secondary
> indexing
> > > > solution what we have done in Huawei at the China Hadoop Conference.
> > You
> > > > can see the presentation from
> > > > http://hbtc2012.hadooper.cn/subject/track4Anoop%20Sam%20John2.pdf
> > > >
> > > >
> > > >
> > > > I would like to hear what others think on this. :)
> > > >
> > > >
> > > >
> > > > -Anoop-
> > > >
> > >
> > >
> > >
> > > --
> > > Andrey.
> > >
> >
> >
> >
> > --
> > Andrey.
> >
>
--
// Jonathan Hsieh (shay)
// Software Engineer, Cloudera
// jon@cloudera.com
Re: HBase - Secondary Index
Posted by Jonathan Hsieh <jo...@cloudera.com>.
I personally feel that we should only add the primitive apis necessary to
make this work (and if non are required, all the better!), and to try to
keep secondary indexing work it as a separate project on top of hbase
because there are many possible valid architectures and implementations.
I have two question areas -- one touched upon in previous messages -- what
logging and how do we get consistency guarantee's do we get bewteen the
index and primary?
The other has to do with scalability. I'm not sure I interpreted the slides
correctly, but from the slides 8 and 13, is the architecture such that each
primary table region has a corresponding index table region?
Is slide 14 a comparison of a full table scan vs the indexed lookups on 4
rs's? What happens if we go up to 20, or 100 rs's? If I'm right about the
per index region per table region, I have a feeling this isn't going to
scale well with a large number of regions (since it would potentially have
to talk to each region and essentially every region server).
Jon.
On Wed, Dec 5, 2012 at 9:54 AM, Anoop John <an...@gmail.com> wrote:
> I mean HBase devs to work on having sec indexing available with the HBase
> distribution... Now I guess many users of HBase implement different kinds
> of sec indexing ..:)
>
> We @Huawei would be happy to provide our support in it as per the interest
> from the community.. :)
>
> -Anoop-
>
> On Wed, Dec 5, 2012 at 9:29 PM, Andrey Stepachev <oc...@gmail.com> wrote:
>
> > Can you explain, what you mean under 'HBase community version with sec
> > indexing in it'. You will wait, until someone implements the same
> algorithm
> > in trunk hbase, or?
> >
> >
> > On Wed, Dec 5, 2012 at 4:55 PM, Anoop Sam John <an...@huawei.com>
> wrote:
> >
> > > No this is not open sourced yet.. As per the interest from the HBase
> > > community we can think of contributing..
> > > It is time to see HBase community version with sec indexing in it
> (IMHO)
> > > :)
> > >
> > > -Anoop-
> > >
> > > ________________________________________
> > > From: Andrey Stepachev [octo47@gmail.com]
> > > Sent: Wednesday, December 05, 2012 5:22 PM
> > > To: dev@hbase.apache.org
> > > Subject: Re: HBase - Secondary Index
> > >
> > > Hi.
> > >
> > > Indexing solution looks tempting.
> > > Are there any plans to open source your solution (or it already open
> and
> > I
> > > can't find it?).
> > >
> > >
> > >
> > >
> > > On Tue, Dec 4, 2012 at 12:10 PM, Anoop Sam John <an...@huawei.com>
> > > wrote:
> > >
> > > > Hi All
> > > >
> > > > Last week I got a chance to present the secondary
> indexing
> > > > solution what we have done in Huawei at the China Hadoop Conference.
> > You
> > > > can see the presentation from
> > > > http://hbtc2012.hadooper.cn/subject/track4Anoop%20Sam%20John2.pdf
> > > >
> > > >
> > > >
> > > > I would like to hear what others think on this. :)
> > > >
> > > >
> > > >
> > > > -Anoop-
> > > >
> > >
> > >
> > >
> > > --
> > > Andrey.
> > >
> >
> >
> >
> > --
> > Andrey.
> >
>
--
// Jonathan Hsieh (shay)
// Software Engineer, Cloudera
// jon@cloudera.com
Re: HBase - Secondary Index
Posted by Anoop John <an...@gmail.com>.
I mean HBase devs to work on having sec indexing available with the HBase
distribution... Now I guess many users of HBase implement different kinds
of sec indexing ..:)
We @Huawei would be happy to provide our support in it as per the interest
from the community.. :)
-Anoop-
On Wed, Dec 5, 2012 at 9:29 PM, Andrey Stepachev <oc...@gmail.com> wrote:
> Can you explain, what you mean under 'HBase community version with sec
> indexing in it'. You will wait, until someone implements the same algorithm
> in trunk hbase, or?
>
>
> On Wed, Dec 5, 2012 at 4:55 PM, Anoop Sam John <an...@huawei.com> wrote:
>
> > No this is not open sourced yet.. As per the interest from the HBase
> > community we can think of contributing..
> > It is time to see HBase community version with sec indexing in it (IMHO)
> > :)
> >
> > -Anoop-
> >
> > ________________________________________
> > From: Andrey Stepachev [octo47@gmail.com]
> > Sent: Wednesday, December 05, 2012 5:22 PM
> > To: dev@hbase.apache.org
> > Subject: Re: HBase - Secondary Index
> >
> > Hi.
> >
> > Indexing solution looks tempting.
> > Are there any plans to open source your solution (or it already open and
> I
> > can't find it?).
> >
> >
> >
> >
> > On Tue, Dec 4, 2012 at 12:10 PM, Anoop Sam John <an...@huawei.com>
> > wrote:
> >
> > > Hi All
> > >
> > > Last week I got a chance to present the secondary indexing
> > > solution what we have done in Huawei at the China Hadoop Conference.
> You
> > > can see the presentation from
> > > http://hbtc2012.hadooper.cn/subject/track4Anoop%20Sam%20John2.pdf
> > >
> > >
> > >
> > > I would like to hear what others think on this. :)
> > >
> > >
> > >
> > > -Anoop-
> > >
> >
> >
> >
> > --
> > Andrey.
> >
>
>
>
> --
> Andrey.
>
Re: HBase - Secondary Index
Posted by Andrey Stepachev <oc...@gmail.com>.
Can you explain, what you mean under 'HBase community version with sec
indexing in it'. You will wait, until someone implements the same algorithm
in trunk hbase, or?
On Wed, Dec 5, 2012 at 4:55 PM, Anoop Sam John <an...@huawei.com> wrote:
> No this is not open sourced yet.. As per the interest from the HBase
> community we can think of contributing..
> It is time to see HBase community version with sec indexing in it (IMHO)
> :)
>
> -Anoop-
>
> ________________________________________
> From: Andrey Stepachev [octo47@gmail.com]
> Sent: Wednesday, December 05, 2012 5:22 PM
> To: dev@hbase.apache.org
> Subject: Re: HBase - Secondary Index
>
> Hi.
>
> Indexing solution looks tempting.
> Are there any plans to open source your solution (or it already open and I
> can't find it?).
>
>
>
>
> On Tue, Dec 4, 2012 at 12:10 PM, Anoop Sam John <an...@huawei.com>
> wrote:
>
> > Hi All
> >
> > Last week I got a chance to present the secondary indexing
> > solution what we have done in Huawei at the China Hadoop Conference. You
> > can see the presentation from
> > http://hbtc2012.hadooper.cn/subject/track4Anoop%20Sam%20John2.pdf
> >
> >
> >
> > I would like to hear what others think on this. :)
> >
> >
> >
> > -Anoop-
> >
>
>
>
> --
> Andrey.
>
--
Andrey.
RE: HBase - Secondary Index
Posted by Anoop Sam John <an...@huawei.com>.
No this is not open sourced yet.. As per the interest from the HBase community we can think of contributing..
It is time to see HBase community version with sec indexing in it (IMHO) :)
-Anoop-
________________________________________
From: Andrey Stepachev [octo47@gmail.com]
Sent: Wednesday, December 05, 2012 5:22 PM
To: dev@hbase.apache.org
Subject: Re: HBase - Secondary Index
Hi.
Indexing solution looks tempting.
Are there any plans to open source your solution (or it already open and I
can't find it?).
On Tue, Dec 4, 2012 at 12:10 PM, Anoop Sam John <an...@huawei.com> wrote:
> Hi All
>
> Last week I got a chance to present the secondary indexing
> solution what we have done in Huawei at the China Hadoop Conference. You
> can see the presentation from
> http://hbtc2012.hadooper.cn/subject/track4Anoop%20Sam%20John2.pdf
>
>
>
> I would like to hear what others think on this. :)
>
>
>
> -Anoop-
>
--
Andrey.
Re: HBase - Secondary Index
Posted by Andrey Stepachev <oc...@gmail.com>.
Hi.
Indexing solution looks tempting.
Are there any plans to open source your solution (or it already open and I
can't find it?).
On Tue, Dec 4, 2012 at 12:10 PM, Anoop Sam John <an...@huawei.com> wrote:
> Hi All
>
> Last week I got a chance to present the secondary indexing
> solution what we have done in Huawei at the China Hadoop Conference. You
> can see the presentation from
> http://hbtc2012.hadooper.cn/subject/track4Anoop%20Sam%20John2.pdf
>
>
>
> I would like to hear what others think on this. :)
>
>
>
> -Anoop-
>
--
Andrey.
Re: HBase - Secondary Index
Posted by anil gupta <an...@gmail.com>.
Hi Michael,
Please find my replies inline.
Thanks,
Anil
On Tue, Dec 18, 2012 at 1:02 AM, Michel Segel <mi...@hotmail.com>wrote:
> Just a couple of questions...
>
> First, since you don't have any natural secondary indices, you can create
> one from a couple of choices. Keeping it simple, you choose an inverted
> table as your index.
>
Reasons for not creating a inverted table:
1. There can be millions of columns corresponding to a rowkey in my
secondary index. In future it can even grow more.
2. While using secondary index, we are also planning to have filtering on
the basis of other non-rowkey columns.
For example: 1 Row of Secondary table might look like this:
Rowkey: cf:PrimarytableRowKey=x, cf:customerFirstName=xyz,
cf:customerAddress=123, Union Sq, LA
My primary table has around 50 columns and in secondary table i duplicate
two columns to used along with secondary index for filtering.
>
> In doing so, you have one column containing all of the row ids for a given
> value.
> This means that it is a simple get().
>
> My question is that since you don't have any formal SQL syntax, how are
> you doing this all server side?
>
As Anoop said, I am not doing the index data scan at the server side. He
scan the index table data back to client and from client doing gets to get
the main table data.
>
>
> Sent from a remote device. Please excuse any typos...
>
> Mike Segel
>
> On Dec 18, 2012, at 2:28 AM, anil gupta <an...@gmail.com> wrote:
>
> > Hi Anoop,
> >
> > Please find my reply inline.
> >
> > Thanks,
> > Anil Gupta
> >
> > On Sun, Dec 16, 2012 at 8:02 PM, Anoop Sam John <an...@huawei.com>
> wrote:
> >
> >> Hi Anil
> >> During the scan, there is no need to fetch any index data
> >> to client side. So there is no need to create any scanner on the index
> >> table at the client side. This happens at the server side.
> >
> >
> >>
> >> For the Scan on the main table with condition on timestamp and customer
> >> id, a scanner to be created with Filters. Yes like normal when there is
> no
> >> secondary index. So this scan from the client will go through all the
> >> regions in the main table.
> >
> >
> > Anil: Do you mean that if the table is spread across 50 region servers in
> > 60 node cluster then we need to send a scan request to all the 50 RS.
> > Right? Doesn't it sounds expensive? IMHO you were not doing this in your
> > solution. Your solution looked cleaner than this since you exactly knew
> > which Node you need to go to for querying while using secondary index due
> > to co-location(due to static begin part for secondary table rowkey) of
> > region of primary table and secondary index table. My problem is little
> > more complicated due to the constraints that: I cannot have a "static
> begin
> > part" in the rowkey of my secondary table.
> >
> > When it scans one particular region say (x,y] on the main table, using
> the
> >> CP we can get the index table region object corresponding to this main
> >> table region from the RS. There is no issue in creating the static
> part of
> >> the rowkey. You know 'x' is the region start key. Then at the server
> side
> >> will create a scanner on the index region directly and here we can
> specify
> >> the startkey. 'x' + <timestamp value> + <customer id>.. Using the
> results
> >> from the index scan we will make reseek on the main region to the exact
> >> rows where the data what we are interested in is available. So there
> wont
> >> be a full region data scan happening.
> >
> >> When in the cases where only timestamp is there but no customer id, it
> >> will be simple again. Create a scanner on the main table with only one
> >> filter. At the CP side the scanner on the index region will get created
> >> with startkey as 'x' + <timestamp value>.. When you create the scan
> >> object and set startRow on that it need not be the full rowkey. It can
> be
> >> part of the rowkey also. Yes like prefix.
> >>
> >> Hope u got it now :)
> > Anil: I hope now we are on same page. Thanks a lot for your valuable time
> > to discuss this stuff.
> >
> >>
> >> -Anoop-
> >> ________________________________________
> >> From: anil gupta [anilgupta84@gmail.com]
> >> Sent: Friday, December 14, 2012 11:31 PM
> >> To: user@hbase.apache.org
> >> Subject: Re: HBase - Secondary Index
> >>
> >> On Fri, Dec 14, 2012 at 12:54 AM, Anoop Sam John <an...@huawei.com>
> >> wrote:
> >>
> >>> Hi Anil,
> >>>
> >>>> 1. In your presentation you mentioned that region of Primary Table and
> >>> Region of Secondary Table are always located on the same region server.
> >> How
> >>> do you achieve it? By using the Primary table rowkey as prefix of
> Rowkey
> >>> of Secondary Table? Will your implementation work if the rowkey of
> >> primary
> >>> table cannot be used as prefix in rowkey of Secondary table( i have
> this
> >>> limitation in my use case)?
> >>> First all there will be same number of regions in both primary and
> index
> >>> tables. All the start/stop keys of the regions also will be same.
> >>> Suppose there are 2 regions on main table say for keys 0-10 and 10-20.
> >>> Then we will create 2 regions in index table also with same key ranges.
> >>> At the master balancing level it is easy to collocate these regions
> >> seeing
> >>> the start and end keys.
> >>> When the selection of the rowkey that will be used in the index table
> is
> >>> the key here.
> >>> What we will do is all the rowkeys in the index table will be prefixed
> >>> with the start key of the region/
> >>> When an entry is added to the main table with rowkey as 5 it will go to
> >>> the 1st region (0-10)
> >>> Now there will be index region with range as 0-10. We will select this
> >>> region to store this index data.
> >>> The row getting added into the index region for this entry will have a
> >>> rowkey 0_x_5
> >>> I am just using '_' as a seperator here just to show this. Actually we
> >>> wont be having any seperator.
> >>> So the rowkeys (in index region) will have a static begin part always.
> >>> Will scan time also we know this part and so the startrow and endrow
> >>> creation for the scan will be possible.. Note that we will store the
> >> actual
> >>> table row key as the last part of the index rowkey itself not as a
> value.
> >>> This is better option in our case of handling the scan index usage also
> >> at
> >>> sever side. There is no index data fetch to client side..
> >>
> >> Anil: My primary table rowkey is customerId+event_id, and my secondary
> >> table rowkey is timestamp+ customerid. In your implementation it seems
> like
> >> for using secondary index the application needs to know about the
> >> "start_key" of the region(static begin part) it wants to query. Right?
> Do
> >> you separately manage the logic of determining the region
> >> "start_key"(static begin part) for a scan?
> >> Also, Its possible that while using secondary index the customerId is
> not
> >> provided. So, i wont be having customer id for all the queries. Hence i
> >> cannot use customer_id as a prefix in rowkey of my Secondary Table.
> >>
> >>>
> >>> I feel your use case perfectly fit with our model
> >> Anil: Somehow i am unable to fit your implementation into my use case
> due
> >> to the constraint of static begin part of rowkey in Secondary table.
> There
> >> seems to be a disconnect. Can you tell me how does my use case fits into
> >> your implementation?
> >>
> >>>
> >>>> 2. Are you using an Endpoint or Observer for building the secondary
> >> index
> >>> table?
> >>> Observer
> >>>
> >>>> 3. "Custom balancer do collocation". Is it a custom load balancer of
> >> HBase
> >>> Master or something else?
> >>> It is a balancer implementation which will be plugged into Master
> >>>
> >>>> 4. Your region split looks interesting. I dont have much info about
> it.
> >>> Can
> >>> you point to some docs on IndexHalfStoreFileReader?
> >>> Sorry I am not able to publish any design doc or code as the company
> has
> >>> not decided to open src the solution yet.
> >>> Any particular query you come acorss pls feel free to aske me :)
> >>> You can see the HalfStoreFileReader class 1st..
> >>>
> >>> -Anoop-
> >>> ________________________________________
> >>> From: anil gupta [anilgupta84@gmail.com]
> >>> Sent: Friday, December 14, 2012 2:11 PM
> >>> To: user@hbase.apache.org
> >>> Subject: Re: HBase - Secondary Index
> >>>
> >>> Hi Anoop,
> >>>
> >>> Nice presentation and seems like a smart implementation. Since the
> >>> presentation only covered bullet points so i have couple of questions
> on
> >>> your implementation. :)
> >>>
> >>> Here is a recap to my implementation and our previous discussion on
> >>> Secondary index:
> >>>
> >>> Here is the link to previous email thread:
> >>> http://search-hadoop.com/m/1zWPMaaRtr .
> >>>
> >>> The secondary index is stored in table "B" as rowkey B -->
> family:<rowkey
> >>> A> . "<rowkey A>" is the column qualifier. Every row in B will only on
> >>> have one column "k" and the value of that column is the rowkey of A.
> >>>
> >>> Suppose i am storing customer events in table A. I have two requirement
> >> for
> >>> data query:
> >>> 1. Query customer events on basis of customer_Id and event_ID.
> >>> 2. Query customer events on basis of event_timestamp and customer_ID.
> >>>
> >>> 70% of querying is done by query#1, so i will create
> >>> <customer_Id><event_ID> as row key of Table A.
> >>> Now, in order to support fast results for query#2, i need to create a
> >>> secondary index on A. I store that secondary index in B, rowkey of B is
> >>> <event_timestamp><customer_ID>.Every row stores the corresponding
> rowkey
> >> of
> >>> A.
> >>>
> >>> HBase Querying approach:
> >>> 1. Scan the secondary table by using prefix filter and startRow to get
> >> the
> >>> list of Rowkeys of Primary table.
> >>> 2. Do a batch get on primary table by using HTable.get(List<Get>)
> method
> >>> using the list of Rowkeys obtained in step1.
> >>>
> >>> The only issue is that in my solution i have at least two RPC calls.
> Once
> >>> each in step1 and step2 above. I want to reduce the number of RPC to 1
> if
> >>> possible.
> >>>
> >>>
> >>> ******Questions on your implementation:*********
> >>>
> >>> 1. In your presentation you mentioned that region of Primary Table and
> >>> Region of Secondary Table are always located on the same region server.
> >> How
> >>> do you achieve it? By using the Primary table rowkey as prefix of
> Rowkey
> >>> of Secondary Table? Will your implementation work if the rowkey of
> >> primary
> >>> table cannot be used as prefix in rowkey of Secondary table( i have
> this
> >>> limitation in my use case)?
> >>> 2. Are you using an Endpoint or Observer for building the secondary
> index
> >>> table?
> >>> 3. "Custom balancer do collocation". Is it a custom load balancer of
> >> HBase
> >>> Master or something else?
> >>> 4. Your region split looks interesting. I dont have much info about it.
> >> Can
> >>> you point to some docs on IndexHalfStoreFileReader?
> >>>
> >>> Thanks,
> >>> Anil Gupta
> >>>
> >>>
> >>>
> >>> On Tue, Dec 4, 2012 at 12:10 AM, Anoop Sam John <an...@huawei.com>
> >>> wrote:
> >>>
> >>>> Hi All
> >>>>
> >>>> Last week I got a chance to present the secondary indexing
> >>>> solution what we have done in Huawei at the China Hadoop Conference.
> >> You
> >>>> can see the presentation from
> >>>> http://hbtc2012.hadooper.cn/subject/track4Anoop%20Sam%20John2.pdf
> >>>>
> >>>>
> >>>>
> >>>> I would like to hear what others think on this. :)
> >>>>
> >>>>
> >>>>
> >>>> -Anoop-
> >>>
> >>>
> >>>
> >>> --
> >>> Thanks & Regards,
> >>> Anil Gupta
> >>
> >>
> >>
> >> --
> >> Thanks & Regards,
> >> Anil Gupta
> >
> >
> >
> > --
> > Thanks & Regards,
> > Anil Gupta
>
--
Thanks & Regards,
Anil Gupta
RE: HBase - Secondary Index
Posted by Anoop Sam John <an...@huawei.com>.
Hi Mike
>My question is that since you don't have any formal SQL syntax, how are you doing this all server side?
I think the question is to Anil.. In his case he is not doing the index data scan at the server side. He scan the index table data back to client and from client doing gets to get the main table data. Correct Anil?
Just making it clear... :)
-Anoop-
________________________________________
From: Michel Segel [michael_segel@hotmail.com]
Sent: Tuesday, December 18, 2012 2:32 PM
To: user@hbase.apache.org
Cc: user@hbase.apache.org
Subject: Re: HBase - Secondary Index
Just a couple of questions...
First, since you don't have any natural secondary indices, you can create one from a couple of choices. Keeping it simple, you choose an inverted table as your index.
In doing so, you have one column containing all of the row ids for a given value.
This means that it is a simple get().
My question is that since you don't have any formal SQL syntax, how are you doing this all server side?
Sent from a remote device. Please excuse any typos...
Mike Segel
On Dec 18, 2012, at 2:28 AM, anil gupta <an...@gmail.com> wrote:
> Hi Anoop,
>
> Please find my reply inline.
>
> Thanks,
> Anil Gupta
>
> On Sun, Dec 16, 2012 at 8:02 PM, Anoop Sam John <an...@huawei.com> wrote:
>
>> Hi Anil
>> During the scan, there is no need to fetch any index data
>> to client side. So there is no need to create any scanner on the index
>> table at the client side. This happens at the server side.
>
>
>>
>> For the Scan on the main table with condition on timestamp and customer
>> id, a scanner to be created with Filters. Yes like normal when there is no
>> secondary index. So this scan from the client will go through all the
>> regions in the main table.
>
>
> Anil: Do you mean that if the table is spread across 50 region servers in
> 60 node cluster then we need to send a scan request to all the 50 RS.
> Right? Doesn't it sounds expensive? IMHO you were not doing this in your
> solution. Your solution looked cleaner than this since you exactly knew
> which Node you need to go to for querying while using secondary index due
> to co-location(due to static begin part for secondary table rowkey) of
> region of primary table and secondary index table. My problem is little
> more complicated due to the constraints that: I cannot have a "static begin
> part" in the rowkey of my secondary table.
>
> When it scans one particular region say (x,y] on the main table, using the
>> CP we can get the index table region object corresponding to this main
>> table region from the RS. There is no issue in creating the static part of
>> the rowkey. You know 'x' is the region start key. Then at the server side
>> will create a scanner on the index region directly and here we can specify
>> the startkey. 'x' + <timestamp value> + <customer id>.. Using the results
>> from the index scan we will make reseek on the main region to the exact
>> rows where the data what we are interested in is available. So there wont
>> be a full region data scan happening.
>
>> When in the cases where only timestamp is there but no customer id, it
>> will be simple again. Create a scanner on the main table with only one
>> filter. At the CP side the scanner on the index region will get created
>> with startkey as 'x' + <timestamp value>.. When you create the scan
>> object and set startRow on that it need not be the full rowkey. It can be
>> part of the rowkey also. Yes like prefix.
>>
>> Hope u got it now :)
> Anil: I hope now we are on same page. Thanks a lot for your valuable time
> to discuss this stuff.
>
>>
>> -Anoop-
>> ________________________________________
>> From: anil gupta [anilgupta84@gmail.com]
>> Sent: Friday, December 14, 2012 11:31 PM
>> To: user@hbase.apache.org
>> Subject: Re: HBase - Secondary Index
>>
>> On Fri, Dec 14, 2012 at 12:54 AM, Anoop Sam John <an...@huawei.com>
>> wrote:
>>
>>> Hi Anil,
>>>
>>>> 1. In your presentation you mentioned that region of Primary Table and
>>> Region of Secondary Table are always located on the same region server.
>> How
>>> do you achieve it? By using the Primary table rowkey as prefix of Rowkey
>>> of Secondary Table? Will your implementation work if the rowkey of
>> primary
>>> table cannot be used as prefix in rowkey of Secondary table( i have this
>>> limitation in my use case)?
>>> First all there will be same number of regions in both primary and index
>>> tables. All the start/stop keys of the regions also will be same.
>>> Suppose there are 2 regions on main table say for keys 0-10 and 10-20.
>>> Then we will create 2 regions in index table also with same key ranges.
>>> At the master balancing level it is easy to collocate these regions
>> seeing
>>> the start and end keys.
>>> When the selection of the rowkey that will be used in the index table is
>>> the key here.
>>> What we will do is all the rowkeys in the index table will be prefixed
>>> with the start key of the region/
>>> When an entry is added to the main table with rowkey as 5 it will go to
>>> the 1st region (0-10)
>>> Now there will be index region with range as 0-10. We will select this
>>> region to store this index data.
>>> The row getting added into the index region for this entry will have a
>>> rowkey 0_x_5
>>> I am just using '_' as a seperator here just to show this. Actually we
>>> wont be having any seperator.
>>> So the rowkeys (in index region) will have a static begin part always.
>>> Will scan time also we know this part and so the startrow and endrow
>>> creation for the scan will be possible.. Note that we will store the
>> actual
>>> table row key as the last part of the index rowkey itself not as a value.
>>> This is better option in our case of handling the scan index usage also
>> at
>>> sever side. There is no index data fetch to client side..
>>
>> Anil: My primary table rowkey is customerId+event_id, and my secondary
>> table rowkey is timestamp+ customerid. In your implementation it seems like
>> for using secondary index the application needs to know about the
>> "start_key" of the region(static begin part) it wants to query. Right? Do
>> you separately manage the logic of determining the region
>> "start_key"(static begin part) for a scan?
>> Also, Its possible that while using secondary index the customerId is not
>> provided. So, i wont be having customer id for all the queries. Hence i
>> cannot use customer_id as a prefix in rowkey of my Secondary Table.
>>
>>>
>>> I feel your use case perfectly fit with our model
>> Anil: Somehow i am unable to fit your implementation into my use case due
>> to the constraint of static begin part of rowkey in Secondary table. There
>> seems to be a disconnect. Can you tell me how does my use case fits into
>> your implementation?
>>
>>>
>>>> 2. Are you using an Endpoint or Observer for building the secondary
>> index
>>> table?
>>> Observer
>>>
>>>> 3. "Custom balancer do collocation". Is it a custom load balancer of
>> HBase
>>> Master or something else?
>>> It is a balancer implementation which will be plugged into Master
>>>
>>>> 4. Your region split looks interesting. I dont have much info about it.
>>> Can
>>> you point to some docs on IndexHalfStoreFileReader?
>>> Sorry I am not able to publish any design doc or code as the company has
>>> not decided to open src the solution yet.
>>> Any particular query you come acorss pls feel free to aske me :)
>>> You can see the HalfStoreFileReader class 1st..
>>>
>>> -Anoop-
>>> ________________________________________
>>> From: anil gupta [anilgupta84@gmail.com]
>>> Sent: Friday, December 14, 2012 2:11 PM
>>> To: user@hbase.apache.org
>>> Subject: Re: HBase - Secondary Index
>>>
>>> Hi Anoop,
>>>
>>> Nice presentation and seems like a smart implementation. Since the
>>> presentation only covered bullet points so i have couple of questions on
>>> your implementation. :)
>>>
>>> Here is a recap to my implementation and our previous discussion on
>>> Secondary index:
>>>
>>> Here is the link to previous email thread:
>>> http://search-hadoop.com/m/1zWPMaaRtr .
>>>
>>> The secondary index is stored in table "B" as rowkey B --> family:<rowkey
>>> A> . "<rowkey A>" is the column qualifier. Every row in B will only on
>>> have one column "k" and the value of that column is the rowkey of A.
>>>
>>> Suppose i am storing customer events in table A. I have two requirement
>> for
>>> data query:
>>> 1. Query customer events on basis of customer_Id and event_ID.
>>> 2. Query customer events on basis of event_timestamp and customer_ID.
>>>
>>> 70% of querying is done by query#1, so i will create
>>> <customer_Id><event_ID> as row key of Table A.
>>> Now, in order to support fast results for query#2, i need to create a
>>> secondary index on A. I store that secondary index in B, rowkey of B is
>>> <event_timestamp><customer_ID>.Every row stores the corresponding rowkey
>> of
>>> A.
>>>
>>> HBase Querying approach:
>>> 1. Scan the secondary table by using prefix filter and startRow to get
>> the
>>> list of Rowkeys of Primary table.
>>> 2. Do a batch get on primary table by using HTable.get(List<Get>) method
>>> using the list of Rowkeys obtained in step1.
>>>
>>> The only issue is that in my solution i have at least two RPC calls. Once
>>> each in step1 and step2 above. I want to reduce the number of RPC to 1 if
>>> possible.
>>>
>>>
>>> ******Questions on your implementation:*********
>>>
>>> 1. In your presentation you mentioned that region of Primary Table and
>>> Region of Secondary Table are always located on the same region server.
>> How
>>> do you achieve it? By using the Primary table rowkey as prefix of Rowkey
>>> of Secondary Table? Will your implementation work if the rowkey of
>> primary
>>> table cannot be used as prefix in rowkey of Secondary table( i have this
>>> limitation in my use case)?
>>> 2. Are you using an Endpoint or Observer for building the secondary index
>>> table?
>>> 3. "Custom balancer do collocation". Is it a custom load balancer of
>> HBase
>>> Master or something else?
>>> 4. Your region split looks interesting. I dont have much info about it.
>> Can
>>> you point to some docs on IndexHalfStoreFileReader?
>>>
>>> Thanks,
>>> Anil Gupta
>>>
>>>
>>>
>>> On Tue, Dec 4, 2012 at 12:10 AM, Anoop Sam John <an...@huawei.com>
>>> wrote:
>>>
>>>> Hi All
>>>>
>>>> Last week I got a chance to present the secondary indexing
>>>> solution what we have done in Huawei at the China Hadoop Conference.
>> You
>>>> can see the presentation from
>>>> http://hbtc2012.hadooper.cn/subject/track4Anoop%20Sam%20John2.pdf
>>>>
>>>>
>>>>
>>>> I would like to hear what others think on this. :)
>>>>
>>>>
>>>>
>>>> -Anoop-
>>>
>>>
>>>
>>> --
>>> Thanks & Regards,
>>> Anil Gupta
>>
>>
>>
>> --
>> Thanks & Regards,
>> Anil Gupta
>
>
>
> --
> Thanks & Regards,
> Anil Gupta
Re: HBase - Secondary Index
Posted by Michel Segel <mi...@hotmail.com>.
Just a couple of questions...
First, since you don't have any natural secondary indices, you can create one from a couple of choices. Keeping it simple, you choose an inverted table as your index.
In doing so, you have one column containing all of the row ids for a given value.
This means that it is a simple get().
My question is that since you don't have any formal SQL syntax, how are you doing this all server side?
Sent from a remote device. Please excuse any typos...
Mike Segel
On Dec 18, 2012, at 2:28 AM, anil gupta <an...@gmail.com> wrote:
> Hi Anoop,
>
> Please find my reply inline.
>
> Thanks,
> Anil Gupta
>
> On Sun, Dec 16, 2012 at 8:02 PM, Anoop Sam John <an...@huawei.com> wrote:
>
>> Hi Anil
>> During the scan, there is no need to fetch any index data
>> to client side. So there is no need to create any scanner on the index
>> table at the client side. This happens at the server side.
>
>
>>
>> For the Scan on the main table with condition on timestamp and customer
>> id, a scanner to be created with Filters. Yes like normal when there is no
>> secondary index. So this scan from the client will go through all the
>> regions in the main table.
>
>
> Anil: Do you mean that if the table is spread across 50 region servers in
> 60 node cluster then we need to send a scan request to all the 50 RS.
> Right? Doesn't it sounds expensive? IMHO you were not doing this in your
> solution. Your solution looked cleaner than this since you exactly knew
> which Node you need to go to for querying while using secondary index due
> to co-location(due to static begin part for secondary table rowkey) of
> region of primary table and secondary index table. My problem is little
> more complicated due to the constraints that: I cannot have a "static begin
> part" in the rowkey of my secondary table.
>
> When it scans one particular region say (x,y] on the main table, using the
>> CP we can get the index table region object corresponding to this main
>> table region from the RS. There is no issue in creating the static part of
>> the rowkey. You know 'x' is the region start key. Then at the server side
>> will create a scanner on the index region directly and here we can specify
>> the startkey. 'x' + <timestamp value> + <customer id>.. Using the results
>> from the index scan we will make reseek on the main region to the exact
>> rows where the data what we are interested in is available. So there wont
>> be a full region data scan happening.
>
>> When in the cases where only timestamp is there but no customer id, it
>> will be simple again. Create a scanner on the main table with only one
>> filter. At the CP side the scanner on the index region will get created
>> with startkey as 'x' + <timestamp value>.. When you create the scan
>> object and set startRow on that it need not be the full rowkey. It can be
>> part of the rowkey also. Yes like prefix.
>>
>> Hope u got it now :)
> Anil: I hope now we are on same page. Thanks a lot for your valuable time
> to discuss this stuff.
>
>>
>> -Anoop-
>> ________________________________________
>> From: anil gupta [anilgupta84@gmail.com]
>> Sent: Friday, December 14, 2012 11:31 PM
>> To: user@hbase.apache.org
>> Subject: Re: HBase - Secondary Index
>>
>> On Fri, Dec 14, 2012 at 12:54 AM, Anoop Sam John <an...@huawei.com>
>> wrote:
>>
>>> Hi Anil,
>>>
>>>> 1. In your presentation you mentioned that region of Primary Table and
>>> Region of Secondary Table are always located on the same region server.
>> How
>>> do you achieve it? By using the Primary table rowkey as prefix of Rowkey
>>> of Secondary Table? Will your implementation work if the rowkey of
>> primary
>>> table cannot be used as prefix in rowkey of Secondary table( i have this
>>> limitation in my use case)?
>>> First all there will be same number of regions in both primary and index
>>> tables. All the start/stop keys of the regions also will be same.
>>> Suppose there are 2 regions on main table say for keys 0-10 and 10-20.
>>> Then we will create 2 regions in index table also with same key ranges.
>>> At the master balancing level it is easy to collocate these regions
>> seeing
>>> the start and end keys.
>>> When the selection of the rowkey that will be used in the index table is
>>> the key here.
>>> What we will do is all the rowkeys in the index table will be prefixed
>>> with the start key of the region/
>>> When an entry is added to the main table with rowkey as 5 it will go to
>>> the 1st region (0-10)
>>> Now there will be index region with range as 0-10. We will select this
>>> region to store this index data.
>>> The row getting added into the index region for this entry will have a
>>> rowkey 0_x_5
>>> I am just using '_' as a seperator here just to show this. Actually we
>>> wont be having any seperator.
>>> So the rowkeys (in index region) will have a static begin part always.
>>> Will scan time also we know this part and so the startrow and endrow
>>> creation for the scan will be possible.. Note that we will store the
>> actual
>>> table row key as the last part of the index rowkey itself not as a value.
>>> This is better option in our case of handling the scan index usage also
>> at
>>> sever side. There is no index data fetch to client side..
>>
>> Anil: My primary table rowkey is customerId+event_id, and my secondary
>> table rowkey is timestamp+ customerid. In your implementation it seems like
>> for using secondary index the application needs to know about the
>> "start_key" of the region(static begin part) it wants to query. Right? Do
>> you separately manage the logic of determining the region
>> "start_key"(static begin part) for a scan?
>> Also, Its possible that while using secondary index the customerId is not
>> provided. So, i wont be having customer id for all the queries. Hence i
>> cannot use customer_id as a prefix in rowkey of my Secondary Table.
>>
>>>
>>> I feel your use case perfectly fit with our model
>> Anil: Somehow i am unable to fit your implementation into my use case due
>> to the constraint of static begin part of rowkey in Secondary table. There
>> seems to be a disconnect. Can you tell me how does my use case fits into
>> your implementation?
>>
>>>
>>>> 2. Are you using an Endpoint or Observer for building the secondary
>> index
>>> table?
>>> Observer
>>>
>>>> 3. "Custom balancer do collocation". Is it a custom load balancer of
>> HBase
>>> Master or something else?
>>> It is a balancer implementation which will be plugged into Master
>>>
>>>> 4. Your region split looks interesting. I dont have much info about it.
>>> Can
>>> you point to some docs on IndexHalfStoreFileReader?
>>> Sorry I am not able to publish any design doc or code as the company has
>>> not decided to open src the solution yet.
>>> Any particular query you come acorss pls feel free to aske me :)
>>> You can see the HalfStoreFileReader class 1st..
>>>
>>> -Anoop-
>>> ________________________________________
>>> From: anil gupta [anilgupta84@gmail.com]
>>> Sent: Friday, December 14, 2012 2:11 PM
>>> To: user@hbase.apache.org
>>> Subject: Re: HBase - Secondary Index
>>>
>>> Hi Anoop,
>>>
>>> Nice presentation and seems like a smart implementation. Since the
>>> presentation only covered bullet points so i have couple of questions on
>>> your implementation. :)
>>>
>>> Here is a recap to my implementation and our previous discussion on
>>> Secondary index:
>>>
>>> Here is the link to previous email thread:
>>> http://search-hadoop.com/m/1zWPMaaRtr .
>>>
>>> The secondary index is stored in table "B" as rowkey B --> family:<rowkey
>>> A> . "<rowkey A>" is the column qualifier. Every row in B will only on
>>> have one column "k" and the value of that column is the rowkey of A.
>>>
>>> Suppose i am storing customer events in table A. I have two requirement
>> for
>>> data query:
>>> 1. Query customer events on basis of customer_Id and event_ID.
>>> 2. Query customer events on basis of event_timestamp and customer_ID.
>>>
>>> 70% of querying is done by query#1, so i will create
>>> <customer_Id><event_ID> as row key of Table A.
>>> Now, in order to support fast results for query#2, i need to create a
>>> secondary index on A. I store that secondary index in B, rowkey of B is
>>> <event_timestamp><customer_ID>.Every row stores the corresponding rowkey
>> of
>>> A.
>>>
>>> HBase Querying approach:
>>> 1. Scan the secondary table by using prefix filter and startRow to get
>> the
>>> list of Rowkeys of Primary table.
>>> 2. Do a batch get on primary table by using HTable.get(List<Get>) method
>>> using the list of Rowkeys obtained in step1.
>>>
>>> The only issue is that in my solution i have at least two RPC calls. Once
>>> each in step1 and step2 above. I want to reduce the number of RPC to 1 if
>>> possible.
>>>
>>>
>>> ******Questions on your implementation:*********
>>>
>>> 1. In your presentation you mentioned that region of Primary Table and
>>> Region of Secondary Table are always located on the same region server.
>> How
>>> do you achieve it? By using the Primary table rowkey as prefix of Rowkey
>>> of Secondary Table? Will your implementation work if the rowkey of
>> primary
>>> table cannot be used as prefix in rowkey of Secondary table( i have this
>>> limitation in my use case)?
>>> 2. Are you using an Endpoint or Observer for building the secondary index
>>> table?
>>> 3. "Custom balancer do collocation". Is it a custom load balancer of
>> HBase
>>> Master or something else?
>>> 4. Your region split looks interesting. I dont have much info about it.
>> Can
>>> you point to some docs on IndexHalfStoreFileReader?
>>>
>>> Thanks,
>>> Anil Gupta
>>>
>>>
>>>
>>> On Tue, Dec 4, 2012 at 12:10 AM, Anoop Sam John <an...@huawei.com>
>>> wrote:
>>>
>>>> Hi All
>>>>
>>>> Last week I got a chance to present the secondary indexing
>>>> solution what we have done in Huawei at the China Hadoop Conference.
>> You
>>>> can see the presentation from
>>>> http://hbtc2012.hadooper.cn/subject/track4Anoop%20Sam%20John2.pdf
>>>>
>>>>
>>>>
>>>> I would like to hear what others think on this. :)
>>>>
>>>>
>>>>
>>>> -Anoop-
>>>
>>>
>>>
>>> --
>>> Thanks & Regards,
>>> Anil Gupta
>>
>>
>>
>> --
>> Thanks & Regards,
>> Anil Gupta
>
>
>
> --
> Thanks & Regards,
> Anil Gupta
Re: HBase - Secondary Index
Posted by anil gupta <an...@gmail.com>.
Hi Anoop,
For my use case, scans will never have primary table rowkey range whenever
i query using secondary index. IMHO, if i am sending the request to all the
RS of table then i am afraid/concerned of too many unnecessary RPC's across
the cluster for every single query based on secondary index. Essentially
everytime it will look like a full table scan but under the hood the CP's
will do the magic using secondary table.Your solution works well when
rowkey range on primary table can be specified.
Unfortunately, i dont have that luxury for now to use "primary table rowkey
range". It seems like i will have to stick to my current solution. However,
it's always good to have a healthy discussion on different approaches. :)
PS: My current secondary index implementation is not yet in production. I
did some preliminary testing and it seems to work fine but i think i need
to do some more testing.
Thanks,
Anil Gupta
On Tue, Dec 18, 2012 at 1:27 AM, Anoop Sam John <an...@huawei.com> wrote:
> Anil:
> If the scan from client side does not specify any rowkey range but
> only the filter condition, yes it will go to all the primary table regions
> for the scan. There 1st it will scan the index table region and seek to
> exact rows in the main table region. If that region is not having any data
> at all corresponding to the filter condition, the entire region will get
> skipped simply.
>
> In a normal scan also, if there is a rowkey range that we can specify,
> then only to specific regions the request will go. In the sec index case of
> ours also it is same..
>
> In a simple way what I can say is for the scan there is no change at all
> wrt the operation that is what is happening at the client side. From the
> meta data to know which all region and RSs to contact, and contacting that
> regions one by one and getting data from that region. Only difference is
> what is happening at the server side. With out index the whole data from
> all the Hfiles will get fetched at the server side and the filter will get
> applied for every row. Only those rows which passes the filter will get
> back to the client side. With index, when the scanning happen at the
> server side, the index data will get scanned 1st from the index region.
> This region will be in the same RS so no extra RPCs. The data to be scanned
> from the index table will be limited.. We can create the start key and stop
> key for that.. Based on the result of the index scan, we will know the
> rowkeys where all the data what we are interested in resides. So reseek
> will happen to those rows and read only those rows. So the time spent at
> the server side for scanning a region will get reduced to a very high value.
>
> Yes but still there will be calls from the client side to the RS for each
> region...
>
> Now I think u might be clear.. In the ppt that I have shared, there also
> it is saying the same thing. It is showing what is happening at the server
> side.
>
> -Anoop-
>
> ________________________________________
> From: anil gupta [anilgupta84@gmail.com]
> Sent: Tuesday, December 18, 2012 1:58 PM
> To: user@hbase.apache.org
> Subject: Re: HBase - Secondary Index
>
> Hi Anoop,
>
> Please find my reply inline.
>
> Thanks,
> Anil Gupta
>
> On Sun, Dec 16, 2012 at 8:02 PM, Anoop Sam John <an...@huawei.com>
> wrote:
>
> > Hi Anil
> > During the scan, there is no need to fetch any index data
> > to client side. So there is no need to create any scanner on the index
> > table at the client side. This happens at the server side.
> >
>
>
> >
> > For the Scan on the main table with condition on timestamp and customer
> > id, a scanner to be created with Filters. Yes like normal when there is
> no
> > secondary index. So this scan from the client will go through all the
> > regions in the main table.
>
>
> Anil: Do you mean that if the table is spread across 50 region servers in
> 60 node cluster then we need to send a scan request to all the 50 RS.
> Right? Doesn't it sounds expensive? IMHO you were not doing this in your
> solution. Your solution looked cleaner than this since you exactly knew
> which Node you need to go to for querying while using secondary index due
> to co-location(due to static begin part for secondary table rowkey) of
> region of primary table and secondary index table. My problem is little
> more complicated due to the constraints that: I cannot have a "static begin
> part" in the rowkey of my secondary table.
>
> When it scans one particular region say (x,y] on the main table, using the
> > CP we can get the index table region object corresponding to this main
> > table region from the RS. There is no issue in creating the static part
> of
> > the rowkey. You know 'x' is the region start key. Then at the server side
> > will create a scanner on the index region directly and here we can
> specify
> > the startkey. 'x' + <timestamp value> + <customer id>.. Using the
> results
> > from the index scan we will make reseek on the main region to the exact
> > rows where the data what we are interested in is available. So there wont
> > be a full region data scan happening.
> >
>
> > When in the cases where only timestamp is there but no customer id, it
> > will be simple again. Create a scanner on the main table with only one
> > filter. At the CP side the scanner on the index region will get created
> > with startkey as 'x' + <timestamp value>.. When you create the scan
> > object and set startRow on that it need not be the full rowkey. It can be
> > part of the rowkey also. Yes like prefix.
> >
> > Hope u got it now :)
> >
> Anil: I hope now we are on same page. Thanks a lot for your valuable time
> to discuss this stuff.
>
> >
> > -Anoop-
> > ________________________________________
> > From: anil gupta [anilgupta84@gmail.com]
> > Sent: Friday, December 14, 2012 11:31 PM
> > To: user@hbase.apache.org
> > Subject: Re: HBase - Secondary Index
> >
> > On Fri, Dec 14, 2012 at 12:54 AM, Anoop Sam John <an...@huawei.com>
> > wrote:
> >
> > > Hi Anil,
> > >
> > > >1. In your presentation you mentioned that region of Primary Table and
> > > Region of Secondary Table are always located on the same region server.
> > How
> > > do you achieve it? By using the Primary table rowkey as prefix of
> Rowkey
> > > of Secondary Table? Will your implementation work if the rowkey of
> > primary
> > > table cannot be used as prefix in rowkey of Secondary table( i have
> this
> > > limitation in my use case)?
> > > First all there will be same number of regions in both primary and
> index
> > > tables. All the start/stop keys of the regions also will be same.
> > > Suppose there are 2 regions on main table say for keys 0-10 and 10-20.
> > > Then we will create 2 regions in index table also with same key
> ranges.
> > > At the master balancing level it is easy to collocate these regions
> > seeing
> > > the start and end keys.
> > > When the selection of the rowkey that will be used in the index table
> is
> > > the key here.
> > > What we will do is all the rowkeys in the index table will be prefixed
> > > with the start key of the region/
> > > When an entry is added to the main table with rowkey as 5 it will go to
> > > the 1st region (0-10)
> > > Now there will be index region with range as 0-10. We will select this
> > > region to store this index data.
> > > The row getting added into the index region for this entry will have a
> > > rowkey 0_x_5
> > > I am just using '_' as a seperator here just to show this. Actually we
> > > wont be having any seperator.
> > > So the rowkeys (in index region) will have a static begin part always.
> > > Will scan time also we know this part and so the startrow and endrow
> > > creation for the scan will be possible.. Note that we will store the
> > actual
> > > table row key as the last part of the index rowkey itself not as a
> value.
> > > This is better option in our case of handling the scan index usage also
> > at
> > > sever side. There is no index data fetch to client side..
> > >
> >
> > Anil: My primary table rowkey is customerId+event_id, and my secondary
> > table rowkey is timestamp+ customerid. In your implementation it seems
> like
> > for using secondary index the application needs to know about the
> > "start_key" of the region(static begin part) it wants to query. Right? Do
> > you separately manage the logic of determining the region
> > "start_key"(static begin part) for a scan?
> > Also, Its possible that while using secondary index the customerId is not
> > provided. So, i wont be having customer id for all the queries. Hence i
> > cannot use customer_id as a prefix in rowkey of my Secondary Table.
> >
> > >
> > > I feel your use case perfectly fit with our model
> > >
> > Anil: Somehow i am unable to fit your implementation into my use case due
> > to the constraint of static begin part of rowkey in Secondary table.
> There
> > seems to be a disconnect. Can you tell me how does my use case fits into
> > your implementation?
> >
> > >
> > > >2. Are you using an Endpoint or Observer for building the secondary
> > index
> > > table?
> > > Observer
> > >
> > > >3. "Custom balancer do collocation". Is it a custom load balancer of
> > HBase
> > > Master or something else?
> > > It is a balancer implementation which will be plugged into Master
> > >
> > > >4. Your region split looks interesting. I dont have much info about
> it.
> > > Can
> > > you point to some docs on IndexHalfStoreFileReader?
> > > Sorry I am not able to publish any design doc or code as the company
> has
> > > not decided to open src the solution yet.
> > > Any particular query you come acorss pls feel free to aske me :)
> > > You can see the HalfStoreFileReader class 1st..
> > >
> > > -Anoop-
> > > ________________________________________
> > > From: anil gupta [anilgupta84@gmail.com]
> > > Sent: Friday, December 14, 2012 2:11 PM
> > > To: user@hbase.apache.org
> > > Subject: Re: HBase - Secondary Index
> > >
> > > Hi Anoop,
> > >
> > > Nice presentation and seems like a smart implementation. Since the
> > > presentation only covered bullet points so i have couple of questions
> on
> > > your implementation. :)
> > >
> > > Here is a recap to my implementation and our previous discussion on
> > > Secondary index:
> > >
> > > Here is the link to previous email thread:
> > > http://search-hadoop.com/m/1zWPMaaRtr .
> > >
> > > The secondary index is stored in table "B" as rowkey B -->
> family:<rowkey
> > > A> . "<rowkey A>" is the column qualifier. Every row in B will only on
> > > have one column "k" and the value of that column is the rowkey of A.
> > >
> > > Suppose i am storing customer events in table A. I have two requirement
> > for
> > > data query:
> > > 1. Query customer events on basis of customer_Id and event_ID.
> > > 2. Query customer events on basis of event_timestamp and customer_ID.
> > >
> > > 70% of querying is done by query#1, so i will create
> > > <customer_Id><event_ID> as row key of Table A.
> > > Now, in order to support fast results for query#2, i need to create a
> > > secondary index on A. I store that secondary index in B, rowkey of B is
> > > <event_timestamp><customer_ID>.Every row stores the corresponding
> rowkey
> > of
> > > A.
> > >
> > > HBase Querying approach:
> > > 1. Scan the secondary table by using prefix filter and startRow to get
> > the
> > > list of Rowkeys of Primary table.
> > > 2. Do a batch get on primary table by using HTable.get(List<Get>)
> method
> > > using the list of Rowkeys obtained in step1.
> > >
> > > The only issue is that in my solution i have at least two RPC calls.
> Once
> > > each in step1 and step2 above. I want to reduce the number of RPC to 1
> if
> > > possible.
> > >
> > >
> > > ******Questions on your implementation:*********
> > >
> > > 1. In your presentation you mentioned that region of Primary Table and
> > > Region of Secondary Table are always located on the same region server.
> > How
> > > do you achieve it? By using the Primary table rowkey as prefix of
> Rowkey
> > > of Secondary Table? Will your implementation work if the rowkey of
> > primary
> > > table cannot be used as prefix in rowkey of Secondary table( i have
> this
> > > limitation in my use case)?
> > > 2. Are you using an Endpoint or Observer for building the secondary
> index
> > > table?
> > > 3. "Custom balancer do collocation". Is it a custom load balancer of
> > HBase
> > > Master or something else?
> > > 4. Your region split looks interesting. I dont have much info about it.
> > Can
> > > you point to some docs on IndexHalfStoreFileReader?
> > >
> > > Thanks,
> > > Anil Gupta
> > >
> > >
> > >
> > > On Tue, Dec 4, 2012 at 12:10 AM, Anoop Sam John <an...@huawei.com>
> > > wrote:
> > >
> > > > Hi All
> > > >
> > > > Last week I got a chance to present the secondary
> indexing
> > > > solution what we have done in Huawei at the China Hadoop Conference.
> > You
> > > > can see the presentation from
> > > > http://hbtc2012.hadooper.cn/subject/track4Anoop%20Sam%20John2.pdf
> > > >
> > > >
> > > >
> > > > I would like to hear what others think on this. :)
> > > >
> > > >
> > > >
> > > > -Anoop-
> > > >
> > >
> > >
> > >
> > > --
> > > Thanks & Regards,
> > > Anil Gupta
> > >
> >
> >
> >
> > --
> > Thanks & Regards,
> > Anil Gupta
> >
>
>
>
> --
> Thanks & Regards,
> Anil Gupta
>
--
Thanks & Regards,
Anil Gupta
RE: HBase - Secondary Index
Posted by Anoop Sam John <an...@huawei.com>.
Anil:
If the scan from client side does not specify any rowkey range but only the filter condition, yes it will go to all the primary table regions for the scan. There 1st it will scan the index table region and seek to exact rows in the main table region. If that region is not having any data at all corresponding to the filter condition, the entire region will get skipped simply.
In a normal scan also, if there is a rowkey range that we can specify, then only to specific regions the request will go. In the sec index case of ours also it is same..
In a simple way what I can say is for the scan there is no change at all wrt the operation that is what is happening at the client side. From the meta data to know which all region and RSs to contact, and contacting that regions one by one and getting data from that region. Only difference is what is happening at the server side. With out index the whole data from all the Hfiles will get fetched at the server side and the filter will get applied for every row. Only those rows which passes the filter will get back to the client side. With index, when the scanning happen at the server side, the index data will get scanned 1st from the index region. This region will be in the same RS so no extra RPCs. The data to be scanned from the index table will be limited.. We can create the start key and stop key for that.. Based on the result of the index scan, we will know the rowkeys where all the data what we are interested in resides. So reseek will happen to those rows and read only those rows. So the time spent at the server side for scanning a region will get reduced to a very high value.
Yes but still there will be calls from the client side to the RS for each region...
Now I think u might be clear.. In the ppt that I have shared, there also it is saying the same thing. It is showing what is happening at the server side.
-Anoop-
________________________________________
From: anil gupta [anilgupta84@gmail.com]
Sent: Tuesday, December 18, 2012 1:58 PM
To: user@hbase.apache.org
Subject: Re: HBase - Secondary Index
Hi Anoop,
Please find my reply inline.
Thanks,
Anil Gupta
On Sun, Dec 16, 2012 at 8:02 PM, Anoop Sam John <an...@huawei.com> wrote:
> Hi Anil
> During the scan, there is no need to fetch any index data
> to client side. So there is no need to create any scanner on the index
> table at the client side. This happens at the server side.
>
>
> For the Scan on the main table with condition on timestamp and customer
> id, a scanner to be created with Filters. Yes like normal when there is no
> secondary index. So this scan from the client will go through all the
> regions in the main table.
Anil: Do you mean that if the table is spread across 50 region servers in
60 node cluster then we need to send a scan request to all the 50 RS.
Right? Doesn't it sounds expensive? IMHO you were not doing this in your
solution. Your solution looked cleaner than this since you exactly knew
which Node you need to go to for querying while using secondary index due
to co-location(due to static begin part for secondary table rowkey) of
region of primary table and secondary index table. My problem is little
more complicated due to the constraints that: I cannot have a "static begin
part" in the rowkey of my secondary table.
When it scans one particular region say (x,y] on the main table, using the
> CP we can get the index table region object corresponding to this main
> table region from the RS. There is no issue in creating the static part of
> the rowkey. You know 'x' is the region start key. Then at the server side
> will create a scanner on the index region directly and here we can specify
> the startkey. 'x' + <timestamp value> + <customer id>.. Using the results
> from the index scan we will make reseek on the main region to the exact
> rows where the data what we are interested in is available. So there wont
> be a full region data scan happening.
>
> When in the cases where only timestamp is there but no customer id, it
> will be simple again. Create a scanner on the main table with only one
> filter. At the CP side the scanner on the index region will get created
> with startkey as 'x' + <timestamp value>.. When you create the scan
> object and set startRow on that it need not be the full rowkey. It can be
> part of the rowkey also. Yes like prefix.
>
> Hope u got it now :)
>
Anil: I hope now we are on same page. Thanks a lot for your valuable time
to discuss this stuff.
>
> -Anoop-
> ________________________________________
> From: anil gupta [anilgupta84@gmail.com]
> Sent: Friday, December 14, 2012 11:31 PM
> To: user@hbase.apache.org
> Subject: Re: HBase - Secondary Index
>
> On Fri, Dec 14, 2012 at 12:54 AM, Anoop Sam John <an...@huawei.com>
> wrote:
>
> > Hi Anil,
> >
> > >1. In your presentation you mentioned that region of Primary Table and
> > Region of Secondary Table are always located on the same region server.
> How
> > do you achieve it? By using the Primary table rowkey as prefix of Rowkey
> > of Secondary Table? Will your implementation work if the rowkey of
> primary
> > table cannot be used as prefix in rowkey of Secondary table( i have this
> > limitation in my use case)?
> > First all there will be same number of regions in both primary and index
> > tables. All the start/stop keys of the regions also will be same.
> > Suppose there are 2 regions on main table say for keys 0-10 and 10-20.
> > Then we will create 2 regions in index table also with same key ranges.
> > At the master balancing level it is easy to collocate these regions
> seeing
> > the start and end keys.
> > When the selection of the rowkey that will be used in the index table is
> > the key here.
> > What we will do is all the rowkeys in the index table will be prefixed
> > with the start key of the region/
> > When an entry is added to the main table with rowkey as 5 it will go to
> > the 1st region (0-10)
> > Now there will be index region with range as 0-10. We will select this
> > region to store this index data.
> > The row getting added into the index region for this entry will have a
> > rowkey 0_x_5
> > I am just using '_' as a seperator here just to show this. Actually we
> > wont be having any seperator.
> > So the rowkeys (in index region) will have a static begin part always.
> > Will scan time also we know this part and so the startrow and endrow
> > creation for the scan will be possible.. Note that we will store the
> actual
> > table row key as the last part of the index rowkey itself not as a value.
> > This is better option in our case of handling the scan index usage also
> at
> > sever side. There is no index data fetch to client side..
> >
>
> Anil: My primary table rowkey is customerId+event_id, and my secondary
> table rowkey is timestamp+ customerid. In your implementation it seems like
> for using secondary index the application needs to know about the
> "start_key" of the region(static begin part) it wants to query. Right? Do
> you separately manage the logic of determining the region
> "start_key"(static begin part) for a scan?
> Also, Its possible that while using secondary index the customerId is not
> provided. So, i wont be having customer id for all the queries. Hence i
> cannot use customer_id as a prefix in rowkey of my Secondary Table.
>
> >
> > I feel your use case perfectly fit with our model
> >
> Anil: Somehow i am unable to fit your implementation into my use case due
> to the constraint of static begin part of rowkey in Secondary table. There
> seems to be a disconnect. Can you tell me how does my use case fits into
> your implementation?
>
> >
> > >2. Are you using an Endpoint or Observer for building the secondary
> index
> > table?
> > Observer
> >
> > >3. "Custom balancer do collocation". Is it a custom load balancer of
> HBase
> > Master or something else?
> > It is a balancer implementation which will be plugged into Master
> >
> > >4. Your region split looks interesting. I dont have much info about it.
> > Can
> > you point to some docs on IndexHalfStoreFileReader?
> > Sorry I am not able to publish any design doc or code as the company has
> > not decided to open src the solution yet.
> > Any particular query you come acorss pls feel free to aske me :)
> > You can see the HalfStoreFileReader class 1st..
> >
> > -Anoop-
> > ________________________________________
> > From: anil gupta [anilgupta84@gmail.com]
> > Sent: Friday, December 14, 2012 2:11 PM
> > To: user@hbase.apache.org
> > Subject: Re: HBase - Secondary Index
> >
> > Hi Anoop,
> >
> > Nice presentation and seems like a smart implementation. Since the
> > presentation only covered bullet points so i have couple of questions on
> > your implementation. :)
> >
> > Here is a recap to my implementation and our previous discussion on
> > Secondary index:
> >
> > Here is the link to previous email thread:
> > http://search-hadoop.com/m/1zWPMaaRtr .
> >
> > The secondary index is stored in table "B" as rowkey B --> family:<rowkey
> > A> . "<rowkey A>" is the column qualifier. Every row in B will only on
> > have one column "k" and the value of that column is the rowkey of A.
> >
> > Suppose i am storing customer events in table A. I have two requirement
> for
> > data query:
> > 1. Query customer events on basis of customer_Id and event_ID.
> > 2. Query customer events on basis of event_timestamp and customer_ID.
> >
> > 70% of querying is done by query#1, so i will create
> > <customer_Id><event_ID> as row key of Table A.
> > Now, in order to support fast results for query#2, i need to create a
> > secondary index on A. I store that secondary index in B, rowkey of B is
> > <event_timestamp><customer_ID>.Every row stores the corresponding rowkey
> of
> > A.
> >
> > HBase Querying approach:
> > 1. Scan the secondary table by using prefix filter and startRow to get
> the
> > list of Rowkeys of Primary table.
> > 2. Do a batch get on primary table by using HTable.get(List<Get>) method
> > using the list of Rowkeys obtained in step1.
> >
> > The only issue is that in my solution i have at least two RPC calls. Once
> > each in step1 and step2 above. I want to reduce the number of RPC to 1 if
> > possible.
> >
> >
> > ******Questions on your implementation:*********
> >
> > 1. In your presentation you mentioned that region of Primary Table and
> > Region of Secondary Table are always located on the same region server.
> How
> > do you achieve it? By using the Primary table rowkey as prefix of Rowkey
> > of Secondary Table? Will your implementation work if the rowkey of
> primary
> > table cannot be used as prefix in rowkey of Secondary table( i have this
> > limitation in my use case)?
> > 2. Are you using an Endpoint or Observer for building the secondary index
> > table?
> > 3. "Custom balancer do collocation". Is it a custom load balancer of
> HBase
> > Master or something else?
> > 4. Your region split looks interesting. I dont have much info about it.
> Can
> > you point to some docs on IndexHalfStoreFileReader?
> >
> > Thanks,
> > Anil Gupta
> >
> >
> >
> > On Tue, Dec 4, 2012 at 12:10 AM, Anoop Sam John <an...@huawei.com>
> > wrote:
> >
> > > Hi All
> > >
> > > Last week I got a chance to present the secondary indexing
> > > solution what we have done in Huawei at the China Hadoop Conference.
> You
> > > can see the presentation from
> > > http://hbtc2012.hadooper.cn/subject/track4Anoop%20Sam%20John2.pdf
> > >
> > >
> > >
> > > I would like to hear what others think on this. :)
> > >
> > >
> > >
> > > -Anoop-
> > >
> >
> >
> >
> > --
> > Thanks & Regards,
> > Anil Gupta
> >
>
>
>
> --
> Thanks & Regards,
> Anil Gupta
>
--
Thanks & Regards,
Anil Gupta
Re: HBase - Secondary Index
Posted by anil gupta <an...@gmail.com>.
Hi Anoop,
Please find my reply inline.
Thanks,
Anil Gupta
On Sun, Dec 16, 2012 at 8:02 PM, Anoop Sam John <an...@huawei.com> wrote:
> Hi Anil
> During the scan, there is no need to fetch any index data
> to client side. So there is no need to create any scanner on the index
> table at the client side. This happens at the server side.
>
>
> For the Scan on the main table with condition on timestamp and customer
> id, a scanner to be created with Filters. Yes like normal when there is no
> secondary index. So this scan from the client will go through all the
> regions in the main table.
Anil: Do you mean that if the table is spread across 50 region servers in
60 node cluster then we need to send a scan request to all the 50 RS.
Right? Doesn't it sounds expensive? IMHO you were not doing this in your
solution. Your solution looked cleaner than this since you exactly knew
which Node you need to go to for querying while using secondary index due
to co-location(due to static begin part for secondary table rowkey) of
region of primary table and secondary index table. My problem is little
more complicated due to the constraints that: I cannot have a "static begin
part" in the rowkey of my secondary table.
When it scans one particular region say (x,y] on the main table, using the
> CP we can get the index table region object corresponding to this main
> table region from the RS. There is no issue in creating the static part of
> the rowkey. You know 'x' is the region start key. Then at the server side
> will create a scanner on the index region directly and here we can specify
> the startkey. 'x' + <timestamp value> + <customer id>.. Using the results
> from the index scan we will make reseek on the main region to the exact
> rows where the data what we are interested in is available. So there wont
> be a full region data scan happening.
>
> When in the cases where only timestamp is there but no customer id, it
> will be simple again. Create a scanner on the main table with only one
> filter. At the CP side the scanner on the index region will get created
> with startkey as 'x' + <timestamp value>.. When you create the scan
> object and set startRow on that it need not be the full rowkey. It can be
> part of the rowkey also. Yes like prefix.
>
> Hope u got it now :)
>
Anil: I hope now we are on same page. Thanks a lot for your valuable time
to discuss this stuff.
>
> -Anoop-
> ________________________________________
> From: anil gupta [anilgupta84@gmail.com]
> Sent: Friday, December 14, 2012 11:31 PM
> To: user@hbase.apache.org
> Subject: Re: HBase - Secondary Index
>
> On Fri, Dec 14, 2012 at 12:54 AM, Anoop Sam John <an...@huawei.com>
> wrote:
>
> > Hi Anil,
> >
> > >1. In your presentation you mentioned that region of Primary Table and
> > Region of Secondary Table are always located on the same region server.
> How
> > do you achieve it? By using the Primary table rowkey as prefix of Rowkey
> > of Secondary Table? Will your implementation work if the rowkey of
> primary
> > table cannot be used as prefix in rowkey of Secondary table( i have this
> > limitation in my use case)?
> > First all there will be same number of regions in both primary and index
> > tables. All the start/stop keys of the regions also will be same.
> > Suppose there are 2 regions on main table say for keys 0-10 and 10-20.
> > Then we will create 2 regions in index table also with same key ranges.
> > At the master balancing level it is easy to collocate these regions
> seeing
> > the start and end keys.
> > When the selection of the rowkey that will be used in the index table is
> > the key here.
> > What we will do is all the rowkeys in the index table will be prefixed
> > with the start key of the region/
> > When an entry is added to the main table with rowkey as 5 it will go to
> > the 1st region (0-10)
> > Now there will be index region with range as 0-10. We will select this
> > region to store this index data.
> > The row getting added into the index region for this entry will have a
> > rowkey 0_x_5
> > I am just using '_' as a seperator here just to show this. Actually we
> > wont be having any seperator.
> > So the rowkeys (in index region) will have a static begin part always.
> > Will scan time also we know this part and so the startrow and endrow
> > creation for the scan will be possible.. Note that we will store the
> actual
> > table row key as the last part of the index rowkey itself not as a value.
> > This is better option in our case of handling the scan index usage also
> at
> > sever side. There is no index data fetch to client side..
> >
>
> Anil: My primary table rowkey is customerId+event_id, and my secondary
> table rowkey is timestamp+ customerid. In your implementation it seems like
> for using secondary index the application needs to know about the
> "start_key" of the region(static begin part) it wants to query. Right? Do
> you separately manage the logic of determining the region
> "start_key"(static begin part) for a scan?
> Also, Its possible that while using secondary index the customerId is not
> provided. So, i wont be having customer id for all the queries. Hence i
> cannot use customer_id as a prefix in rowkey of my Secondary Table.
>
> >
> > I feel your use case perfectly fit with our model
> >
> Anil: Somehow i am unable to fit your implementation into my use case due
> to the constraint of static begin part of rowkey in Secondary table. There
> seems to be a disconnect. Can you tell me how does my use case fits into
> your implementation?
>
> >
> > >2. Are you using an Endpoint or Observer for building the secondary
> index
> > table?
> > Observer
> >
> > >3. "Custom balancer do collocation". Is it a custom load balancer of
> HBase
> > Master or something else?
> > It is a balancer implementation which will be plugged into Master
> >
> > >4. Your region split looks interesting. I dont have much info about it.
> > Can
> > you point to some docs on IndexHalfStoreFileReader?
> > Sorry I am not able to publish any design doc or code as the company has
> > not decided to open src the solution yet.
> > Any particular query you come acorss pls feel free to aske me :)
> > You can see the HalfStoreFileReader class 1st..
> >
> > -Anoop-
> > ________________________________________
> > From: anil gupta [anilgupta84@gmail.com]
> > Sent: Friday, December 14, 2012 2:11 PM
> > To: user@hbase.apache.org
> > Subject: Re: HBase - Secondary Index
> >
> > Hi Anoop,
> >
> > Nice presentation and seems like a smart implementation. Since the
> > presentation only covered bullet points so i have couple of questions on
> > your implementation. :)
> >
> > Here is a recap to my implementation and our previous discussion on
> > Secondary index:
> >
> > Here is the link to previous email thread:
> > http://search-hadoop.com/m/1zWPMaaRtr .
> >
> > The secondary index is stored in table "B" as rowkey B --> family:<rowkey
> > A> . "<rowkey A>" is the column qualifier. Every row in B will only on
> > have one column "k" and the value of that column is the rowkey of A.
> >
> > Suppose i am storing customer events in table A. I have two requirement
> for
> > data query:
> > 1. Query customer events on basis of customer_Id and event_ID.
> > 2. Query customer events on basis of event_timestamp and customer_ID.
> >
> > 70% of querying is done by query#1, so i will create
> > <customer_Id><event_ID> as row key of Table A.
> > Now, in order to support fast results for query#2, i need to create a
> > secondary index on A. I store that secondary index in B, rowkey of B is
> > <event_timestamp><customer_ID>.Every row stores the corresponding rowkey
> of
> > A.
> >
> > HBase Querying approach:
> > 1. Scan the secondary table by using prefix filter and startRow to get
> the
> > list of Rowkeys of Primary table.
> > 2. Do a batch get on primary table by using HTable.get(List<Get>) method
> > using the list of Rowkeys obtained in step1.
> >
> > The only issue is that in my solution i have at least two RPC calls. Once
> > each in step1 and step2 above. I want to reduce the number of RPC to 1 if
> > possible.
> >
> >
> > ******Questions on your implementation:*********
> >
> > 1. In your presentation you mentioned that region of Primary Table and
> > Region of Secondary Table are always located on the same region server.
> How
> > do you achieve it? By using the Primary table rowkey as prefix of Rowkey
> > of Secondary Table? Will your implementation work if the rowkey of
> primary
> > table cannot be used as prefix in rowkey of Secondary table( i have this
> > limitation in my use case)?
> > 2. Are you using an Endpoint or Observer for building the secondary index
> > table?
> > 3. "Custom balancer do collocation". Is it a custom load balancer of
> HBase
> > Master or something else?
> > 4. Your region split looks interesting. I dont have much info about it.
> Can
> > you point to some docs on IndexHalfStoreFileReader?
> >
> > Thanks,
> > Anil Gupta
> >
> >
> >
> > On Tue, Dec 4, 2012 at 12:10 AM, Anoop Sam John <an...@huawei.com>
> > wrote:
> >
> > > Hi All
> > >
> > > Last week I got a chance to present the secondary indexing
> > > solution what we have done in Huawei at the China Hadoop Conference.
> You
> > > can see the presentation from
> > > http://hbtc2012.hadooper.cn/subject/track4Anoop%20Sam%20John2.pdf
> > >
> > >
> > >
> > > I would like to hear what others think on this. :)
> > >
> > >
> > >
> > > -Anoop-
> > >
> >
> >
> >
> > --
> > Thanks & Regards,
> > Anil Gupta
> >
>
>
>
> --
> Thanks & Regards,
> Anil Gupta
>
--
Thanks & Regards,
Anil Gupta
RE: HBase - Secondary Index
Posted by Anoop Sam John <an...@huawei.com>.
Hi Anil
During the scan, there is no need to fetch any index data to client side. So there is no need to create any scanner on the index table at the client side. This happens at the server side.
For the Scan on the main table with condition on timestamp and customer id, a scanner to be created with Filters. Yes like normal when there is no secondary index. So this scan from the client will go through all the regions in the main table. When it scans one particular region say (x,y] on the main table, using the CP we can get the index table region object corresponding to this main table region from the RS. There is no issue in creating the static part of the rowkey. You know 'x' is the region start key. Then at the server side will create a scanner on the index region directly and here we can specify the startkey. 'x' + <timestamp value> + <customer id>.. Using the results from the index scan we will make reseek on the main region to the exact rows where the data what we are interested in is available. So there wont be a full region data scan happening.
When in the cases where only timestamp is there but no customer id, it will be simple again. Create a scanner on the main table with only one filter. At the CP side the scanner on the index region will get created with startkey as 'x' + <timestamp value>.. When you create the scan object and set startRow on that it need not be the full rowkey. It can be part of the rowkey also. Yes like prefix.
Hope u got it now :)
-Anoop-
________________________________________
From: anil gupta [anilgupta84@gmail.com]
Sent: Friday, December 14, 2012 11:31 PM
To: user@hbase.apache.org
Subject: Re: HBase - Secondary Index
On Fri, Dec 14, 2012 at 12:54 AM, Anoop Sam John <an...@huawei.com> wrote:
> Hi Anil,
>
> >1. In your presentation you mentioned that region of Primary Table and
> Region of Secondary Table are always located on the same region server. How
> do you achieve it? By using the Primary table rowkey as prefix of Rowkey
> of Secondary Table? Will your implementation work if the rowkey of primary
> table cannot be used as prefix in rowkey of Secondary table( i have this
> limitation in my use case)?
> First all there will be same number of regions in both primary and index
> tables. All the start/stop keys of the regions also will be same.
> Suppose there are 2 regions on main table say for keys 0-10 and 10-20.
> Then we will create 2 regions in index table also with same key ranges.
> At the master balancing level it is easy to collocate these regions seeing
> the start and end keys.
> When the selection of the rowkey that will be used in the index table is
> the key here.
> What we will do is all the rowkeys in the index table will be prefixed
> with the start key of the region/
> When an entry is added to the main table with rowkey as 5 it will go to
> the 1st region (0-10)
> Now there will be index region with range as 0-10. We will select this
> region to store this index data.
> The row getting added into the index region for this entry will have a
> rowkey 0_x_5
> I am just using '_' as a seperator here just to show this. Actually we
> wont be having any seperator.
> So the rowkeys (in index region) will have a static begin part always.
> Will scan time also we know this part and so the startrow and endrow
> creation for the scan will be possible.. Note that we will store the actual
> table row key as the last part of the index rowkey itself not as a value.
> This is better option in our case of handling the scan index usage also at
> sever side. There is no index data fetch to client side..
>
Anil: My primary table rowkey is customerId+event_id, and my secondary
table rowkey is timestamp+ customerid. In your implementation it seems like
for using secondary index the application needs to know about the
"start_key" of the region(static begin part) it wants to query. Right? Do
you separately manage the logic of determining the region
"start_key"(static begin part) for a scan?
Also, Its possible that while using secondary index the customerId is not
provided. So, i wont be having customer id for all the queries. Hence i
cannot use customer_id as a prefix in rowkey of my Secondary Table.
>
> I feel your use case perfectly fit with our model
>
Anil: Somehow i am unable to fit your implementation into my use case due
to the constraint of static begin part of rowkey in Secondary table. There
seems to be a disconnect. Can you tell me how does my use case fits into
your implementation?
>
> >2. Are you using an Endpoint or Observer for building the secondary index
> table?
> Observer
>
> >3. "Custom balancer do collocation". Is it a custom load balancer of HBase
> Master or something else?
> It is a balancer implementation which will be plugged into Master
>
> >4. Your region split looks interesting. I dont have much info about it.
> Can
> you point to some docs on IndexHalfStoreFileReader?
> Sorry I am not able to publish any design doc or code as the company has
> not decided to open src the solution yet.
> Any particular query you come acorss pls feel free to aske me :)
> You can see the HalfStoreFileReader class 1st..
>
> -Anoop-
> ________________________________________
> From: anil gupta [anilgupta84@gmail.com]
> Sent: Friday, December 14, 2012 2:11 PM
> To: user@hbase.apache.org
> Subject: Re: HBase - Secondary Index
>
> Hi Anoop,
>
> Nice presentation and seems like a smart implementation. Since the
> presentation only covered bullet points so i have couple of questions on
> your implementation. :)
>
> Here is a recap to my implementation and our previous discussion on
> Secondary index:
>
> Here is the link to previous email thread:
> http://search-hadoop.com/m/1zWPMaaRtr .
>
> The secondary index is stored in table "B" as rowkey B --> family:<rowkey
> A> . "<rowkey A>" is the column qualifier. Every row in B will only on
> have one column "k" and the value of that column is the rowkey of A.
>
> Suppose i am storing customer events in table A. I have two requirement for
> data query:
> 1. Query customer events on basis of customer_Id and event_ID.
> 2. Query customer events on basis of event_timestamp and customer_ID.
>
> 70% of querying is done by query#1, so i will create
> <customer_Id><event_ID> as row key of Table A.
> Now, in order to support fast results for query#2, i need to create a
> secondary index on A. I store that secondary index in B, rowkey of B is
> <event_timestamp><customer_ID>.Every row stores the corresponding rowkey of
> A.
>
> HBase Querying approach:
> 1. Scan the secondary table by using prefix filter and startRow to get the
> list of Rowkeys of Primary table.
> 2. Do a batch get on primary table by using HTable.get(List<Get>) method
> using the list of Rowkeys obtained in step1.
>
> The only issue is that in my solution i have at least two RPC calls. Once
> each in step1 and step2 above. I want to reduce the number of RPC to 1 if
> possible.
>
>
> ******Questions on your implementation:*********
>
> 1. In your presentation you mentioned that region of Primary Table and
> Region of Secondary Table are always located on the same region server. How
> do you achieve it? By using the Primary table rowkey as prefix of Rowkey
> of Secondary Table? Will your implementation work if the rowkey of primary
> table cannot be used as prefix in rowkey of Secondary table( i have this
> limitation in my use case)?
> 2. Are you using an Endpoint or Observer for building the secondary index
> table?
> 3. "Custom balancer do collocation". Is it a custom load balancer of HBase
> Master or something else?
> 4. Your region split looks interesting. I dont have much info about it. Can
> you point to some docs on IndexHalfStoreFileReader?
>
> Thanks,
> Anil Gupta
>
>
>
> On Tue, Dec 4, 2012 at 12:10 AM, Anoop Sam John <an...@huawei.com>
> wrote:
>
> > Hi All
> >
> > Last week I got a chance to present the secondary indexing
> > solution what we have done in Huawei at the China Hadoop Conference. You
> > can see the presentation from
> > http://hbtc2012.hadooper.cn/subject/track4Anoop%20Sam%20John2.pdf
> >
> >
> >
> > I would like to hear what others think on this. :)
> >
> >
> >
> > -Anoop-
> >
>
>
>
> --
> Thanks & Regards,
> Anil Gupta
>
--
Thanks & Regards,
Anil Gupta
Re: HBase - Secondary Index
Posted by anil gupta <an...@gmail.com>.
On Fri, Dec 14, 2012 at 12:54 AM, Anoop Sam John <an...@huawei.com> wrote:
> Hi Anil,
>
> >1. In your presentation you mentioned that region of Primary Table and
> Region of Secondary Table are always located on the same region server. How
> do you achieve it? By using the Primary table rowkey as prefix of Rowkey
> of Secondary Table? Will your implementation work if the rowkey of primary
> table cannot be used as prefix in rowkey of Secondary table( i have this
> limitation in my use case)?
> First all there will be same number of regions in both primary and index
> tables. All the start/stop keys of the regions also will be same.
> Suppose there are 2 regions on main table say for keys 0-10 and 10-20.
> Then we will create 2 regions in index table also with same key ranges.
> At the master balancing level it is easy to collocate these regions seeing
> the start and end keys.
> When the selection of the rowkey that will be used in the index table is
> the key here.
> What we will do is all the rowkeys in the index table will be prefixed
> with the start key of the region/
> When an entry is added to the main table with rowkey as 5 it will go to
> the 1st region (0-10)
> Now there will be index region with range as 0-10. We will select this
> region to store this index data.
> The row getting added into the index region for this entry will have a
> rowkey 0_x_5
> I am just using '_' as a seperator here just to show this. Actually we
> wont be having any seperator.
> So the rowkeys (in index region) will have a static begin part always.
> Will scan time also we know this part and so the startrow and endrow
> creation for the scan will be possible.. Note that we will store the actual
> table row key as the last part of the index rowkey itself not as a value.
> This is better option in our case of handling the scan index usage also at
> sever side. There is no index data fetch to client side..
>
Anil: My primary table rowkey is customerId+event_id, and my secondary
table rowkey is timestamp+ customerid. In your implementation it seems like
for using secondary index the application needs to know about the
"start_key" of the region(static begin part) it wants to query. Right? Do
you separately manage the logic of determining the region
"start_key"(static begin part) for a scan?
Also, Its possible that while using secondary index the customerId is not
provided. So, i wont be having customer id for all the queries. Hence i
cannot use customer_id as a prefix in rowkey of my Secondary Table.
>
> I feel your use case perfectly fit with our model
>
Anil: Somehow i am unable to fit your implementation into my use case due
to the constraint of static begin part of rowkey in Secondary table. There
seems to be a disconnect. Can you tell me how does my use case fits into
your implementation?
>
> >2. Are you using an Endpoint or Observer for building the secondary index
> table?
> Observer
>
> >3. "Custom balancer do collocation". Is it a custom load balancer of HBase
> Master or something else?
> It is a balancer implementation which will be plugged into Master
>
> >4. Your region split looks interesting. I dont have much info about it.
> Can
> you point to some docs on IndexHalfStoreFileReader?
> Sorry I am not able to publish any design doc or code as the company has
> not decided to open src the solution yet.
> Any particular query you come acorss pls feel free to aske me :)
> You can see the HalfStoreFileReader class 1st..
>
> -Anoop-
> ________________________________________
> From: anil gupta [anilgupta84@gmail.com]
> Sent: Friday, December 14, 2012 2:11 PM
> To: user@hbase.apache.org
> Subject: Re: HBase - Secondary Index
>
> Hi Anoop,
>
> Nice presentation and seems like a smart implementation. Since the
> presentation only covered bullet points so i have couple of questions on
> your implementation. :)
>
> Here is a recap to my implementation and our previous discussion on
> Secondary index:
>
> Here is the link to previous email thread:
> http://search-hadoop.com/m/1zWPMaaRtr .
>
> The secondary index is stored in table "B" as rowkey B --> family:<rowkey
> A> . "<rowkey A>" is the column qualifier. Every row in B will only on
> have one column "k" and the value of that column is the rowkey of A.
>
> Suppose i am storing customer events in table A. I have two requirement for
> data query:
> 1. Query customer events on basis of customer_Id and event_ID.
> 2. Query customer events on basis of event_timestamp and customer_ID.
>
> 70% of querying is done by query#1, so i will create
> <customer_Id><event_ID> as row key of Table A.
> Now, in order to support fast results for query#2, i need to create a
> secondary index on A. I store that secondary index in B, rowkey of B is
> <event_timestamp><customer_ID>.Every row stores the corresponding rowkey of
> A.
>
> HBase Querying approach:
> 1. Scan the secondary table by using prefix filter and startRow to get the
> list of Rowkeys of Primary table.
> 2. Do a batch get on primary table by using HTable.get(List<Get>) method
> using the list of Rowkeys obtained in step1.
>
> The only issue is that in my solution i have at least two RPC calls. Once
> each in step1 and step2 above. I want to reduce the number of RPC to 1 if
> possible.
>
>
> ******Questions on your implementation:*********
>
> 1. In your presentation you mentioned that region of Primary Table and
> Region of Secondary Table are always located on the same region server. How
> do you achieve it? By using the Primary table rowkey as prefix of Rowkey
> of Secondary Table? Will your implementation work if the rowkey of primary
> table cannot be used as prefix in rowkey of Secondary table( i have this
> limitation in my use case)?
> 2. Are you using an Endpoint or Observer for building the secondary index
> table?
> 3. "Custom balancer do collocation". Is it a custom load balancer of HBase
> Master or something else?
> 4. Your region split looks interesting. I dont have much info about it. Can
> you point to some docs on IndexHalfStoreFileReader?
>
> Thanks,
> Anil Gupta
>
>
>
> On Tue, Dec 4, 2012 at 12:10 AM, Anoop Sam John <an...@huawei.com>
> wrote:
>
> > Hi All
> >
> > Last week I got a chance to present the secondary indexing
> > solution what we have done in Huawei at the China Hadoop Conference. You
> > can see the presentation from
> > http://hbtc2012.hadooper.cn/subject/track4Anoop%20Sam%20John2.pdf
> >
> >
> >
> > I would like to hear what others think on this. :)
> >
> >
> >
> > -Anoop-
> >
>
>
>
> --
> Thanks & Regards,
> Anil Gupta
>
--
Thanks & Regards,
Anil Gupta
Re: HBase - Secondary Index
Posted by Mohit Anchlia <mo...@gmail.com>.
It makes sense to use inverted indexes when you have unique index columns.
But if you have columns that are evenly distributed then parallel search
makes more sense. It just depends on cardinality of your indexed columns.
On Tue, Jan 8, 2013 at 5:28 PM, anil gupta <an...@gmail.com> wrote:
> +1 on Lars comment.
>
> Either the client gets the rowkey from secondary table and then gets the
> real data from Primary Table. ** OR ** Send the request to all the RS(or
> region) hosting a region of primary table.
>
> Anoop is using the latter mechanism. Both the mechanism have their pros and
> cons. IMO, there is no outright winner.
>
> ~Anil Gupta
>
> On Tue, Jan 8, 2013 at 4:30 PM, lars hofhansl <la...@apache.org> wrote:
>
> > Different use cases.
> >
> >
> > For global point queries you want exactly what you said below.
> > For range scans across many rows you want Anoop's design. As usually it
> > depends.
> >
> >
> > The tradeoff is bringing a lot of unnecessary data to the client vs
> having
> > to contact each region (or at least each region server).
> >
> >
> > -- Lars
> >
> >
> >
> > ________________________________
> > From: Michael Segel <mi...@hotmail.com>
> > To: user@hbase.apache.org
> > Sent: Tuesday, January 8, 2013 6:33 AM
> > Subject: Re: HBase - Secondary Index
> >
> > So if you're using an inverted table / index why on earth are you doing
> it
> > at the region level?
> >
> > I've tried to explain this to others over 6 months ago and its not really
> > a good idea.
> >
> > You're over complicating this and you will end up creating performance
> > bottlenecks when your secondary index is completely orthogonal to your
> row
> > key.
> >
> > To give you an example...
> >
> > Suppose you're CCCIS and you have a large database of auto insurance
> > claims that you've acquired over the years from your Pathways product.
> >
> > Your primary key would be a combination of the Insurance Company's ID and
> > their internal claim ID for the individual claim.
> > Your row would be all of the data associated to that claim.
> >
> > So now lets say you want to find the average cost to repair a front end
> > collision of an S80 Volvo.
> > The make and model of the car would be orthogonal to the initial key.
> This
> > means that the result set containing insurance records for Front End
> > collisions of S80 Volvos would be most likely evenly distributed across
> the
> > cluster's regions.
> >
> > If you used a series of inverted tables, you would be able to use a
> series
> > of get()s to get the result set from each index and then find their
> > intersections. (Note that you could also put them in sort order so that
> the
> > intersections would be fairly straight forward to find.
> >
> > Doing this at the region level isn't so simple.
> >
> > So I have to again ask why go through and over complicate things?
> >
> > Just saying...
> >
> > On Jan 7, 2013, at 7:49 AM, Anoop Sam John <an...@huawei.com> wrote:
> >
> > > Hi,
> > > It is inverted index based on column(s) value(s)
> > > It will be region wise indexing. Can work when some one knows the
> rowkey
> > range or NOT.
> > >
> > > -Anoop-
> > > ________________________________________
> > > From: Mohit Anchlia [mohitanchlia@gmail.com]
> > > Sent: Monday, January 07, 2013 9:47 AM
> > > To: user@hbase.apache.org
> > > Subject: Re: HBase - Secondary Index
> > >
> > > Hi Anoop,
> > >
> > > Am I correct in understanding that this indexing mechanism is only
> > > applicable when you know the row key? It's not an inverted index truly
> > > based on the column value.
> > >
> > > Mohit
> > > On Sun, Jan 6, 2013 at 7:48 PM, Anoop Sam John <an...@huawei.com>
> > wrote:
> > >
> > >> Hi Adrien
> > >> We are making the consistency btw the main table and
> > >> index table and the roll back mentioned below etc using the CP hooks.
> > The
> > >> current hooks were not enough for those though.. I am in the process
> of
> > >> trying to contribute those new hooks, core changes etc now... Once
> all
> > are
> > >> done I will be able to explain in details..
> > >>
> > >> -Anoop-
> > >> ________________________________________
> > >> From: Adrien Mogenet [adrien.mogenet@gmail.com]
> > >> Sent: Monday, January 07, 2013 2:00 AM
> > >> To: user@hbase.apache.org
> > >> Subject: Re: HBase - Secondary Index
> > >>
> > >> Nice topic, perhaps one of the most important for 2013 :-)
> > >> I still don't get how you're ensuring consistency between index table
> > and
> > >> main table, without an external component (such as
> > bookkeeper/zookeeper).
> > >> What's the exact write path in your situation when inserting data ?
> > >> (WAL/RegionObserver, pre/post put/WALedit...)
> > >>
> > >> The underlying question is about how you're ensuring that WALEdit in
> > Index
> > >> and Main tables are perfectly sync'ed, and how you 're able to
> rollback
> > in
> > >> case of issue in both WAL ?
> > >>
> > >>
> > >> On Fri, Dec 28, 2012 at 11:55 AM, Shengjie Min <ke...@gmail.com>
> > >> wrote:
> > >>
> > >>>> Yes as you say when the no of rows to be returned is becoming more
> and
> > >>> more the latency will be becoming more. seeks within an HFile block
> is
> > >>> some what expensive op now. (Not much but still) The new encoding
> > >>> prefix
> > >>> trie will be a huge bonus here. There the seeks will be flying.. [Ted
> > >> also
> > >>> presented this in the Hadoop China] Thanks to Matt... :) I am
> trying
> > to
> > >>> measure the scan performance with this new encoding . Trying to >back
> > >> port
> > >>> a simple patch for 94 version just for testing... Yes when the no
> of
> > >>> results to be returned is more and more any index will become less
> > >>> performing as per my study :)
> > >>>
> > >>> yes, you are right, I guess it's just a drawback of any index
> approach.
> > >>> Thanks for the explanation.
> > >>>
> > >>> Shengjie
> > >>>
> > >>> On 28 December 2012 04:14, Anoop Sam John <an...@huawei.com>
> wrote:
> > >>>
> > >>>>> Do you have link to that presentation?
> > >>>>
> > >>>> http://hbtc2012.hadooper.cn/subject/track4TedYu4.pdf
> > >>>>
> > >>>> -Anoop-
> > >>>>
> > >>>> ________________________________________
> > >>>> From: Mohit Anchlia [mohitanchlia@gmail.com]
> > >>>> Sent: Friday, December 28, 2012 9:12 AM
> > >>>> To: user@hbase.apache.org
> > >>>> Subject: Re: HBase - Secondary Index
> > >>>>
> > >>>> On Thu, Dec 27, 2012 at 7:33 PM, Anoop Sam John <anoopsj@huawei.com
> >
> > >>>> wrote:
> > >>>>
> > >>>>> Yes as you say when the no of rows to be returned is becoming more
> > >> and
> > >>>>> more the latency will be becoming more. seeks within an HFile
> block
> > >> is
> > >>>>> some what expensive op now. (Not much but still) The new encoding
> > >>> prefix
> > >>>>> trie will be a huge bonus here. There the seeks will be flying..
> [Ted
> > >>>> also
> > >>>>> presented this in the Hadoop China] Thanks to Matt... :) I am
> > >> trying
> > >>> to
> > >>>>> measure the scan performance with this new encoding . Trying to
> back
> > >>>> port a
> > >>>>> simple patch for 94 version just for testing... Yes when the no
> of
> > >>>>> results to be returned is more and more any index will become less
> > >>>>> performing as per my study :)
> > >>>>>
> > >>>>> Do you have link to that presentation?
> > >>>>
> > >>>>
> > >>>>>> btw, quick question- in your presentation, the scale there is
> > >> seconds
> > >>> or
> > >>>>> mill-seconds:)
> > >>>>>
> > >>>>> It is seconds. Dont consider the exact values. What is the % of
> > >>> increase
> > >>>>> in latency is important :) Those were not high end machines.
> > >>>>>
> > >>>>> -Anoop-
> > >>>>> ________________________________________
> > >>>>> From: Shengjie Min [kelvin.msj@gmail.com]
> > >>>>> Sent: Thursday, December 27, 2012 9:59 PM
> > >>>>> To: user@hbase.apache.org
> > >>>>> Subject: Re: HBase - Secondary Index
> > >>>>>
> > >>>>>> Didnt follow u completely here. There wont be any get()
> happening..
> > >>> As
> > >>>>> the
> > >>>>>> exact rowkey in a region we get from the index table, we can seek
> to
> > >>> the
> > >>>>>> exact position and return that row.
> > >>>>>
> > >>>>> Sorry, When I misused "get()" here, I meant seeking. Yes, if it's
> > >> just
> > >>>>> small number of rows returned, this works perfect. As you said you
> > >> will
> > >>>> get
> > >>>>> the exact rowkey positions per region, and simply seek them. I was
> > >>> trying
> > >>>>> to work out the case that when the number of result rows increases
> > >>>>> massively. Like in Anil's case, he wants to do a scan query against
> > >> the
> > >>>>> 2ndary index(timestamp): "select all rows from timestamp1 to
> > >>> timestamp2"
> > >>>>> given no customerId provided. During that time period, he might
> have
> > >> a
> > >>>> big
> > >>>>> chunk of rows from different customerIds. The index table returns a
> > >> lot
> > >>>> of
> > >>>>> rowkey positions for different customerIds (I believe they are
> > >>> scattered
> > >>>> in
> > >>>>> different regions), then you end up seeking all different positions
> > >> in
> > >>>>> different regions and return all the rows needed. According to your
> > >>>>> presentation page14 - Performance Test Results (Scan), without
> index,
> > >>>> it's
> > >>>>> a linear increase as result rows # increases. on the other hand,
> with
> > >>>>> index, time spent climbs up way quicker than the case without
> index.
> > >>>>>
> > >>>>> btw, quick question- in your presentation, the scale there is
> seconds
> > >>> or
> > >>>>> mill-seconds:)
> > >>>>>
> > >>>>> - Shengjie
> > >>>>>
> > >>>>>
> > >>>>> On 27 December 2012 15:54, Anoop John <an...@gmail.com>
> wrote:
> > >>>>>
> > >>>>>>> how the massive number of get() is going to
> > >>>>>> perform againt the main table
> > >>>>>>
> > >>>>>> Didnt follow u completely here. There wont be any get()
> happening..
> > >>> As
> > >>>>> the
> > >>>>>> exact rowkey in a region we get from the index table, we can seek
> > >> to
> > >>>> the
> > >>>>>> exact position and return that row.
> > >>>>>>
> > >>>>>> -Anoop-
> > >>>>>>
> > >>>>>> On Thu, Dec 27, 2012 at 6:37 PM, Shengjie Min <
> > >> kelvin.msj@gmail.com>
> > >>>>>> wrote:
> > >>>>>>
> > >>>>>>> how the massive number of get() is going to
> > >>>>>>> perform againt the main table
> > >>>>>>>
> > >>>>>>
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>> --
> > >>>>> All the best,
> > >>>>> Shengjie Min
> > >>>>>
> > >>>>
> > >>>
> > >>>
> > >>>
> > >>> --
> > >>> All the best,
> > >>> Shengjie Min
> > >>>
> > >>
> > >>
> > >>
> > >> --
> > >> Adrien Mogenet
> > >> 06.59.16.64.22
> > >> http://www.mogenet.me
> > >>
>
>
>
>
> --
> Thanks & Regards,
> Anil Gupta
>
Re: HBase - Secondary Index
Posted by Michel Segel <mi...@hotmail.com>.
You haven't provided a use case...
You are defending a design without showing an example of how it its implemented.
Without a concrete example, it's impossible to determine if there is a design flaw in the initial design.
Sorry, but I don't think that enough thought has been done...
I'm trying to understand your reasoning, but without a concrete example... It's kind of hard.
Sent from a remote device. Please excuse any typos...
Mike Segel
On Jan 8, 2013, at 9:22 PM, Anoop Sam John <an...@huawei.com> wrote:
> Totally agree with Lars. The design came up as per our usage and data distribution style etc.
> Also the put performance we were not able to compromise. That is why the region collocation based region based indexing design came :)
> Also as we are having the indexing and index usage every thing happening at server side, there is no need for any change in the client part depending on what type of client u use. Java code or REST APIs or any thing. Also MR based parallel scans any thing can be comparably easy I feel as there is absolutely no changes needed at client side. :)
>
> As Anil said there will be pros and cons for every way and which one suits your usage, needs to be adopted. :)
>
> -Anoop-
> ________________________________________
> From: anil gupta [anilgupta84@gmail.com]
> Sent: Wednesday, January 09, 2013 6:58 AM
> To: user@hbase.apache.org; lars hofhansl
> Subject: Re: HBase - Secondary Index
>
> +1 on Lars comment.
>
> Either the client gets the rowkey from secondary table and then gets the
> real data from Primary Table. ** OR ** Send the request to all the RS(or
> region) hosting a region of primary table.
>
> Anoop is using the latter mechanism. Both the mechanism have their pros and
> cons. IMO, there is no outright winner.
>
> ~Anil Gupta
>
> On Tue, Jan 8, 2013 at 4:30 PM, lars hofhansl <la...@apache.org> wrote:
>
>> Different use cases.
>>
>>
>> For global point queries you want exactly what you said below.
>> For range scans across many rows you want Anoop's design. As usually it
>> depends.
>
Re: HBase - Secondary Index
Posted by Michel Segel <mi...@hotmail.com>.
I suggest you think more about the problem... Whenever I hear someone talk about unnecessary RPC calls in a distributed system.... Well, I get skeptical. Especially in light of 10GBe.
Sloppy code is one thing. Being myopic is another.
This is why I am asking for a more concrete use case. Lars makes a point that there is a wide variety of potential use cases. However trying to solve a bad use case with something that could have a negative impact on overall performance for other use cases... well, I'd like to avoid that if it were possible.
I think I can give you a couple of examples...
First if you had a use case where you really needed a distributed OLTP database, I'd say HBase wasn't the right tool and you should look at Informix's XPS engine,provided IBM still sells it... The point being
Second, HBase sits on top of HDFS. Because of design issues in HDFS, like not having R/W access to files, there are limitations to HBase where we have to deal with things like compactions.
I point this out because its a design constraint that impacts the solutions we design using HBase.
In these two examples, we would want to question the use of HBase as part of the solution. This is why I'm asking for a use case of where indexing at the region level makes sense.
It sounds like the idea is to use the secondary index as a filter on a range scan. Using an inverted table, the columns are in sort order. So that it would be easier and lighter to fetch only the columns within the range of the query. Simple extension to using a secondary inverted table for your index.
Just saying...
Sent from a remote device. Please excuse any typos...
Mike Segel
On Jan 8, 2013, at 10:11 PM, ramkrishna vasudevan <ra...@gmail.com> wrote:
> As far as i can see its more related to using the coprocessor framework in
> this soln that helps us in a great way to avoid unnecessary RPC calls when
> we go with Region level indexing.
>
> Regards
> Ram
>
> On Wed, Jan 9, 2013 at 8:52 AM, Anoop Sam John <an...@huawei.com> wrote:
>
>> Totally agree with Lars. The design came up as per our usage and data
>> distribution style etc.
>> Also the put performance we were not able to compromise. That is why the
>> region collocation based region based indexing design came :)
>> Also as we are having the indexing and index usage every thing happening
>> at server side, there is no need for any change in the client part
>> depending on what type of client u use. Java code or REST APIs or any
>> thing. Also MR based parallel scans any thing can be comparably easy I
>> feel as there is absolutely no changes needed at client side. :)
>>
>> As Anil said there will be pros and cons for every way and which one suits
>> your usage, needs to be adopted. :)
>>
>> -Anoop-
>> ________________________________________
>> From: anil gupta [anilgupta84@gmail.com]
>> Sent: Wednesday, January 09, 2013 6:58 AM
>> To: user@hbase.apache.org; lars hofhansl
>> Subject: Re: HBase - Secondary Index
>>
>> +1 on Lars comment.
>>
>> Either the client gets the rowkey from secondary table and then gets the
>> real data from Primary Table. ** OR ** Send the request to all the RS(or
>> region) hosting a region of primary table.
>>
>> Anoop is using the latter mechanism. Both the mechanism have their pros and
>> cons. IMO, there is no outright winner.
>>
>> ~Anil Gupta
>>
>> On Tue, Jan 8, 2013 at 4:30 PM, lars hofhansl <la...@apache.org> wrote:
>>
>>> Different use cases.
>>>
>>>
>>> For global point queries you want exactly what you said below.
>>> For range scans across many rows you want Anoop's design. As usually it
>>> depends.
>>>
>>>
>>> The tradeoff is bringing a lot of unnecessary data to the client vs
>> having
>>> to contact each region (or at least each region server).
>>>
>>>
>>> -- Lars
>>>
>>>
>>>
>>> ________________________________
>>> From: Michael Segel <mi...@hotmail.com>
>>> To: user@hbase.apache.org
>>> Sent: Tuesday, January 8, 2013 6:33 AM
>>> Subject: Re: HBase - Secondary Index
>>>
>>> So if you're using an inverted table / index why on earth are you doing
>> it
>>> at the region level?
>>>
>>> I've tried to explain this to others over 6 months ago and its not really
>>> a good idea.
>>>
>>> You're over complicating this and you will end up creating performance
>>> bottlenecks when your secondary index is completely orthogonal to your
>> row
>>> key.
>>>
>>> To give you an example...
>>>
>>> Suppose you're CCCIS and you have a large database of auto insurance
>>> claims that you've acquired over the years from your Pathways product.
>>>
>>> Your primary key would be a combination of the Insurance Company's ID and
>>> their internal claim ID for the individual claim.
>>> Your row would be all of the data associated to that claim.
>>>
>>> So now lets say you want to find the average cost to repair a front end
>>> collision of an S80 Volvo.
>>> The make and model of the car would be orthogonal to the initial key.
>> This
>>> means that the result set containing insurance records for Front End
>>> collisions of S80 Volvos would be most likely evenly distributed across
>> the
>>> cluster's regions.
>>>
>>> If you used a series of inverted tables, you would be able to use a
>> series
>>> of get()s to get the result set from each index and then find their
>>> intersections. (Note that you could also put them in sort order so that
>> the
>>> intersections would be fairly straight forward to find.
>>>
>>> Doing this at the region level isn't so simple.
>>>
>>> So I have to again ask why go through and over complicate things?
>>>
>>> Just saying...
>>>
>>> On Jan 7, 2013, at 7:49 AM, Anoop Sam John <an...@huawei.com> wrote:
>>>
>>>> Hi,
>>>> It is inverted index based on column(s) value(s)
>>>> It will be region wise indexing. Can work when some one knows the
>> rowkey
>>> range or NOT.
>>>>
>>>> -Anoop-
>>>> ________________________________________
>>>> From: Mohit Anchlia [mohitanchlia@gmail.com]
>>>> Sent: Monday, January 07, 2013 9:47 AM
>>>> To: user@hbase.apache.org
>>>> Subject: Re: HBase - Secondary Index
>>>>
>>>> Hi Anoop,
>>>>
>>>> Am I correct in understanding that this indexing mechanism is only
>>>> applicable when you know the row key? It's not an inverted index truly
>>>> based on the column value.
>>>>
>>>> Mohit
>>>> On Sun, Jan 6, 2013 at 7:48 PM, Anoop Sam John <an...@huawei.com>
>>> wrote:
>>>>
>>>>> Hi Adrien
>>>>> We are making the consistency btw the main table and
>>>>> index table and the roll back mentioned below etc using the CP hooks.
>>> The
>>>>> current hooks were not enough for those though.. I am in the process
>> of
>>>>> trying to contribute those new hooks, core changes etc now... Once
>> all
>>> are
>>>>> done I will be able to explain in details..
>>>>>
>>>>> -Anoop-
>>>>> ________________________________________
>>>>> From: Adrien Mogenet [adrien.mogenet@gmail.com]
>>>>> Sent: Monday, January 07, 2013 2:00 AM
>>>>> To: user@hbase.apache.org
>>>>> Subject: Re: HBase - Secondary Index
>>>>>
>>>>> Nice topic, perhaps one of the most important for 2013 :-)
>>>>> I still don't get how you're ensuring consistency between index table
>>> and
>>>>> main table, without an external component (such as
>>> bookkeeper/zookeeper).
>>>>> What's the exact write path in your situation when inserting data ?
>>>>> (WAL/RegionObserver, pre/post put/WALedit...)
>>>>>
>>>>> The underlying question is about how you're ensuring that WALEdit in
>>> Index
>>>>> and Main tables are perfectly sync'ed, and how you 're able to
>> rollback
>>> in
>>>>> case of issue in both WAL ?
>>>>>
>>>>>
>>>>> On Fri, Dec 28, 2012 at 11:55 AM, Shengjie Min <ke...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>>> Yes as you say when the no of rows to be returned is becoming more
>> and
>>>>>> more the latency will be becoming more. seeks within an HFile block
>> is
>>>>>> some what expensive op now. (Not much but still) The new encoding
>>>>>> prefix
>>>>>> trie will be a huge bonus here. There the seeks will be flying.. [Ted
>>>>> also
>>>>>> presented this in the Hadoop China] Thanks to Matt... :) I am
>> trying
>>> to
>>>>>> measure the scan performance with this new encoding . Trying to >back
>>>>> port
>>>>>> a simple patch for 94 version just for testing... Yes when the no
>> of
>>>>>> results to be returned is more and more any index will become less
>>>>>> performing as per my study :)
>>>>>>
>>>>>> yes, you are right, I guess it's just a drawback of any index
>> approach.
>>>>>> Thanks for the explanation.
>>>>>>
>>>>>> Shengjie
>>>>>>
>>>>>> On 28 December 2012 04:14, Anoop Sam John <an...@huawei.com>
>> wrote:
>>>>>>
>>>>>>>> Do you have link to that presentation?
>>>>>>>
>>>>>>> http://hbtc2012.hadooper.cn/subject/track4TedYu4.pdf
>>>>>>>
>>>>>>> -Anoop-
>>>>>>>
>>>>>>> ________________________________________
>>>>>>> From: Mohit Anchlia [mohitanchlia@gmail.com]
>>>>>>> Sent: Friday, December 28, 2012 9:12 AM
>>>>>>> To: user@hbase.apache.org
>>>>>>> Subject: Re: HBase - Secondary Index
>>>>>>>
>>>>>>> On Thu, Dec 27, 2012 at 7:33 PM, Anoop Sam John <anoopsj@huawei.com
>>>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Yes as you say when the no of rows to be returned is becoming more
>>>>> and
>>>>>>>> more the latency will be becoming more. seeks within an HFile
>> block
>>>>> is
>>>>>>>> some what expensive op now. (Not much but still) The new encoding
>>>>>> prefix
>>>>>>>> trie will be a huge bonus here. There the seeks will be flying..
>> [Ted
>>>>>>> also
>>>>>>>> presented this in the Hadoop China] Thanks to Matt... :) I am
>>>>> trying
>>>>>> to
>>>>>>>> measure the scan performance with this new encoding . Trying to
>> back
>>>>>>> port a
>>>>>>>> simple patch for 94 version just for testing... Yes when the no
>> of
>>>>>>>> results to be returned is more and more any index will become less
>>>>>>>> performing as per my study :)
>>>>>>>>
>>>>>>>> Do you have link to that presentation?
>>>>>>>
>>>>>>>
>>>>>>>>> btw, quick question- in your presentation, the scale there is
>>>>> seconds
>>>>>> or
>>>>>>>> mill-seconds:)
>>>>>>>>
>>>>>>>> It is seconds. Dont consider the exact values. What is the % of
>>>>>> increase
>>>>>>>> in latency is important :) Those were not high end machines.
>>>>>>>>
>>>>>>>> -Anoop-
>>>>>>>> ________________________________________
>>>>>>>> From: Shengjie Min [kelvin.msj@gmail.com]
>>>>>>>> Sent: Thursday, December 27, 2012 9:59 PM
>>>>>>>> To: user@hbase.apache.org
>>>>>>>> Subject: Re: HBase - Secondary Index
>>>>>>>>
>>>>>>>>> Didnt follow u completely here. There wont be any get()
>> happening..
>>>>>> As
>>>>>>>> the
>>>>>>>>> exact rowkey in a region we get from the index table, we can seek
>> to
>>>>>> the
>>>>>>>>> exact position and return that row.
>>>>>>>>
>>>>>>>> Sorry, When I misused "get()" here, I meant seeking. Yes, if it's
>>>>> just
>>>>>>>> small number of rows returned, this works perfect. As you said you
>>>>> will
>>>>>>> get
>>>>>>>> the exact rowkey positions per region, and simply seek them. I was
>>>>>> trying
>>>>>>>> to work out the case that when the number of result rows increases
>>>>>>>> massively. Like in Anil's case, he wants to do a scan query against
>>>>> the
>>>>>>>> 2ndary index(timestamp): "select all rows from timestamp1 to
>>>>>> timestamp2"
>>>>>>>> given no customerId provided. During that time period, he might
>> have
>>>>> a
>>>>>>> big
>>>>>>>> chunk of rows from different customerIds. The index table returns a
>>>>> lot
>>>>>>> of
>>>>>>>> rowkey positions for different customerIds (I believe they are
>>>>>> scattered
>>>>>>> in
>>>>>>>> different regions), then you end up seeking all different positions
>>>>> in
>>>>>>>> different regions and return all the rows needed. According to your
>>>>>>>> presentation page14 - Performance Test Results (Scan), without
>> index,
>>>>>>> it's
>>>>>>>> a linear increase as result rows # increases. on the other hand,
>> with
>>>>>>>> index, time spent climbs up way quicker than the case without
>> index.
>>>>>>>>
>>>>>>>> btw, quick question- in your presentation, the scale there is
>> seconds
>>>>>> or
>>>>>>>> mill-seconds:)
>>>>>>>>
>>>>>>>> - Shengjie
>>>>>>>>
>>>>>>>>
>>>>>>>> On 27 December 2012 15:54, Anoop John <an...@gmail.com>
>> wrote:
>>>>>>>>
>>>>>>>>>> how the massive number of get() is going to
>>>>>>>>> perform againt the main table
>>>>>>>>>
>>>>>>>>> Didnt follow u completely here. There wont be any get()
>> happening..
>>>>>> As
>>>>>>>> the
>>>>>>>>> exact rowkey in a region we get from the index table, we can seek
>>>>> to
>>>>>>> the
>>>>>>>>> exact position and return that row.
>>>>>>>>>
>>>>>>>>> -Anoop-
>>>>>>>>>
>>>>>>>>> On Thu, Dec 27, 2012 at 6:37 PM, Shengjie Min <
>>>>> kelvin.msj@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> how the massive number of get() is going to
>>>>>>>>>> perform againt the main table
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> All the best,
>>>>>>>> Shengjie Min
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> All the best,
>>>>>> Shengjie Min
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Adrien Mogenet
>>>>> 06.59.16.64.22
>>>>> http://www.mogenet.me
>>
>>
>>
>>
>> --
>> Thanks & Regards,
>> Anil Gupta
>>
Re: HBase - Secondary Index
Posted by ramkrishna vasudevan <ra...@gmail.com>.
As far as i can see its more related to using the coprocessor framework in
this soln that helps us in a great way to avoid unnecessary RPC calls when
we go with Region level indexing.
Regards
Ram
On Wed, Jan 9, 2013 at 8:52 AM, Anoop Sam John <an...@huawei.com> wrote:
> Totally agree with Lars. The design came up as per our usage and data
> distribution style etc.
> Also the put performance we were not able to compromise. That is why the
> region collocation based region based indexing design came :)
> Also as we are having the indexing and index usage every thing happening
> at server side, there is no need for any change in the client part
> depending on what type of client u use. Java code or REST APIs or any
> thing. Also MR based parallel scans any thing can be comparably easy I
> feel as there is absolutely no changes needed at client side. :)
>
> As Anil said there will be pros and cons for every way and which one suits
> your usage, needs to be adopted. :)
>
> -Anoop-
> ________________________________________
> From: anil gupta [anilgupta84@gmail.com]
> Sent: Wednesday, January 09, 2013 6:58 AM
> To: user@hbase.apache.org; lars hofhansl
> Subject: Re: HBase - Secondary Index
>
> +1 on Lars comment.
>
> Either the client gets the rowkey from secondary table and then gets the
> real data from Primary Table. ** OR ** Send the request to all the RS(or
> region) hosting a region of primary table.
>
> Anoop is using the latter mechanism. Both the mechanism have their pros and
> cons. IMO, there is no outright winner.
>
> ~Anil Gupta
>
> On Tue, Jan 8, 2013 at 4:30 PM, lars hofhansl <la...@apache.org> wrote:
>
> > Different use cases.
> >
> >
> > For global point queries you want exactly what you said below.
> > For range scans across many rows you want Anoop's design. As usually it
> > depends.
> >
> >
> > The tradeoff is bringing a lot of unnecessary data to the client vs
> having
> > to contact each region (or at least each region server).
> >
> >
> > -- Lars
> >
> >
> >
> > ________________________________
> > From: Michael Segel <mi...@hotmail.com>
> > To: user@hbase.apache.org
> > Sent: Tuesday, January 8, 2013 6:33 AM
> > Subject: Re: HBase - Secondary Index
> >
> > So if you're using an inverted table / index why on earth are you doing
> it
> > at the region level?
> >
> > I've tried to explain this to others over 6 months ago and its not really
> > a good idea.
> >
> > You're over complicating this and you will end up creating performance
> > bottlenecks when your secondary index is completely orthogonal to your
> row
> > key.
> >
> > To give you an example...
> >
> > Suppose you're CCCIS and you have a large database of auto insurance
> > claims that you've acquired over the years from your Pathways product.
> >
> > Your primary key would be a combination of the Insurance Company's ID and
> > their internal claim ID for the individual claim.
> > Your row would be all of the data associated to that claim.
> >
> > So now lets say you want to find the average cost to repair a front end
> > collision of an S80 Volvo.
> > The make and model of the car would be orthogonal to the initial key.
> This
> > means that the result set containing insurance records for Front End
> > collisions of S80 Volvos would be most likely evenly distributed across
> the
> > cluster's regions.
> >
> > If you used a series of inverted tables, you would be able to use a
> series
> > of get()s to get the result set from each index and then find their
> > intersections. (Note that you could also put them in sort order so that
> the
> > intersections would be fairly straight forward to find.
> >
> > Doing this at the region level isn't so simple.
> >
> > So I have to again ask why go through and over complicate things?
> >
> > Just saying...
> >
> > On Jan 7, 2013, at 7:49 AM, Anoop Sam John <an...@huawei.com> wrote:
> >
> > > Hi,
> > > It is inverted index based on column(s) value(s)
> > > It will be region wise indexing. Can work when some one knows the
> rowkey
> > range or NOT.
> > >
> > > -Anoop-
> > > ________________________________________
> > > From: Mohit Anchlia [mohitanchlia@gmail.com]
> > > Sent: Monday, January 07, 2013 9:47 AM
> > > To: user@hbase.apache.org
> > > Subject: Re: HBase - Secondary Index
> > >
> > > Hi Anoop,
> > >
> > > Am I correct in understanding that this indexing mechanism is only
> > > applicable when you know the row key? It's not an inverted index truly
> > > based on the column value.
> > >
> > > Mohit
> > > On Sun, Jan 6, 2013 at 7:48 PM, Anoop Sam John <an...@huawei.com>
> > wrote:
> > >
> > >> Hi Adrien
> > >> We are making the consistency btw the main table and
> > >> index table and the roll back mentioned below etc using the CP hooks.
> > The
> > >> current hooks were not enough for those though.. I am in the process
> of
> > >> trying to contribute those new hooks, core changes etc now... Once
> all
> > are
> > >> done I will be able to explain in details..
> > >>
> > >> -Anoop-
> > >> ________________________________________
> > >> From: Adrien Mogenet [adrien.mogenet@gmail.com]
> > >> Sent: Monday, January 07, 2013 2:00 AM
> > >> To: user@hbase.apache.org
> > >> Subject: Re: HBase - Secondary Index
> > >>
> > >> Nice topic, perhaps one of the most important for 2013 :-)
> > >> I still don't get how you're ensuring consistency between index table
> > and
> > >> main table, without an external component (such as
> > bookkeeper/zookeeper).
> > >> What's the exact write path in your situation when inserting data ?
> > >> (WAL/RegionObserver, pre/post put/WALedit...)
> > >>
> > >> The underlying question is about how you're ensuring that WALEdit in
> > Index
> > >> and Main tables are perfectly sync'ed, and how you 're able to
> rollback
> > in
> > >> case of issue in both WAL ?
> > >>
> > >>
> > >> On Fri, Dec 28, 2012 at 11:55 AM, Shengjie Min <ke...@gmail.com>
> > >> wrote:
> > >>
> > >>>> Yes as you say when the no of rows to be returned is becoming more
> and
> > >>> more the latency will be becoming more. seeks within an HFile block
> is
> > >>> some what expensive op now. (Not much but still) The new encoding
> > >>> prefix
> > >>> trie will be a huge bonus here. There the seeks will be flying.. [Ted
> > >> also
> > >>> presented this in the Hadoop China] Thanks to Matt... :) I am
> trying
> > to
> > >>> measure the scan performance with this new encoding . Trying to >back
> > >> port
> > >>> a simple patch for 94 version just for testing... Yes when the no
> of
> > >>> results to be returned is more and more any index will become less
> > >>> performing as per my study :)
> > >>>
> > >>> yes, you are right, I guess it's just a drawback of any index
> approach.
> > >>> Thanks for the explanation.
> > >>>
> > >>> Shengjie
> > >>>
> > >>> On 28 December 2012 04:14, Anoop Sam John <an...@huawei.com>
> wrote:
> > >>>
> > >>>>> Do you have link to that presentation?
> > >>>>
> > >>>> http://hbtc2012.hadooper.cn/subject/track4TedYu4.pdf
> > >>>>
> > >>>> -Anoop-
> > >>>>
> > >>>> ________________________________________
> > >>>> From: Mohit Anchlia [mohitanchlia@gmail.com]
> > >>>> Sent: Friday, December 28, 2012 9:12 AM
> > >>>> To: user@hbase.apache.org
> > >>>> Subject: Re: HBase - Secondary Index
> > >>>>
> > >>>> On Thu, Dec 27, 2012 at 7:33 PM, Anoop Sam John <anoopsj@huawei.com
> >
> > >>>> wrote:
> > >>>>
> > >>>>> Yes as you say when the no of rows to be returned is becoming more
> > >> and
> > >>>>> more the latency will be becoming more. seeks within an HFile
> block
> > >> is
> > >>>>> some what expensive op now. (Not much but still) The new encoding
> > >>> prefix
> > >>>>> trie will be a huge bonus here. There the seeks will be flying..
> [Ted
> > >>>> also
> > >>>>> presented this in the Hadoop China] Thanks to Matt... :) I am
> > >> trying
> > >>> to
> > >>>>> measure the scan performance with this new encoding . Trying to
> back
> > >>>> port a
> > >>>>> simple patch for 94 version just for testing... Yes when the no
> of
> > >>>>> results to be returned is more and more any index will become less
> > >>>>> performing as per my study :)
> > >>>>>
> > >>>>> Do you have link to that presentation?
> > >>>>
> > >>>>
> > >>>>>> btw, quick question- in your presentation, the scale there is
> > >> seconds
> > >>> or
> > >>>>> mill-seconds:)
> > >>>>>
> > >>>>> It is seconds. Dont consider the exact values. What is the % of
> > >>> increase
> > >>>>> in latency is important :) Those were not high end machines.
> > >>>>>
> > >>>>> -Anoop-
> > >>>>> ________________________________________
> > >>>>> From: Shengjie Min [kelvin.msj@gmail.com]
> > >>>>> Sent: Thursday, December 27, 2012 9:59 PM
> > >>>>> To: user@hbase.apache.org
> > >>>>> Subject: Re: HBase - Secondary Index
> > >>>>>
> > >>>>>> Didnt follow u completely here. There wont be any get()
> happening..
> > >>> As
> > >>>>> the
> > >>>>>> exact rowkey in a region we get from the index table, we can seek
> to
> > >>> the
> > >>>>>> exact position and return that row.
> > >>>>>
> > >>>>> Sorry, When I misused "get()" here, I meant seeking. Yes, if it's
> > >> just
> > >>>>> small number of rows returned, this works perfect. As you said you
> > >> will
> > >>>> get
> > >>>>> the exact rowkey positions per region, and simply seek them. I was
> > >>> trying
> > >>>>> to work out the case that when the number of result rows increases
> > >>>>> massively. Like in Anil's case, he wants to do a scan query against
> > >> the
> > >>>>> 2ndary index(timestamp): "select all rows from timestamp1 to
> > >>> timestamp2"
> > >>>>> given no customerId provided. During that time period, he might
> have
> > >> a
> > >>>> big
> > >>>>> chunk of rows from different customerIds. The index table returns a
> > >> lot
> > >>>> of
> > >>>>> rowkey positions for different customerIds (I believe they are
> > >>> scattered
> > >>>> in
> > >>>>> different regions), then you end up seeking all different positions
> > >> in
> > >>>>> different regions and return all the rows needed. According to your
> > >>>>> presentation page14 - Performance Test Results (Scan), without
> index,
> > >>>> it's
> > >>>>> a linear increase as result rows # increases. on the other hand,
> with
> > >>>>> index, time spent climbs up way quicker than the case without
> index.
> > >>>>>
> > >>>>> btw, quick question- in your presentation, the scale there is
> seconds
> > >>> or
> > >>>>> mill-seconds:)
> > >>>>>
> > >>>>> - Shengjie
> > >>>>>
> > >>>>>
> > >>>>> On 27 December 2012 15:54, Anoop John <an...@gmail.com>
> wrote:
> > >>>>>
> > >>>>>>> how the massive number of get() is going to
> > >>>>>> perform againt the main table
> > >>>>>>
> > >>>>>> Didnt follow u completely here. There wont be any get()
> happening..
> > >>> As
> > >>>>> the
> > >>>>>> exact rowkey in a region we get from the index table, we can seek
> > >> to
> > >>>> the
> > >>>>>> exact position and return that row.
> > >>>>>>
> > >>>>>> -Anoop-
> > >>>>>>
> > >>>>>> On Thu, Dec 27, 2012 at 6:37 PM, Shengjie Min <
> > >> kelvin.msj@gmail.com>
> > >>>>>> wrote:
> > >>>>>>
> > >>>>>>> how the massive number of get() is going to
> > >>>>>>> perform againt the main table
> > >>>>>>>
> > >>>>>>
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>> --
> > >>>>> All the best,
> > >>>>> Shengjie Min
> > >>>>>
> > >>>>
> > >>>
> > >>>
> > >>>
> > >>> --
> > >>> All the best,
> > >>> Shengjie Min
> > >>>
> > >>
> > >>
> > >>
> > >> --
> > >> Adrien Mogenet
> > >> 06.59.16.64.22
> > >> http://www.mogenet.me
> > >>
>
>
>
>
> --
> Thanks & Regards,
> Anil Gupta
>
RE: HBase - Secondary Index
Posted by Anoop Sam John <an...@huawei.com>.
Totally agree with Lars. The design came up as per our usage and data distribution style etc.
Also the put performance we were not able to compromise. That is why the region collocation based region based indexing design came :)
Also as we are having the indexing and index usage every thing happening at server side, there is no need for any change in the client part depending on what type of client u use. Java code or REST APIs or any thing. Also MR based parallel scans any thing can be comparably easy I feel as there is absolutely no changes needed at client side. :)
As Anil said there will be pros and cons for every way and which one suits your usage, needs to be adopted. :)
-Anoop-
________________________________________
From: anil gupta [anilgupta84@gmail.com]
Sent: Wednesday, January 09, 2013 6:58 AM
To: user@hbase.apache.org; lars hofhansl
Subject: Re: HBase - Secondary Index
+1 on Lars comment.
Either the client gets the rowkey from secondary table and then gets the
real data from Primary Table. ** OR ** Send the request to all the RS(or
region) hosting a region of primary table.
Anoop is using the latter mechanism. Both the mechanism have their pros and
cons. IMO, there is no outright winner.
~Anil Gupta
On Tue, Jan 8, 2013 at 4:30 PM, lars hofhansl <la...@apache.org> wrote:
> Different use cases.
>
>
> For global point queries you want exactly what you said below.
> For range scans across many rows you want Anoop's design. As usually it
> depends.
>
>
> The tradeoff is bringing a lot of unnecessary data to the client vs having
> to contact each region (or at least each region server).
>
>
> -- Lars
>
>
>
> ________________________________
> From: Michael Segel <mi...@hotmail.com>
> To: user@hbase.apache.org
> Sent: Tuesday, January 8, 2013 6:33 AM
> Subject: Re: HBase - Secondary Index
>
> So if you're using an inverted table / index why on earth are you doing it
> at the region level?
>
> I've tried to explain this to others over 6 months ago and its not really
> a good idea.
>
> You're over complicating this and you will end up creating performance
> bottlenecks when your secondary index is completely orthogonal to your row
> key.
>
> To give you an example...
>
> Suppose you're CCCIS and you have a large database of auto insurance
> claims that you've acquired over the years from your Pathways product.
>
> Your primary key would be a combination of the Insurance Company's ID and
> their internal claim ID for the individual claim.
> Your row would be all of the data associated to that claim.
>
> So now lets say you want to find the average cost to repair a front end
> collision of an S80 Volvo.
> The make and model of the car would be orthogonal to the initial key. This
> means that the result set containing insurance records for Front End
> collisions of S80 Volvos would be most likely evenly distributed across the
> cluster's regions.
>
> If you used a series of inverted tables, you would be able to use a series
> of get()s to get the result set from each index and then find their
> intersections. (Note that you could also put them in sort order so that the
> intersections would be fairly straight forward to find.
>
> Doing this at the region level isn't so simple.
>
> So I have to again ask why go through and over complicate things?
>
> Just saying...
>
> On Jan 7, 2013, at 7:49 AM, Anoop Sam John <an...@huawei.com> wrote:
>
> > Hi,
> > It is inverted index based on column(s) value(s)
> > It will be region wise indexing. Can work when some one knows the rowkey
> range or NOT.
> >
> > -Anoop-
> > ________________________________________
> > From: Mohit Anchlia [mohitanchlia@gmail.com]
> > Sent: Monday, January 07, 2013 9:47 AM
> > To: user@hbase.apache.org
> > Subject: Re: HBase - Secondary Index
> >
> > Hi Anoop,
> >
> > Am I correct in understanding that this indexing mechanism is only
> > applicable when you know the row key? It's not an inverted index truly
> > based on the column value.
> >
> > Mohit
> > On Sun, Jan 6, 2013 at 7:48 PM, Anoop Sam John <an...@huawei.com>
> wrote:
> >
> >> Hi Adrien
> >> We are making the consistency btw the main table and
> >> index table and the roll back mentioned below etc using the CP hooks.
> The
> >> current hooks were not enough for those though.. I am in the process of
> >> trying to contribute those new hooks, core changes etc now... Once all
> are
> >> done I will be able to explain in details..
> >>
> >> -Anoop-
> >> ________________________________________
> >> From: Adrien Mogenet [adrien.mogenet@gmail.com]
> >> Sent: Monday, January 07, 2013 2:00 AM
> >> To: user@hbase.apache.org
> >> Subject: Re: HBase - Secondary Index
> >>
> >> Nice topic, perhaps one of the most important for 2013 :-)
> >> I still don't get how you're ensuring consistency between index table
> and
> >> main table, without an external component (such as
> bookkeeper/zookeeper).
> >> What's the exact write path in your situation when inserting data ?
> >> (WAL/RegionObserver, pre/post put/WALedit...)
> >>
> >> The underlying question is about how you're ensuring that WALEdit in
> Index
> >> and Main tables are perfectly sync'ed, and how you 're able to rollback
> in
> >> case of issue in both WAL ?
> >>
> >>
> >> On Fri, Dec 28, 2012 at 11:55 AM, Shengjie Min <ke...@gmail.com>
> >> wrote:
> >>
> >>>> Yes as you say when the no of rows to be returned is becoming more and
> >>> more the latency will be becoming more. seeks within an HFile block is
> >>> some what expensive op now. (Not much but still) The new encoding
> >>> prefix
> >>> trie will be a huge bonus here. There the seeks will be flying.. [Ted
> >> also
> >>> presented this in the Hadoop China] Thanks to Matt... :) I am trying
> to
> >>> measure the scan performance with this new encoding . Trying to >back
> >> port
> >>> a simple patch for 94 version just for testing... Yes when the no of
> >>> results to be returned is more and more any index will become less
> >>> performing as per my study :)
> >>>
> >>> yes, you are right, I guess it's just a drawback of any index approach.
> >>> Thanks for the explanation.
> >>>
> >>> Shengjie
> >>>
> >>> On 28 December 2012 04:14, Anoop Sam John <an...@huawei.com> wrote:
> >>>
> >>>>> Do you have link to that presentation?
> >>>>
> >>>> http://hbtc2012.hadooper.cn/subject/track4TedYu4.pdf
> >>>>
> >>>> -Anoop-
> >>>>
> >>>> ________________________________________
> >>>> From: Mohit Anchlia [mohitanchlia@gmail.com]
> >>>> Sent: Friday, December 28, 2012 9:12 AM
> >>>> To: user@hbase.apache.org
> >>>> Subject: Re: HBase - Secondary Index
> >>>>
> >>>> On Thu, Dec 27, 2012 at 7:33 PM, Anoop Sam John <an...@huawei.com>
> >>>> wrote:
> >>>>
> >>>>> Yes as you say when the no of rows to be returned is becoming more
> >> and
> >>>>> more the latency will be becoming more. seeks within an HFile block
> >> is
> >>>>> some what expensive op now. (Not much but still) The new encoding
> >>> prefix
> >>>>> trie will be a huge bonus here. There the seeks will be flying.. [Ted
> >>>> also
> >>>>> presented this in the Hadoop China] Thanks to Matt... :) I am
> >> trying
> >>> to
> >>>>> measure the scan performance with this new encoding . Trying to back
> >>>> port a
> >>>>> simple patch for 94 version just for testing... Yes when the no of
> >>>>> results to be returned is more and more any index will become less
> >>>>> performing as per my study :)
> >>>>>
> >>>>> Do you have link to that presentation?
> >>>>
> >>>>
> >>>>>> btw, quick question- in your presentation, the scale there is
> >> seconds
> >>> or
> >>>>> mill-seconds:)
> >>>>>
> >>>>> It is seconds. Dont consider the exact values. What is the % of
> >>> increase
> >>>>> in latency is important :) Those were not high end machines.
> >>>>>
> >>>>> -Anoop-
> >>>>> ________________________________________
> >>>>> From: Shengjie Min [kelvin.msj@gmail.com]
> >>>>> Sent: Thursday, December 27, 2012 9:59 PM
> >>>>> To: user@hbase.apache.org
> >>>>> Subject: Re: HBase - Secondary Index
> >>>>>
> >>>>>> Didnt follow u completely here. There wont be any get() happening..
> >>> As
> >>>>> the
> >>>>>> exact rowkey in a region we get from the index table, we can seek to
> >>> the
> >>>>>> exact position and return that row.
> >>>>>
> >>>>> Sorry, When I misused "get()" here, I meant seeking. Yes, if it's
> >> just
> >>>>> small number of rows returned, this works perfect. As you said you
> >> will
> >>>> get
> >>>>> the exact rowkey positions per region, and simply seek them. I was
> >>> trying
> >>>>> to work out the case that when the number of result rows increases
> >>>>> massively. Like in Anil's case, he wants to do a scan query against
> >> the
> >>>>> 2ndary index(timestamp): "select all rows from timestamp1 to
> >>> timestamp2"
> >>>>> given no customerId provided. During that time period, he might have
> >> a
> >>>> big
> >>>>> chunk of rows from different customerIds. The index table returns a
> >> lot
> >>>> of
> >>>>> rowkey positions for different customerIds (I believe they are
> >>> scattered
> >>>> in
> >>>>> different regions), then you end up seeking all different positions
> >> in
> >>>>> different regions and return all the rows needed. According to your
> >>>>> presentation page14 - Performance Test Results (Scan), without index,
> >>>> it's
> >>>>> a linear increase as result rows # increases. on the other hand, with
> >>>>> index, time spent climbs up way quicker than the case without index.
> >>>>>
> >>>>> btw, quick question- in your presentation, the scale there is seconds
> >>> or
> >>>>> mill-seconds:)
> >>>>>
> >>>>> - Shengjie
> >>>>>
> >>>>>
> >>>>> On 27 December 2012 15:54, Anoop John <an...@gmail.com> wrote:
> >>>>>
> >>>>>>> how the massive number of get() is going to
> >>>>>> perform againt the main table
> >>>>>>
> >>>>>> Didnt follow u completely here. There wont be any get() happening..
> >>> As
> >>>>> the
> >>>>>> exact rowkey in a region we get from the index table, we can seek
> >> to
> >>>> the
> >>>>>> exact position and return that row.
> >>>>>>
> >>>>>> -Anoop-
> >>>>>>
> >>>>>> On Thu, Dec 27, 2012 at 6:37 PM, Shengjie Min <
> >> kelvin.msj@gmail.com>
> >>>>>> wrote:
> >>>>>>
> >>>>>>> how the massive number of get() is going to
> >>>>>>> perform againt the main table
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> --
> >>>>> All the best,
> >>>>> Shengjie Min
> >>>>>
> >>>>
> >>>
> >>>
> >>>
> >>> --
> >>> All the best,
> >>> Shengjie Min
> >>>
> >>
> >>
> >>
> >> --
> >> Adrien Mogenet
> >> 06.59.16.64.22
> >> http://www.mogenet.me
> >>
--
Thanks & Regards,
Anil Gupta
Re: HBase - Secondary Index
Posted by anil gupta <an...@gmail.com>.
+1 on Lars comment.
Either the client gets the rowkey from secondary table and then gets the
real data from Primary Table. ** OR ** Send the request to all the RS(or
region) hosting a region of primary table.
Anoop is using the latter mechanism. Both the mechanism have their pros and
cons. IMO, there is no outright winner.
~Anil Gupta
On Tue, Jan 8, 2013 at 4:30 PM, lars hofhansl <la...@apache.org> wrote:
> Different use cases.
>
>
> For global point queries you want exactly what you said below.
> For range scans across many rows you want Anoop's design. As usually it
> depends.
>
>
> The tradeoff is bringing a lot of unnecessary data to the client vs having
> to contact each region (or at least each region server).
>
>
> -- Lars
>
>
>
> ________________________________
> From: Michael Segel <mi...@hotmail.com>
> To: user@hbase.apache.org
> Sent: Tuesday, January 8, 2013 6:33 AM
> Subject: Re: HBase - Secondary Index
>
> So if you're using an inverted table / index why on earth are you doing it
> at the region level?
>
> I've tried to explain this to others over 6 months ago and its not really
> a good idea.
>
> You're over complicating this and you will end up creating performance
> bottlenecks when your secondary index is completely orthogonal to your row
> key.
>
> To give you an example...
>
> Suppose you're CCCIS and you have a large database of auto insurance
> claims that you've acquired over the years from your Pathways product.
>
> Your primary key would be a combination of the Insurance Company's ID and
> their internal claim ID for the individual claim.
> Your row would be all of the data associated to that claim.
>
> So now lets say you want to find the average cost to repair a front end
> collision of an S80 Volvo.
> The make and model of the car would be orthogonal to the initial key. This
> means that the result set containing insurance records for Front End
> collisions of S80 Volvos would be most likely evenly distributed across the
> cluster's regions.
>
> If you used a series of inverted tables, you would be able to use a series
> of get()s to get the result set from each index and then find their
> intersections. (Note that you could also put them in sort order so that the
> intersections would be fairly straight forward to find.
>
> Doing this at the region level isn't so simple.
>
> So I have to again ask why go through and over complicate things?
>
> Just saying...
>
> On Jan 7, 2013, at 7:49 AM, Anoop Sam John <an...@huawei.com> wrote:
>
> > Hi,
> > It is inverted index based on column(s) value(s)
> > It will be region wise indexing. Can work when some one knows the rowkey
> range or NOT.
> >
> > -Anoop-
> > ________________________________________
> > From: Mohit Anchlia [mohitanchlia@gmail.com]
> > Sent: Monday, January 07, 2013 9:47 AM
> > To: user@hbase.apache.org
> > Subject: Re: HBase - Secondary Index
> >
> > Hi Anoop,
> >
> > Am I correct in understanding that this indexing mechanism is only
> > applicable when you know the row key? It's not an inverted index truly
> > based on the column value.
> >
> > Mohit
> > On Sun, Jan 6, 2013 at 7:48 PM, Anoop Sam John <an...@huawei.com>
> wrote:
> >
> >> Hi Adrien
> >> We are making the consistency btw the main table and
> >> index table and the roll back mentioned below etc using the CP hooks.
> The
> >> current hooks were not enough for those though.. I am in the process of
> >> trying to contribute those new hooks, core changes etc now... Once all
> are
> >> done I will be able to explain in details..
> >>
> >> -Anoop-
> >> ________________________________________
> >> From: Adrien Mogenet [adrien.mogenet@gmail.com]
> >> Sent: Monday, January 07, 2013 2:00 AM
> >> To: user@hbase.apache.org
> >> Subject: Re: HBase - Secondary Index
> >>
> >> Nice topic, perhaps one of the most important for 2013 :-)
> >> I still don't get how you're ensuring consistency between index table
> and
> >> main table, without an external component (such as
> bookkeeper/zookeeper).
> >> What's the exact write path in your situation when inserting data ?
> >> (WAL/RegionObserver, pre/post put/WALedit...)
> >>
> >> The underlying question is about how you're ensuring that WALEdit in
> Index
> >> and Main tables are perfectly sync'ed, and how you 're able to rollback
> in
> >> case of issue in both WAL ?
> >>
> >>
> >> On Fri, Dec 28, 2012 at 11:55 AM, Shengjie Min <ke...@gmail.com>
> >> wrote:
> >>
> >>>> Yes as you say when the no of rows to be returned is becoming more and
> >>> more the latency will be becoming more. seeks within an HFile block is
> >>> some what expensive op now. (Not much but still) The new encoding
> >>> prefix
> >>> trie will be a huge bonus here. There the seeks will be flying.. [Ted
> >> also
> >>> presented this in the Hadoop China] Thanks to Matt... :) I am trying
> to
> >>> measure the scan performance with this new encoding . Trying to >back
> >> port
> >>> a simple patch for 94 version just for testing... Yes when the no of
> >>> results to be returned is more and more any index will become less
> >>> performing as per my study :)
> >>>
> >>> yes, you are right, I guess it's just a drawback of any index approach.
> >>> Thanks for the explanation.
> >>>
> >>> Shengjie
> >>>
> >>> On 28 December 2012 04:14, Anoop Sam John <an...@huawei.com> wrote:
> >>>
> >>>>> Do you have link to that presentation?
> >>>>
> >>>> http://hbtc2012.hadooper.cn/subject/track4TedYu4.pdf
> >>>>
> >>>> -Anoop-
> >>>>
> >>>> ________________________________________
> >>>> From: Mohit Anchlia [mohitanchlia@gmail.com]
> >>>> Sent: Friday, December 28, 2012 9:12 AM
> >>>> To: user@hbase.apache.org
> >>>> Subject: Re: HBase - Secondary Index
> >>>>
> >>>> On Thu, Dec 27, 2012 at 7:33 PM, Anoop Sam John <an...@huawei.com>
> >>>> wrote:
> >>>>
> >>>>> Yes as you say when the no of rows to be returned is becoming more
> >> and
> >>>>> more the latency will be becoming more. seeks within an HFile block
> >> is
> >>>>> some what expensive op now. (Not much but still) The new encoding
> >>> prefix
> >>>>> trie will be a huge bonus here. There the seeks will be flying.. [Ted
> >>>> also
> >>>>> presented this in the Hadoop China] Thanks to Matt... :) I am
> >> trying
> >>> to
> >>>>> measure the scan performance with this new encoding . Trying to back
> >>>> port a
> >>>>> simple patch for 94 version just for testing... Yes when the no of
> >>>>> results to be returned is more and more any index will become less
> >>>>> performing as per my study :)
> >>>>>
> >>>>> Do you have link to that presentation?
> >>>>
> >>>>
> >>>>>> btw, quick question- in your presentation, the scale there is
> >> seconds
> >>> or
> >>>>> mill-seconds:)
> >>>>>
> >>>>> It is seconds. Dont consider the exact values. What is the % of
> >>> increase
> >>>>> in latency is important :) Those were not high end machines.
> >>>>>
> >>>>> -Anoop-
> >>>>> ________________________________________
> >>>>> From: Shengjie Min [kelvin.msj@gmail.com]
> >>>>> Sent: Thursday, December 27, 2012 9:59 PM
> >>>>> To: user@hbase.apache.org
> >>>>> Subject: Re: HBase - Secondary Index
> >>>>>
> >>>>>> Didnt follow u completely here. There wont be any get() happening..
> >>> As
> >>>>> the
> >>>>>> exact rowkey in a region we get from the index table, we can seek to
> >>> the
> >>>>>> exact position and return that row.
> >>>>>
> >>>>> Sorry, When I misused "get()" here, I meant seeking. Yes, if it's
> >> just
> >>>>> small number of rows returned, this works perfect. As you said you
> >> will
> >>>> get
> >>>>> the exact rowkey positions per region, and simply seek them. I was
> >>> trying
> >>>>> to work out the case that when the number of result rows increases
> >>>>> massively. Like in Anil's case, he wants to do a scan query against
> >> the
> >>>>> 2ndary index(timestamp): "select all rows from timestamp1 to
> >>> timestamp2"
> >>>>> given no customerId provided. During that time period, he might have
> >> a
> >>>> big
> >>>>> chunk of rows from different customerIds. The index table returns a
> >> lot
> >>>> of
> >>>>> rowkey positions for different customerIds (I believe they are
> >>> scattered
> >>>> in
> >>>>> different regions), then you end up seeking all different positions
> >> in
> >>>>> different regions and return all the rows needed. According to your
> >>>>> presentation page14 - Performance Test Results (Scan), without index,
> >>>> it's
> >>>>> a linear increase as result rows # increases. on the other hand, with
> >>>>> index, time spent climbs up way quicker than the case without index.
> >>>>>
> >>>>> btw, quick question- in your presentation, the scale there is seconds
> >>> or
> >>>>> mill-seconds:)
> >>>>>
> >>>>> - Shengjie
> >>>>>
> >>>>>
> >>>>> On 27 December 2012 15:54, Anoop John <an...@gmail.com> wrote:
> >>>>>
> >>>>>>> how the massive number of get() is going to
> >>>>>> perform againt the main table
> >>>>>>
> >>>>>> Didnt follow u completely here. There wont be any get() happening..
> >>> As
> >>>>> the
> >>>>>> exact rowkey in a region we get from the index table, we can seek
> >> to
> >>>> the
> >>>>>> exact position and return that row.
> >>>>>>
> >>>>>> -Anoop-
> >>>>>>
> >>>>>> On Thu, Dec 27, 2012 at 6:37 PM, Shengjie Min <
> >> kelvin.msj@gmail.com>
> >>>>>> wrote:
> >>>>>>
> >>>>>>> how the massive number of get() is going to
> >>>>>>> perform againt the main table
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> --
> >>>>> All the best,
> >>>>> Shengjie Min
> >>>>>
> >>>>
> >>>
> >>>
> >>>
> >>> --
> >>> All the best,
> >>> Shengjie Min
> >>>
> >>
> >>
> >>
> >> --
> >> Adrien Mogenet
> >> 06.59.16.64.22
> >> http://www.mogenet.me
> >>
--
Thanks & Regards,
Anil Gupta
Re: HBase - Secondary Index
Posted by Michel Segel <mi...@hotmail.com>.
Can you provide a use case?
Sent from a remote device. Please excuse any typos...
Mike Segel
On Jan 8, 2013, at 6:30 PM, lars hofhansl <la...@apache.org> wrote:
> Different use cases.
>
>
> For global point queries you want exactly what you said below.
> For range scans across many rows you want Anoop's design. As usually it depends.
>
>
> The tradeoff is bringing a lot of unnecessary data to the client vs having to contact each region (or at least each region server).
>
>
> -- Lars
>
>
>
> ________________________________
> From: Michael Segel <mi...@hotmail.com>
> To: user@hbase.apache.org
> Sent: Tuesday, January 8, 2013 6:33 AM
> Subject: Re: HBase - Secondary Index
>
> So if you're using an inverted table / index why on earth are you doing it at the region level?
>
> I've tried to explain this to others over 6 months ago and its not really a good idea.
>
> You're over complicating this and you will end up creating performance bottlenecks when your secondary index is completely orthogonal to your row key.
>
> To give you an example...
>
> Suppose you're CCCIS and you have a large database of auto insurance claims that you've acquired over the years from your Pathways product.
>
> Your primary key would be a combination of the Insurance Company's ID and their internal claim ID for the individual claim.
> Your row would be all of the data associated to that claim.
>
> So now lets say you want to find the average cost to repair a front end collision of an S80 Volvo.
> The make and model of the car would be orthogonal to the initial key. This means that the result set containing insurance records for Front End collisions of S80 Volvos would be most likely evenly distributed across the cluster's regions.
>
> If you used a series of inverted tables, you would be able to use a series of get()s to get the result set from each index and then find their intersections. (Note that you could also put them in sort order so that the intersections would be fairly straight forward to find.
>
> Doing this at the region level isn't so simple.
>
> So I have to again ask why go through and over complicate things?
>
> Just saying...
>
> On Jan 7, 2013, at 7:49 AM, Anoop Sam John <an...@huawei.com> wrote:
>
>> Hi,
>> It is inverted index based on column(s) value(s)
>> It will be region wise indexing. Can work when some one knows the rowkey range or NOT.
>>
>> -Anoop-
>> ________________________________________
>> From: Mohit Anchlia [mohitanchlia@gmail.com]
>> Sent: Monday, January 07, 2013 9:47 AM
>> To: user@hbase.apache.org
>> Subject: Re: HBase - Secondary Index
>>
>> Hi Anoop,
>>
>> Am I correct in understanding that this indexing mechanism is only
>> applicable when you know the row key? It's not an inverted index truly
>> based on the column value.
>>
>> Mohit
>> On Sun, Jan 6, 2013 at 7:48 PM, Anoop Sam John <an...@huawei.com> wrote:
>>
>>> Hi Adrien
>>> We are making the consistency btw the main table and
>>> index table and the roll back mentioned below etc using the CP hooks. The
>>> current hooks were not enough for those though.. I am in the process of
>>> trying to contribute those new hooks, core changes etc now... Once all are
>>> done I will be able to explain in details..
>>>
>>> -Anoop-
>>> ________________________________________
>>> From: Adrien Mogenet [adrien.mogenet@gmail.com]
>>> Sent: Monday, January 07, 2013 2:00 AM
>>> To: user@hbase.apache.org
>>> Subject: Re: HBase - Secondary Index
>>>
>>> Nice topic, perhaps one of the most important for 2013 :-)
>>> I still don't get how you're ensuring consistency between index table and
>>> main table, without an external component (such as bookkeeper/zookeeper).
>>> What's the exact write path in your situation when inserting data ?
>>> (WAL/RegionObserver, pre/post put/WALedit...)
>>>
>>> The underlying question is about how you're ensuring that WALEdit in Index
>>> and Main tables are perfectly sync'ed, and how you 're able to rollback in
>>> case of issue in both WAL ?
>>>
>>>
>>> On Fri, Dec 28, 2012 at 11:55 AM, Shengjie Min <ke...@gmail.com>
>>> wrote:
>>>
>>>>> Yes as you say when the no of rows to be returned is becoming more and
>>>> more the latency will be becoming more. seeks within an HFile block is
>>>> some what expensive op now. (Not much but still) The new encoding
>>>> prefix
>>>> trie will be a huge bonus here. There the seeks will be flying.. [Ted
>>> also
>>>> presented this in the Hadoop China] Thanks to Matt... :) I am trying to
>>>> measure the scan performance with this new encoding . Trying to >back
>>> port
>>>> a simple patch for 94 version just for testing... Yes when the no of
>>>> results to be returned is more and more any index will become less
>>>> performing as per my study :)
>>>>
>>>> yes, you are right, I guess it's just a drawback of any index approach.
>>>> Thanks for the explanation.
>>>>
>>>> Shengjie
>>>>
>>>> On 28 December 2012 04:14, Anoop Sam John <an...@huawei.com> wrote:
>>>>
>>>>>> Do you have link to that presentation?
>>>>>
>>>>> http://hbtc2012.hadooper.cn/subject/track4TedYu4.pdf
>>>>>
>>>>> -Anoop-
>>>>>
>>>>> ________________________________________
>>>>> From: Mohit Anchlia [mohitanchlia@gmail.com]
>>>>> Sent: Friday, December 28, 2012 9:12 AM
>>>>> To: user@hbase.apache.org
>>>>> Subject: Re: HBase - Secondary Index
>>>>>
>>>>> On Thu, Dec 27, 2012 at 7:33 PM, Anoop Sam John <an...@huawei.com>
>>>>> wrote:
>>>>>
>>>>>> Yes as you say when the no of rows to be returned is becoming more
>>> and
>>>>>> more the latency will be becoming more. seeks within an HFile block
>>> is
>>>>>> some what expensive op now. (Not much but still) The new encoding
>>>> prefix
>>>>>> trie will be a huge bonus here. There the seeks will be flying.. [Ted
>>>>> also
>>>>>> presented this in the Hadoop China] Thanks to Matt... :) I am
>>> trying
>>>> to
>>>>>> measure the scan performance with this new encoding . Trying to back
>>>>> port a
>>>>>> simple patch for 94 version just for testing... Yes when the no of
>>>>>> results to be returned is more and more any index will become less
>>>>>> performing as per my study :)
>>>>>>
>>>>>> Do you have link to that presentation?
>>>>>
>>>>>
>>>>>>> btw, quick question- in your presentation, the scale there is
>>> seconds
>>>> or
>>>>>> mill-seconds:)
>>>>>>
>>>>>> It is seconds. Dont consider the exact values. What is the % of
>>>> increase
>>>>>> in latency is important :) Those were not high end machines.
>>>>>>
>>>>>> -Anoop-
>>>>>> ________________________________________
>>>>>> From: Shengjie Min [kelvin.msj@gmail.com]
>>>>>> Sent: Thursday, December 27, 2012 9:59 PM
>>>>>> To: user@hbase.apache.org
>>>>>> Subject: Re: HBase - Secondary Index
>>>>>>
>>>>>>> Didnt follow u completely here. There wont be any get() happening..
>>>> As
>>>>>> the
>>>>>>> exact rowkey in a region we get from the index table, we can seek to
>>>> the
>>>>>>> exact position and return that row.
>>>>>>
>>>>>> Sorry, When I misused "get()" here, I meant seeking. Yes, if it's
>>> just
>>>>>> small number of rows returned, this works perfect. As you said you
>>> will
>>>>> get
>>>>>> the exact rowkey positions per region, and simply seek them. I was
>>>> trying
>>>>>> to work out the case that when the number of result rows increases
>>>>>> massively. Like in Anil's case, he wants to do a scan query against
>>> the
>>>>>> 2ndary index(timestamp): "select all rows from timestamp1 to
>>>> timestamp2"
>>>>>> given no customerId provided. During that time period, he might have
>>> a
>>>>> big
>>>>>> chunk of rows from different customerIds. The index table returns a
>>> lot
>>>>> of
>>>>>> rowkey positions for different customerIds (I believe they are
>>>> scattered
>>>>> in
>>>>>> different regions), then you end up seeking all different positions
>>> in
>>>>>> different regions and return all the rows needed. According to your
>>>>>> presentation page14 - Performance Test Results (Scan), without index,
>>>>> it's
>>>>>> a linear increase as result rows # increases. on the other hand, with
>>>>>> index, time spent climbs up way quicker than the case without index.
>>>>>>
>>>>>> btw, quick question- in your presentation, the scale there is seconds
>>>> or
>>>>>> mill-seconds:)
>>>>>>
>>>>>> - Shengjie
>>>>>>
>>>>>>
>>>>>> On 27 December 2012 15:54, Anoop John <an...@gmail.com> wrote:
>>>>>>
>>>>>>>> how the massive number of get() is going to
>>>>>>> perform againt the main table
>>>>>>>
>>>>>>> Didnt follow u completely here. There wont be any get() happening..
>>>> As
>>>>>> the
>>>>>>> exact rowkey in a region we get from the index table, we can seek
>>> to
>>>>> the
>>>>>>> exact position and return that row.
>>>>>>>
>>>>>>> -Anoop-
>>>>>>>
>>>>>>> On Thu, Dec 27, 2012 at 6:37 PM, Shengjie Min <
>>> kelvin.msj@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> how the massive number of get() is going to
>>>>>>>> perform againt the main table
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> All the best,
>>>>>> Shengjie Min
>>>>
>>>>
>>>>
>>>> --
>>>> All the best,
>>>> Shengjie Min
>>>
>>>
>>>
>>> --
>>> Adrien Mogenet
>>> 06.59.16.64.22
>>> http://www.mogenet.me
Re: HBase - Secondary Index
Posted by Michel Segel <mi...@hotmail.com>.
Sorry, this makes no sense...
You are doing a range scan, I get that...
Consider that in an inverted table as your index, each column would be your rowkey which will be in a sort order.
Extend get() to take in a range pair as parameters and limit the result set returned to those columns which fall within your range...
Problem solved. Right?
The RPC and network traffic is kept to a minimum and you are still solving the underlying use case with cleaner code.
Just saying...
Sent from a remote device. Please excuse any typos...
Mike Segel
On Jan 8, 2013, at 6:30 PM, lars hofhansl <la...@apache.org> wrote:
> Different use cases.
>
>
> For global point queries you want exactly what you said below.
> For range scans across many rows you want Anoop's design. As usually it depends.
>
>
> The tradeoff is bringing a lot of unnecessary data to the client vs having to contact each region (or at least each region server).
>
>
> -- Lars
>
>
>
> ________________________________
> From: Michael Segel <mi...@hotmail.com>
> To: user@hbase.apache.org
> Sent: Tuesday, January 8, 2013 6:33 AM
> Subject: Re: HBase - Secondary Index
>
> So if you're using an inverted table / index why on earth are you doing it at the region level?
>
> I've tried to explain this to others over 6 months ago and its not really a good idea.
>
> You're over complicating this and you will end up creating performance bottlenecks when your secondary index is completely orthogonal to your row key.
>
> To give you an example...
>
> Suppose you're CCCIS and you have a large database of auto insurance claims that you've acquired over the years from your Pathways product.
>
> Your primary key would be a combination of the Insurance Company's ID and their internal claim ID for the individual claim.
> Your row would be all of the data associated to that claim.
>
> So now lets say you want to find the average cost to repair a front end collision of an S80 Volvo.
> The make and model of the car would be orthogonal to the initial key. This means that the result set containing insurance records for Front End collisions of S80 Volvos would be most likely evenly distributed across the cluster's regions.
>
> If you used a series of inverted tables, you would be able to use a series of get()s to get the result set from each index and then find their intersections. (Note that you could also put them in sort order so that the intersections would be fairly straight forward to find.
>
> Doing this at the region level isn't so simple.
>
> So I have to again ask why go through and over complicate things?
>
> Just saying...
>
> On Jan 7, 2013, at 7:49 AM, Anoop Sam John <an...@huawei.com> wrote:
>
>> Hi,
>> It is inverted index based on column(s) value(s)
>> It will be region wise indexing. Can work when some one knows the rowkey range or NOT.
>>
>> -Anoop-
>> ________________________________________
>> From: Mohit Anchlia [mohitanchlia@gmail.com]
>> Sent: Monday, January 07, 2013 9:47 AM
>> To: user@hbase.apache.org
>> Subject: Re: HBase - Secondary Index
>>
>> Hi Anoop,
>>
>> Am I correct in understanding that this indexing mechanism is only
>> applicable when you know the row key? It's not an inverted index truly
>> based on the column value.
>>
>> Mohit
>> On Sun, Jan 6, 2013 at 7:48 PM, Anoop Sam John <an...@huawei.com> wrote:
>>
>>> Hi Adrien
>>> We are making the consistency btw the main table and
>>> index table and the roll back mentioned below etc using the CP hooks. The
>>> current hooks were not enough for those though.. I am in the process of
>>> trying to contribute those new hooks, core changes etc now... Once all are
>>> done I will be able to explain in details..
>>>
>>> -Anoop-
>>> ________________________________________
>>> From: Adrien Mogenet [adrien.mogenet@gmail.com]
>>> Sent: Monday, January 07, 2013 2:00 AM
>>> To: user@hbase.apache.org
>>> Subject: Re: HBase - Secondary Index
>>>
>>> Nice topic, perhaps one of the most important for 2013 :-)
>>> I still don't get how you're ensuring consistency between index table and
>>> main table, without an external component (such as bookkeeper/zookeeper).
>>> What's the exact write path in your situation when inserting data ?
>>> (WAL/RegionObserver, pre/post put/WALedit...)
>>>
>>> The underlying question is about how you're ensuring that WALEdit in Index
>>> and Main tables are perfectly sync'ed, and how you 're able to rollback in
>>> case of issue in both WAL ?
>>>
>>>
>>> On Fri, Dec 28, 2012 at 11:55 AM, Shengjie Min <ke...@gmail.com>
>>> wrote:
>>>
>>>>> Yes as you say when the no of rows to be returned is becoming more and
>>>> more the latency will be becoming more. seeks within an HFile block is
>>>> some what expensive op now. (Not much but still) The new encoding
>>>> prefix
>>>> trie will be a huge bonus here. There the seeks will be flying.. [Ted
>>> also
>>>> presented this in the Hadoop China] Thanks to Matt... :) I am trying to
>>>> measure the scan performance with this new encoding . Trying to >back
>>> port
>>>> a simple patch for 94 version just for testing... Yes when the no of
>>>> results to be returned is more and more any index will become less
>>>> performing as per my study :)
>>>>
>>>> yes, you are right, I guess it's just a drawback of any index approach.
>>>> Thanks for the explanation.
>>>>
>>>> Shengjie
>>>>
>>>> On 28 December 2012 04:14, Anoop Sam John <an...@huawei.com> wrote:
>>>>
>>>>>> Do you have link to that presentation?
>>>>>
>>>>> http://hbtc2012.hadooper.cn/subject/track4TedYu4.pdf
>>>>>
>>>>> -Anoop-
>>>>>
>>>>> ________________________________________
>>>>> From: Mohit Anchlia [mohitanchlia@gmail.com]
>>>>> Sent: Friday, December 28, 2012 9:12 AM
>>>>> To: user@hbase.apache.org
>>>>> Subject: Re: HBase - Secondary Index
>>>>>
>>>>> On Thu, Dec 27, 2012 at 7:33 PM, Anoop Sam John <an...@huawei.com>
>>>>> wrote:
>>>>>
>>>>>> Yes as you say when the no of rows to be returned is becoming more
>>> and
>>>>>> more the latency will be becoming more. seeks within an HFile block
>>> is
>>>>>> some what expensive op now. (Not much but still) The new encoding
>>>> prefix
>>>>>> trie will be a huge bonus here. There the seeks will be flying.. [Ted
>>>>> also
>>>>>> presented this in the Hadoop China] Thanks to Matt... :) I am
>>> trying
>>>> to
>>>>>> measure the scan performance with this new encoding . Trying to back
>>>>> port a
>>>>>> simple patch for 94 version just for testing... Yes when the no of
>>>>>> results to be returned is more and more any index will become less
>>>>>> performing as per my study :)
>>>>>>
>>>>>> Do you have link to that presentation?
>>>>>
>>>>>
>>>>>>> btw, quick question- in your presentation, the scale there is
>>> seconds
>>>> or
>>>>>> mill-seconds:)
>>>>>>
>>>>>> It is seconds. Dont consider the exact values. What is the % of
>>>> increase
>>>>>> in latency is important :) Those were not high end machines.
>>>>>>
>>>>>> -Anoop-
>>>>>> ________________________________________
>>>>>> From: Shengjie Min [kelvin.msj@gmail.com]
>>>>>> Sent: Thursday, December 27, 2012 9:59 PM
>>>>>> To: user@hbase.apache.org
>>>>>> Subject: Re: HBase - Secondary Index
>>>>>>
>>>>>>> Didnt follow u completely here. There wont be any get() happening..
>>>> As
>>>>>> the
>>>>>>> exact rowkey in a region we get from the index table, we can seek to
>>>> the
>>>>>>> exact position and return that row.
>>>>>>
>>>>>> Sorry, When I misused "get()" here, I meant seeking. Yes, if it's
>>> just
>>>>>> small number of rows returned, this works perfect. As you said you
>>> will
>>>>> get
>>>>>> the exact rowkey positions per region, and simply seek them. I was
>>>> trying
>>>>>> to work out the case that when the number of result rows increases
>>>>>> massively. Like in Anil's case, he wants to do a scan query against
>>> the
>>>>>> 2ndary index(timestamp): "select all rows from timestamp1 to
>>>> timestamp2"
>>>>>> given no customerId provided. During that time period, he might have
>>> a
>>>>> big
>>>>>> chunk of rows from different customerIds. The index table returns a
>>> lot
>>>>> of
>>>>>> rowkey positions for different customerIds (I believe they are
>>>> scattered
>>>>> in
>>>>>> different regions), then you end up seeking all different positions
>>> in
>>>>>> different regions and return all the rows needed. According to your
>>>>>> presentation page14 - Performance Test Results (Scan), without index,
>>>>> it's
>>>>>> a linear increase as result rows # increases. on the other hand, with
>>>>>> index, time spent climbs up way quicker than the case without index.
>>>>>>
>>>>>> btw, quick question- in your presentation, the scale there is seconds
>>>> or
>>>>>> mill-seconds:)
>>>>>>
>>>>>> - Shengjie
>>>>>>
>>>>>>
>>>>>> On 27 December 2012 15:54, Anoop John <an...@gmail.com> wrote:
>>>>>>
>>>>>>>> how the massive number of get() is going to
>>>>>>> perform againt the main table
>>>>>>>
>>>>>>> Didnt follow u completely here. There wont be any get() happening..
>>>> As
>>>>>> the
>>>>>>> exact rowkey in a region we get from the index table, we can seek
>>> to
>>>>> the
>>>>>>> exact position and return that row.
>>>>>>>
>>>>>>> -Anoop-
>>>>>>>
>>>>>>> On Thu, Dec 27, 2012 at 6:37 PM, Shengjie Min <
>>> kelvin.msj@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> how the massive number of get() is going to
>>>>>>>> perform againt the main table
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> All the best,
>>>>>> Shengjie Min
>>>>
>>>>
>>>>
>>>> --
>>>> All the best,
>>>> Shengjie Min
>>>
>>>
>>>
>>> --
>>> Adrien Mogenet
>>> 06.59.16.64.22
>>> http://www.mogenet.me
Re: HBase - Secondary Index
Posted by lars hofhansl <la...@apache.org>.
Different use cases.
For global point queries you want exactly what you said below.
For range scans across many rows you want Anoop's design. As usually it depends.
The tradeoff is bringing a lot of unnecessary data to the client vs having to contact each region (or at least each region server).
-- Lars
________________________________
From: Michael Segel <mi...@hotmail.com>
To: user@hbase.apache.org
Sent: Tuesday, January 8, 2013 6:33 AM
Subject: Re: HBase - Secondary Index
So if you're using an inverted table / index why on earth are you doing it at the region level?
I've tried to explain this to others over 6 months ago and its not really a good idea.
You're over complicating this and you will end up creating performance bottlenecks when your secondary index is completely orthogonal to your row key.
To give you an example...
Suppose you're CCCIS and you have a large database of auto insurance claims that you've acquired over the years from your Pathways product.
Your primary key would be a combination of the Insurance Company's ID and their internal claim ID for the individual claim.
Your row would be all of the data associated to that claim.
So now lets say you want to find the average cost to repair a front end collision of an S80 Volvo.
The make and model of the car would be orthogonal to the initial key. This means that the result set containing insurance records for Front End collisions of S80 Volvos would be most likely evenly distributed across the cluster's regions.
If you used a series of inverted tables, you would be able to use a series of get()s to get the result set from each index and then find their intersections. (Note that you could also put them in sort order so that the intersections would be fairly straight forward to find.
Doing this at the region level isn't so simple.
So I have to again ask why go through and over complicate things?
Just saying...
On Jan 7, 2013, at 7:49 AM, Anoop Sam John <an...@huawei.com> wrote:
> Hi,
> It is inverted index based on column(s) value(s)
> It will be region wise indexing. Can work when some one knows the rowkey range or NOT.
>
> -Anoop-
> ________________________________________
> From: Mohit Anchlia [mohitanchlia@gmail.com]
> Sent: Monday, January 07, 2013 9:47 AM
> To: user@hbase.apache.org
> Subject: Re: HBase - Secondary Index
>
> Hi Anoop,
>
> Am I correct in understanding that this indexing mechanism is only
> applicable when you know the row key? It's not an inverted index truly
> based on the column value.
>
> Mohit
> On Sun, Jan 6, 2013 at 7:48 PM, Anoop Sam John <an...@huawei.com> wrote:
>
>> Hi Adrien
>> We are making the consistency btw the main table and
>> index table and the roll back mentioned below etc using the CP hooks. The
>> current hooks were not enough for those though.. I am in the process of
>> trying to contribute those new hooks, core changes etc now... Once all are
>> done I will be able to explain in details..
>>
>> -Anoop-
>> ________________________________________
>> From: Adrien Mogenet [adrien.mogenet@gmail.com]
>> Sent: Monday, January 07, 2013 2:00 AM
>> To: user@hbase.apache.org
>> Subject: Re: HBase - Secondary Index
>>
>> Nice topic, perhaps one of the most important for 2013 :-)
>> I still don't get how you're ensuring consistency between index table and
>> main table, without an external component (such as bookkeeper/zookeeper).
>> What's the exact write path in your situation when inserting data ?
>> (WAL/RegionObserver, pre/post put/WALedit...)
>>
>> The underlying question is about how you're ensuring that WALEdit in Index
>> and Main tables are perfectly sync'ed, and how you 're able to rollback in
>> case of issue in both WAL ?
>>
>>
>> On Fri, Dec 28, 2012 at 11:55 AM, Shengjie Min <ke...@gmail.com>
>> wrote:
>>
>>>> Yes as you say when the no of rows to be returned is becoming more and
>>> more the latency will be becoming more. seeks within an HFile block is
>>> some what expensive op now. (Not much but still) The new encoding
>>> prefix
>>> trie will be a huge bonus here. There the seeks will be flying.. [Ted
>> also
>>> presented this in the Hadoop China] Thanks to Matt... :) I am trying to
>>> measure the scan performance with this new encoding . Trying to >back
>> port
>>> a simple patch for 94 version just for testing... Yes when the no of
>>> results to be returned is more and more any index will become less
>>> performing as per my study :)
>>>
>>> yes, you are right, I guess it's just a drawback of any index approach.
>>> Thanks for the explanation.
>>>
>>> Shengjie
>>>
>>> On 28 December 2012 04:14, Anoop Sam John <an...@huawei.com> wrote:
>>>
>>>>> Do you have link to that presentation?
>>>>
>>>> http://hbtc2012.hadooper.cn/subject/track4TedYu4.pdf
>>>>
>>>> -Anoop-
>>>>
>>>> ________________________________________
>>>> From: Mohit Anchlia [mohitanchlia@gmail.com]
>>>> Sent: Friday, December 28, 2012 9:12 AM
>>>> To: user@hbase.apache.org
>>>> Subject: Re: HBase - Secondary Index
>>>>
>>>> On Thu, Dec 27, 2012 at 7:33 PM, Anoop Sam John <an...@huawei.com>
>>>> wrote:
>>>>
>>>>> Yes as you say when the no of rows to be returned is becoming more
>> and
>>>>> more the latency will be becoming more. seeks within an HFile block
>> is
>>>>> some what expensive op now. (Not much but still) The new encoding
>>> prefix
>>>>> trie will be a huge bonus here. There the seeks will be flying.. [Ted
>>>> also
>>>>> presented this in the Hadoop China] Thanks to Matt... :) I am
>> trying
>>> to
>>>>> measure the scan performance with this new encoding . Trying to back
>>>> port a
>>>>> simple patch for 94 version just for testing... Yes when the no of
>>>>> results to be returned is more and more any index will become less
>>>>> performing as per my study :)
>>>>>
>>>>> Do you have link to that presentation?
>>>>
>>>>
>>>>>> btw, quick question- in your presentation, the scale there is
>> seconds
>>> or
>>>>> mill-seconds:)
>>>>>
>>>>> It is seconds. Dont consider the exact values. What is the % of
>>> increase
>>>>> in latency is important :) Those were not high end machines.
>>>>>
>>>>> -Anoop-
>>>>> ________________________________________
>>>>> From: Shengjie Min [kelvin.msj@gmail.com]
>>>>> Sent: Thursday, December 27, 2012 9:59 PM
>>>>> To: user@hbase.apache.org
>>>>> Subject: Re: HBase - Secondary Index
>>>>>
>>>>>> Didnt follow u completely here. There wont be any get() happening..
>>> As
>>>>> the
>>>>>> exact rowkey in a region we get from the index table, we can seek to
>>> the
>>>>>> exact position and return that row.
>>>>>
>>>>> Sorry, When I misused "get()" here, I meant seeking. Yes, if it's
>> just
>>>>> small number of rows returned, this works perfect. As you said you
>> will
>>>> get
>>>>> the exact rowkey positions per region, and simply seek them. I was
>>> trying
>>>>> to work out the case that when the number of result rows increases
>>>>> massively. Like in Anil's case, he wants to do a scan query against
>> the
>>>>> 2ndary index(timestamp): "select all rows from timestamp1 to
>>> timestamp2"
>>>>> given no customerId provided. During that time period, he might have
>> a
>>>> big
>>>>> chunk of rows from different customerIds. The index table returns a
>> lot
>>>> of
>>>>> rowkey positions for different customerIds (I believe they are
>>> scattered
>>>> in
>>>>> different regions), then you end up seeking all different positions
>> in
>>>>> different regions and return all the rows needed. According to your
>>>>> presentation page14 - Performance Test Results (Scan), without index,
>>>> it's
>>>>> a linear increase as result rows # increases. on the other hand, with
>>>>> index, time spent climbs up way quicker than the case without index.
>>>>>
>>>>> btw, quick question- in your presentation, the scale there is seconds
>>> or
>>>>> mill-seconds:)
>>>>>
>>>>> - Shengjie
>>>>>
>>>>>
>>>>> On 27 December 2012 15:54, Anoop John <an...@gmail.com> wrote:
>>>>>
>>>>>>> how the massive number of get() is going to
>>>>>> perform againt the main table
>>>>>>
>>>>>> Didnt follow u completely here. There wont be any get() happening..
>>> As
>>>>> the
>>>>>> exact rowkey in a region we get from the index table, we can seek
>> to
>>>> the
>>>>>> exact position and return that row.
>>>>>>
>>>>>> -Anoop-
>>>>>>
>>>>>> On Thu, Dec 27, 2012 at 6:37 PM, Shengjie Min <
>> kelvin.msj@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> how the massive number of get() is going to
>>>>>>> perform againt the main table
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> All the best,
>>>>> Shengjie Min
>>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> All the best,
>>> Shengjie Min
>>>
>>
>>
>>
>> --
>> Adrien Mogenet
>> 06.59.16.64.22
>> http://www.mogenet.me
>>
Re: HBase - Secondary Index
Posted by Asaf Mesika <as...@gmail.com>.
I guess one reason is the the amount of data traveling. In your design, you
have to query a secondary index table, read all the matched original table
row keys, send them back to the client, and then issue a special scan that
retrieves only those row keys values. In his example, he retrieved 2% of
the data which was around 10 million records, which is around 1 GB
according his key size (800 bytes). That's a lot of bytes being transferred
and throttling your switches. In hi design you read the rowkeys locally,
thus able to apply the rest of the filters, and may eventually return just
100 key values which matches to those extra filters. Thus saving tons of
bandwidth and lots of rpc calls.
In your example, and using his design, you can treat each region as mini
table, each indexing its own data.
Having a secondary indexing solution which also supports join like any
RDBMS does as yet to be found since its fairly complex.
On Tuesday, January 8, 2013, Michael Segel wrote:
> So if you're using an inverted table / index why on earth are you doing it
> at the region level?
>
> I've tried to explain this to others over 6 months ago and its not really
> a good idea.
>
> You're over complicating this and you will end up creating performance
> bottlenecks when your secondary index is completely orthogonal to your row
> key.
>
> To give you an example...
>
> Suppose you're CCCIS and you have a large database of auto insurance
> claims that you've acquired over the years from your Pathways product.
>
> Your primary key would be a combination of the Insurance Company's ID and
> their internal claim ID for the individual claim.
> Your row would be all of the data associated to that claim.
>
> So now lets say you want to find the average cost to repair a front end
> collision of an S80 Volvo.
> The make and model of the car would be orthogonal to the initial key. This
> means that the result set containing insurance records for Front End
> collisions of S80 Volvos would be most likely evenly distributed across the
> cluster's regions.
>
> If you used a series of inverted tables, you would be able to use a series
> of get()s to get the result set from each index and then find their
> intersections. (Note that you could also put them in sort order so that the
> intersections would be fairly straight forward to find.
>
> Doing this at the region level isn't so simple.
>
> So I have to again ask why go through and over complicate things?
>
> Just saying...
>
> On Jan 7, 2013, at 7:49 AM, Anoop Sam John <an...@huawei.com> wrote:
>
> > Hi,
> > It is inverted index based on column(s) value(s)
> > It will be region wise indexing. Can work when some one knows the rowkey
> range or NOT.
> >
> > -Anoop-
> > ________________________________________
> > From: Mohit Anchlia [mohitanchlia@gmail.com]
> > Sent: Monday, January 07, 2013 9:47 AM
> > To: user@hbase.apache.org
> > Subject: Re: HBase - Secondary Index
> >
> > Hi Anoop,
> >
> > Am I correct in understanding that this indexing mechanism is only
> > applicable when you know the row key? It's not an inverted index truly
> > based on the column value.
> >
> > Mohit
> > On Sun, Jan 6, 2013 at 7:48 PM, Anoop Sam John <an...@huawei.com>
> wrote:
> >
> >> Hi Adrien
> >> We are making the consistency btw the main table and
> >> index table and the roll back mentioned below etc using the CP hooks.
> The
> >> current hooks were not enough for those though.. I am in the process of
> >> trying to contribute those new hooks, core changes etc now... Once all
> are
> >> done I will be able to explain in details..
> >>
> >> -Anoop-
> >> ________________________________________
> >> From: Adrien Mogenet [adrien.mogenet@gmail.com]
> >> Sent: Monday, January 07, 2013 2:00 AM
> >> To: user@hbase.apache.org
> >> Subject: Re: HBase - Secondary Index
> >>
> >> Nice topic, perhaps one of the most important for 2013 :-)
> >> I still don't get how you're ensuring consistency between index table
> and
> >> main table, without an external component (such as
> bookkeeper/zookeeper).
> >> What's the exact write path in your situation when inserting data ?
> >> (WAL/RegionObserver, pre/post put/WALedit...)
> >>
> >> The underlying question is about how you're ensuring that WALEdit in
> Index
> >> and Main tables are perfectly sync'ed, and how you 're able to rollback
> in
> >> case of issue in both WAL ?
> >>
> >>
> >> On Fri, Dec 28, 2012 at 11:55 AM, Shengjie Min <ke...@gmail.com>
> >> wrote:
> >>
> >>>> Yes as you say when the no of rows to be returned is becoming more and
> >>> more the latency will be becoming more. seeks within an HFile block is
> >>> some what expensive op now. (Not much but still) The new encoding
> >>> prefix
> >>> trie will be a huge bonus here. There the seeks will be flying.. [Ted
> >> also
> >>> presented this in the Hadoop China] Thanks to Matt... :) I am trying
> to
> >>> measure the scan performance with this new encoding . Trying to >back
> >> port
> >>> a simple patch for 94 version just for testing... Yes when the no of
> >>> results to be returned is more and more any index will become less
> >>> performing as per my study :)
> >>>
> >>> yes, you are right, I guess it's just a drawback of any index approach.
> >>> Thanks for the explanation.
> >>>
> >>> Shengjie
> >>>
> >>> On 28 December 2012 04:14, Anoop Sam John <an...@huawei.com> wrote:
> >>>
> >>>>> Do you have link to that presentation?
> >>>>
> >>>> http://hbtc2012.hadooper.cn/subject/track4TedYu4.pdf
> >>>>
> >>>> -Anoop-
> >>>>
> >>>> ________________________________________
> >>>> From: Mohit Anchlia [mohitanchlia@gmail.com]
> >>>> Sent: Friday, December 28, 2012 9:12 AM
> >>>> To: user@hbase.apache.org
> >>>
Re: HBase - Secondary Index
Posted by Michael Segel <mi...@hotmail.com>.
So if you're using an inverted table / index why on earth are you doing it at the region level?
I've tried to explain this to others over 6 months ago and its not really a good idea.
You're over complicating this and you will end up creating performance bottlenecks when your secondary index is completely orthogonal to your row key.
To give you an example...
Suppose you're CCCIS and you have a large database of auto insurance claims that you've acquired over the years from your Pathways product.
Your primary key would be a combination of the Insurance Company's ID and their internal claim ID for the individual claim.
Your row would be all of the data associated to that claim.
So now lets say you want to find the average cost to repair a front end collision of an S80 Volvo.
The make and model of the car would be orthogonal to the initial key. This means that the result set containing insurance records for Front End collisions of S80 Volvos would be most likely evenly distributed across the cluster's regions.
If you used a series of inverted tables, you would be able to use a series of get()s to get the result set from each index and then find their intersections. (Note that you could also put them in sort order so that the intersections would be fairly straight forward to find.
Doing this at the region level isn't so simple.
So I have to again ask why go through and over complicate things?
Just saying...
On Jan 7, 2013, at 7:49 AM, Anoop Sam John <an...@huawei.com> wrote:
> Hi,
> It is inverted index based on column(s) value(s)
> It will be region wise indexing. Can work when some one knows the rowkey range or NOT.
>
> -Anoop-
> ________________________________________
> From: Mohit Anchlia [mohitanchlia@gmail.com]
> Sent: Monday, January 07, 2013 9:47 AM
> To: user@hbase.apache.org
> Subject: Re: HBase - Secondary Index
>
> Hi Anoop,
>
> Am I correct in understanding that this indexing mechanism is only
> applicable when you know the row key? It's not an inverted index truly
> based on the column value.
>
> Mohit
> On Sun, Jan 6, 2013 at 7:48 PM, Anoop Sam John <an...@huawei.com> wrote:
>
>> Hi Adrien
>> We are making the consistency btw the main table and
>> index table and the roll back mentioned below etc using the CP hooks. The
>> current hooks were not enough for those though.. I am in the process of
>> trying to contribute those new hooks, core changes etc now... Once all are
>> done I will be able to explain in details..
>>
>> -Anoop-
>> ________________________________________
>> From: Adrien Mogenet [adrien.mogenet@gmail.com]
>> Sent: Monday, January 07, 2013 2:00 AM
>> To: user@hbase.apache.org
>> Subject: Re: HBase - Secondary Index
>>
>> Nice topic, perhaps one of the most important for 2013 :-)
>> I still don't get how you're ensuring consistency between index table and
>> main table, without an external component (such as bookkeeper/zookeeper).
>> What's the exact write path in your situation when inserting data ?
>> (WAL/RegionObserver, pre/post put/WALedit...)
>>
>> The underlying question is about how you're ensuring that WALEdit in Index
>> and Main tables are perfectly sync'ed, and how you 're able to rollback in
>> case of issue in both WAL ?
>>
>>
>> On Fri, Dec 28, 2012 at 11:55 AM, Shengjie Min <ke...@gmail.com>
>> wrote:
>>
>>>> Yes as you say when the no of rows to be returned is becoming more and
>>> more the latency will be becoming more. seeks within an HFile block is
>>> some what expensive op now. (Not much but still) The new encoding
>>> prefix
>>> trie will be a huge bonus here. There the seeks will be flying.. [Ted
>> also
>>> presented this in the Hadoop China] Thanks to Matt... :) I am trying to
>>> measure the scan performance with this new encoding . Trying to >back
>> port
>>> a simple patch for 94 version just for testing... Yes when the no of
>>> results to be returned is more and more any index will become less
>>> performing as per my study :)
>>>
>>> yes, you are right, I guess it's just a drawback of any index approach.
>>> Thanks for the explanation.
>>>
>>> Shengjie
>>>
>>> On 28 December 2012 04:14, Anoop Sam John <an...@huawei.com> wrote:
>>>
>>>>> Do you have link to that presentation?
>>>>
>>>> http://hbtc2012.hadooper.cn/subject/track4TedYu4.pdf
>>>>
>>>> -Anoop-
>>>>
>>>> ________________________________________
>>>> From: Mohit Anchlia [mohitanchlia@gmail.com]
>>>> Sent: Friday, December 28, 2012 9:12 AM
>>>> To: user@hbase.apache.org
>>>> Subject: Re: HBase - Secondary Index
>>>>
>>>> On Thu, Dec 27, 2012 at 7:33 PM, Anoop Sam John <an...@huawei.com>
>>>> wrote:
>>>>
>>>>> Yes as you say when the no of rows to be returned is becoming more
>> and
>>>>> more the latency will be becoming more. seeks within an HFile block
>> is
>>>>> some what expensive op now. (Not much but still) The new encoding
>>> prefix
>>>>> trie will be a huge bonus here. There the seeks will be flying.. [Ted
>>>> also
>>>>> presented this in the Hadoop China] Thanks to Matt... :) I am
>> trying
>>> to
>>>>> measure the scan performance with this new encoding . Trying to back
>>>> port a
>>>>> simple patch for 94 version just for testing... Yes when the no of
>>>>> results to be returned is more and more any index will become less
>>>>> performing as per my study :)
>>>>>
>>>>> Do you have link to that presentation?
>>>>
>>>>
>>>>>> btw, quick question- in your presentation, the scale there is
>> seconds
>>> or
>>>>> mill-seconds:)
>>>>>
>>>>> It is seconds. Dont consider the exact values. What is the % of
>>> increase
>>>>> in latency is important :) Those were not high end machines.
>>>>>
>>>>> -Anoop-
>>>>> ________________________________________
>>>>> From: Shengjie Min [kelvin.msj@gmail.com]
>>>>> Sent: Thursday, December 27, 2012 9:59 PM
>>>>> To: user@hbase.apache.org
>>>>> Subject: Re: HBase - Secondary Index
>>>>>
>>>>>> Didnt follow u completely here. There wont be any get() happening..
>>> As
>>>>> the
>>>>>> exact rowkey in a region we get from the index table, we can seek to
>>> the
>>>>>> exact position and return that row.
>>>>>
>>>>> Sorry, When I misused "get()" here, I meant seeking. Yes, if it's
>> just
>>>>> small number of rows returned, this works perfect. As you said you
>> will
>>>> get
>>>>> the exact rowkey positions per region, and simply seek them. I was
>>> trying
>>>>> to work out the case that when the number of result rows increases
>>>>> massively. Like in Anil's case, he wants to do a scan query against
>> the
>>>>> 2ndary index(timestamp): "select all rows from timestamp1 to
>>> timestamp2"
>>>>> given no customerId provided. During that time period, he might have
>> a
>>>> big
>>>>> chunk of rows from different customerIds. The index table returns a
>> lot
>>>> of
>>>>> rowkey positions for different customerIds (I believe they are
>>> scattered
>>>> in
>>>>> different regions), then you end up seeking all different positions
>> in
>>>>> different regions and return all the rows needed. According to your
>>>>> presentation page14 - Performance Test Results (Scan), without index,
>>>> it's
>>>>> a linear increase as result rows # increases. on the other hand, with
>>>>> index, time spent climbs up way quicker than the case without index.
>>>>>
>>>>> btw, quick question- in your presentation, the scale there is seconds
>>> or
>>>>> mill-seconds:)
>>>>>
>>>>> - Shengjie
>>>>>
>>>>>
>>>>> On 27 December 2012 15:54, Anoop John <an...@gmail.com> wrote:
>>>>>
>>>>>>> how the massive number of get() is going to
>>>>>> perform againt the main table
>>>>>>
>>>>>> Didnt follow u completely here. There wont be any get() happening..
>>> As
>>>>> the
>>>>>> exact rowkey in a region we get from the index table, we can seek
>> to
>>>> the
>>>>>> exact position and return that row.
>>>>>>
>>>>>> -Anoop-
>>>>>>
>>>>>> On Thu, Dec 27, 2012 at 6:37 PM, Shengjie Min <
>> kelvin.msj@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> how the massive number of get() is going to
>>>>>>> perform againt the main table
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> All the best,
>>>>> Shengjie Min
>>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> All the best,
>>> Shengjie Min
>>>
>>
>>
>>
>> --
>> Adrien Mogenet
>> 06.59.16.64.22
>> http://www.mogenet.me
>>
RE: HBase - Secondary Index
Posted by Anoop Sam John <an...@huawei.com>.
Hi,
It is inverted index based on column(s) value(s)
It will be region wise indexing. Can work when some one knows the rowkey range or NOT.
-Anoop-
________________________________________
From: Mohit Anchlia [mohitanchlia@gmail.com]
Sent: Monday, January 07, 2013 9:47 AM
To: user@hbase.apache.org
Subject: Re: HBase - Secondary Index
Hi Anoop,
Am I correct in understanding that this indexing mechanism is only
applicable when you know the row key? It's not an inverted index truly
based on the column value.
Mohit
On Sun, Jan 6, 2013 at 7:48 PM, Anoop Sam John <an...@huawei.com> wrote:
> Hi Adrien
> We are making the consistency btw the main table and
> index table and the roll back mentioned below etc using the CP hooks. The
> current hooks were not enough for those though.. I am in the process of
> trying to contribute those new hooks, core changes etc now... Once all are
> done I will be able to explain in details..
>
> -Anoop-
> ________________________________________
> From: Adrien Mogenet [adrien.mogenet@gmail.com]
> Sent: Monday, January 07, 2013 2:00 AM
> To: user@hbase.apache.org
> Subject: Re: HBase - Secondary Index
>
> Nice topic, perhaps one of the most important for 2013 :-)
> I still don't get how you're ensuring consistency between index table and
> main table, without an external component (such as bookkeeper/zookeeper).
> What's the exact write path in your situation when inserting data ?
> (WAL/RegionObserver, pre/post put/WALedit...)
>
> The underlying question is about how you're ensuring that WALEdit in Index
> and Main tables are perfectly sync'ed, and how you 're able to rollback in
> case of issue in both WAL ?
>
>
> On Fri, Dec 28, 2012 at 11:55 AM, Shengjie Min <ke...@gmail.com>
> wrote:
>
> > >Yes as you say when the no of rows to be returned is becoming more and
> > more the latency will be becoming more. seeks within an HFile block is
> > some what expensive op now. (Not much but still) The new encoding
> >prefix
> > trie will be a huge bonus here. There the seeks will be flying.. [Ted
> also
> > presented this in the Hadoop China] Thanks to Matt... :) I am trying to
> > measure the scan performance with this new encoding . Trying to >back
> port
> > a simple patch for 94 version just for testing... Yes when the no of
> > results to be returned is more and more any index will become less
> > performing as per my study :)
> >
> > yes, you are right, I guess it's just a drawback of any index approach.
> > Thanks for the explanation.
> >
> > Shengjie
> >
> > On 28 December 2012 04:14, Anoop Sam John <an...@huawei.com> wrote:
> >
> > > > Do you have link to that presentation?
> > >
> > > http://hbtc2012.hadooper.cn/subject/track4TedYu4.pdf
> > >
> > > -Anoop-
> > >
> > > ________________________________________
> > > From: Mohit Anchlia [mohitanchlia@gmail.com]
> > > Sent: Friday, December 28, 2012 9:12 AM
> > > To: user@hbase.apache.org
> > > Subject: Re: HBase - Secondary Index
> > >
> > > On Thu, Dec 27, 2012 at 7:33 PM, Anoop Sam John <an...@huawei.com>
> > > wrote:
> > >
> > > > Yes as you say when the no of rows to be returned is becoming more
> and
> > > > more the latency will be becoming more. seeks within an HFile block
> is
> > > > some what expensive op now. (Not much but still) The new encoding
> > prefix
> > > > trie will be a huge bonus here. There the seeks will be flying.. [Ted
> > > also
> > > > presented this in the Hadoop China] Thanks to Matt... :) I am
> trying
> > to
> > > > measure the scan performance with this new encoding . Trying to back
> > > port a
> > > > simple patch for 94 version just for testing... Yes when the no of
> > > > results to be returned is more and more any index will become less
> > > > performing as per my study :)
> > > >
> > > > Do you have link to that presentation?
> > >
> > >
> > > > >btw, quick question- in your presentation, the scale there is
> seconds
> > or
> > > > mill-seconds:)
> > > >
> > > > It is seconds. Dont consider the exact values. What is the % of
> > increase
> > > > in latency is important :) Those were not high end machines.
> > > >
> > > > -Anoop-
> > > > ________________________________________
> > > > From: Shengjie Min [kelvin.msj@gmail.com]
> > > > Sent: Thursday, December 27, 2012 9:59 PM
> > > > To: user@hbase.apache.org
> > > > Subject: Re: HBase - Secondary Index
> > > >
> > > > >Didnt follow u completely here. There wont be any get() happening..
> > As
> > > > the
> > > > >exact rowkey in a region we get from the index table, we can seek to
> > the
> > > > >exact position and return that row.
> > > >
> > > > Sorry, When I misused "get()" here, I meant seeking. Yes, if it's
> just
> > > > small number of rows returned, this works perfect. As you said you
> will
> > > get
> > > > the exact rowkey positions per region, and simply seek them. I was
> > trying
> > > > to work out the case that when the number of result rows increases
> > > > massively. Like in Anil's case, he wants to do a scan query against
> the
> > > > 2ndary index(timestamp): "select all rows from timestamp1 to
> > timestamp2"
> > > > given no customerId provided. During that time period, he might have
> a
> > > big
> > > > chunk of rows from different customerIds. The index table returns a
> lot
> > > of
> > > > rowkey positions for different customerIds (I believe they are
> > scattered
> > > in
> > > > different regions), then you end up seeking all different positions
> in
> > > > different regions and return all the rows needed. According to your
> > > > presentation page14 - Performance Test Results (Scan), without index,
> > > it's
> > > > a linear increase as result rows # increases. on the other hand, with
> > > > index, time spent climbs up way quicker than the case without index.
> > > >
> > > > btw, quick question- in your presentation, the scale there is seconds
> > or
> > > > mill-seconds:)
> > > >
> > > > - Shengjie
> > > >
> > > >
> > > > On 27 December 2012 15:54, Anoop John <an...@gmail.com> wrote:
> > > >
> > > > > >how the massive number of get() is going to
> > > > > perform againt the main table
> > > > >
> > > > > Didnt follow u completely here. There wont be any get() happening..
> > As
> > > > the
> > > > > exact rowkey in a region we get from the index table, we can seek
> to
> > > the
> > > > > exact position and return that row.
> > > > >
> > > > > -Anoop-
> > > > >
> > > > > On Thu, Dec 27, 2012 at 6:37 PM, Shengjie Min <
> kelvin.msj@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > how the massive number of get() is going to
> > > > > > perform againt the main table
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > All the best,
> > > > Shengjie Min
> > > >
> > >
> >
> >
> >
> > --
> > All the best,
> > Shengjie Min
> >
>
>
>
> --
> Adrien Mogenet
> 06.59.16.64.22
> http://www.mogenet.me
>
Re: HBase - Secondary Index
Posted by Mohit Anchlia <mo...@gmail.com>.
Hi Anoop,
Am I correct in understanding that this indexing mechanism is only
applicable when you know the row key? It's not an inverted index truly
based on the column value.
Mohit
On Sun, Jan 6, 2013 at 7:48 PM, Anoop Sam John <an...@huawei.com> wrote:
> Hi Adrien
> We are making the consistency btw the main table and
> index table and the roll back mentioned below etc using the CP hooks. The
> current hooks were not enough for those though.. I am in the process of
> trying to contribute those new hooks, core changes etc now... Once all are
> done I will be able to explain in details..
>
> -Anoop-
> ________________________________________
> From: Adrien Mogenet [adrien.mogenet@gmail.com]
> Sent: Monday, January 07, 2013 2:00 AM
> To: user@hbase.apache.org
> Subject: Re: HBase - Secondary Index
>
> Nice topic, perhaps one of the most important for 2013 :-)
> I still don't get how you're ensuring consistency between index table and
> main table, without an external component (such as bookkeeper/zookeeper).
> What's the exact write path in your situation when inserting data ?
> (WAL/RegionObserver, pre/post put/WALedit...)
>
> The underlying question is about how you're ensuring that WALEdit in Index
> and Main tables are perfectly sync'ed, and how you 're able to rollback in
> case of issue in both WAL ?
>
>
> On Fri, Dec 28, 2012 at 11:55 AM, Shengjie Min <ke...@gmail.com>
> wrote:
>
> > >Yes as you say when the no of rows to be returned is becoming more and
> > more the latency will be becoming more. seeks within an HFile block is
> > some what expensive op now. (Not much but still) The new encoding
> >prefix
> > trie will be a huge bonus here. There the seeks will be flying.. [Ted
> also
> > presented this in the Hadoop China] Thanks to Matt... :) I am trying to
> > measure the scan performance with this new encoding . Trying to >back
> port
> > a simple patch for 94 version just for testing... Yes when the no of
> > results to be returned is more and more any index will become less
> > performing as per my study :)
> >
> > yes, you are right, I guess it's just a drawback of any index approach.
> > Thanks for the explanation.
> >
> > Shengjie
> >
> > On 28 December 2012 04:14, Anoop Sam John <an...@huawei.com> wrote:
> >
> > > > Do you have link to that presentation?
> > >
> > > http://hbtc2012.hadooper.cn/subject/track4TedYu4.pdf
> > >
> > > -Anoop-
> > >
> > > ________________________________________
> > > From: Mohit Anchlia [mohitanchlia@gmail.com]
> > > Sent: Friday, December 28, 2012 9:12 AM
> > > To: user@hbase.apache.org
> > > Subject: Re: HBase - Secondary Index
> > >
> > > On Thu, Dec 27, 2012 at 7:33 PM, Anoop Sam John <an...@huawei.com>
> > > wrote:
> > >
> > > > Yes as you say when the no of rows to be returned is becoming more
> and
> > > > more the latency will be becoming more. seeks within an HFile block
> is
> > > > some what expensive op now. (Not much but still) The new encoding
> > prefix
> > > > trie will be a huge bonus here. There the seeks will be flying.. [Ted
> > > also
> > > > presented this in the Hadoop China] Thanks to Matt... :) I am
> trying
> > to
> > > > measure the scan performance with this new encoding . Trying to back
> > > port a
> > > > simple patch for 94 version just for testing... Yes when the no of
> > > > results to be returned is more and more any index will become less
> > > > performing as per my study :)
> > > >
> > > > Do you have link to that presentation?
> > >
> > >
> > > > >btw, quick question- in your presentation, the scale there is
> seconds
> > or
> > > > mill-seconds:)
> > > >
> > > > It is seconds. Dont consider the exact values. What is the % of
> > increase
> > > > in latency is important :) Those were not high end machines.
> > > >
> > > > -Anoop-
> > > > ________________________________________
> > > > From: Shengjie Min [kelvin.msj@gmail.com]
> > > > Sent: Thursday, December 27, 2012 9:59 PM
> > > > To: user@hbase.apache.org
> > > > Subject: Re: HBase - Secondary Index
> > > >
> > > > >Didnt follow u completely here. There wont be any get() happening..
> > As
> > > > the
> > > > >exact rowkey in a region we get from the index table, we can seek to
> > the
> > > > >exact position and return that row.
> > > >
> > > > Sorry, When I misused "get()" here, I meant seeking. Yes, if it's
> just
> > > > small number of rows returned, this works perfect. As you said you
> will
> > > get
> > > > the exact rowkey positions per region, and simply seek them. I was
> > trying
> > > > to work out the case that when the number of result rows increases
> > > > massively. Like in Anil's case, he wants to do a scan query against
> the
> > > > 2ndary index(timestamp): "select all rows from timestamp1 to
> > timestamp2"
> > > > given no customerId provided. During that time period, he might have
> a
> > > big
> > > > chunk of rows from different customerIds. The index table returns a
> lot
> > > of
> > > > rowkey positions for different customerIds (I believe they are
> > scattered
> > > in
> > > > different regions), then you end up seeking all different positions
> in
> > > > different regions and return all the rows needed. According to your
> > > > presentation page14 - Performance Test Results (Scan), without index,
> > > it's
> > > > a linear increase as result rows # increases. on the other hand, with
> > > > index, time spent climbs up way quicker than the case without index.
> > > >
> > > > btw, quick question- in your presentation, the scale there is seconds
> > or
> > > > mill-seconds:)
> > > >
> > > > - Shengjie
> > > >
> > > >
> > > > On 27 December 2012 15:54, Anoop John <an...@gmail.com> wrote:
> > > >
> > > > > >how the massive number of get() is going to
> > > > > perform againt the main table
> > > > >
> > > > > Didnt follow u completely here. There wont be any get() happening..
> > As
> > > > the
> > > > > exact rowkey in a region we get from the index table, we can seek
> to
> > > the
> > > > > exact position and return that row.
> > > > >
> > > > > -Anoop-
> > > > >
> > > > > On Thu, Dec 27, 2012 at 6:37 PM, Shengjie Min <
> kelvin.msj@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > how the massive number of get() is going to
> > > > > > perform againt the main table
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > All the best,
> > > > Shengjie Min
> > > >
> > >
> >
> >
> >
> > --
> > All the best,
> > Shengjie Min
> >
>
>
>
> --
> Adrien Mogenet
> 06.59.16.64.22
> http://www.mogenet.me
>
RE: HBase - Secondary Index
Posted by Anoop Sam John <an...@huawei.com>.
Hi Adrien
We are making the consistency btw the main table and index table and the roll back mentioned below etc using the CP hooks. The current hooks were not enough for those though.. I am in the process of trying to contribute those new hooks, core changes etc now... Once all are done I will be able to explain in details..
-Anoop-
________________________________________
From: Adrien Mogenet [adrien.mogenet@gmail.com]
Sent: Monday, January 07, 2013 2:00 AM
To: user@hbase.apache.org
Subject: Re: HBase - Secondary Index
Nice topic, perhaps one of the most important for 2013 :-)
I still don't get how you're ensuring consistency between index table and
main table, without an external component (such as bookkeeper/zookeeper).
What's the exact write path in your situation when inserting data ?
(WAL/RegionObserver, pre/post put/WALedit...)
The underlying question is about how you're ensuring that WALEdit in Index
and Main tables are perfectly sync'ed, and how you 're able to rollback in
case of issue in both WAL ?
On Fri, Dec 28, 2012 at 11:55 AM, Shengjie Min <ke...@gmail.com> wrote:
> >Yes as you say when the no of rows to be returned is becoming more and
> more the latency will be becoming more. seeks within an HFile block is
> some what expensive op now. (Not much but still) The new encoding >prefix
> trie will be a huge bonus here. There the seeks will be flying.. [Ted also
> presented this in the Hadoop China] Thanks to Matt... :) I am trying to
> measure the scan performance with this new encoding . Trying to >back port
> a simple patch for 94 version just for testing... Yes when the no of
> results to be returned is more and more any index will become less
> performing as per my study :)
>
> yes, you are right, I guess it's just a drawback of any index approach.
> Thanks for the explanation.
>
> Shengjie
>
> On 28 December 2012 04:14, Anoop Sam John <an...@huawei.com> wrote:
>
> > > Do you have link to that presentation?
> >
> > http://hbtc2012.hadooper.cn/subject/track4TedYu4.pdf
> >
> > -Anoop-
> >
> > ________________________________________
> > From: Mohit Anchlia [mohitanchlia@gmail.com]
> > Sent: Friday, December 28, 2012 9:12 AM
> > To: user@hbase.apache.org
> > Subject: Re: HBase - Secondary Index
> >
> > On Thu, Dec 27, 2012 at 7:33 PM, Anoop Sam John <an...@huawei.com>
> > wrote:
> >
> > > Yes as you say when the no of rows to be returned is becoming more and
> > > more the latency will be becoming more. seeks within an HFile block is
> > > some what expensive op now. (Not much but still) The new encoding
> prefix
> > > trie will be a huge bonus here. There the seeks will be flying.. [Ted
> > also
> > > presented this in the Hadoop China] Thanks to Matt... :) I am trying
> to
> > > measure the scan performance with this new encoding . Trying to back
> > port a
> > > simple patch for 94 version just for testing... Yes when the no of
> > > results to be returned is more and more any index will become less
> > > performing as per my study :)
> > >
> > > Do you have link to that presentation?
> >
> >
> > > >btw, quick question- in your presentation, the scale there is seconds
> or
> > > mill-seconds:)
> > >
> > > It is seconds. Dont consider the exact values. What is the % of
> increase
> > > in latency is important :) Those were not high end machines.
> > >
> > > -Anoop-
> > > ________________________________________
> > > From: Shengjie Min [kelvin.msj@gmail.com]
> > > Sent: Thursday, December 27, 2012 9:59 PM
> > > To: user@hbase.apache.org
> > > Subject: Re: HBase - Secondary Index
> > >
> > > >Didnt follow u completely here. There wont be any get() happening..
> As
> > > the
> > > >exact rowkey in a region we get from the index table, we can seek to
> the
> > > >exact position and return that row.
> > >
> > > Sorry, When I misused "get()" here, I meant seeking. Yes, if it's just
> > > small number of rows returned, this works perfect. As you said you will
> > get
> > > the exact rowkey positions per region, and simply seek them. I was
> trying
> > > to work out the case that when the number of result rows increases
> > > massively. Like in Anil's case, he wants to do a scan query against the
> > > 2ndary index(timestamp): "select all rows from timestamp1 to
> timestamp2"
> > > given no customerId provided. During that time period, he might have a
> > big
> > > chunk of rows from different customerIds. The index table returns a lot
> > of
> > > rowkey positions for different customerIds (I believe they are
> scattered
> > in
> > > different regions), then you end up seeking all different positions in
> > > different regions and return all the rows needed. According to your
> > > presentation page14 - Performance Test Results (Scan), without index,
> > it's
> > > a linear increase as result rows # increases. on the other hand, with
> > > index, time spent climbs up way quicker than the case without index.
> > >
> > > btw, quick question- in your presentation, the scale there is seconds
> or
> > > mill-seconds:)
> > >
> > > - Shengjie
> > >
> > >
> > > On 27 December 2012 15:54, Anoop John <an...@gmail.com> wrote:
> > >
> > > > >how the massive number of get() is going to
> > > > perform againt the main table
> > > >
> > > > Didnt follow u completely here. There wont be any get() happening..
> As
> > > the
> > > > exact rowkey in a region we get from the index table, we can seek to
> > the
> > > > exact position and return that row.
> > > >
> > > > -Anoop-
> > > >
> > > > On Thu, Dec 27, 2012 at 6:37 PM, Shengjie Min <ke...@gmail.com>
> > > > wrote:
> > > >
> > > > > how the massive number of get() is going to
> > > > > perform againt the main table
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > All the best,
> > > Shengjie Min
> > >
> >
>
>
>
> --
> All the best,
> Shengjie Min
>
--
Adrien Mogenet
06.59.16.64.22
http://www.mogenet.me
Re: HBase - Secondary Index
Posted by anil gupta <an...@gmail.com>.
@Mohit: Here is the jira for prefix compression discussed here:
https://issues.apache.org/jira/browse/HBASE-4676
HTH,
Anil Gupta
On Sun, Jan 6, 2013 at 12:40 PM, Adrien Mogenet <ad...@gmail.com>wrote:
> Are your talking about Data block encoding of K/V ?
> https://issues.apache.org/jira/browse/HBASE-4218
>
>
> On Sun, Jan 6, 2013 at 9:36 PM, Mohit Anchlia <mohitanchlia@gmail.com
> >wrote:
>
> > Does anyone has any links or information to the new prefix encoding
> feature
> > in HBase that's being referred to in this mail?
> >
> > On Sun, Jan 6, 2013 at 12:30 PM, Adrien Mogenet <
> adrien.mogenet@gmail.com
> > >wrote:
> >
> > > Nice topic, perhaps one of the most important for 2013 :-)
> > > I still don't get how you're ensuring consistency between index table
> and
> > > main table, without an external component (such as
> bookkeeper/zookeeper).
> > > What's the exact write path in your situation when inserting data ?
> > > (WAL/RegionObserver, pre/post put/WALedit...)
> > >
> > > The underlying question is about how you're ensuring that WALEdit in
> > Index
> > > and Main tables are perfectly sync'ed, and how you 're able to rollback
> > in
> > > case of issue in both WAL ?
> > >
> > >
> > > On Fri, Dec 28, 2012 at 11:55 AM, Shengjie Min <ke...@gmail.com>
> > > wrote:
> > >
> > > > >Yes as you say when the no of rows to be returned is becoming more
> and
> > > > more the latency will be becoming more. seeks within an HFile block
> is
> > > > some what expensive op now. (Not much but still) The new encoding
> > > >prefix
> > > > trie will be a huge bonus here. There the seeks will be flying.. [Ted
> > > also
> > > > presented this in the Hadoop China] Thanks to Matt... :) I am
> trying
> > to
> > > > measure the scan performance with this new encoding . Trying to >back
> > > port
> > > > a simple patch for 94 version just for testing... Yes when the no
> of
> > > > results to be returned is more and more any index will become less
> > > > performing as per my study :)
> > > >
> > > > yes, you are right, I guess it's just a drawback of any index
> approach.
> > > > Thanks for the explanation.
> > > >
> > > > Shengjie
> > > >
> > > > On 28 December 2012 04:14, Anoop Sam John <an...@huawei.com>
> wrote:
> > > >
> > > > > > Do you have link to that presentation?
> > > > >
> > > > > http://hbtc2012.hadooper.cn/subject/track4TedYu4.pdf
> > > > >
> > > > > -Anoop-
> > > > >
> > > > > ________________________________________
> > > > > From: Mohit Anchlia [mohitanchlia@gmail.com]
> > > > > Sent: Friday, December 28, 2012 9:12 AM
> > > > > To: user@hbase.apache.org
> > > > > Subject: Re: HBase - Secondary Index
> > > > >
> > > > > On Thu, Dec 27, 2012 at 7:33 PM, Anoop Sam John <
> anoopsj@huawei.com>
> > > > > wrote:
> > > > >
> > > > > > Yes as you say when the no of rows to be returned is becoming
> more
> > > and
> > > > > > more the latency will be becoming more. seeks within an HFile
> > block
> > > is
> > > > > > some what expensive op now. (Not much but still) The new
> encoding
> > > > prefix
> > > > > > trie will be a huge bonus here. There the seeks will be flying..
> > [Ted
> > > > > also
> > > > > > presented this in the Hadoop China] Thanks to Matt... :) I am
> > > trying
> > > > to
> > > > > > measure the scan performance with this new encoding . Trying to
> > back
> > > > > port a
> > > > > > simple patch for 94 version just for testing... Yes when the no
> > of
> > > > > > results to be returned is more and more any index will become
> less
> > > > > > performing as per my study :)
> > > > > >
> > > > > > Do you have link to that presentation?
> > > > >
> > > > >
> > > > > > >btw, quick question- in your presentation, the scale there is
> > > seconds
> > > > or
> > > > > > mill-seconds:)
> > > > > >
> > > > > > It is seconds. Dont consider the exact values. What is the % of
> > > > increase
> > > > > > in latency is important :) Those were not high end machines.
> > > > > >
> > > > > > -Anoop-
> > > > > > ________________________________________
> > > > > > From: Shengjie Min [kelvin.msj@gmail.com]
> > > > > > Sent: Thursday, December 27, 2012 9:59 PM
> > > > > > To: user@hbase.apache.org
> > > > > > Subject: Re: HBase - Secondary Index
> > > > > >
> > > > > > >Didnt follow u completely here. There wont be any get()
> > happening..
> > > > As
> > > > > > the
> > > > > > >exact rowkey in a region we get from the index table, we can
> seek
> > to
> > > > the
> > > > > > >exact position and return that row.
> > > > > >
> > > > > > Sorry, When I misused "get()" here, I meant seeking. Yes, if it's
> > > just
> > > > > > small number of rows returned, this works perfect. As you said
> you
> > > will
> > > > > get
> > > > > > the exact rowkey positions per region, and simply seek them. I
> was
> > > > trying
> > > > > > to work out the case that when the number of result rows
> increases
> > > > > > massively. Like in Anil's case, he wants to do a scan query
> against
> > > the
> > > > > > 2ndary index(timestamp): "select all rows from timestamp1 to
> > > > timestamp2"
> > > > > > given no customerId provided. During that time period, he might
> > have
> > > a
> > > > > big
> > > > > > chunk of rows from different customerIds. The index table
> returns a
> > > lot
> > > > > of
> > > > > > rowkey positions for different customerIds (I believe they are
> > > > scattered
> > > > > in
> > > > > > different regions), then you end up seeking all different
> positions
> > > in
> > > > > > different regions and return all the rows needed. According to
> your
> > > > > > presentation page14 - Performance Test Results (Scan), without
> > index,
> > > > > it's
> > > > > > a linear increase as result rows # increases. on the other hand,
> > with
> > > > > > index, time spent climbs up way quicker than the case without
> > index.
> > > > > >
> > > > > > btw, quick question- in your presentation, the scale there is
> > seconds
> > > > or
> > > > > > mill-seconds:)
> > > > > >
> > > > > > - Shengjie
> > > > > >
> > > > > >
> > > > > > On 27 December 2012 15:54, Anoop John <an...@gmail.com>
> > wrote:
> > > > > >
> > > > > > > >how the massive number of get() is going to
> > > > > > > perform againt the main table
> > > > > > >
> > > > > > > Didnt follow u completely here. There wont be any get()
> > happening..
> > > > As
> > > > > > the
> > > > > > > exact rowkey in a region we get from the index table, we can
> seek
> > > to
> > > > > the
> > > > > > > exact position and return that row.
> > > > > > >
> > > > > > > -Anoop-
> > > > > > >
> > > > > > > On Thu, Dec 27, 2012 at 6:37 PM, Shengjie Min <
> > > kelvin.msj@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > how the massive number of get() is going to
> > > > > > > > perform againt the main table
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > All the best,
> > > > > > Shengjie Min
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > All the best,
> > > > Shengjie Min
> > > >
> > >
> > >
> > >
> > > --
> > > Adrien Mogenet
> > > 06.59.16.64.22
> > > http://www.mogenet.me
> > >
> >
>
>
>
> --
> Adrien Mogenet
> 06.59.16.64.22
> http://www.mogenet.me
>
--
Thanks & Regards,
Anil Gupta
Re: HBase - Secondary Index
Posted by Adrien Mogenet <ad...@gmail.com>.
Are your talking about Data block encoding of K/V ?
https://issues.apache.org/jira/browse/HBASE-4218
On Sun, Jan 6, 2013 at 9:36 PM, Mohit Anchlia <mo...@gmail.com>wrote:
> Does anyone has any links or information to the new prefix encoding feature
> in HBase that's being referred to in this mail?
>
> On Sun, Jan 6, 2013 at 12:30 PM, Adrien Mogenet <adrien.mogenet@gmail.com
> >wrote:
>
> > Nice topic, perhaps one of the most important for 2013 :-)
> > I still don't get how you're ensuring consistency between index table and
> > main table, without an external component (such as bookkeeper/zookeeper).
> > What's the exact write path in your situation when inserting data ?
> > (WAL/RegionObserver, pre/post put/WALedit...)
> >
> > The underlying question is about how you're ensuring that WALEdit in
> Index
> > and Main tables are perfectly sync'ed, and how you 're able to rollback
> in
> > case of issue in both WAL ?
> >
> >
> > On Fri, Dec 28, 2012 at 11:55 AM, Shengjie Min <ke...@gmail.com>
> > wrote:
> >
> > > >Yes as you say when the no of rows to be returned is becoming more and
> > > more the latency will be becoming more. seeks within an HFile block is
> > > some what expensive op now. (Not much but still) The new encoding
> > >prefix
> > > trie will be a huge bonus here. There the seeks will be flying.. [Ted
> > also
> > > presented this in the Hadoop China] Thanks to Matt... :) I am trying
> to
> > > measure the scan performance with this new encoding . Trying to >back
> > port
> > > a simple patch for 94 version just for testing... Yes when the no of
> > > results to be returned is more and more any index will become less
> > > performing as per my study :)
> > >
> > > yes, you are right, I guess it's just a drawback of any index approach.
> > > Thanks for the explanation.
> > >
> > > Shengjie
> > >
> > > On 28 December 2012 04:14, Anoop Sam John <an...@huawei.com> wrote:
> > >
> > > > > Do you have link to that presentation?
> > > >
> > > > http://hbtc2012.hadooper.cn/subject/track4TedYu4.pdf
> > > >
> > > > -Anoop-
> > > >
> > > > ________________________________________
> > > > From: Mohit Anchlia [mohitanchlia@gmail.com]
> > > > Sent: Friday, December 28, 2012 9:12 AM
> > > > To: user@hbase.apache.org
> > > > Subject: Re: HBase - Secondary Index
> > > >
> > > > On Thu, Dec 27, 2012 at 7:33 PM, Anoop Sam John <an...@huawei.com>
> > > > wrote:
> > > >
> > > > > Yes as you say when the no of rows to be returned is becoming more
> > and
> > > > > more the latency will be becoming more. seeks within an HFile
> block
> > is
> > > > > some what expensive op now. (Not much but still) The new encoding
> > > prefix
> > > > > trie will be a huge bonus here. There the seeks will be flying..
> [Ted
> > > > also
> > > > > presented this in the Hadoop China] Thanks to Matt... :) I am
> > trying
> > > to
> > > > > measure the scan performance with this new encoding . Trying to
> back
> > > > port a
> > > > > simple patch for 94 version just for testing... Yes when the no
> of
> > > > > results to be returned is more and more any index will become less
> > > > > performing as per my study :)
> > > > >
> > > > > Do you have link to that presentation?
> > > >
> > > >
> > > > > >btw, quick question- in your presentation, the scale there is
> > seconds
> > > or
> > > > > mill-seconds:)
> > > > >
> > > > > It is seconds. Dont consider the exact values. What is the % of
> > > increase
> > > > > in latency is important :) Those were not high end machines.
> > > > >
> > > > > -Anoop-
> > > > > ________________________________________
> > > > > From: Shengjie Min [kelvin.msj@gmail.com]
> > > > > Sent: Thursday, December 27, 2012 9:59 PM
> > > > > To: user@hbase.apache.org
> > > > > Subject: Re: HBase - Secondary Index
> > > > >
> > > > > >Didnt follow u completely here. There wont be any get()
> happening..
> > > As
> > > > > the
> > > > > >exact rowkey in a region we get from the index table, we can seek
> to
> > > the
> > > > > >exact position and return that row.
> > > > >
> > > > > Sorry, When I misused "get()" here, I meant seeking. Yes, if it's
> > just
> > > > > small number of rows returned, this works perfect. As you said you
> > will
> > > > get
> > > > > the exact rowkey positions per region, and simply seek them. I was
> > > trying
> > > > > to work out the case that when the number of result rows increases
> > > > > massively. Like in Anil's case, he wants to do a scan query against
> > the
> > > > > 2ndary index(timestamp): "select all rows from timestamp1 to
> > > timestamp2"
> > > > > given no customerId provided. During that time period, he might
> have
> > a
> > > > big
> > > > > chunk of rows from different customerIds. The index table returns a
> > lot
> > > > of
> > > > > rowkey positions for different customerIds (I believe they are
> > > scattered
> > > > in
> > > > > different regions), then you end up seeking all different positions
> > in
> > > > > different regions and return all the rows needed. According to your
> > > > > presentation page14 - Performance Test Results (Scan), without
> index,
> > > > it's
> > > > > a linear increase as result rows # increases. on the other hand,
> with
> > > > > index, time spent climbs up way quicker than the case without
> index.
> > > > >
> > > > > btw, quick question- in your presentation, the scale there is
> seconds
> > > or
> > > > > mill-seconds:)
> > > > >
> > > > > - Shengjie
> > > > >
> > > > >
> > > > > On 27 December 2012 15:54, Anoop John <an...@gmail.com>
> wrote:
> > > > >
> > > > > > >how the massive number of get() is going to
> > > > > > perform againt the main table
> > > > > >
> > > > > > Didnt follow u completely here. There wont be any get()
> happening..
> > > As
> > > > > the
> > > > > > exact rowkey in a region we get from the index table, we can seek
> > to
> > > > the
> > > > > > exact position and return that row.
> > > > > >
> > > > > > -Anoop-
> > > > > >
> > > > > > On Thu, Dec 27, 2012 at 6:37 PM, Shengjie Min <
> > kelvin.msj@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > how the massive number of get() is going to
> > > > > > > perform againt the main table
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > All the best,
> > > > > Shengjie Min
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > All the best,
> > > Shengjie Min
> > >
> >
> >
> >
> > --
> > Adrien Mogenet
> > 06.59.16.64.22
> > http://www.mogenet.me
> >
>
--
Adrien Mogenet
06.59.16.64.22
http://www.mogenet.me
Re: HBase - Secondary Index
Posted by Mohit Anchlia <mo...@gmail.com>.
Does anyone has any links or information to the new prefix encoding feature
in HBase that's being referred to in this mail?
On Sun, Jan 6, 2013 at 12:30 PM, Adrien Mogenet <ad...@gmail.com>wrote:
> Nice topic, perhaps one of the most important for 2013 :-)
> I still don't get how you're ensuring consistency between index table and
> main table, without an external component (such as bookkeeper/zookeeper).
> What's the exact write path in your situation when inserting data ?
> (WAL/RegionObserver, pre/post put/WALedit...)
>
> The underlying question is about how you're ensuring that WALEdit in Index
> and Main tables are perfectly sync'ed, and how you 're able to rollback in
> case of issue in both WAL ?
>
>
> On Fri, Dec 28, 2012 at 11:55 AM, Shengjie Min <ke...@gmail.com>
> wrote:
>
> > >Yes as you say when the no of rows to be returned is becoming more and
> > more the latency will be becoming more. seeks within an HFile block is
> > some what expensive op now. (Not much but still) The new encoding
> >prefix
> > trie will be a huge bonus here. There the seeks will be flying.. [Ted
> also
> > presented this in the Hadoop China] Thanks to Matt... :) I am trying to
> > measure the scan performance with this new encoding . Trying to >back
> port
> > a simple patch for 94 version just for testing... Yes when the no of
> > results to be returned is more and more any index will become less
> > performing as per my study :)
> >
> > yes, you are right, I guess it's just a drawback of any index approach.
> > Thanks for the explanation.
> >
> > Shengjie
> >
> > On 28 December 2012 04:14, Anoop Sam John <an...@huawei.com> wrote:
> >
> > > > Do you have link to that presentation?
> > >
> > > http://hbtc2012.hadooper.cn/subject/track4TedYu4.pdf
> > >
> > > -Anoop-
> > >
> > > ________________________________________
> > > From: Mohit Anchlia [mohitanchlia@gmail.com]
> > > Sent: Friday, December 28, 2012 9:12 AM
> > > To: user@hbase.apache.org
> > > Subject: Re: HBase - Secondary Index
> > >
> > > On Thu, Dec 27, 2012 at 7:33 PM, Anoop Sam John <an...@huawei.com>
> > > wrote:
> > >
> > > > Yes as you say when the no of rows to be returned is becoming more
> and
> > > > more the latency will be becoming more. seeks within an HFile block
> is
> > > > some what expensive op now. (Not much but still) The new encoding
> > prefix
> > > > trie will be a huge bonus here. There the seeks will be flying.. [Ted
> > > also
> > > > presented this in the Hadoop China] Thanks to Matt... :) I am
> trying
> > to
> > > > measure the scan performance with this new encoding . Trying to back
> > > port a
> > > > simple patch for 94 version just for testing... Yes when the no of
> > > > results to be returned is more and more any index will become less
> > > > performing as per my study :)
> > > >
> > > > Do you have link to that presentation?
> > >
> > >
> > > > >btw, quick question- in your presentation, the scale there is
> seconds
> > or
> > > > mill-seconds:)
> > > >
> > > > It is seconds. Dont consider the exact values. What is the % of
> > increase
> > > > in latency is important :) Those were not high end machines.
> > > >
> > > > -Anoop-
> > > > ________________________________________
> > > > From: Shengjie Min [kelvin.msj@gmail.com]
> > > > Sent: Thursday, December 27, 2012 9:59 PM
> > > > To: user@hbase.apache.org
> > > > Subject: Re: HBase - Secondary Index
> > > >
> > > > >Didnt follow u completely here. There wont be any get() happening..
> > As
> > > > the
> > > > >exact rowkey in a region we get from the index table, we can seek to
> > the
> > > > >exact position and return that row.
> > > >
> > > > Sorry, When I misused "get()" here, I meant seeking. Yes, if it's
> just
> > > > small number of rows returned, this works perfect. As you said you
> will
> > > get
> > > > the exact rowkey positions per region, and simply seek them. I was
> > trying
> > > > to work out the case that when the number of result rows increases
> > > > massively. Like in Anil's case, he wants to do a scan query against
> the
> > > > 2ndary index(timestamp): "select all rows from timestamp1 to
> > timestamp2"
> > > > given no customerId provided. During that time period, he might have
> a
> > > big
> > > > chunk of rows from different customerIds. The index table returns a
> lot
> > > of
> > > > rowkey positions for different customerIds (I believe they are
> > scattered
> > > in
> > > > different regions), then you end up seeking all different positions
> in
> > > > different regions and return all the rows needed. According to your
> > > > presentation page14 - Performance Test Results (Scan), without index,
> > > it's
> > > > a linear increase as result rows # increases. on the other hand, with
> > > > index, time spent climbs up way quicker than the case without index.
> > > >
> > > > btw, quick question- in your presentation, the scale there is seconds
> > or
> > > > mill-seconds:)
> > > >
> > > > - Shengjie
> > > >
> > > >
> > > > On 27 December 2012 15:54, Anoop John <an...@gmail.com> wrote:
> > > >
> > > > > >how the massive number of get() is going to
> > > > > perform againt the main table
> > > > >
> > > > > Didnt follow u completely here. There wont be any get() happening..
> > As
> > > > the
> > > > > exact rowkey in a region we get from the index table, we can seek
> to
> > > the
> > > > > exact position and return that row.
> > > > >
> > > > > -Anoop-
> > > > >
> > > > > On Thu, Dec 27, 2012 at 6:37 PM, Shengjie Min <
> kelvin.msj@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > how the massive number of get() is going to
> > > > > > perform againt the main table
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > All the best,
> > > > Shengjie Min
> > > >
> > >
> >
> >
> >
> > --
> > All the best,
> > Shengjie Min
> >
>
>
>
> --
> Adrien Mogenet
> 06.59.16.64.22
> http://www.mogenet.me
>
Re: HBase - Secondary Index
Posted by Adrien Mogenet <ad...@gmail.com>.
Nice topic, perhaps one of the most important for 2013 :-)
I still don't get how you're ensuring consistency between index table and
main table, without an external component (such as bookkeeper/zookeeper).
What's the exact write path in your situation when inserting data ?
(WAL/RegionObserver, pre/post put/WALedit...)
The underlying question is about how you're ensuring that WALEdit in Index
and Main tables are perfectly sync'ed, and how you 're able to rollback in
case of issue in both WAL ?
On Fri, Dec 28, 2012 at 11:55 AM, Shengjie Min <ke...@gmail.com> wrote:
> >Yes as you say when the no of rows to be returned is becoming more and
> more the latency will be becoming more. seeks within an HFile block is
> some what expensive op now. (Not much but still) The new encoding >prefix
> trie will be a huge bonus here. There the seeks will be flying.. [Ted also
> presented this in the Hadoop China] Thanks to Matt... :) I am trying to
> measure the scan performance with this new encoding . Trying to >back port
> a simple patch for 94 version just for testing... Yes when the no of
> results to be returned is more and more any index will become less
> performing as per my study :)
>
> yes, you are right, I guess it's just a drawback of any index approach.
> Thanks for the explanation.
>
> Shengjie
>
> On 28 December 2012 04:14, Anoop Sam John <an...@huawei.com> wrote:
>
> > > Do you have link to that presentation?
> >
> > http://hbtc2012.hadooper.cn/subject/track4TedYu4.pdf
> >
> > -Anoop-
> >
> > ________________________________________
> > From: Mohit Anchlia [mohitanchlia@gmail.com]
> > Sent: Friday, December 28, 2012 9:12 AM
> > To: user@hbase.apache.org
> > Subject: Re: HBase - Secondary Index
> >
> > On Thu, Dec 27, 2012 at 7:33 PM, Anoop Sam John <an...@huawei.com>
> > wrote:
> >
> > > Yes as you say when the no of rows to be returned is becoming more and
> > > more the latency will be becoming more. seeks within an HFile block is
> > > some what expensive op now. (Not much but still) The new encoding
> prefix
> > > trie will be a huge bonus here. There the seeks will be flying.. [Ted
> > also
> > > presented this in the Hadoop China] Thanks to Matt... :) I am trying
> to
> > > measure the scan performance with this new encoding . Trying to back
> > port a
> > > simple patch for 94 version just for testing... Yes when the no of
> > > results to be returned is more and more any index will become less
> > > performing as per my study :)
> > >
> > > Do you have link to that presentation?
> >
> >
> > > >btw, quick question- in your presentation, the scale there is seconds
> or
> > > mill-seconds:)
> > >
> > > It is seconds. Dont consider the exact values. What is the % of
> increase
> > > in latency is important :) Those were not high end machines.
> > >
> > > -Anoop-
> > > ________________________________________
> > > From: Shengjie Min [kelvin.msj@gmail.com]
> > > Sent: Thursday, December 27, 2012 9:59 PM
> > > To: user@hbase.apache.org
> > > Subject: Re: HBase - Secondary Index
> > >
> > > >Didnt follow u completely here. There wont be any get() happening..
> As
> > > the
> > > >exact rowkey in a region we get from the index table, we can seek to
> the
> > > >exact position and return that row.
> > >
> > > Sorry, When I misused "get()" here, I meant seeking. Yes, if it's just
> > > small number of rows returned, this works perfect. As you said you will
> > get
> > > the exact rowkey positions per region, and simply seek them. I was
> trying
> > > to work out the case that when the number of result rows increases
> > > massively. Like in Anil's case, he wants to do a scan query against the
> > > 2ndary index(timestamp): "select all rows from timestamp1 to
> timestamp2"
> > > given no customerId provided. During that time period, he might have a
> > big
> > > chunk of rows from different customerIds. The index table returns a lot
> > of
> > > rowkey positions for different customerIds (I believe they are
> scattered
> > in
> > > different regions), then you end up seeking all different positions in
> > > different regions and return all the rows needed. According to your
> > > presentation page14 - Performance Test Results (Scan), without index,
> > it's
> > > a linear increase as result rows # increases. on the other hand, with
> > > index, time spent climbs up way quicker than the case without index.
> > >
> > > btw, quick question- in your presentation, the scale there is seconds
> or
> > > mill-seconds:)
> > >
> > > - Shengjie
> > >
> > >
> > > On 27 December 2012 15:54, Anoop John <an...@gmail.com> wrote:
> > >
> > > > >how the massive number of get() is going to
> > > > perform againt the main table
> > > >
> > > > Didnt follow u completely here. There wont be any get() happening..
> As
> > > the
> > > > exact rowkey in a region we get from the index table, we can seek to
> > the
> > > > exact position and return that row.
> > > >
> > > > -Anoop-
> > > >
> > > > On Thu, Dec 27, 2012 at 6:37 PM, Shengjie Min <ke...@gmail.com>
> > > > wrote:
> > > >
> > > > > how the massive number of get() is going to
> > > > > perform againt the main table
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > All the best,
> > > Shengjie Min
> > >
> >
>
>
>
> --
> All the best,
> Shengjie Min
>
--
Adrien Mogenet
06.59.16.64.22
http://www.mogenet.me
Re: HBase - Secondary Index
Posted by Shengjie Min <ke...@gmail.com>.
>Yes as you say when the no of rows to be returned is becoming more and
more the latency will be becoming more. seeks within an HFile block is
some what expensive op now. (Not much but still) The new encoding >prefix
trie will be a huge bonus here. There the seeks will be flying.. [Ted also
presented this in the Hadoop China] Thanks to Matt... :) I am trying to
measure the scan performance with this new encoding . Trying to >back port
a simple patch for 94 version just for testing... Yes when the no of
results to be returned is more and more any index will become less
performing as per my study :)
yes, you are right, I guess it's just a drawback of any index approach.
Thanks for the explanation.
Shengjie
On 28 December 2012 04:14, Anoop Sam John <an...@huawei.com> wrote:
> > Do you have link to that presentation?
>
> http://hbtc2012.hadooper.cn/subject/track4TedYu4.pdf
>
> -Anoop-
>
> ________________________________________
> From: Mohit Anchlia [mohitanchlia@gmail.com]
> Sent: Friday, December 28, 2012 9:12 AM
> To: user@hbase.apache.org
> Subject: Re: HBase - Secondary Index
>
> On Thu, Dec 27, 2012 at 7:33 PM, Anoop Sam John <an...@huawei.com>
> wrote:
>
> > Yes as you say when the no of rows to be returned is becoming more and
> > more the latency will be becoming more. seeks within an HFile block is
> > some what expensive op now. (Not much but still) The new encoding prefix
> > trie will be a huge bonus here. There the seeks will be flying.. [Ted
> also
> > presented this in the Hadoop China] Thanks to Matt... :) I am trying to
> > measure the scan performance with this new encoding . Trying to back
> port a
> > simple patch for 94 version just for testing... Yes when the no of
> > results to be returned is more and more any index will become less
> > performing as per my study :)
> >
> > Do you have link to that presentation?
>
>
> > >btw, quick question- in your presentation, the scale there is seconds or
> > mill-seconds:)
> >
> > It is seconds. Dont consider the exact values. What is the % of increase
> > in latency is important :) Those were not high end machines.
> >
> > -Anoop-
> > ________________________________________
> > From: Shengjie Min [kelvin.msj@gmail.com]
> > Sent: Thursday, December 27, 2012 9:59 PM
> > To: user@hbase.apache.org
> > Subject: Re: HBase - Secondary Index
> >
> > >Didnt follow u completely here. There wont be any get() happening.. As
> > the
> > >exact rowkey in a region we get from the index table, we can seek to the
> > >exact position and return that row.
> >
> > Sorry, When I misused "get()" here, I meant seeking. Yes, if it's just
> > small number of rows returned, this works perfect. As you said you will
> get
> > the exact rowkey positions per region, and simply seek them. I was trying
> > to work out the case that when the number of result rows increases
> > massively. Like in Anil's case, he wants to do a scan query against the
> > 2ndary index(timestamp): "select all rows from timestamp1 to timestamp2"
> > given no customerId provided. During that time period, he might have a
> big
> > chunk of rows from different customerIds. The index table returns a lot
> of
> > rowkey positions for different customerIds (I believe they are scattered
> in
> > different regions), then you end up seeking all different positions in
> > different regions and return all the rows needed. According to your
> > presentation page14 - Performance Test Results (Scan), without index,
> it's
> > a linear increase as result rows # increases. on the other hand, with
> > index, time spent climbs up way quicker than the case without index.
> >
> > btw, quick question- in your presentation, the scale there is seconds or
> > mill-seconds:)
> >
> > - Shengjie
> >
> >
> > On 27 December 2012 15:54, Anoop John <an...@gmail.com> wrote:
> >
> > > >how the massive number of get() is going to
> > > perform againt the main table
> > >
> > > Didnt follow u completely here. There wont be any get() happening.. As
> > the
> > > exact rowkey in a region we get from the index table, we can seek to
> the
> > > exact position and return that row.
> > >
> > > -Anoop-
> > >
> > > On Thu, Dec 27, 2012 at 6:37 PM, Shengjie Min <ke...@gmail.com>
> > > wrote:
> > >
> > > > how the massive number of get() is going to
> > > > perform againt the main table
> > > >
> > >
> >
> >
> >
> > --
> > All the best,
> > Shengjie Min
> >
>
--
All the best,
Shengjie Min
RE: HBase - Secondary Index
Posted by Anoop Sam John <an...@huawei.com>.
> Do you have link to that presentation?
http://hbtc2012.hadooper.cn/subject/track4TedYu4.pdf
-Anoop-
________________________________________
From: Mohit Anchlia [mohitanchlia@gmail.com]
Sent: Friday, December 28, 2012 9:12 AM
To: user@hbase.apache.org
Subject: Re: HBase - Secondary Index
On Thu, Dec 27, 2012 at 7:33 PM, Anoop Sam John <an...@huawei.com> wrote:
> Yes as you say when the no of rows to be returned is becoming more and
> more the latency will be becoming more. seeks within an HFile block is
> some what expensive op now. (Not much but still) The new encoding prefix
> trie will be a huge bonus here. There the seeks will be flying.. [Ted also
> presented this in the Hadoop China] Thanks to Matt... :) I am trying to
> measure the scan performance with this new encoding . Trying to back port a
> simple patch for 94 version just for testing... Yes when the no of
> results to be returned is more and more any index will become less
> performing as per my study :)
>
> Do you have link to that presentation?
> >btw, quick question- in your presentation, the scale there is seconds or
> mill-seconds:)
>
> It is seconds. Dont consider the exact values. What is the % of increase
> in latency is important :) Those were not high end machines.
>
> -Anoop-
> ________________________________________
> From: Shengjie Min [kelvin.msj@gmail.com]
> Sent: Thursday, December 27, 2012 9:59 PM
> To: user@hbase.apache.org
> Subject: Re: HBase - Secondary Index
>
> >Didnt follow u completely here. There wont be any get() happening.. As
> the
> >exact rowkey in a region we get from the index table, we can seek to the
> >exact position and return that row.
>
> Sorry, When I misused "get()" here, I meant seeking. Yes, if it's just
> small number of rows returned, this works perfect. As you said you will get
> the exact rowkey positions per region, and simply seek them. I was trying
> to work out the case that when the number of result rows increases
> massively. Like in Anil's case, he wants to do a scan query against the
> 2ndary index(timestamp): "select all rows from timestamp1 to timestamp2"
> given no customerId provided. During that time period, he might have a big
> chunk of rows from different customerIds. The index table returns a lot of
> rowkey positions for different customerIds (I believe they are scattered in
> different regions), then you end up seeking all different positions in
> different regions and return all the rows needed. According to your
> presentation page14 - Performance Test Results (Scan), without index, it's
> a linear increase as result rows # increases. on the other hand, with
> index, time spent climbs up way quicker than the case without index.
>
> btw, quick question- in your presentation, the scale there is seconds or
> mill-seconds:)
>
> - Shengjie
>
>
> On 27 December 2012 15:54, Anoop John <an...@gmail.com> wrote:
>
> > >how the massive number of get() is going to
> > perform againt the main table
> >
> > Didnt follow u completely here. There wont be any get() happening.. As
> the
> > exact rowkey in a region we get from the index table, we can seek to the
> > exact position and return that row.
> >
> > -Anoop-
> >
> > On Thu, Dec 27, 2012 at 6:37 PM, Shengjie Min <ke...@gmail.com>
> > wrote:
> >
> > > how the massive number of get() is going to
> > > perform againt the main table
> > >
> >
>
>
>
> --
> All the best,
> Shengjie Min
>
Re: HBase - Secondary Index
Posted by Mohit Anchlia <mo...@gmail.com>.
On Thu, Dec 27, 2012 at 7:33 PM, Anoop Sam John <an...@huawei.com> wrote:
> Yes as you say when the no of rows to be returned is becoming more and
> more the latency will be becoming more. seeks within an HFile block is
> some what expensive op now. (Not much but still) The new encoding prefix
> trie will be a huge bonus here. There the seeks will be flying.. [Ted also
> presented this in the Hadoop China] Thanks to Matt... :) I am trying to
> measure the scan performance with this new encoding . Trying to back port a
> simple patch for 94 version just for testing... Yes when the no of
> results to be returned is more and more any index will become less
> performing as per my study :)
>
> Do you have link to that presentation?
> >btw, quick question- in your presentation, the scale there is seconds or
> mill-seconds:)
>
> It is seconds. Dont consider the exact values. What is the % of increase
> in latency is important :) Those were not high end machines.
>
> -Anoop-
> ________________________________________
> From: Shengjie Min [kelvin.msj@gmail.com]
> Sent: Thursday, December 27, 2012 9:59 PM
> To: user@hbase.apache.org
> Subject: Re: HBase - Secondary Index
>
> >Didnt follow u completely here. There wont be any get() happening.. As
> the
> >exact rowkey in a region we get from the index table, we can seek to the
> >exact position and return that row.
>
> Sorry, When I misused "get()" here, I meant seeking. Yes, if it's just
> small number of rows returned, this works perfect. As you said you will get
> the exact rowkey positions per region, and simply seek them. I was trying
> to work out the case that when the number of result rows increases
> massively. Like in Anil's case, he wants to do a scan query against the
> 2ndary index(timestamp): "select all rows from timestamp1 to timestamp2"
> given no customerId provided. During that time period, he might have a big
> chunk of rows from different customerIds. The index table returns a lot of
> rowkey positions for different customerIds (I believe they are scattered in
> different regions), then you end up seeking all different positions in
> different regions and return all the rows needed. According to your
> presentation page14 - Performance Test Results (Scan), without index, it's
> a linear increase as result rows # increases. on the other hand, with
> index, time spent climbs up way quicker than the case without index.
>
> btw, quick question- in your presentation, the scale there is seconds or
> mill-seconds:)
>
> - Shengjie
>
>
> On 27 December 2012 15:54, Anoop John <an...@gmail.com> wrote:
>
> > >how the massive number of get() is going to
> > perform againt the main table
> >
> > Didnt follow u completely here. There wont be any get() happening.. As
> the
> > exact rowkey in a region we get from the index table, we can seek to the
> > exact position and return that row.
> >
> > -Anoop-
> >
> > On Thu, Dec 27, 2012 at 6:37 PM, Shengjie Min <ke...@gmail.com>
> > wrote:
> >
> > > how the massive number of get() is going to
> > > perform againt the main table
> > >
> >
>
>
>
> --
> All the best,
> Shengjie Min
>
RE: HBase - Secondary Index
Posted by Anoop Sam John <an...@huawei.com>.
Yes as you say when the no of rows to be returned is becoming more and more the latency will be becoming more. seeks within an HFile block is some what expensive op now. (Not much but still) The new encoding prefix trie will be a huge bonus here. There the seeks will be flying.. [Ted also presented this in the Hadoop China] Thanks to Matt... :) I am trying to measure the scan performance with this new encoding . Trying to back port a simple patch for 94 version just for testing... Yes when the no of results to be returned is more and more any index will become less performing as per my study :)
>btw, quick question- in your presentation, the scale there is seconds or
mill-seconds:)
It is seconds. Dont consider the exact values. What is the % of increase in latency is important :) Those were not high end machines.
-Anoop-
________________________________________
From: Shengjie Min [kelvin.msj@gmail.com]
Sent: Thursday, December 27, 2012 9:59 PM
To: user@hbase.apache.org
Subject: Re: HBase - Secondary Index
>Didnt follow u completely here. There wont be any get() happening.. As the
>exact rowkey in a region we get from the index table, we can seek to the
>exact position and return that row.
Sorry, When I misused "get()" here, I meant seeking. Yes, if it's just
small number of rows returned, this works perfect. As you said you will get
the exact rowkey positions per region, and simply seek them. I was trying
to work out the case that when the number of result rows increases
massively. Like in Anil's case, he wants to do a scan query against the
2ndary index(timestamp): "select all rows from timestamp1 to timestamp2"
given no customerId provided. During that time period, he might have a big
chunk of rows from different customerIds. The index table returns a lot of
rowkey positions for different customerIds (I believe they are scattered in
different regions), then you end up seeking all different positions in
different regions and return all the rows needed. According to your
presentation page14 - Performance Test Results (Scan), without index, it's
a linear increase as result rows # increases. on the other hand, with
index, time spent climbs up way quicker than the case without index.
btw, quick question- in your presentation, the scale there is seconds or
mill-seconds:)
- Shengjie
On 27 December 2012 15:54, Anoop John <an...@gmail.com> wrote:
> >how the massive number of get() is going to
> perform againt the main table
>
> Didnt follow u completely here. There wont be any get() happening.. As the
> exact rowkey in a region we get from the index table, we can seek to the
> exact position and return that row.
>
> -Anoop-
>
> On Thu, Dec 27, 2012 at 6:37 PM, Shengjie Min <ke...@gmail.com>
> wrote:
>
> > how the massive number of get() is going to
> > perform againt the main table
> >
>
--
All the best,
Shengjie Min
Re: HBase - Secondary Index
Posted by Shengjie Min <ke...@gmail.com>.
>Didnt follow u completely here. There wont be any get() happening.. As the
>exact rowkey in a region we get from the index table, we can seek to the
>exact position and return that row.
Sorry, When I misused "get()" here, I meant seeking. Yes, if it's just
small number of rows returned, this works perfect. As you said you will get
the exact rowkey positions per region, and simply seek them. I was trying
to work out the case that when the number of result rows increases
massively. Like in Anil's case, he wants to do a scan query against the
2ndary index(timestamp): "select all rows from timestamp1 to timestamp2"
given no customerId provided. During that time period, he might have a big
chunk of rows from different customerIds. The index table returns a lot of
rowkey positions for different customerIds (I believe they are scattered in
different regions), then you end up seeking all different positions in
different regions and return all the rows needed. According to your
presentation page14 - Performance Test Results (Scan), without index, it's
a linear increase as result rows # increases. on the other hand, with
index, time spent climbs up way quicker than the case without index.
btw, quick question- in your presentation, the scale there is seconds or
mill-seconds:)
- Shengjie
On 27 December 2012 15:54, Anoop John <an...@gmail.com> wrote:
> >how the massive number of get() is going to
> perform againt the main table
>
> Didnt follow u completely here. There wont be any get() happening.. As the
> exact rowkey in a region we get from the index table, we can seek to the
> exact position and return that row.
>
> -Anoop-
>
> On Thu, Dec 27, 2012 at 6:37 PM, Shengjie Min <ke...@gmail.com>
> wrote:
>
> > how the massive number of get() is going to
> > perform againt the main table
> >
>
--
All the best,
Shengjie Min
Re: HBase - Secondary Index
Posted by ramkrishna vasudevan <ra...@gmail.com>.
As per the design there is no get() operation at all. Incase of equals
query nothing is cached in memory.
For Range may be we need to cache some intermediate result.
Regards
Ram
On Thu, Dec 27, 2012 at 9:24 PM, Anoop John <an...@gmail.com> wrote:
> >how the massive number of get() is going to
> perform againt the main table
>
> Didnt follow u completely here. There wont be any get() happening.. As the
> exact rowkey in a region we get from the index table, we can seek to the
> exact position and return that row.
>
> -Anoop-
>
> On Thu, Dec 27, 2012 at 6:37 PM, Shengjie Min <ke...@gmail.com>
> wrote:
>
> > how the massive number of get() is going to
> > perform againt the main table
> >
>
Re: HBase - Secondary Index
Posted by Anoop John <an...@gmail.com>.
>how the massive number of get() is going to
perform againt the main table
Didnt follow u completely here. There wont be any get() happening.. As the
exact rowkey in a region we get from the index table, we can seek to the
exact position and return that row.
-Anoop-
On Thu, Dec 27, 2012 at 6:37 PM, Shengjie Min <ke...@gmail.com> wrote:
> how the massive number of get() is going to
> perform againt the main table
>
Re: HBase - Secondary Index
Posted by Shengjie Min <ke...@gmail.com>.
Thanks, Anoop, That makes sense. Hope you guys make this open source soon.
This model seems working ok if the resultSet is not huge, you get the main
keys and re-seek exact rows from the main table. But One thing concerns me
a little bit is that after querying the index table, you get a resultSet of
the main keys is very big, how the massive number of get() is going to
perform againt the main table, becoz potentially the results can be
scattered to all different regions.
- Shengjie
On 27 December 2012 11:30, Anoop Sam John <an...@huawei.com> wrote:
>
> >What happens when regions get splitted ? do you update the startkey on the
> index table?
>
> We have a custom HalfStoreFileReader to read the split index region data.
> This reader will change the rowkey it returns with replacing the startkey
> part.
> After a split immediately HBase will initiate a compaction and the
> compation uses this new reader. So the rowkey coming out will be a changed
> one and thus the newly written HFiles will have the changed rowkey. Also a
> normal read (as part of scan) during this time uses this new reader and so
> we will always get the rowkey in the expected format.. :) Hope I make it
> clear for you.
>
> -Anoop-
> ________________________________________
> From: Shengjie Min [kelvin.msj@gmail.com]
> Sent: Thursday, December 27, 2012 4:53 PM
> To: user@hbase.apache.org
> Subject: Re: HBase - Secondary Index
>
> Hi Anoop,
>
> >First all there will be same number of regions in both primary and index
> tables. All the start/stop keys of the regions also will be same.
> >Suppose there are 2 regions on main table say for keys 0-10 and 10-20.
> Then we will create 2 regions in index table also with same key ranges.
> >At the master balancing level it is easy to collocate these regions seeing
> the start and end keys.
> >When the selection of the rowkey that will be used in the index table is
> the key here.
> >What we will do is all the rowkeys in the index table will be prefixed
> with the start key of the region/
> >When an entry is added to the main table with rowkey as 5 it will go to
> the 1st region (0-10)
> >Now there will be index region with range as 0-10. We will select this
> region to store this index data.
> >The row getting added into the index region for this entry will have a
> rowkey 0_x_5
> >I am just using '_' as a seperator here just to show this. Actually we
> wont be having any seperator.
> >So the rowkeys (in index region) will have a static begin part always.
> Will scan time also we know this part and so the startrow and endrow
> creation for the scan will be possible.. Note that we will store the actual
> table row >key as the last part of the index rowkey itself not as a value.
> >This is better option in our case of handling the scan index usage also at
> sever side. There is no index data fetch to client side..
>
> What happens when regions get splitted ? do you update the startkey on the
> index table?
>
> -Shengjie
>
>
> On 14 December 2012 08:54, Anoop Sam John <an...@huawei.com> wrote:
>
> > Hi Anil,
> >
> > >1. In your presentation you mentioned that region of Primary Table and
> > Region of Secondary Table are always located on the same region server.
> How
> > do you achieve it? By using the Primary table rowkey as prefix of Rowkey
> > of Secondary Table? Will your implementation work if the rowkey of
> primary
> > table cannot be used as prefix in rowkey of Secondary table( i have this
> > limitation in my use case)?
> > First all there will be same number of regions in both primary and index
> > tables. All the start/stop keys of the regions also will be same.
> > Suppose there are 2 regions on main table say for keys 0-10 and 10-20.
> > Then we will create 2 regions in index table also with same key ranges.
> > At the master balancing level it is easy to collocate these regions
> seeing
> > the start and end keys.
> > When the selection of the rowkey that will be used in the index table is
> > the key here.
> > What we will do is all the rowkeys in the index table will be prefixed
> > with the start key of the region/
> > When an entry is added to the main table with rowkey as 5 it will go to
> > the 1st region (0-10)
> > Now there will be index region with range as 0-10. We will select this
> > region to store this index data.
> > The row getting added into the index region for this entry will have a
> > rowkey 0_x_5
> > I am just using '_' as a seperator here just to show this. Actually we
> > wont be having any seperator.
> > So the rowkeys (in index region) will have a static begin part always.
> > Will scan time also we know this part and so the startrow and endrow
> > creation for the scan will be possible.. Note that we will store the
> actual
> > table row key as the last part of the index rowkey itself not as a value.
> > This is better option in our case of handling the scan index usage also
> at
> > sever side. There is no index data fetch to client side..
> >
> > I feel your use case perfectly fit with our model
> >
> > >2. Are you using an Endpoint or Observer for building the secondary
> index
> > table?
> > Observer
> >
> > >3. "Custom balancer do collocation". Is it a custom load balancer of
> HBase
> > Master or something else?
> > It is a balancer implementation which will be plugged into Master
> >
> > >4. Your region split looks interesting. I dont have much info about it.
> > Can
> > you point to some docs on IndexHalfStoreFileReader?
> > Sorry I am not able to publish any design doc or code as the company has
> > not decided to open src the solution yet.
> > Any particular query you come acorss pls feel free to aske me :)
> > You can see the HalfStoreFileReader class 1st..
> >
> > -Anoop-
> > ________________________________________
> > From: anil gupta [anilgupta84@gmail.com]
> > Sent: Friday, December 14, 2012 2:11 PM
> > To: user@hbase.apache.org
> > Subject: Re: HBase - Secondary Index
> >
> > Hi Anoop,
> >
> > Nice presentation and seems like a smart implementation. Since the
> > presentation only covered bullet points so i have couple of questions on
> > your implementation. :)
> >
> > Here is a recap to my implementation and our previous discussion on
> > Secondary index:
> >
> > Here is the link to previous email thread:
> > http://search-hadoop.com/m/1zWPMaaRtr .
> >
> > The secondary index is stored in table "B" as rowkey B --> family:<rowkey
> > A> . "<rowkey A>" is the column qualifier. Every row in B will only on
> > have one column "k" and the value of that column is the rowkey of A.
> >
> > Suppose i am storing customer events in table A. I have two requirement
> for
> > data query:
> > 1. Query customer events on basis of customer_Id and event_ID.
> > 2. Query customer events on basis of event_timestamp and customer_ID.
> >
> > 70% of querying is done by query#1, so i will create
> > <customer_Id><event_ID> as row key of Table A.
> > Now, in order to support fast results for query#2, i need to create a
> > secondary index on A. I store that secondary index in B, rowkey of B is
> > <event_timestamp><customer_ID>.Every row stores the corresponding rowkey
> of
> > A.
> >
> > HBase Querying approach:
> > 1. Scan the secondary table by using prefix filter and startRow to get
> the
> > list of Rowkeys of Primary table.
> > 2. Do a batch get on primary table by using HTable.get(List<Get>) method
> > using the list of Rowkeys obtained in step1.
> >
> > The only issue is that in my solution i have at least two RPC calls. Once
> > each in step1 and step2 above. I want to reduce the number of RPC to 1 if
> > possible.
> >
> >
> > ******Questions on your implementation:*********
> >
> > 1. In your presentation you mentioned that region of Primary Table and
> > Region of Secondary Table are always located on the same region server.
> How
> > do you achieve it? By using the Primary table rowkey as prefix of Rowkey
> > of Secondary Table? Will your implementation work if the rowkey of
> primary
> > table cannot be used as prefix in rowkey of Secondary table( i have this
> > limitation in my use case)?
> > 2. Are you using an Endpoint or Observer for building the secondary index
> > table?
> > 3. "Custom balancer do collocation". Is it a custom load balancer of
> HBase
> > Master or something else?
> > 4. Your region split looks interesting. I dont have much info about it.
> Can
> > you point to some docs on IndexHalfStoreFileReader?
> >
> > Thanks,
> > Anil Gupta
> >
> >
> >
> > On Tue, Dec 4, 2012 at 12:10 AM, Anoop Sam John <an...@huawei.com>
> > wrote:
> >
> > > Hi All
> > >
> > > Last week I got a chance to present the secondary indexing
> > > solution what we have done in Huawei at the China Hadoop Conference.
> You
> > > can see the presentation from
> > > http://hbtc2012.hadooper.cn/subject/track4Anoop%20Sam%20John2.pdf
> > >
> > >
> > >
> > > I would like to hear what others think on this. :)
> > >
> > >
> > >
> > > -Anoop-
> > >
> >
> >
> >
> > --
> > Thanks & Regards,
> > Anil Gupta
> >
>
>
>
> --
> All the best,
> Shengjie Min
>
--
All the best,
Shengjie Min
RE: HBase - Secondary Index
Posted by Anoop Sam John <an...@huawei.com>.
>What happens when regions get splitted ? do you update the startkey on the
index table?
We have a custom HalfStoreFileReader to read the split index region data. This reader will change the rowkey it returns with replacing the startkey part.
After a split immediately HBase will initiate a compaction and the compation uses this new reader. So the rowkey coming out will be a changed one and thus the newly written HFiles will have the changed rowkey. Also a normal read (as part of scan) during this time uses this new reader and so we will always get the rowkey in the expected format.. :) Hope I make it clear for you.
-Anoop-
________________________________________
From: Shengjie Min [kelvin.msj@gmail.com]
Sent: Thursday, December 27, 2012 4:53 PM
To: user@hbase.apache.org
Subject: Re: HBase - Secondary Index
Hi Anoop,
>First all there will be same number of regions in both primary and index
tables. All the start/stop keys of the regions also will be same.
>Suppose there are 2 regions on main table say for keys 0-10 and 10-20.
Then we will create 2 regions in index table also with same key ranges.
>At the master balancing level it is easy to collocate these regions seeing
the start and end keys.
>When the selection of the rowkey that will be used in the index table is
the key here.
>What we will do is all the rowkeys in the index table will be prefixed
with the start key of the region/
>When an entry is added to the main table with rowkey as 5 it will go to
the 1st region (0-10)
>Now there will be index region with range as 0-10. We will select this
region to store this index data.
>The row getting added into the index region for this entry will have a
rowkey 0_x_5
>I am just using '_' as a seperator here just to show this. Actually we
wont be having any seperator.
>So the rowkeys (in index region) will have a static begin part always.
Will scan time also we know this part and so the startrow and endrow
creation for the scan will be possible.. Note that we will store the actual
table row >key as the last part of the index rowkey itself not as a value.
>This is better option in our case of handling the scan index usage also at
sever side. There is no index data fetch to client side..
What happens when regions get splitted ? do you update the startkey on the
index table?
-Shengjie
On 14 December 2012 08:54, Anoop Sam John <an...@huawei.com> wrote:
> Hi Anil,
>
> >1. In your presentation you mentioned that region of Primary Table and
> Region of Secondary Table are always located on the same region server. How
> do you achieve it? By using the Primary table rowkey as prefix of Rowkey
> of Secondary Table? Will your implementation work if the rowkey of primary
> table cannot be used as prefix in rowkey of Secondary table( i have this
> limitation in my use case)?
> First all there will be same number of regions in both primary and index
> tables. All the start/stop keys of the regions also will be same.
> Suppose there are 2 regions on main table say for keys 0-10 and 10-20.
> Then we will create 2 regions in index table also with same key ranges.
> At the master balancing level it is easy to collocate these regions seeing
> the start and end keys.
> When the selection of the rowkey that will be used in the index table is
> the key here.
> What we will do is all the rowkeys in the index table will be prefixed
> with the start key of the region/
> When an entry is added to the main table with rowkey as 5 it will go to
> the 1st region (0-10)
> Now there will be index region with range as 0-10. We will select this
> region to store this index data.
> The row getting added into the index region for this entry will have a
> rowkey 0_x_5
> I am just using '_' as a seperator here just to show this. Actually we
> wont be having any seperator.
> So the rowkeys (in index region) will have a static begin part always.
> Will scan time also we know this part and so the startrow and endrow
> creation for the scan will be possible.. Note that we will store the actual
> table row key as the last part of the index rowkey itself not as a value.
> This is better option in our case of handling the scan index usage also at
> sever side. There is no index data fetch to client side..
>
> I feel your use case perfectly fit with our model
>
> >2. Are you using an Endpoint or Observer for building the secondary index
> table?
> Observer
>
> >3. "Custom balancer do collocation". Is it a custom load balancer of HBase
> Master or something else?
> It is a balancer implementation which will be plugged into Master
>
> >4. Your region split looks interesting. I dont have much info about it.
> Can
> you point to some docs on IndexHalfStoreFileReader?
> Sorry I am not able to publish any design doc or code as the company has
> not decided to open src the solution yet.
> Any particular query you come acorss pls feel free to aske me :)
> You can see the HalfStoreFileReader class 1st..
>
> -Anoop-
> ________________________________________
> From: anil gupta [anilgupta84@gmail.com]
> Sent: Friday, December 14, 2012 2:11 PM
> To: user@hbase.apache.org
> Subject: Re: HBase - Secondary Index
>
> Hi Anoop,
>
> Nice presentation and seems like a smart implementation. Since the
> presentation only covered bullet points so i have couple of questions on
> your implementation. :)
>
> Here is a recap to my implementation and our previous discussion on
> Secondary index:
>
> Here is the link to previous email thread:
> http://search-hadoop.com/m/1zWPMaaRtr .
>
> The secondary index is stored in table "B" as rowkey B --> family:<rowkey
> A> . "<rowkey A>" is the column qualifier. Every row in B will only on
> have one column "k" and the value of that column is the rowkey of A.
>
> Suppose i am storing customer events in table A. I have two requirement for
> data query:
> 1. Query customer events on basis of customer_Id and event_ID.
> 2. Query customer events on basis of event_timestamp and customer_ID.
>
> 70% of querying is done by query#1, so i will create
> <customer_Id><event_ID> as row key of Table A.
> Now, in order to support fast results for query#2, i need to create a
> secondary index on A. I store that secondary index in B, rowkey of B is
> <event_timestamp><customer_ID>.Every row stores the corresponding rowkey of
> A.
>
> HBase Querying approach:
> 1. Scan the secondary table by using prefix filter and startRow to get the
> list of Rowkeys of Primary table.
> 2. Do a batch get on primary table by using HTable.get(List<Get>) method
> using the list of Rowkeys obtained in step1.
>
> The only issue is that in my solution i have at least two RPC calls. Once
> each in step1 and step2 above. I want to reduce the number of RPC to 1 if
> possible.
>
>
> ******Questions on your implementation:*********
>
> 1. In your presentation you mentioned that region of Primary Table and
> Region of Secondary Table are always located on the same region server. How
> do you achieve it? By using the Primary table rowkey as prefix of Rowkey
> of Secondary Table? Will your implementation work if the rowkey of primary
> table cannot be used as prefix in rowkey of Secondary table( i have this
> limitation in my use case)?
> 2. Are you using an Endpoint or Observer for building the secondary index
> table?
> 3. "Custom balancer do collocation". Is it a custom load balancer of HBase
> Master or something else?
> 4. Your region split looks interesting. I dont have much info about it. Can
> you point to some docs on IndexHalfStoreFileReader?
>
> Thanks,
> Anil Gupta
>
>
>
> On Tue, Dec 4, 2012 at 12:10 AM, Anoop Sam John <an...@huawei.com>
> wrote:
>
> > Hi All
> >
> > Last week I got a chance to present the secondary indexing
> > solution what we have done in Huawei at the China Hadoop Conference. You
> > can see the presentation from
> > http://hbtc2012.hadooper.cn/subject/track4Anoop%20Sam%20John2.pdf
> >
> >
> >
> > I would like to hear what others think on this. :)
> >
> >
> >
> > -Anoop-
> >
>
>
>
> --
> Thanks & Regards,
> Anil Gupta
>
--
All the best,
Shengjie Min
Re: HBase - Secondary Index
Posted by Shengjie Min <ke...@gmail.com>.
Hi Anoop,
>First all there will be same number of regions in both primary and index
tables. All the start/stop keys of the regions also will be same.
>Suppose there are 2 regions on main table say for keys 0-10 and 10-20.
Then we will create 2 regions in index table also with same key ranges.
>At the master balancing level it is easy to collocate these regions seeing
the start and end keys.
>When the selection of the rowkey that will be used in the index table is
the key here.
>What we will do is all the rowkeys in the index table will be prefixed
with the start key of the region/
>When an entry is added to the main table with rowkey as 5 it will go to
the 1st region (0-10)
>Now there will be index region with range as 0-10. We will select this
region to store this index data.
>The row getting added into the index region for this entry will have a
rowkey 0_x_5
>I am just using '_' as a seperator here just to show this. Actually we
wont be having any seperator.
>So the rowkeys (in index region) will have a static begin part always.
Will scan time also we know this part and so the startrow and endrow
creation for the scan will be possible.. Note that we will store the actual
table row >key as the last part of the index rowkey itself not as a value.
>This is better option in our case of handling the scan index usage also at
sever side. There is no index data fetch to client side..
What happens when regions get splitted ? do you update the startkey on the
index table?
-Shengjie
On 14 December 2012 08:54, Anoop Sam John <an...@huawei.com> wrote:
> Hi Anil,
>
> >1. In your presentation you mentioned that region of Primary Table and
> Region of Secondary Table are always located on the same region server. How
> do you achieve it? By using the Primary table rowkey as prefix of Rowkey
> of Secondary Table? Will your implementation work if the rowkey of primary
> table cannot be used as prefix in rowkey of Secondary table( i have this
> limitation in my use case)?
> First all there will be same number of regions in both primary and index
> tables. All the start/stop keys of the regions also will be same.
> Suppose there are 2 regions on main table say for keys 0-10 and 10-20.
> Then we will create 2 regions in index table also with same key ranges.
> At the master balancing level it is easy to collocate these regions seeing
> the start and end keys.
> When the selection of the rowkey that will be used in the index table is
> the key here.
> What we will do is all the rowkeys in the index table will be prefixed
> with the start key of the region/
> When an entry is added to the main table with rowkey as 5 it will go to
> the 1st region (0-10)
> Now there will be index region with range as 0-10. We will select this
> region to store this index data.
> The row getting added into the index region for this entry will have a
> rowkey 0_x_5
> I am just using '_' as a seperator here just to show this. Actually we
> wont be having any seperator.
> So the rowkeys (in index region) will have a static begin part always.
> Will scan time also we know this part and so the startrow and endrow
> creation for the scan will be possible.. Note that we will store the actual
> table row key as the last part of the index rowkey itself not as a value.
> This is better option in our case of handling the scan index usage also at
> sever side. There is no index data fetch to client side..
>
> I feel your use case perfectly fit with our model
>
> >2. Are you using an Endpoint or Observer for building the secondary index
> table?
> Observer
>
> >3. "Custom balancer do collocation". Is it a custom load balancer of HBase
> Master or something else?
> It is a balancer implementation which will be plugged into Master
>
> >4. Your region split looks interesting. I dont have much info about it.
> Can
> you point to some docs on IndexHalfStoreFileReader?
> Sorry I am not able to publish any design doc or code as the company has
> not decided to open src the solution yet.
> Any particular query you come acorss pls feel free to aske me :)
> You can see the HalfStoreFileReader class 1st..
>
> -Anoop-
> ________________________________________
> From: anil gupta [anilgupta84@gmail.com]
> Sent: Friday, December 14, 2012 2:11 PM
> To: user@hbase.apache.org
> Subject: Re: HBase - Secondary Index
>
> Hi Anoop,
>
> Nice presentation and seems like a smart implementation. Since the
> presentation only covered bullet points so i have couple of questions on
> your implementation. :)
>
> Here is a recap to my implementation and our previous discussion on
> Secondary index:
>
> Here is the link to previous email thread:
> http://search-hadoop.com/m/1zWPMaaRtr .
>
> The secondary index is stored in table "B" as rowkey B --> family:<rowkey
> A> . "<rowkey A>" is the column qualifier. Every row in B will only on
> have one column "k" and the value of that column is the rowkey of A.
>
> Suppose i am storing customer events in table A. I have two requirement for
> data query:
> 1. Query customer events on basis of customer_Id and event_ID.
> 2. Query customer events on basis of event_timestamp and customer_ID.
>
> 70% of querying is done by query#1, so i will create
> <customer_Id><event_ID> as row key of Table A.
> Now, in order to support fast results for query#2, i need to create a
> secondary index on A. I store that secondary index in B, rowkey of B is
> <event_timestamp><customer_ID>.Every row stores the corresponding rowkey of
> A.
>
> HBase Querying approach:
> 1. Scan the secondary table by using prefix filter and startRow to get the
> list of Rowkeys of Primary table.
> 2. Do a batch get on primary table by using HTable.get(List<Get>) method
> using the list of Rowkeys obtained in step1.
>
> The only issue is that in my solution i have at least two RPC calls. Once
> each in step1 and step2 above. I want to reduce the number of RPC to 1 if
> possible.
>
>
> ******Questions on your implementation:*********
>
> 1. In your presentation you mentioned that region of Primary Table and
> Region of Secondary Table are always located on the same region server. How
> do you achieve it? By using the Primary table rowkey as prefix of Rowkey
> of Secondary Table? Will your implementation work if the rowkey of primary
> table cannot be used as prefix in rowkey of Secondary table( i have this
> limitation in my use case)?
> 2. Are you using an Endpoint or Observer for building the secondary index
> table?
> 3. "Custom balancer do collocation". Is it a custom load balancer of HBase
> Master or something else?
> 4. Your region split looks interesting. I dont have much info about it. Can
> you point to some docs on IndexHalfStoreFileReader?
>
> Thanks,
> Anil Gupta
>
>
>
> On Tue, Dec 4, 2012 at 12:10 AM, Anoop Sam John <an...@huawei.com>
> wrote:
>
> > Hi All
> >
> > Last week I got a chance to present the secondary indexing
> > solution what we have done in Huawei at the China Hadoop Conference. You
> > can see the presentation from
> > http://hbtc2012.hadooper.cn/subject/track4Anoop%20Sam%20John2.pdf
> >
> >
> >
> > I would like to hear what others think on this. :)
> >
> >
> >
> > -Anoop-
> >
>
>
>
> --
> Thanks & Regards,
> Anil Gupta
>
--
All the best,
Shengjie Min
Re: HBase - Secondary Index
Posted by ramkrishna vasudevan <ra...@gmail.com>.
Nice explanation Anoop. :)
No prefix filters will be needed to query the secondary index table. As
Anoop told
<Startkey><IndexName>-> Static part
<Value>-> Main table rowkey value
<Actualrowkey>Actual rowkey.
So you just need to set a start row with <StartKey><IndexName><Value>...
This will give you the Actual rowkey as it is part of the rowkey..
Just use this rowkey on the primary table...You get the exact row needed...
All are server side...Nothing comes to the client till the final actual row
key is fetched..
Regards
Ram
On Fri, Dec 14, 2012 at 2:24 PM, Anoop Sam John <an...@huawei.com> wrote:
> Hi Anil,
>
> >1. In your presentation you mentioned that region of Primary Table and
> Region of Secondary Table are always located on the same region server. How
> do you achieve it? By using the Primary table rowkey as prefix of Rowkey
> of Secondary Table? Will your implementation work if the rowkey of primary
> table cannot be used as prefix in rowkey of Secondary table( i have this
> limitation in my use case)?
> First all there will be same number of regions in both primary and index
> tables. All the start/stop keys of the regions also will be same.
> Suppose there are 2 regions on main table say for keys 0-10 and 10-20.
> Then we will create 2 regions in index table also with same key ranges.
> At the master balancing level it is easy to collocate these regions seeing
> the start and end keys.
> When the selection of the rowkey that will be used in the index table is
> the key here.
> What we will do is all the rowkeys in the index table will be prefixed
> with the start key of the region/
> When an entry is added to the main table with rowkey as 5 it will go to
> the 1st region (0-10)
> Now there will be index region with range as 0-10. We will select this
> region to store this index data.
> The row getting added into the index region for this entry will have a
> rowkey 0_x_5
> I am just using '_' as a seperator here just to show this. Actually we
> wont be having any seperator.
> So the rowkeys (in index region) will have a static begin part always.
> Will scan time also we know this part and so the startrow and endrow
> creation for the scan will be possible.. Note that we will store the actual
> table row key as the last part of the index rowkey itself not as a value.
> This is better option in our case of handling the scan index usage also at
> sever side. There is no index data fetch to client side..
>
> I feel your use case perfectly fit with our model
>
> >2. Are you using an Endpoint or Observer for building the secondary index
> table?
> Observer
>
> >3. "Custom balancer do collocation". Is it a custom load balancer of HBase
> Master or something else?
> It is a balancer implementation which will be plugged into Master
>
> >4. Your region split looks interesting. I dont have much info about it.
> Can
> you point to some docs on IndexHalfStoreFileReader?
> Sorry I am not able to publish any design doc or code as the company has
> not decided to open src the solution yet.
> Any particular query you come acorss pls feel free to aske me :)
> You can see the HalfStoreFileReader class 1st..
>
> -Anoop-
> ________________________________________
> From: anil gupta [anilgupta84@gmail.com]
> Sent: Friday, December 14, 2012 2:11 PM
> To: user@hbase.apache.org
> Subject: Re: HBase - Secondary Index
>
> Hi Anoop,
>
> Nice presentation and seems like a smart implementation. Since the
> presentation only covered bullet points so i have couple of questions on
> your implementation. :)
>
> Here is a recap to my implementation and our previous discussion on
> Secondary index:
>
> Here is the link to previous email thread:
> http://search-hadoop.com/m/1zWPMaaRtr .
>
> The secondary index is stored in table "B" as rowkey B --> family:<rowkey
> A> . "<rowkey A>" is the column qualifier. Every row in B will only on
> have one column "k" and the value of that column is the rowkey of A.
>
> Suppose i am storing customer events in table A. I have two requirement for
> data query:
> 1. Query customer events on basis of customer_Id and event_ID.
> 2. Query customer events on basis of event_timestamp and customer_ID.
>
> 70% of querying is done by query#1, so i will create
> <customer_Id><event_ID> as row key of Table A.
> Now, in order to support fast results for query#2, i need to create a
> secondary index on A. I store that secondary index in B, rowkey of B is
> <event_timestamp><customer_ID>.Every row stores the corresponding rowkey of
> A.
>
> HBase Querying approach:
> 1. Scan the secondary table by using prefix filter and startRow to get the
> list of Rowkeys of Primary table.
> 2. Do a batch get on primary table by using HTable.get(List<Get>) method
> using the list of Rowkeys obtained in step1.
>
> The only issue is that in my solution i have at least two RPC calls. Once
> each in step1 and step2 above. I want to reduce the number of RPC to 1 if
> possible.
>
>
> ******Questions on your implementation:*********
>
> 1. In your presentation you mentioned that region of Primary Table and
> Region of Secondary Table are always located on the same region server. How
> do you achieve it? By using the Primary table rowkey as prefix of Rowkey
> of Secondary Table? Will your implementation work if the rowkey of primary
> table cannot be used as prefix in rowkey of Secondary table( i have this
> limitation in my use case)?
> 2. Are you using an Endpoint or Observer for building the secondary index
> table?
> 3. "Custom balancer do collocation". Is it a custom load balancer of HBase
> Master or something else?
> 4. Your region split looks interesting. I dont have much info about it. Can
> you point to some docs on IndexHalfStoreFileReader?
>
> Thanks,
> Anil Gupta
>
>
>
> On Tue, Dec 4, 2012 at 12:10 AM, Anoop Sam John <an...@huawei.com>
> wrote:
>
> > Hi All
> >
> > Last week I got a chance to present the secondary indexing
> > solution what we have done in Huawei at the China Hadoop Conference. You
> > can see the presentation from
> > http://hbtc2012.hadooper.cn/subject/track4Anoop%20Sam%20John2.pdf
> >
> >
> >
> > I would like to hear what others think on this. :)
> >
> >
> >
> > -Anoop-
> >
>
>
>
> --
> Thanks & Regards,
> Anil Gupta
>
RE: HBase - Secondary Index
Posted by Anoop Sam John <an...@huawei.com>.
Hi Anil,
>1. In your presentation you mentioned that region of Primary Table and
Region of Secondary Table are always located on the same region server. How
do you achieve it? By using the Primary table rowkey as prefix of Rowkey
of Secondary Table? Will your implementation work if the rowkey of primary
table cannot be used as prefix in rowkey of Secondary table( i have this
limitation in my use case)?
First all there will be same number of regions in both primary and index tables. All the start/stop keys of the regions also will be same.
Suppose there are 2 regions on main table say for keys 0-10 and 10-20. Then we will create 2 regions in index table also with same key ranges.
At the master balancing level it is easy to collocate these regions seeing the start and end keys.
When the selection of the rowkey that will be used in the index table is the key here.
What we will do is all the rowkeys in the index table will be prefixed with the start key of the region/
When an entry is added to the main table with rowkey as 5 it will go to the 1st region (0-10)
Now there will be index region with range as 0-10. We will select this region to store this index data.
The row getting added into the index region for this entry will have a rowkey 0_x_5
I am just using '_' as a seperator here just to show this. Actually we wont be having any seperator.
So the rowkeys (in index region) will have a static begin part always. Will scan time also we know this part and so the startrow and endrow creation for the scan will be possible.. Note that we will store the actual table row key as the last part of the index rowkey itself not as a value.
This is better option in our case of handling the scan index usage also at sever side. There is no index data fetch to client side..
I feel your use case perfectly fit with our model
>2. Are you using an Endpoint or Observer for building the secondary index
table?
Observer
>3. "Custom balancer do collocation". Is it a custom load balancer of HBase
Master or something else?
It is a balancer implementation which will be plugged into Master
>4. Your region split looks interesting. I dont have much info about it. Can
you point to some docs on IndexHalfStoreFileReader?
Sorry I am not able to publish any design doc or code as the company has not decided to open src the solution yet.
Any particular query you come acorss pls feel free to aske me :)
You can see the HalfStoreFileReader class 1st..
-Anoop-
________________________________________
From: anil gupta [anilgupta84@gmail.com]
Sent: Friday, December 14, 2012 2:11 PM
To: user@hbase.apache.org
Subject: Re: HBase - Secondary Index
Hi Anoop,
Nice presentation and seems like a smart implementation. Since the
presentation only covered bullet points so i have couple of questions on
your implementation. :)
Here is a recap to my implementation and our previous discussion on
Secondary index:
Here is the link to previous email thread:
http://search-hadoop.com/m/1zWPMaaRtr .
The secondary index is stored in table "B" as rowkey B --> family:<rowkey
A> . "<rowkey A>" is the column qualifier. Every row in B will only on
have one column "k" and the value of that column is the rowkey of A.
Suppose i am storing customer events in table A. I have two requirement for
data query:
1. Query customer events on basis of customer_Id and event_ID.
2. Query customer events on basis of event_timestamp and customer_ID.
70% of querying is done by query#1, so i will create
<customer_Id><event_ID> as row key of Table A.
Now, in order to support fast results for query#2, i need to create a
secondary index on A. I store that secondary index in B, rowkey of B is
<event_timestamp><customer_ID>.Every row stores the corresponding rowkey of
A.
HBase Querying approach:
1. Scan the secondary table by using prefix filter and startRow to get the
list of Rowkeys of Primary table.
2. Do a batch get on primary table by using HTable.get(List<Get>) method
using the list of Rowkeys obtained in step1.
The only issue is that in my solution i have at least two RPC calls. Once
each in step1 and step2 above. I want to reduce the number of RPC to 1 if
possible.
******Questions on your implementation:*********
1. In your presentation you mentioned that region of Primary Table and
Region of Secondary Table are always located on the same region server. How
do you achieve it? By using the Primary table rowkey as prefix of Rowkey
of Secondary Table? Will your implementation work if the rowkey of primary
table cannot be used as prefix in rowkey of Secondary table( i have this
limitation in my use case)?
2. Are you using an Endpoint or Observer for building the secondary index
table?
3. "Custom balancer do collocation". Is it a custom load balancer of HBase
Master or something else?
4. Your region split looks interesting. I dont have much info about it. Can
you point to some docs on IndexHalfStoreFileReader?
Thanks,
Anil Gupta
On Tue, Dec 4, 2012 at 12:10 AM, Anoop Sam John <an...@huawei.com> wrote:
> Hi All
>
> Last week I got a chance to present the secondary indexing
> solution what we have done in Huawei at the China Hadoop Conference. You
> can see the presentation from
> http://hbtc2012.hadooper.cn/subject/track4Anoop%20Sam%20John2.pdf
>
>
>
> I would like to hear what others think on this. :)
>
>
>
> -Anoop-
>
--
Thanks & Regards,
Anil Gupta
Re: HBase - Secondary Index
Posted by anil gupta <an...@gmail.com>.
Hi Anoop,
Nice presentation and seems like a smart implementation. Since the
presentation only covered bullet points so i have couple of questions on
your implementation. :)
Here is a recap to my implementation and our previous discussion on
Secondary index:
Here is the link to previous email thread:
http://search-hadoop.com/m/1zWPMaaRtr .
The secondary index is stored in table "B" as rowkey B --> family:<rowkey
A> . "<rowkey A>" is the column qualifier. Every row in B will only on
have one column "k" and the value of that column is the rowkey of A.
Suppose i am storing customer events in table A. I have two requirement for
data query:
1. Query customer events on basis of customer_Id and event_ID.
2. Query customer events on basis of event_timestamp and customer_ID.
70% of querying is done by query#1, so i will create
<customer_Id><event_ID> as row key of Table A.
Now, in order to support fast results for query#2, i need to create a
secondary index on A. I store that secondary index in B, rowkey of B is
<event_timestamp><customer_ID>.Every row stores the corresponding rowkey of
A.
HBase Querying approach:
1. Scan the secondary table by using prefix filter and startRow to get the
list of Rowkeys of Primary table.
2. Do a batch get on primary table by using HTable.get(List<Get>) method
using the list of Rowkeys obtained in step1.
The only issue is that in my solution i have at least two RPC calls. Once
each in step1 and step2 above. I want to reduce the number of RPC to 1 if
possible.
******Questions on your implementation:*********
1. In your presentation you mentioned that region of Primary Table and
Region of Secondary Table are always located on the same region server. How
do you achieve it? By using the Primary table rowkey as prefix of Rowkey
of Secondary Table? Will your implementation work if the rowkey of primary
table cannot be used as prefix in rowkey of Secondary table( i have this
limitation in my use case)?
2. Are you using an Endpoint or Observer for building the secondary index
table?
3. "Custom balancer do collocation". Is it a custom load balancer of HBase
Master or something else?
4. Your region split looks interesting. I dont have much info about it. Can
you point to some docs on IndexHalfStoreFileReader?
Thanks,
Anil Gupta
On Tue, Dec 4, 2012 at 12:10 AM, Anoop Sam John <an...@huawei.com> wrote:
> Hi All
>
> Last week I got a chance to present the secondary indexing
> solution what we have done in Huawei at the China Hadoop Conference. You
> can see the presentation from
> http://hbtc2012.hadooper.cn/subject/track4Anoop%20Sam%20John2.pdf
>
>
>
> I would like to hear what others think on this. :)
>
>
>
> -Anoop-
>
--
Thanks & Regards,
Anil Gupta
Re: HBase - Secondary Index
Posted by ramkrishna vasudevan <ra...@gmail.com>.
Also the WalObserver hooks were also used to ensure that the append to WAL
happens thro them.
Regards
Ram
On Wed, Dec 5, 2012 at 4:58 PM, ramkrishna vasudevan <
ramkrishna.s.vasudevan@gmail.com> wrote:
> Thanks Anoop for the reply. I was planning to reply if not from you.
>
> Regards
> Raml
>
>
> On Wed, Dec 5, 2012 at 4:34 PM, Anoop Sam John <an...@huawei.com> wrote:
>
>> Hi Jan
>> Yes we guarentee the consistency between user table and index
>> table. The put operation will be handled as a transactional way so as to
>> make sure the data is added to both tables or reverted back from both. Some
>> new CP hooks we have added for this obviously.
>>
>> -Anoop-
>> ________________________________________
>> From: Jan Van Besien [janvb@ngdata.com]
>> Sent: Wednesday, December 05, 2012 12:54 AM
>> To: dev@hbase.apache.org
>> Subject: Re: HBase - Secondary Index
>>
>> Hi,
>>
>> On 12/04/2012 09:10 AM, Anoop Sam John wrote:
>> > I would like to hear what others think on this. :)
>>
>> I found it interesting to read your approach on how the indexes can be
>> used to speed up existing scan operations.
>>
>> I couldn't find anything in your presentation though about whether your
>> implementation makes any guarantees to ensure the source table and the
>> index table are always (eventually) in sync.
>>
>> What if data is inserted in the source table and then the region server
>> crashes (before the coprocessor is executed)? Will the index be out of
>> sync? Do you have a mechanisme in place to detect and restore this
>> situation?
>>
>> thanks
>> Jan
>>
>
>
Re: HBase - Secondary Index
Posted by ramkrishna vasudevan <ra...@gmail.com>.
Thanks Anoop for the reply. I was planning to reply if not from you.
Regards
Raml
On Wed, Dec 5, 2012 at 4:34 PM, Anoop Sam John <an...@huawei.com> wrote:
> Hi Jan
> Yes we guarentee the consistency between user table and index
> table. The put operation will be handled as a transactional way so as to
> make sure the data is added to both tables or reverted back from both. Some
> new CP hooks we have added for this obviously.
>
> -Anoop-
> ________________________________________
> From: Jan Van Besien [janvb@ngdata.com]
> Sent: Wednesday, December 05, 2012 12:54 AM
> To: dev@hbase.apache.org
> Subject: Re: HBase - Secondary Index
>
> Hi,
>
> On 12/04/2012 09:10 AM, Anoop Sam John wrote:
> > I would like to hear what others think on this. :)
>
> I found it interesting to read your approach on how the indexes can be
> used to speed up existing scan operations.
>
> I couldn't find anything in your presentation though about whether your
> implementation makes any guarantees to ensure the source table and the
> index table are always (eventually) in sync.
>
> What if data is inserted in the source table and then the region server
> crashes (before the coprocessor is executed)? Will the index be out of
> sync? Do you have a mechanisme in place to detect and restore this
> situation?
>
> thanks
> Jan
>
Re: HBase - Secondary Index
Posted by ramkrishna vasudevan <ra...@gmail.com>.
I would like to reply to this...As previously i was part of the impl of
this..Hope Anoop corrects me if i am going wrong here..
.
However if the update to the index table (after a succesful update of the
source table) fails for some other reason (without a crash of the region
server), the HLog will not be replayed..
As you understood the WALEdit is done for the index region and also for the
main region using WAL hooks. The next step includes the addition to the
memstore..
So the KVs needs to be added to the memstore of both the main region and
index region. What type of failure do you foresee here?
You think of flushes or something that could fail?
If you see the Put() api code once the WAL edit is successfull there is no
need to rollback also. Just after the memstore addition happens for the
main table new hooks were added to make an entry for the index table.
Ideally here both should pass. Also some addtional work was done inorder
to take into account the MVCC part. So that a flush of the main region or
index region does not affect an incoming put or vice versa.
Anyway Anoop can answer to this more specifically as i dont have access to
the src code anymore.
Regards
Ram
On Wed, Dec 5, 2012 at 8:42 PM, Jan Van Besien <ja...@ngdata.com> wrote:
> On 12/05/2012 12:04 PM, Anoop Sam John wrote:
>
>> Yes we guarentee the consistency between user table and index
>> table. The put operation will be handled as a transactional way so as to
>> make sure the data is added to both tables or reverted back from both. Some
>> new CP hooks we have added for this obviously.
>>
>
> Would you be interested in sharing how exactly you guarantee this
> consistency? What are the CP hooks that you added and how exactly are they
> used?
>
> I can currently only guess how it could work in your implementation. For
> examply it could be that an update is a single WALEdit, which results in an
> update to both the source and index table. If the region server crashes
> between the update to the source and the index table, the HLog will be
> replied and thus you will have a chance to recover. However if the update
> to the index table (after a succesful update of the source table) fails for
> some other reason (without a crash of the region server), the HLog will not
> be replayed..
>
>
> Anyway, the above is just one assumption of your implementation could
> work. If you could share more details of the actual implementation, this
> would be helpful.
>
> Thanks
> Jan
>
Re: HBase - Secondary Index
Posted by Jan Van Besien <ja...@ngdata.com>.
On 12/05/2012 12:04 PM, Anoop Sam John wrote:
> Yes we guarentee the consistency between user table and index table. The put operation will be handled as a transactional way so as to make sure the data is added to both tables or reverted back from both. Some new CP hooks we have added for this obviously.
Would you be interested in sharing how exactly you guarantee this
consistency? What are the CP hooks that you added and how exactly are
they used?
I can currently only guess how it could work in your implementation. For
examply it could be that an update is a single WALEdit, which results in
an update to both the source and index table. If the region server
crashes between the update to the source and the index table, the HLog
will be replied and thus you will have a chance to recover. However if
the update to the index table (after a succesful update of the source
table) fails for some other reason (without a crash of the region
server), the HLog will not be replayed..
Anyway, the above is just one assumption of your implementation could
work. If you could share more details of the actual implementation, this
would be helpful.
Thanks
Jan
RE: HBase - Secondary Index
Posted by Anoop Sam John <an...@huawei.com>.
Hi Jan
Yes we guarentee the consistency between user table and index table. The put operation will be handled as a transactional way so as to make sure the data is added to both tables or reverted back from both. Some new CP hooks we have added for this obviously.
-Anoop-
________________________________________
From: Jan Van Besien [janvb@ngdata.com]
Sent: Wednesday, December 05, 2012 12:54 AM
To: dev@hbase.apache.org
Subject: Re: HBase - Secondary Index
Hi,
On 12/04/2012 09:10 AM, Anoop Sam John wrote:
> I would like to hear what others think on this. :)
I found it interesting to read your approach on how the indexes can be
used to speed up existing scan operations.
I couldn't find anything in your presentation though about whether your
implementation makes any guarantees to ensure the source table and the
index table are always (eventually) in sync.
What if data is inserted in the source table and then the region server
crashes (before the coprocessor is executed)? Will the index be out of
sync? Do you have a mechanisme in place to detect and restore this
situation?
thanks
Jan
Re: HBase - Secondary Index
Posted by Jan Van Besien <ja...@ngdata.com>.
Hi,
On 12/04/2012 09:10 AM, Anoop Sam John wrote:
> I would like to hear what others think on this. :)
I found it interesting to read your approach on how the indexes can be
used to speed up existing scan operations.
I couldn't find anything in your presentation though about whether your
implementation makes any guarantees to ensure the source table and the
index table are always (eventually) in sync.
What if data is inserted in the source table and then the region server
crashes (before the coprocessor is executed)? Will the index be out of
sync? Do you have a mechanisme in place to detect and restore this
situation?
thanks
Jan
Re: HBase - Secondary Index
Posted by Nick Dimiduk <nd...@gmail.com>.
Hi Anoop,
Your presentation has garnered quite a bit of community interest. Have you
considered providing your implementation to the community, perhaps in an
HBase-contrib module?
Thanks,
Nick
On Tue, Dec 4, 2012 at 12:10 AM, Anoop Sam John <an...@huawei.com> wrote:
> Hi All
>
> Last week I got a chance to present the secondary indexing
> solution what we have done in Huawei at the China Hadoop Conference. You
> can see the presentation from
> http://hbtc2012.hadooper.cn/subject/track4Anoop%20Sam%20John2.pdf
>
>
>
> I would like to hear what others think on this. :)
>
>
>
> -Anoop-
>