You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Jamie Cockrill <ja...@gmail.com> on 2010/07/07 10:15:44 UTC
Secondary indexes in 0.89
Dear all,
I've installed HBase from the Cloudera repository, using the
karmic-cdh3 distribution, which evidently gives me hbase version
0.89.20100621+17. I'm particularly interested in creating a few
secondary indexes and was initially using the following blog as an
example: (http://rajeev1982.blogspot.com/2009/06/secondary-indexes-in-hbase.html).
Various classes used in this example no longer appear to exist and
having investigated a bit further, by comparing the API docs for
0.20.5 and 0.89.20100621, it looks like those packages were removed.
I'm guessing that the HBase Versioning page is still correct in that
new features are being actively developed on the 0.89 branch and the
0.20 branch is the stable release version.
Have those classes been superseded by something else? I did find, in
the package page for org.apache.hadoop.hbase.client
(http://hbase.apache.org/docs/r0.89.20100621/apidocs/org/apache/hadoop/hbase/client/package-frame.html)
a link at the bottom called 'Table Indexed HBase', but it is broken.
I'd rather not go through the pain of trying to create something
similar myself, or have to re-install my cluster from scratch, and by
hand with a downloaded version of the 0.20.5 branch. However, if thats
the only answer then I suppose it'll have to be!
Thanks,
Jamie
Re: Secondary indexes in 0.89
Posted by Todd Lipcon <to...@cloudera.com>.
On Wed, Jul 7, 2010 at 6:44 AM, Michael Segel <mi...@hotmail.com>wrote:
>
> Todd,
>
> It seems that I'm not the only one looking at this issue. ;-)
>
> Secondary indexes are going to be an issue to many people who adopt HBase
> and MapReduce.
>
> I'd hope that the keepers of HBase rethink their decision to push contrib
> out to Github.
>
>
Hey Mike,
I think everyone in the community acknowledges the importance of secondary
indexes - clearly they're a useful construct, and long term, we absolutely
need them to be fully supported. However, right now the top priority is to
stabilize core, and having the contrib packages part of it was detrimental
to that goal (eg a bunch of performance regressions were introduced in an
earlier dot release of the 0.20 branch due to changes for indexing)
As is the case with every software team in the world, we have limited
resources available to us, and right now, I think the resources are best
served by stabilizing and improving core. Having the contribs out on github
means it's easy for others to pick up the projects and maintain them against
HBase trunk. Maybe you, Jamie, and a few others who need indexes can get
together and help us out here?
Thanks,
-Todd
-Mike
>
>
> > From: todd@cloudera.com
> > Date: Wed, 7 Jul 2010 01:23:34 -0700
> > Subject: Re: Secondary indexes in 0.89
> > To: user@hbase.apache.org
> >
> > Hey Jamie,
> >
> > Since the secondary indexing package isn't part of HBase core, its
> progress
> > may fall behind the core releases, depending on the activity of the
> > maintainers. Because of that, we recently moved contrib packages like
> > secondary indexing out of the HBase release onto github. The
> transactional
> > indexed hbase package can be found here:
> >
> > http://github.com/hbase-trx/hbase-transactional-tableindexed
> >
> > <http://github.com/hbase-trx/hbase-transactional-tableindexed
> >Unfortunately,
> > it hasn't yet been updated to work with 0.89/trunk. Since it's on github,
> > you should definitely feel free to pick it up, or file an issue for the
> > maintainers. If you do decide to pick up the contrib and update it to
> > compile against trunk, feel free to ask on-list if you need any help
> > understanding where things may have changed.
> >
> > Thanks
> > -Todd
> >
> >
> > On Wed, Jul 7, 2010 at 1:15 AM, Jamie Cockrill <jamie.cockrill@gmail.com
> >wrote:
> >
> > > Dear all,
> > >
> > > I've installed HBase from the Cloudera repository, using the
> > > karmic-cdh3 distribution, which evidently gives me hbase version
> > > 0.89.20100621+17. I'm particularly interested in creating a few
> > > secondary indexes and was initially using the following blog as an
> > > example: (
> > > http://rajeev1982.blogspot.com/2009/06/secondary-indexes-in-hbase.html
> ).
> > > Various classes used in this example no longer appear to exist and
> > > having investigated a bit further, by comparing the API docs for
> > > 0.20.5 and 0.89.20100621, it looks like those packages were removed.
> > > I'm guessing that the HBase Versioning page is still correct in that
> > > new features are being actively developed on the 0.89 branch and the
> > > 0.20 branch is the stable release version.
> > >
> > > Have those classes been superseded by something else? I did find, in
> > > the package page for org.apache.hadoop.hbase.client
> > > (
> > >
> http://hbase.apache.org/docs/r0.89.20100621/apidocs/org/apache/hadoop/hbase/client/package-frame.html
> > > )
> > > a link at the bottom called 'Table Indexed HBase', but it is broken.
> > > I'd rather not go through the pain of trying to create something
> > > similar myself, or have to re-install my cluster from scratch, and by
> > > hand with a downloaded version of the 0.20.5 branch. However, if thats
> > > the only answer then I suppose it'll have to be!
> > >
> > > Thanks,
> > >
> > > Jamie
> > >
> >
> >
> >
> > --
> > Todd Lipcon
> > Software Engineer, Cloudera
>
> _________________________________________________________________
> The New Busy is not the too busy. Combine all your e-mail accounts with
> Hotmail.
>
> http://www.windowslive.com/campaign/thenewbusy?tile=multiaccount&ocid=PID28326::T:WLMTAGL:ON:WL:en-US:WM_HMP:042010_4
>
--
Todd Lipcon
Software Engineer, Cloudera
RE: Secondary indexes in 0.89
Posted by Michael Segel <mi...@hotmail.com>.
Todd,
It seems that I'm not the only one looking at this issue. ;-)
Secondary indexes are going to be an issue to many people who adopt HBase and MapReduce.
I'd hope that the keepers of HBase rethink their decision to push contrib out to Github.
-Mike
> From: todd@cloudera.com
> Date: Wed, 7 Jul 2010 01:23:34 -0700
> Subject: Re: Secondary indexes in 0.89
> To: user@hbase.apache.org
>
> Hey Jamie,
>
> Since the secondary indexing package isn't part of HBase core, its progress
> may fall behind the core releases, depending on the activity of the
> maintainers. Because of that, we recently moved contrib packages like
> secondary indexing out of the HBase release onto github. The transactional
> indexed hbase package can be found here:
>
> http://github.com/hbase-trx/hbase-transactional-tableindexed
>
> <http://github.com/hbase-trx/hbase-transactional-tableindexed>Unfortunately,
> it hasn't yet been updated to work with 0.89/trunk. Since it's on github,
> you should definitely feel free to pick it up, or file an issue for the
> maintainers. If you do decide to pick up the contrib and update it to
> compile against trunk, feel free to ask on-list if you need any help
> understanding where things may have changed.
>
> Thanks
> -Todd
>
>
> On Wed, Jul 7, 2010 at 1:15 AM, Jamie Cockrill <ja...@gmail.com>wrote:
>
> > Dear all,
> >
> > I've installed HBase from the Cloudera repository, using the
> > karmic-cdh3 distribution, which evidently gives me hbase version
> > 0.89.20100621+17. I'm particularly interested in creating a few
> > secondary indexes and was initially using the following blog as an
> > example: (
> > http://rajeev1982.blogspot.com/2009/06/secondary-indexes-in-hbase.html).
> > Various classes used in this example no longer appear to exist and
> > having investigated a bit further, by comparing the API docs for
> > 0.20.5 and 0.89.20100621, it looks like those packages were removed.
> > I'm guessing that the HBase Versioning page is still correct in that
> > new features are being actively developed on the 0.89 branch and the
> > 0.20 branch is the stable release version.
> >
> > Have those classes been superseded by something else? I did find, in
> > the package page for org.apache.hadoop.hbase.client
> > (
> > http://hbase.apache.org/docs/r0.89.20100621/apidocs/org/apache/hadoop/hbase/client/package-frame.html
> > )
> > a link at the bottom called 'Table Indexed HBase', but it is broken.
> > I'd rather not go through the pain of trying to create something
> > similar myself, or have to re-install my cluster from scratch, and by
> > hand with a downloaded version of the 0.20.5 branch. However, if thats
> > the only answer then I suppose it'll have to be!
> >
> > Thanks,
> >
> > Jamie
> >
>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
_________________________________________________________________
The New Busy is not the too busy. Combine all your e-mail accounts with Hotmail.
http://www.windowslive.com/campaign/thenewbusy?tile=multiaccount&ocid=PID28326::T:WLMTAGL:ON:WL:en-US:WM_HMP:042010_4
Re: Secondary indexes in 0.89
Posted by Samuru Jackson <sa...@googlemail.com>.
Hi,
I'm wondering how your design of the key for your index looks like?!
My own inital implementation for an inverted index is to create for
each distinct column a separate table.
Example (key - Col : Col : Col)
A - Id1 : Id2 : Id3
B - Id4 : Id5 : Id6
C - Id7 : Id8 : Id9
This is convenient, however, this can lead to problems as the number
of columns (holding the referenced key) can grow extremely large.
As far as I understand the contributed Indexer in HBase it maintains
the Indexes in this way (only had a quick look on it):
AId1
AId2
AId3
BId4
BId5
BId6
CId7
CId8
CId9
I would really like to hear some opinions on this?!
/SJ
which then contains the value as the key and the related columns
contain the keys to the occurences. The drawback is that the colu
On Fri, Jul 23, 2010 at 1:44 PM, Luke Forehand
<lu...@networkedinsights.com> wrote:
> Jamie Cockrill <ja...@...> writes:
>
>>
>> Luke,
>>
>> Apologies no, I've been rather sidelined by another issue at the
>> moment. It's always the same, you get to playing with something
>> interesting and you get pulled off to fight fires somewhere else. Once
>> I get back on the case I'll have a look, however it someone did
>> previously mention another library built by the guys building Lily
>> that seems to aim to achieve the same goal. Reposted again here:
>>
>> http://lilycms.org/lily/roadmap/sketchbook/hbaseindexes.html
>>
>> I've not had time to look at it in detail, but it might be a good
>> starting point to get something up and going quickly if that's what
>> you're after.
>>
>> Ta,
>>
>> Jamie
>>
>> On 23 July 2010 16:58, Luke Forehand
>> <lu...@...> wrote:
>> > Jamie Cockrill <ja...@...> writes:
>> >
>
> Jamie,
>
> Thanks for the lead, I'm taking a look at the hbaseindex src now. I'm now
> leaning toward writing and maintaining my own secondary index rather than use
> the contrib IndexedTable stuff. With IndexedTable I don't have enough control
> over the index row key construction, and that is important depending on how I
> want to scan/filter the indexed table. Also, writing the index table from an
> existing huge table with IndexedTableAdmin takes too long and would be better
> suited as a Map Reduce job. These are just a few observations I've made after a
> somewhat cursory glance at the code.
>
> -Luke
>
>
Re: Secondary indexes in 0.89
Posted by Stack <st...@duboce.net>.
On Mon, Jul 26, 2010 at 6:42 PM, Samuru Jackson
<sa...@googlemail.com> wrote:
> Hi,
>
> Maybe it has been discussed already, but I don't understand why this
> featuer is only located in the contib. I think that this is a really
> important feature that should be included and maintained.
>
It used to be in contrib. It has since been moved out of the hbase
project and up into github.
The secondary index feature was evolving at a different rate to core
and because of this was seen by hbase devs as a drag on forward
progress (Current core hbase devs don't seem to have a need of
secondary indices -- or were maintaining secondary indices otherwise).
Also, coming from the other direction, its better if those interested
in this feature not have to go via hbase committers getting their code
committed. The hbase devs were not well versed in its workings --- as
implemented, secondary indices were atop transactional hbase, which
makes sense, but its a nuance that takes a little work grokking -- so
we weren't adding anything by being in the way of a commit and could
be slow to consider new patches. So hbase devs were a drag on
secondary index progress.
As soon as there is evidence of a good developer draft behind a
secondary index implementation, we'll add it to core.
St.Ack
Re: Secondary indexes in 0.89
Posted by Samuru Jackson <sa...@googlemail.com>.
Hi,
Maybe it has been discussed already, but I don't understand why this
featuer is only located in the contib. I think that this is a really
important feature that should be included and maintained.
I guess in most use-cases you will need to index your columns....
/SJ
On Mon, Jul 26, 2010 at 4:53 PM, Matthew Smillie
<ma...@troove.net> wrote:
> Hello Luke, Jamie, others.
>
> Sorry I missed this discussion; here's (as best I can) answers to a few of
> the previous questions about the contributed indexing:
>
> 0. Work has been slow on hbase-trx (which includes both transaction and
> indexing contrib) due to recent changes here. I'm doing my best to dive
> in/catch up. Definitely fork it, download it, file issues, and the usual.
> Best way to ping us directly is via github or hbase-dev@troove.net - I do
> try and monitor the hbase lists, but (obviously) don't catch everything.
>
> 1. The current code at hbase-trx relies on an in-progress patch to this
> issue:
> https://issues.apache.org/jira/browse/HBASE-2641
>
> Which factors log splitting out of HLog, and adds instance methods (which we
> need to override for the transactional log). This is only a requirement for
> the transactional aspect and indexing should work using calls to the
> existing static methods in trunk, but I have not tested this approach. If
> this would be of immediate use, I could probably push a branch to github in
> pretty short order; ping me directly if so. (I'd be happy to take a patch,
> too)
>
> 2. Controlling keys: you should be able to implement IndexKeyGenerator for a
> significant degree of control over, er, key generation. The default
> implementation (SimpleIndexKeyGenerator) functions as Samuru describes
> here: http://search-hadoop.com/m/U1H6O1M6dpr/Secondary+indexes+in+0.89
>
> best regards,
> Matthew Smillie
> Troove Technologies, Canada.
>
Re: Secondary indexes in 0.89
Posted by Matthew Smillie <ma...@troove.net>.
Hello Luke, Jamie, others.
Sorry I missed this discussion; here's (as best I can) answers to a few of
the previous questions about the contributed indexing:
0. Work has been slow on hbase-trx (which includes both transaction and
indexing contrib) due to recent changes here. I'm doing my best to dive
in/catch up. Definitely fork it, download it, file issues, and the usual.
Best way to ping us directly is via github or hbase-dev@troove.net - I do
try and monitor the hbase lists, but (obviously) don't catch everything.
1. The current code at hbase-trx relies on an in-progress patch to this
issue:
https://issues.apache.org/jira/browse/HBASE-2641
Which factors log splitting out of HLog, and adds instance methods (which we
need to override for the transactional log). This is only a requirement for
the transactional aspect and indexing should work using calls to the
existing static methods in trunk, but I have not tested this approach. If
this would be of immediate use, I could probably push a branch to github in
pretty short order; ping me directly if so. (I'd be happy to take a patch,
too)
2. Controlling keys: you should be able to implement IndexKeyGenerator for a
significant degree of control over, er, key generation. The default
implementation (SimpleIndexKeyGenerator) functions as Samuru describes
here: http://search-hadoop.com/m/U1H6O1M6dpr/Secondary+indexes+in+0.89
best regards,
Matthew Smillie
Troove Technologies, Canada.
Re: Secondary indexes in 0.89
Posted by Luke Forehand <lu...@networkedinsights.com>.
Stack <st...@...> writes:
>
> Luke: Where you looking? St.Ack
>
> On Fri, Jul 23, 2010 at 10:44 AM, Luke Forehand
> <lu...@...> wrote:
> > want to scan/filter the indexed table. Also, writing the index table from an
> > existing huge table with IndexedTableAdmin takes too long and would be better
> > suited as a Map Reduce job. These are just a few observations I've made
> > after a somewhat cursory glance at the code.
> >
I was looking at the IndexedTableAdmin class over at github. The private
method reIndexTable gets called at the end of addIndex method, but I would
prefer to remove that call and then let a Mapper Job write the index
table.
I might have been wrong about not having enough control over the index row
key construction since I didn't look closely.
In any case I've been trying two strategies for the index table: One is to
write a column for each master column's row key so that these row keys are
"grouped" by a single row.
The second and what I think may be faster is to have the index table's row key
be a composite of the master table's column + row key. Then you could use a
PrefixFilter to get specific results.
-Luke
Re: Secondary indexes in 0.89
Posted by Stack <st...@duboce.net>.
Luke: Where you looking? St.Ack
On Fri, Jul 23, 2010 at 10:44 AM, Luke Forehand
<lu...@networkedinsights.com> wrote:
> Jamie Cockrill <ja...@...> writes:
>
>>
>> Luke,
>>
>> Apologies no, I've been rather sidelined by another issue at the
>> moment. It's always the same, you get to playing with something
>> interesting and you get pulled off to fight fires somewhere else. Once
>> I get back on the case I'll have a look, however it someone did
>> previously mention another library built by the guys building Lily
>> that seems to aim to achieve the same goal. Reposted again here:
>>
>> http://lilycms.org/lily/roadmap/sketchbook/hbaseindexes.html
>>
>> I've not had time to look at it in detail, but it might be a good
>> starting point to get something up and going quickly if that's what
>> you're after.
>>
>> Ta,
>>
>> Jamie
>>
>> On 23 July 2010 16:58, Luke Forehand
>> <lu...@...> wrote:
>> > Jamie Cockrill <ja...@...> writes:
>> >
>
> Jamie,
>
> Thanks for the lead, I'm taking a look at the hbaseindex src now. I'm now
> leaning toward writing and maintaining my own secondary index rather than use
> the contrib IndexedTable stuff. With IndexedTable I don't have enough control
> over the index row key construction, and that is important depending on how I
> want to scan/filter the indexed table. Also, writing the index table from an
> existing huge table with IndexedTableAdmin takes too long and would be better
> suited as a Map Reduce job. These are just a few observations I've made after a
> somewhat cursory glance at the code.
>
> -Luke
>
>
Re: Secondary indexes in 0.89
Posted by Luke Forehand <lu...@networkedinsights.com>.
Jamie Cockrill <ja...@...> writes:
>
> Luke,
>
> Apologies no, I've been rather sidelined by another issue at the
> moment. It's always the same, you get to playing with something
> interesting and you get pulled off to fight fires somewhere else. Once
> I get back on the case I'll have a look, however it someone did
> previously mention another library built by the guys building Lily
> that seems to aim to achieve the same goal. Reposted again here:
>
> http://lilycms.org/lily/roadmap/sketchbook/hbaseindexes.html
>
> I've not had time to look at it in detail, but it might be a good
> starting point to get something up and going quickly if that's what
> you're after.
>
> Ta,
>
> Jamie
>
> On 23 July 2010 16:58, Luke Forehand
> <lu...@...> wrote:
> > Jamie Cockrill <ja...@...> writes:
> >
Jamie,
Thanks for the lead, I'm taking a look at the hbaseindex src now. I'm now
leaning toward writing and maintaining my own secondary index rather than use
the contrib IndexedTable stuff. With IndexedTable I don't have enough control
over the index row key construction, and that is important depending on how I
want to scan/filter the indexed table. Also, writing the index table from an
existing huge table with IndexedTableAdmin takes too long and would be better
suited as a Map Reduce job. These are just a few observations I've made after a
somewhat cursory glance at the code.
-Luke
Re: Secondary indexes in 0.89
Posted by Jamie Cockrill <ja...@gmail.com>.
Luke,
Apologies no, I've been rather sidelined by another issue at the
moment. It's always the same, you get to playing with something
interesting and you get pulled off to fight fires somewhere else. Once
I get back on the case I'll have a look, however it someone did
previously mention another library built by the guys building Lily
that seems to aim to achieve the same goal. Reposted again here:
http://lilycms.org/lily/roadmap/sketchbook/hbaseindexes.html
I've not had time to look at it in detail, but it might be a good
starting point to get something up and going quickly if that's what
you're after.
Ta,
Jamie
On 23 July 2010 16:58, Luke Forehand
<lu...@networkedinsights.com> wrote:
> Jamie Cockrill <ja...@...> writes:
>
>>
>> Hi Todd,
>>
>> Thanks for that, I'm a bit new to HBase at the minute, so I might
>> refrain from making any changes just yet, however, if I get some spare
>> time I may compare 0.20 and 0.89 and see what the functional
>> differences are. Ignoring the tests, the bulk of the issues appear to
>> be changed method/constructor signatures and the absence of
>> HLogSplitter. Also, the POM doesn't have hadoop or zookeeper as
>> dependencies, yet depends on their classes.
>>
>> Thanks for the pointers. I'm only trialing HBase at the moment, so in
>> the short-term I guess down-grading is the only option! However, if we
>> do decide to go with HBase, I'll definitely look into seeing if it's
>> possible to make the required changes.
>>
>> Jamie
>>
>> PS, i'm working against hadoop 0.20.2 at the moment.
>>
>> On 7 July 2010 09:23, Todd Lipcon <to...@...> wrote:
>> > Hey Jamie,
>> >
>> > Since the secondary indexing package isn't part of HBase core, its progress
>> > may fall behind the core releases, depending on the activity of the
>> > maintainers. Because of that, we recently moved contrib packages like
>> > secondary indexing out of the HBase release onto github. The transactional
>> > indexed hbase package can be found here:
>> >
>> > http://github.com/hbase-trx/hbase-transactional-tableindexed
>> >
>> > <http://github.com/hbase-trx/hbase-transactional-tableindexed>Unfortunately,
>> > it hasn't yet been updated to work with 0.89/trunk. Since it's on github,
>> > you should definitely feel free to pick it up, or file an issue for the
>> > maintainers. If you do decide to pick up the contrib and update it to
>> > compile against trunk, feel free to ask on-list if you need any help
>> > understanding where things may have changed.
>> >
>> > --
>> > Todd Lipcon
>> > Software Engineer, Cloudera
>> >
>>
>>
>
> Jamie / Todd,
>
> I'm attempting to do this exact same thing. Jamie, I've encountered your issues
> this morning. I have one compilation error left to fix and it's related to the
> HLogSplitter class missing. I also avoided refactoring the unit tests. I'm
> currently tracking down the necessary code for HLogSplitter and refactoring...
> Have you made further progress since this post? fyi - I have no idea how well
> this refactor will actually work with hbase 0.89 but am crossing my fingers.
>
> Luke Forehand
> Networked Insights
> http://www.networkedinsights.com
>
>
>
Re: Secondary indexes in 0.89
Posted by Luke Forehand <lu...@networkedinsights.com>.
Jamie Cockrill <ja...@...> writes:
>
> Hi Todd,
>
> Thanks for that, I'm a bit new to HBase at the minute, so I might
> refrain from making any changes just yet, however, if I get some spare
> time I may compare 0.20 and 0.89 and see what the functional
> differences are. Ignoring the tests, the bulk of the issues appear to
> be changed method/constructor signatures and the absence of
> HLogSplitter. Also, the POM doesn't have hadoop or zookeeper as
> dependencies, yet depends on their classes.
>
> Thanks for the pointers. I'm only trialing HBase at the moment, so in
> the short-term I guess down-grading is the only option! However, if we
> do decide to go with HBase, I'll definitely look into seeing if it's
> possible to make the required changes.
>
> Jamie
>
> PS, i'm working against hadoop 0.20.2 at the moment.
>
> On 7 July 2010 09:23, Todd Lipcon <to...@...> wrote:
> > Hey Jamie,
> >
> > Since the secondary indexing package isn't part of HBase core, its progress
> > may fall behind the core releases, depending on the activity of the
> > maintainers. Because of that, we recently moved contrib packages like
> > secondary indexing out of the HBase release onto github. The transactional
> > indexed hbase package can be found here:
> >
> > http://github.com/hbase-trx/hbase-transactional-tableindexed
> >
> > <http://github.com/hbase-trx/hbase-transactional-tableindexed>Unfortunately,
> > it hasn't yet been updated to work with 0.89/trunk. Since it's on github,
> > you should definitely feel free to pick it up, or file an issue for the
> > maintainers. If you do decide to pick up the contrib and update it to
> > compile against trunk, feel free to ask on-list if you need any help
> > understanding where things may have changed.
> >
> > --
> > Todd Lipcon
> > Software Engineer, Cloudera
> >
>
>
Jamie / Todd,
I'm attempting to do this exact same thing. Jamie, I've encountered your issues
this morning. I have one compilation error left to fix and it's related to the
HLogSplitter class missing. I also avoided refactoring the unit tests. I'm
currently tracking down the necessary code for HLogSplitter and refactoring...
Have you made further progress since this post? fyi - I have no idea how well
this refactor will actually work with hbase 0.89 but am crossing my fingers.
Luke Forehand
Networked Insights
http://www.networkedinsights.com
Re: Secondary indexes in 0.89
Posted by Jamie Cockrill <ja...@gmail.com>.
Hi Todd,
Thanks for that, I'm a bit new to HBase at the minute, so I might
refrain from making any changes just yet, however, if I get some spare
time I may compare 0.20 and 0.89 and see what the functional
differences are. Ignoring the tests, the bulk of the issues appear to
be changed method/constructor signatures and the absence of
HLogSplitter. Also, the POM doesn't have hadoop or zookeeper as
dependencies, yet depends on their classes.
Thanks for the pointers. I'm only trialing HBase at the moment, so in
the short-term I guess down-grading is the only option! However, if we
do decide to go with HBase, I'll definitely look into seeing if it's
possible to make the required changes.
Jamie
PS, i'm working against hadoop 0.20.2 at the moment.
On 7 July 2010 09:23, Todd Lipcon <to...@cloudera.com> wrote:
> Hey Jamie,
>
> Since the secondary indexing package isn't part of HBase core, its progress
> may fall behind the core releases, depending on the activity of the
> maintainers. Because of that, we recently moved contrib packages like
> secondary indexing out of the HBase release onto github. The transactional
> indexed hbase package can be found here:
>
> http://github.com/hbase-trx/hbase-transactional-tableindexed
>
> <http://github.com/hbase-trx/hbase-transactional-tableindexed>Unfortunately,
> it hasn't yet been updated to work with 0.89/trunk. Since it's on github,
> you should definitely feel free to pick it up, or file an issue for the
> maintainers. If you do decide to pick up the contrib and update it to
> compile against trunk, feel free to ask on-list if you need any help
> understanding where things may have changed.
>
> Thanks
> -Todd
>
>
> On Wed, Jul 7, 2010 at 1:15 AM, Jamie Cockrill <ja...@gmail.com>wrote:
>
>> Dear all,
>>
>> I've installed HBase from the Cloudera repository, using the
>> karmic-cdh3 distribution, which evidently gives me hbase version
>> 0.89.20100621+17. I'm particularly interested in creating a few
>> secondary indexes and was initially using the following blog as an
>> example: (
>> http://rajeev1982.blogspot.com/2009/06/secondary-indexes-in-hbase.html).
>> Various classes used in this example no longer appear to exist and
>> having investigated a bit further, by comparing the API docs for
>> 0.20.5 and 0.89.20100621, it looks like those packages were removed.
>> I'm guessing that the HBase Versioning page is still correct in that
>> new features are being actively developed on the 0.89 branch and the
>> 0.20 branch is the stable release version.
>>
>> Have those classes been superseded by something else? I did find, in
>> the package page for org.apache.hadoop.hbase.client
>> (
>> http://hbase.apache.org/docs/r0.89.20100621/apidocs/org/apache/hadoop/hbase/client/package-frame.html
>> )
>> a link at the bottom called 'Table Indexed HBase', but it is broken.
>> I'd rather not go through the pain of trying to create something
>> similar myself, or have to re-install my cluster from scratch, and by
>> hand with a downloaded version of the 0.20.5 branch. However, if thats
>> the only answer then I suppose it'll have to be!
>>
>> Thanks,
>>
>> Jamie
>>
>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>
Re: Secondary indexes in 0.89
Posted by Todd Lipcon <to...@cloudera.com>.
Hey Jamie,
Since the secondary indexing package isn't part of HBase core, its progress
may fall behind the core releases, depending on the activity of the
maintainers. Because of that, we recently moved contrib packages like
secondary indexing out of the HBase release onto github. The transactional
indexed hbase package can be found here:
http://github.com/hbase-trx/hbase-transactional-tableindexed
<http://github.com/hbase-trx/hbase-transactional-tableindexed>Unfortunately,
it hasn't yet been updated to work with 0.89/trunk. Since it's on github,
you should definitely feel free to pick it up, or file an issue for the
maintainers. If you do decide to pick up the contrib and update it to
compile against trunk, feel free to ask on-list if you need any help
understanding where things may have changed.
Thanks
-Todd
On Wed, Jul 7, 2010 at 1:15 AM, Jamie Cockrill <ja...@gmail.com>wrote:
> Dear all,
>
> I've installed HBase from the Cloudera repository, using the
> karmic-cdh3 distribution, which evidently gives me hbase version
> 0.89.20100621+17. I'm particularly interested in creating a few
> secondary indexes and was initially using the following blog as an
> example: (
> http://rajeev1982.blogspot.com/2009/06/secondary-indexes-in-hbase.html).
> Various classes used in this example no longer appear to exist and
> having investigated a bit further, by comparing the API docs for
> 0.20.5 and 0.89.20100621, it looks like those packages were removed.
> I'm guessing that the HBase Versioning page is still correct in that
> new features are being actively developed on the 0.89 branch and the
> 0.20 branch is the stable release version.
>
> Have those classes been superseded by something else? I did find, in
> the package page for org.apache.hadoop.hbase.client
> (
> http://hbase.apache.org/docs/r0.89.20100621/apidocs/org/apache/hadoop/hbase/client/package-frame.html
> )
> a link at the bottom called 'Table Indexed HBase', but it is broken.
> I'd rather not go through the pain of trying to create something
> similar myself, or have to re-install my cluster from scratch, and by
> hand with a downloaded version of the 0.20.5 branch. However, if thats
> the only answer then I suppose it'll have to be!
>
> Thanks,
>
> Jamie
>
--
Todd Lipcon
Software Engineer, Cloudera
Re: Secondary indexes in 0.89
Posted by Jamie Cockrill <ja...@gmail.com>.
Neil,
Thanks for that, I'll give it a look tomorrow!
Jamie
On 7 July 2010 21:25, <no...@yahoo.com> wrote:
> ----- Original Message ----
>> From: Jamie Cockrill <ja...@gmail.com>
>>
>> I've installed HBase from the Cloudera repository
>> ...
>> Various classes used in this example no longer appear to exist and
>> having investigated a bit further, by comparing the API docs for
>> 0.20.5 and 0.89.20100621, it looks like those packages were removed.
>
> Too funny. I was just about to post a very similar message. I also installed
> CDH3 yesterday, with the intent of diving straight into secondary indexes. Our
> prototype is only just now evolving to need them, so we hadn't used them yet.
> And then after spending most of the day getting CDH3 up and running (I'm still
> finding the cluster configuration a bit challenging) I was a bit surprised to
> find they were gone. Shame on me for making the assumption they'd be there, even
> thinking they'd be improved, but it is what it is.
>
> I understand and appreciate Todd's explanation of the situation. Thanks for the
> info. I'm only just beginning to get my head wrapped around HBase myself, so I'm
> not yet comfortable working on those libraries myself. But I'd like to throw a
> "me too" vote on getting secondary indexes back in the product ASAP.
>
> In the meantime, this morning I found, downloaded, and successfully ran the
> sample code for this against 0.89:
>
> http://lilycms.org/lily/roadmap/sketchbook/hbaseindexes.html
>
> It ran with no changes, and looks reasonably straight-forward, so I'm going to
> start playing with it more and see if it will work for our needs. Jamie, you may
> want to check it out.
>
> Neal
>
Re: Secondary indexes in 0.89
Posted by no...@yahoo.com.
----- Original Message ----
> From: Jamie Cockrill <ja...@gmail.com>
>
> I've installed HBase from the Cloudera repository
> ...
> Various classes used in this example no longer appear to exist and
> having investigated a bit further, by comparing the API docs for
> 0.20.5 and 0.89.20100621, it looks like those packages were removed.
Too funny. I was just about to post a very similar message. I also installed
CDH3 yesterday, with the intent of diving straight into secondary indexes. Our
prototype is only just now evolving to need them, so we hadn't used them yet.
And then after spending most of the day getting CDH3 up and running (I'm still
finding the cluster configuration a bit challenging) I was a bit surprised to
find they were gone. Shame on me for making the assumption they'd be there, even
thinking they'd be improved, but it is what it is.
I understand and appreciate Todd's explanation of the situation. Thanks for the
info. I'm only just beginning to get my head wrapped around HBase myself, so I'm
not yet comfortable working on those libraries myself. But I'd like to throw a
"me too" vote on getting secondary indexes back in the product ASAP.
In the meantime, this morning I found, downloaded, and successfully ran the
sample code for this against 0.89:
http://lilycms.org/lily/roadmap/sketchbook/hbaseindexes.html
It ran with no changes, and looks reasonably straight-forward, so I'm going to
start playing with it more and see if it will work for our needs. Jamie, you may
want to check it out.
Neal