HBASE 0.90.4 and indexing

[Ulrich Staudinger <us...@activequant.org>]

Hi there,

I have been using hbase some years ago and returned to dig deeper into it.
Great progress.

My goal is to add secondary indexes for various column fields in my main
column family. I tried some of the packages, such as Lily and Pigy, but
found them as not adequate for my use case. I took hbase-trx-0.90.0RC3,
imported the source from GIT into my workspace, fixed the compilation
errors with hbase-0.90.4, configured hbase through hbase-site.xml, set the
regionserver.class and impl properties to
   <property>
        <name>hbase.regionserver.class</name>
        <value>org.apache.hadoop.hbase.ipc.IndexedRegionInterface</value>
    </property>

    <property>
        <name>hbase.regionserver.impl</name>

<value>org.apache.hadoop.hbase.regionserver.tableindexed.IndexedRegionServer</value>
    </property>


... also learned that there must not be a whitespace around the class
names, 'cause whitespace is not getting trimmed, and now run into these
exceptions:
2011-11-18 14:58:50,774 ERROR
org.apache.hadoop.hbase.regionserver.transactional.CleanOldTransactionsChore:
Caught exception
java.lang.ClassCastException: org.apache.hadoop.hbase.regionserver.HRegion
cannot be cast to
org.apache.hadoop.hbase.regionserver.transactional.TransactionalRegion
        at
org.apache.hadoop.hbase.regionserver.transactional.CleanOldTransactionsChore.chore(CleanOldTransactionsChore.java:40)
        at org.apache.hadoop.hbase.Chore.run(Chore.java:66)
2011-11-18 14:58:50,783 DEBUG
org.apache.hadoop.hbase.io.hfile.LruBlockCache: LRU Stats: total=958.3 KB,
free=197.79 MB, max=198.73 MB, blocks=3, accesses=38, hits=31,
hitRatio=81.57%%, cachingAccesses=34, cachingHits=31,
cachingHitsRatio=91.17%%, evictions=0, evicted=0, evictedPerRun=NaN
2011-11-18 14:58:54,001 INFO org.apache.zookeeper.server.ZooKeeperServer:
Expiring session 0x133b6e206ea0006, timeout of 40000ms exceeded
2011-11-18 14:58:54,001 INFO
org.apache.zookeeper.server.PrepRequestProcessor: Processed session
termination for sessionid: 0x133b6e206ea0006
2011-11-18 14:58:54,539 DEBUG org.apache.hadoop.hbase.master.LoadBalancer:
Server information: baikal.local,47912,1321623226625=5
2011-11-18 14:58:54,539 INFO org.apache.hadoop.hbase.master.LoadBalancer:
Skipping load balancing.  servers=1 regions=5 average=5.0 mostloaded=5
leastloaded=5
2011-11-18 14:58:54,550 DEBUG
org.apache.hadoop.hbase.master.CatalogJanitor: Scanned 3 catalog row(s) and
gc'd 0 unreferenced parent region(s)
2011-11-18 14:59:50,774 ERROR
org.apache.hadoop.hbase.regionserver.transactional.CleanOldTransactionsChore:
Caught exception
java.lang.ClassCastException: org.apache.hadoop.hbase.regionserver.HRegion
cannot be cast to
org.apache.hadoop.hbase.regionserver.transactional.TransactionalRegion
        at
org.apache.hadoop.hbase.regionserver.transactional.CleanOldTransactionsChore.chore(CleanOldTransactionsChore.java:40)
        at org.apache.hadoop.hbase.Chore.run(Chore.java:66)


Now I decided to leave it as it is and ask on the mailing list.


1) Why is the transactional package and the indexed* functionality not part
of base hbase any more?
2) What's the best (most developer friendly) way to add additional indexes?
3) Is there an hbase-trx package somewhere, which is compatible with one of
the hbase-0.90x releases?


Thanks,
Ulrich

Re: HBASE 0.90.4 and indexing

[Stack <st...@duboce.net>]

On Fri, Nov 18, 2011 at 6:02 AM, Ulrich Staudinger
<us...@activequant.org> wrote:
> 1) Why is the transactional package and the indexed* functionality not part
> of base hbase any more?

Because we didn't think this functionality part of core.  We also
wanted to remove friction; both core dev friction caused by our having
to ripple changes out into a code base few of us knew much about and
then friction for the trans/index maintainers as the only way for them
to get their changes in was by going via a committer.

> 2) What's the best (most developer friendly) way to add additional indexes?

Up to this, build it yourself unfortunately.  A one-size fits all
scalable, secondary indexing soln. is a bit of an undertaking it
seems.

> 3) Is there an hbase-trx package somewhere, which is compatible with one of
> the hbase-0.90x releases?
>

James' github is the most recent that I know of.  Have you tried writing him?

Good on you Ulrich,
St.Ack