You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Jamie Johnson <je...@gmail.com> on 2009/05/16 22:03:34 UTC

hbase & lucene

I have seen several pages (most over a year old) which make reference to
building lucene indexes against hbase, is there any updated documentation
which can be used to do this, or an old document which is still valid?  I am
new to the hbase world so any help on this would be greatly appreciated.

Jamie

Re: hbase & lucene

Posted by tim robertson <ti...@gmail.com>.
Hi,

So I am new to this as well, but I think you would have to handle this
in your insertion code manually.  I believe work is underway to have
indexes on the columns natively in HBase (I presume not lucene
though).

Regarding anything after a date, perhaps you can use the timestamp for
the row and then open a scanner with the filter "timestamp > x"?

Cheers

Tim


On Sun, May 17, 2009 at 2:31 AM, Jamie Johnson <je...@gmail.com> wrote:
> Thanks, I am looking into this perhaps my question should have been a little
> different though.
>
> If I am starting from scratch, nothing in the DB and want to keep a Lucene
> index for information I add in hbase is there a way to build the index on
> insertion instead of running a map reduce task after it has been inserted?
> Or is there a way to specify only to index files after a certain date so
> this task can be run periodically instead of having to reindex everything?
>
> Again I am very new to this so any help would be appreciated.
>
> On Sat, May 16, 2009 at 4:31 PM, tim robertson <ti...@gmail.com>wrote:
>
>> There is an IndexTableReduce class in the HBase source that you might
>> want to look at.
>>
>> Here is a basic example of usage:
>>
>>                BuildTableIndex bti = new BuildTableIndex();
>>                JobConf conf = new JobConf(TestBuildLucene.class);
>>                conf = bti.createJob(conf, 1, 1, "/tmp/lucene-hbase",
>> "occurrence",
>> "raw:CatalogueNo");
>>                try {
>>                        JobClient.runJob(conf);
>>                } catch (IOException e) {
>>                        e.printStackTrace();
>>                }
>>
>>                // do a term query
>>                try {
>>                        IndexReader reader =
>> IndexReader.open("/tmp/lucene-hbase/part-00000");
>>
>> Cheers,
>> Tim
>>
>>
>>
>>
>> On Sat, May 16, 2009 at 10:03 PM, Jamie Johnson <je...@gmail.com> wrote:
>> > I have seen several pages (most over a year old) which make reference to
>> > building lucene indexes against hbase, is there any updated documentation
>> > which can be used to do this, or an old document which is still valid?  I
>> am
>> > new to the hbase world so any help on this would be greatly appreciated.
>> >
>> > Jamie
>> >
>>
>

Re: hbase & lucene

Posted by Andrew Purtell <ap...@apache.org>.
Yes. See https://issues.apache.org/jira/browse/HBASE-1432




________________________________
From: Yair Even-Zohar <ya...@audiencescience.com>
To: hbase-user@hadoop.apache.org
Sent: Sunday, May 17, 2009 11:13:29 AM
Subject: RE: hbase & lucene

On a similar note, I have just created Lucene index from the column
names. That is, I modified the reduce part in IndexTableReduce.java just
a little bit. To do so, I had to modify BuildTableIndex.java too. 

I also had to duplicate LuceneDocumentWrapper.java nad
IndexOutputFormat.java (due to its dependency on  LuceneDocumentWrapper)
because LuceneDocumentWrapper is not public.
Is there any chance it will be public in 0.20?

Thanks
-Yair

-----Original Message-----
From: saint.ack@gmail.com [mailto:saint.ack@gmail.com] On Behalf Of
stack
Sent: Sunday, May 17, 2009 7:42 PM
To: hbase-user@hadoop.apache.org
Subject: Re: hbase & lucene

On Sat, May 16, 2009 at 5:31 PM, Jamie Johnson <je...@gmail.com>
wrote:

> Thanks, I am looking into this perhaps my question should have been a
> little
> different though.
>
> If I am starting from scratch, nothing in the DB and want to keep a
Lucene
> index for information I add in hbase is there a way to build the index
on
> insertion instead of running a map reduce task after it has been
inserted?


You could study the experimental transactional hbase -- THBase -- and
its
derivative indexed hbase where here the indexed refers to upkeep of
secondary indices.  They are subclasses of hbase core.   They include
some
doc. on how to set them up in the javadoc.  See how an update is amended
to
both insert into an hbase table and to do another action all as part of
the
one insert operation.

St.Ack



      

RE: hbase & lucene

Posted by Yair Even-Zohar <ya...@audiencescience.com>.
On a similar note, I have just created Lucene index from the column
names. That is, I modified the reduce part in IndexTableReduce.java just
a little bit. To do so, I had to modify BuildTableIndex.java too. 

I also had to duplicate LuceneDocumentWrapper.java nad
IndexOutputFormat.java (due to its dependency on  LuceneDocumentWrapper)
because LuceneDocumentWrapper is not public.
Is there any chance it will be public in 0.20?

Thanks
-Yair

-----Original Message-----
From: saint.ack@gmail.com [mailto:saint.ack@gmail.com] On Behalf Of
stack
Sent: Sunday, May 17, 2009 7:42 PM
To: hbase-user@hadoop.apache.org
Subject: Re: hbase & lucene

On Sat, May 16, 2009 at 5:31 PM, Jamie Johnson <je...@gmail.com>
wrote:

> Thanks, I am looking into this perhaps my question should have been a
> little
> different though.
>
> If I am starting from scratch, nothing in the DB and want to keep a
Lucene
> index for information I add in hbase is there a way to build the index
on
> insertion instead of running a map reduce task after it has been
inserted?


You could study the experimental transactional hbase -- THBase -- and
its
derivative indexed hbase where here the indexed refers to upkeep of
secondary indices.  They are subclasses of hbase core.   They include
some
doc. on how to set them up in the javadoc.  See how an update is amended
to
both insert into an hbase table and to do another action all as part of
the
one insert operation.

St.Ack

Re: hbase & lucene

Posted by stack <st...@duboce.net>.
http://hadoop.apache.org/hbase/docs/r0.19.2/api/org/apache/hadoop/hbase/client/tableindexed/package-summary.html#package_description

St.Ack

On Sun, May 17, 2009 at 6:10 PM, Jamie Johnson <je...@gmail.com> wrote:

> Are there any details for Indexed Hbase or Indexed Hbase?  I have done a
> quick google and have not come up with anything.
>
> On Sun, May 17, 2009 at 12:41 PM, stack <st...@duboce.net> wrote:
>
> > On Sat, May 16, 2009 at 5:31 PM, Jamie Johnson <je...@gmail.com>
> wrote:
> >
> > > Thanks, I am looking into this perhaps my question should have been a
> > > little
> > > different though.
> > >
> > > If I am starting from scratch, nothing in the DB and want to keep a
> > Lucene
> > > index for information I add in hbase is there a way to build the index
> on
> > > insertion instead of running a map reduce task after it has been
> > inserted?
> >
> >
> > You could study the experimental transactional hbase -- THBase -- and its
> > derivative indexed hbase where here the indexed refers to upkeep of
> > secondary indices.  They are subclasses of hbase core.   They include
> some
> > doc. on how to set them up in the javadoc.  See how an update is amended
> to
> > both insert into an hbase table and to do another action all as part of
> the
> > one insert operation.
> >
> > St.Ack
> >
>

Re: hbase & lucene

Posted by Jamie Johnson <je...@gmail.com>.
Are there any details for Indexed Hbase or Indexed Hbase?  I have done a
quick google and have not come up with anything.

On Sun, May 17, 2009 at 12:41 PM, stack <st...@duboce.net> wrote:

> On Sat, May 16, 2009 at 5:31 PM, Jamie Johnson <je...@gmail.com> wrote:
>
> > Thanks, I am looking into this perhaps my question should have been a
> > little
> > different though.
> >
> > If I am starting from scratch, nothing in the DB and want to keep a
> Lucene
> > index for information I add in hbase is there a way to build the index on
> > insertion instead of running a map reduce task after it has been
> inserted?
>
>
> You could study the experimental transactional hbase -- THBase -- and its
> derivative indexed hbase where here the indexed refers to upkeep of
> secondary indices.  They are subclasses of hbase core.   They include some
> doc. on how to set them up in the javadoc.  See how an update is amended to
> both insert into an hbase table and to do another action all as part of the
> one insert operation.
>
> St.Ack
>

Re: hbase & lucene

Posted by stack <st...@duboce.net>.
On Sat, May 16, 2009 at 5:31 PM, Jamie Johnson <je...@gmail.com> wrote:

> Thanks, I am looking into this perhaps my question should have been a
> little
> different though.
>
> If I am starting from scratch, nothing in the DB and want to keep a Lucene
> index for information I add in hbase is there a way to build the index on
> insertion instead of running a map reduce task after it has been inserted?


You could study the experimental transactional hbase -- THBase -- and its
derivative indexed hbase where here the indexed refers to upkeep of
secondary indices.  They are subclasses of hbase core.   They include some
doc. on how to set them up in the javadoc.  See how an update is amended to
both insert into an hbase table and to do another action all as part of the
one insert operation.

St.Ack

Re: hbase & lucene

Posted by Jamie Johnson <je...@gmail.com>.
Thanks, I am looking into this perhaps my question should have been a little
different though.

If I am starting from scratch, nothing in the DB and want to keep a Lucene
index for information I add in hbase is there a way to build the index on
insertion instead of running a map reduce task after it has been inserted?
Or is there a way to specify only to index files after a certain date so
this task can be run periodically instead of having to reindex everything?

Again I am very new to this so any help would be appreciated.

On Sat, May 16, 2009 at 4:31 PM, tim robertson <ti...@gmail.com>wrote:

> There is an IndexTableReduce class in the HBase source that you might
> want to look at.
>
> Here is a basic example of usage:
>
>                BuildTableIndex bti = new BuildTableIndex();
>                JobConf conf = new JobConf(TestBuildLucene.class);
>                conf = bti.createJob(conf, 1, 1, "/tmp/lucene-hbase",
> "occurrence",
> "raw:CatalogueNo");
>                try {
>                        JobClient.runJob(conf);
>                } catch (IOException e) {
>                        e.printStackTrace();
>                }
>
>                // do a term query
>                try {
>                        IndexReader reader =
> IndexReader.open("/tmp/lucene-hbase/part-00000");
>
> Cheers,
> Tim
>
>
>
>
> On Sat, May 16, 2009 at 10:03 PM, Jamie Johnson <je...@gmail.com> wrote:
> > I have seen several pages (most over a year old) which make reference to
> > building lucene indexes against hbase, is there any updated documentation
> > which can be used to do this, or an old document which is still valid?  I
> am
> > new to the hbase world so any help on this would be greatly appreciated.
> >
> > Jamie
> >
>

Re: hbase & lucene

Posted by tim robertson <ti...@gmail.com>.
There is an IndexTableReduce class in the HBase source that you might
want to look at.

Here is a basic example of usage:

		BuildTableIndex bti = new BuildTableIndex();
		JobConf conf = new JobConf(TestBuildLucene.class);
		conf = bti.createJob(conf, 1, 1, "/tmp/lucene-hbase", "occurrence",
"raw:CatalogueNo");
		try {
			JobClient.runJob(conf);
		} catch (IOException e) {
			e.printStackTrace();
		}
		
		// do a term query
		try {
			IndexReader reader = IndexReader.open("/tmp/lucene-hbase/part-00000");

Cheers,
Tim




On Sat, May 16, 2009 at 10:03 PM, Jamie Johnson <je...@gmail.com> wrote:
> I have seen several pages (most over a year old) which make reference to
> building lucene indexes against hbase, is there any updated documentation
> which can be used to do this, or an old document which is still valid?  I am
> new to the hbase world so any help on this would be greatly appreciated.
>
> Jamie
>