You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by John Wang <jo...@gmail.com> on 2008/01/31 01:42:51 UTC
appending field to an existing index
Hi all:
We have a large index and it is difficult to reindex.
We want to add another field to the index without reindexing, e.g. just
create a new inverted index, dictionary files etc.
How feasible is it to add this to lucene?
Thanks
-John
Re: appending field to an existing index
Posted by John Wang <jo...@gmail.com>.
I was actually thinking of creating a separate index with only the extra
field and them modify the other index (change some files etc.)
Sounds hacky. Dunno if its possible.
Thanks
-john
On Jan 31, 2008 9:36 AM, Erick Erickson <er...@gmail.com> wrote:
> As always, "it depends". You can try to reconstruct the doc from an index,
> see Luke. But depending upon you you indexed things, it may be more
> or less lossy. I remember this was discussed recently, you might have
> some luck if you search the archive.
>
> But it may be very, very expensive to reconstruct all the docs from an
> index, see how long Luke takes to do this on your index.....
>
> If you stored all the fields in addition to indexing them, then there's
> no problem since the stored fields *are* what you indexed.
>
> How did you create your original index?
>
> Erick
>
> On Jan 31, 2008 12:28 PM, John Wang <jo...@gmail.com> wrote:
>
> > I have to keep one index though. Is there a way to reproduce an index
> from
> > an indexReader?
> >
> > -John
> >
> > On Jan 31, 2008 1:30 AM, Michael McCandless <lu...@mikemccandless.com>
> > wrote:
> >
> > >
> > > Just beware, though, that with ParallelReader you must ensure that
> > > the internal docIDs of both indices remain "aligned" over time.
> > >
> > > If you never do deletions, then that happens for free.
> > >
> > > If you do deletions, then, you must change IndexWriter to buffer by
> > > doc count (same doc count for all writhers), and, you must change the
> > > merge policy to LogDocMergePolicy. You also might need to switch to
> > > SerialMergeScheduler if you ever allow readers to refresh against the
> > > indices while a writer is still open. If you don't do that (ie
> > > reader can only re-open after a writer is closed) then it's safe to
> > > stick with ConcurrentMergeScheduler.
> > >
> > > Mike
> > >
> > > On Jan 31, 2008, at 4:23 AM, Doron Cohen wrote:
> > >
> > > > This may help:
> > > > http://www.nabble.com/Updating-Lucene-Index-with-Unstored-fields-
> > > > tt15188818.html#a15188818
> > > >
> > > > Doron
> > > >
> > > > On Thu, Jan 31, 2008 at 2:42 AM, John Wang <jo...@gmail.com>
> > > > wrote:
> > > >
> > > >> Hi all:
> > > >>
> > > >> We have a large index and it is difficult to reindex.
> > > >>
> > > >> We want to add another field to the index without reindexing,
> > > >> e.g. just
> > > >> create a new inverted index, dictionary files etc.
> > > >>
> > > >> How feasible is it to add this to lucene?
> > > >>
> > > >>
> > > >> Thanks
> > > >>
> > > >> -John
> > > >>
> > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > > For additional commands, e-mail: java-user-help@lucene.apache.org
> > >
> > >
> >
>
Re: appending field to an existing index
Posted by Andrzej Bialecki <ab...@getopt.org>.
Erick Erickson wrote:
> As always, "it depends". You can try to reconstruct the doc from an index,
> see Luke. But depending upon you you indexed things, it may be more
> or less lossy. I remember this was discussed recently, you might have
> some luck if you search the archive.
>
> But it may be very, very expensive to reconstruct all the docs from an
> index, see how long Luke takes to do this on your index.....
It wouldn't be much more expensive to reconstruct all unstored fields in
all docs - Luke has to traverse all posting lists anyway. This "bulk
restore" is not implemented (yet?).
--
Best regards,
Andrzej Bialecki <><
___. ___ ___ ___ _ _ __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web
___|||__|| \| || | Embedded Unix, System Integration
http://www.sigram.com Contact: info at sigram dot com
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: appending field to an existing index
Posted by Erick Erickson <er...@gmail.com>.
As always, "it depends". You can try to reconstruct the doc from an index,
see Luke. But depending upon you you indexed things, it may be more
or less lossy. I remember this was discussed recently, you might have
some luck if you search the archive.
But it may be very, very expensive to reconstruct all the docs from an
index, see how long Luke takes to do this on your index.....
If you stored all the fields in addition to indexing them, then there's
no problem since the stored fields *are* what you indexed.
How did you create your original index?
Erick
On Jan 31, 2008 12:28 PM, John Wang <jo...@gmail.com> wrote:
> I have to keep one index though. Is there a way to reproduce an index from
> an indexReader?
>
> -John
>
> On Jan 31, 2008 1:30 AM, Michael McCandless <lu...@mikemccandless.com>
> wrote:
>
> >
> > Just beware, though, that with ParallelReader you must ensure that
> > the internal docIDs of both indices remain "aligned" over time.
> >
> > If you never do deletions, then that happens for free.
> >
> > If you do deletions, then, you must change IndexWriter to buffer by
> > doc count (same doc count for all writhers), and, you must change the
> > merge policy to LogDocMergePolicy. You also might need to switch to
> > SerialMergeScheduler if you ever allow readers to refresh against the
> > indices while a writer is still open. If you don't do that (ie
> > reader can only re-open after a writer is closed) then it's safe to
> > stick with ConcurrentMergeScheduler.
> >
> > Mike
> >
> > On Jan 31, 2008, at 4:23 AM, Doron Cohen wrote:
> >
> > > This may help:
> > > http://www.nabble.com/Updating-Lucene-Index-with-Unstored-fields-
> > > tt15188818.html#a15188818
> > >
> > > Doron
> > >
> > > On Thu, Jan 31, 2008 at 2:42 AM, John Wang <jo...@gmail.com>
> > > wrote:
> > >
> > >> Hi all:
> > >>
> > >> We have a large index and it is difficult to reindex.
> > >>
> > >> We want to add another field to the index without reindexing,
> > >> e.g. just
> > >> create a new inverted index, dictionary files etc.
> > >>
> > >> How feasible is it to add this to lucene?
> > >>
> > >>
> > >> Thanks
> > >>
> > >> -John
> > >>
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
> >
>
Re: appending field to an existing index
Posted by Michael McCandless <lu...@mikemccandless.com>.
Woops -- disregard this! Chris' approach will work. And, we don't
need addParallelIndices!
Mike
Michael McCandless wrote:
>
> I don't think this will do the right thing in this case, because
> addIndexes "appends" the documents from all indices together.
>
> Whereas John wants to "merge in" a new field into all docs in an
> existing index.
>
> Really what we need is a new "addParallellIndices" method. I
> think, conceptually, it's really not that hard to do ("it's just
> software"), especially if the fields are completely orthogonal
> (which is the case here, I think)? We would need a variant of
> SegmentMerger, that instead of appending everything, would "splice"
> the indices together, doc by doc. The posting lists should be
> especially simple since each field is contiguous in the prx/frq files.
>
> Mike
>
> Chris Hostetter wrote:
>
>>
>> : I have to keep one index though. Is there a way to reproduce an
>> index from
>> : an indexReader?
>>
>> asuming you have indexes that work in conjunction with eachther they
>> way you want when using ParallelReader, you should (in theory) be
>> able to
>> use...
>>
>> ParallelReader r = ...;
>> IndexWriter w = new IndexWRiter(...);
>> w.addIndexes(new IndexReader[] { r });
>> w.close();
>>
>> ...to "combine" those indexes into a single physical index. but i've
>> never personally tried it. there may be quirks about
>> ParallelReader that
>> prevent this from working.
>>
>>
>> -Hoss
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: appending field to an existing index
Posted by Michael McCandless <lu...@mikemccandless.com>.
I don't think this will do the right thing in this case, because
addIndexes "appends" the documents from all indices together.
Whereas John wants to "merge in" a new field into all docs in an
existing index.
Really what we need is a new "addParallellIndices" method. I think,
conceptually, it's really not that hard to do ("it's just software"),
especially if the fields are completely orthogonal (which is the case
here, I think)? We would need a variant of SegmentMerger, that
instead of appending everything, would "splice" the indices together,
doc by doc. The posting lists should be especially simple since each
field is contiguous in the prx/frq files.
Mike
Chris Hostetter wrote:
>
> : I have to keep one index though. Is there a way to reproduce an
> index from
> : an indexReader?
>
> asuming you have indexes that work in conjunction with eachther they
> way you want when using ParallelReader, you should (in theory) be
> able to
> use...
>
> ParallelReader r = ...;
> IndexWriter w = new IndexWRiter(...);
> w.addIndexes(new IndexReader[] { r });
> w.close();
>
> ...to "combine" those indexes into a single physical index. but i've
> never personally tried it. there may be quirks about
> ParallelReader that
> prevent this from working.
>
>
> -Hoss
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: appending field to an existing index
Posted by Cam Bazz <ca...@gmail.com>.
Hello,
I have read the parallel reader doc. It says it must have the same number of
documents as the other index.
When we are using a writer - searcher combination, how can we integrate this
parallel reader into game.
Simply, I have some documents, and I just like to mark them, in an efficient
way. is there a way?
I was having problem with deletes from my index, since I could not delete by
a query, I had to store an extra field that was the combination of the two
fields I had to delete with.
So I was thinking of integrating another index for marking them. but it
appears this parallel reader is kind of heavy for the job, since it has to
contain the same number of documents as the original index.
Best.
On Feb 1, 2008 12:02 AM, Chris Hostetter <ho...@fucit.org> wrote:
>
> : I have to keep one index though. Is there a way to reproduce an index
> from
> : an indexReader?
>
> asuming you have indexes that work in conjunction with eachther they
> way you want when using ParallelReader, you should (in theory) be able to
> use...
>
> ParallelReader r = ...;
> IndexWriter w = new IndexWRiter(...);
> w.addIndexes(new IndexReader[] { r });
> w.close();
>
> ...to "combine" those indexes into a single physical index. but i've
> never personally tried it. there may be quirks about ParallelReader that
> prevent this from working.
>
>
> -Hoss
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
Re: appending field to an existing index
Posted by Michael McCandless <lu...@mikemccandless.com>.
John Wang wrote:
> Thanks Chris! This works with lucene2.1 and greater (breaks with
> lucene2.0)
>
> A follow up question, anyway to rename and to delete a field.
>
> Rename field: I tried to use a hex editor to edit the .fnm file
> (screwed up
> my index, but I was careful and followed the index format)
> Deleting a field: ideas?
Maybe this?
http://www.markmail.org/message/vaohqx6bmh3smkrk
Mike
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: appending field to an existing index
Posted by John Wang <jo...@gmail.com>.
Thanks Chris! This works with lucene2.1 and greater (breaks with lucene2.0)
A follow up question, anyway to rename and to delete a field.
Rename field: I tried to use a hex editor to edit the .fnm file (screwed up
my index, but I was careful and followed the index format)
Deleting a field: ideas?
Thanks
-John
On Jan 31, 2008 2:02 PM, Chris Hostetter <ho...@fucit.org> wrote:
>
> : I have to keep one index though. Is there a way to reproduce an index
> from
> : an indexReader?
>
> asuming you have indexes that work in conjunction with eachther they
> way you want when using ParallelReader, you should (in theory) be able to
> use...
>
> ParallelReader r = ...;
> IndexWriter w = new IndexWRiter(...);
> w.addIndexes(new IndexReader[] { r });
> w.close();
>
> ...to "combine" those indexes into a single physical index. but i've
> never personally tried it. there may be quirks about ParallelReader that
> prevent this from working.
>
>
> -Hoss
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
Re: appending field to an existing index
Posted by Chris Hostetter <ho...@fucit.org>.
: I have to keep one index though. Is there a way to reproduce an index from
: an indexReader?
asuming you have indexes that work in conjunction with eachther they
way you want when using ParallelReader, you should (in theory) be able to
use...
ParallelReader r = ...;
IndexWriter w = new IndexWRiter(...);
w.addIndexes(new IndexReader[] { r });
w.close();
...to "combine" those indexes into a single physical index. but i've
never personally tried it. there may be quirks about ParallelReader that
prevent this from working.
-Hoss
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: appending field to an existing index
Posted by John Wang <jo...@gmail.com>.
I have to keep one index though. Is there a way to reproduce an index from
an indexReader?
-John
On Jan 31, 2008 1:30 AM, Michael McCandless <lu...@mikemccandless.com>
wrote:
>
> Just beware, though, that with ParallelReader you must ensure that
> the internal docIDs of both indices remain "aligned" over time.
>
> If you never do deletions, then that happens for free.
>
> If you do deletions, then, you must change IndexWriter to buffer by
> doc count (same doc count for all writhers), and, you must change the
> merge policy to LogDocMergePolicy. You also might need to switch to
> SerialMergeScheduler if you ever allow readers to refresh against the
> indices while a writer is still open. If you don't do that (ie
> reader can only re-open after a writer is closed) then it's safe to
> stick with ConcurrentMergeScheduler.
>
> Mike
>
> On Jan 31, 2008, at 4:23 AM, Doron Cohen wrote:
>
> > This may help:
> > http://www.nabble.com/Updating-Lucene-Index-with-Unstored-fields-
> > tt15188818.html#a15188818
> >
> > Doron
> >
> > On Thu, Jan 31, 2008 at 2:42 AM, John Wang <jo...@gmail.com>
> > wrote:
> >
> >> Hi all:
> >>
> >> We have a large index and it is difficult to reindex.
> >>
> >> We want to add another field to the index without reindexing,
> >> e.g. just
> >> create a new inverted index, dictionary files etc.
> >>
> >> How feasible is it to add this to lucene?
> >>
> >>
> >> Thanks
> >>
> >> -John
> >>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
Re: appending field to an existing index
Posted by Michael McCandless <lu...@mikemccandless.com>.
Just beware, though, that with ParallelReader you must ensure that
the internal docIDs of both indices remain "aligned" over time.
If you never do deletions, then that happens for free.
If you do deletions, then, you must change IndexWriter to buffer by
doc count (same doc count for all writhers), and, you must change the
merge policy to LogDocMergePolicy. You also might need to switch to
SerialMergeScheduler if you ever allow readers to refresh against the
indices while a writer is still open. If you don't do that (ie
reader can only re-open after a writer is closed) then it's safe to
stick with ConcurrentMergeScheduler.
Mike
On Jan 31, 2008, at 4:23 AM, Doron Cohen wrote:
> This may help:
> http://www.nabble.com/Updating-Lucene-Index-with-Unstored-fields-
> tt15188818.html#a15188818
>
> Doron
>
> On Thu, Jan 31, 2008 at 2:42 AM, John Wang <jo...@gmail.com>
> wrote:
>
>> Hi all:
>>
>> We have a large index and it is difficult to reindex.
>>
>> We want to add another field to the index without reindexing,
>> e.g. just
>> create a new inverted index, dictionary files etc.
>>
>> How feasible is it to add this to lucene?
>>
>>
>> Thanks
>>
>> -John
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: appending field to an existing index
Posted by Doron Cohen <cd...@gmail.com>.
This may help:
http://www.nabble.com/Updating-Lucene-Index-with-Unstored-fields-tt15188818.html#a15188818
Doron
On Thu, Jan 31, 2008 at 2:42 AM, John Wang <jo...@gmail.com> wrote:
> Hi all:
>
> We have a large index and it is difficult to reindex.
>
> We want to add another field to the index without reindexing, e.g. just
> create a new inverted index, dictionary files etc.
>
> How feasible is it to add this to lucene?
>
>
> Thanks
>
> -John
>
Re: appending field to an existing index
Posted by Erik Hatcher <er...@ehatchersolutions.com>.
One option is to index the new field in sync (same index order) into
a new index, and search using a ParallelIndexReader.
Erik
On Jan 30, 2008, at 7:42 PM, John Wang wrote:
> Hi all:
>
> We have a large index and it is difficult to reindex.
>
> We want to add another field to the index without reindexing,
> e.g. just
> create a new inverted index, dictionary files etc.
>
> How feasible is it to add this to lucene?
>
>
> Thanks
>
> -John
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org