You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Stanislav Jordanov <st...@sirma.bg> on 2006/12/01 14:11:33 UTC
an alternative to optimize?
Guys,
I've already asked this question but nobody answered:
Suppose we have a relatively big index which is continuously updated -
i.e. new docs get added while some of the old docs get deleted.
For pragmatic reasons we have a restriction on maxMergeDocs so that
segment files don't get enormously big.
Consider now a segment of max size (i.e. containing maxMergeDocs docs
hence not eligible for a merge)
It is possible that (as time passes) this segment will have more and
more of its docs deleted.
But as it is not merge-able it will remain the same size and with lots
of "wholes" in it which is bad for performance.
The only way that I am aware of to correct this problem is to invoke
index optimization, which has several drawbacks:
1. it takes a while to optimize a big index.
2. the optimization process always produces a index comprising of a
single (extremely) large segment.
We can live with 1.
But 2 is undesirable.
Is there a way to "optimize" (in terms of purging its deleted docs) an
index or a single segment
without ending up with a single segment index?
Best,
Stanislav
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: an alternative to optimize?
Posted by Yonik Seeley <yo...@apache.org>.
On 12/1/06, Grant Ingersoll <gs...@apache.org> wrote:
> Are the search statistics the same for the MultiReader? That is,
> would a search on a MultiReader over several small indexes
> necessarily return the same ranking as a single IndexReader on an
> optimized reader? Would they return the same actual scores?
>
> Just curious, I haven't tried MultiReader, but the code seems like it
> should, at least based on my quick glance at it.
You use a MultiReader whenever you open a multi-segment index.
idf factors can change in the presence of deleted documents.
-Yonik
http://incubator.apache.org/solr Solr, the open-source Lucene search server
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: an alternative to optimize?
Posted by Grant Ingersoll <gs...@apache.org>.
Are the search statistics the same for the MultiReader? That is,
would a search on a MultiReader over several small indexes
necessarily return the same ranking as a single IndexReader on an
optimized reader? Would they return the same actual scores?
Just curious, I haven't tried MultiReader, but the code seems like it
should, at least based on my quick glance at it.
-Grant
On Dec 1, 2006, at 2:59 PM, Chris Hostetter wrote:
>
> : I haven't tried it, but according to http://lucene.apache.org/java/
> : docs/fileformats.html, each segment is a complete sub index. I
> : _wonder_ if you couldn't manage your own merges by using
> : IndexWriter.addIndexes() where you load each segment in separately
> : (this may mean copying the segments to other directories, but I am
>
> more specificly: opening an IndexReader on a an index with multiple
> segments is functionally the same as opening a MultiReader on several
> indexes with single segments.
>
> so instead of having one index, and carefully controlling
> maxMergeDocs,
> you could manually create many small indexes, all of which you
> continiously optimize (both to remove deletions, and to keep the
> index at
> a single segment) which you then explicly open a multi reader on.
>
>
>
>
> -Hoss
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
--------------------------
Grant Ingersoll
Center for Natural Language Processing
http://www.cnlp.org
Read the Lucene Java FAQ at http://wiki.apache.org/jakarta-lucene/
LuceneFAQ
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: an alternative to optimize?
Posted by Chris Hostetter <ho...@fucit.org>.
: I haven't tried it, but according to http://lucene.apache.org/java/
: docs/fileformats.html, each segment is a complete sub index. I
: _wonder_ if you couldn't manage your own merges by using
: IndexWriter.addIndexes() where you load each segment in separately
: (this may mean copying the segments to other directories, but I am
more specificly: opening an IndexReader on a an index with multiple
segments is functionally the same as opening a MultiReader on several
indexes with single segments.
so instead of having one index, and carefully controlling maxMergeDocs,
you could manually create many small indexes, all of which you
continiously optimize (both to remove deletions, and to keep the index at
a single segment) which you then explicly open a multi reader on.
-Hoss
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: an alternative to optimize?
Posted by Grant Ingersoll <gs...@apache.org>.
I haven't tried it, but according to http://lucene.apache.org/java/
docs/fileformats.html, each segment is a complete sub index. I
_wonder_ if you couldn't manage your own merges by using
IndexWriter.addIndexes() where you load each segment in separately
(this may mean copying the segments to other directories, but I am
not sure). Another option would be to modify Lucene to expose the
merge functionality.
This is pure speculation at this point, but I know the capabilities
exist (as all optimize does is merge segments until there is one
segment) so it seems like it should be possible.
-Grant
On Dec 1, 2006, at 8:11 AM, Stanislav Jordanov wrote:
> Guys,
>
> I've already asked this question but nobody answered:
>
> Suppose we have a relatively big index which is continuously
> updated - i.e. new docs get added while some of the old docs get
> deleted.
> For pragmatic reasons we have a restriction on maxMergeDocs so that
> segment files don't get enormously big.
> Consider now a segment of max size (i.e. containing maxMergeDocs
> docs hence not eligible for a merge)
> It is possible that (as time passes) this segment will have more
> and more of its docs deleted.
> But as it is not merge-able it will remain the same size and with
> lots of "wholes" in it which is bad for performance.
> The only way that I am aware of to correct this problem is to
> invoke index optimization, which has several drawbacks:
> 1. it takes a while to optimize a big index.
> 2. the optimization process always produces a index comprising of a
> single (extremely) large segment.
> We can live with 1.
> But 2 is undesirable.
> Is there a way to "optimize" (in terms of purging its deleted docs)
> an index or a single segment
> without ending up with a single segment index?
>
> Best,
> Stanislav
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
--------------------------
Grant Ingersoll
Center for Natural Language Processing
http://www.cnlp.org
Read the Lucene Java FAQ at http://wiki.apache.org/jakarta-lucene/
LuceneFAQ
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org