You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by vivek sar <vi...@gmail.com> on 2008/03/31 01:42:15 UTC
java.lang.IllegalArgumentException: Segment is too large
Hi,
I'm using 2.3.0 Lucene build and have following merge parameters,
mergeFactor = 100
maxMergeDocs = 99999
maxBufferedDocs = 10000
maxRAMBufferSizeMB = 200
After running with this setting for a month without problem all the
sudden I'm getting following exception,
java.lang.IllegalArgumentException: Segment is too large (100001 docs
vs max docs 99999)
at org.apache.lucene.index.LogMergePolicy.findMerges(LogMergePolicy.java:274)
at org.apache.lucene.index.IndexWriter.updatePendingMerges(IndexWriter.java:1824)
at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:1799)
at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:1795)
at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:1791)
at org.apache.lucene.index.IndexWriter.addIndexesNoOptimize(IndexWriter.java:2231)
I thought Lucene should control the number of documents in a segment
during merge. Is there a different configuration I need to set? Should
I leave the maxMergeDocs to its default value, Integer#MAX_VALUE?
Thanks,
-vivek
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: java.lang.IllegalArgumentException: Segment is too large
Posted by Michael McCandless <lu...@mikemccandless.com>.
OK, I opened LUCENE-1254 and committed the fix to trunk & (upcoming)
2.3.2.
Mike
Yonik Seeley wrote:
> On Mon, Mar 31, 2008 at 5:19 AM, Michael McCandless
> <lu...@mikemccandless.com> wrote:
>> I think we should remove those checks and allow addIndexesNoOptimize
>> to import and index even if it has segments over this limit. I'll
>> open an issue.
>
> +1
>
> -Yonik
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: java.lang.IllegalArgumentException: Segment is too large
Posted by Yonik Seeley <yo...@apache.org>.
On Mon, Mar 31, 2008 at 5:19 AM, Michael McCandless
<lu...@mikemccandless.com> wrote:
> I think we should remove those checks and allow addIndexesNoOptimize
> to import and index even if it has segments over this limit. I'll
> open an issue.
+1
-Yonik
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: java.lang.IllegalArgumentException: Segment is too large
Posted by Michael McCandless <lu...@mikemccandless.com>.
This happens when addIndexesNoOptimize finds a segment that's larger
than maxMergeDocs in the index(es) it's given.
If you leave maxMergeDocs at Integer.MAX_VALUE it will fix that.
Though really it's being a little too pedantic because that setting
(maxMergeDocs) sets the maximum size of a segment that's may be
selected for merging. So it's easily possible to have segments in
your index that are larger than this (eg by merging segments that
were all just a bit smaller than the limit).
I think we should remove those checks and allow addIndexesNoOptimize
to import and index even if it has segments over this limit. I'll
open an issue.
Mike
vivek sar wrote:
> Hi,
>
> I'm using 2.3.0 Lucene build and have following merge parameters,
>
> mergeFactor = 100
> maxMergeDocs = 99999
> maxBufferedDocs = 10000
> maxRAMBufferSizeMB = 200
>
>
> After running with this setting for a month without problem all the
> sudden I'm getting following exception,
>
> java.lang.IllegalArgumentException: Segment is too large (100001 docs
> vs max docs 99999)
> at org.apache.lucene.index.LogMergePolicy.findMerges
> (LogMergePolicy.java:274)
> at org.apache.lucene.index.IndexWriter.updatePendingMerges
> (IndexWriter.java:1824)
> at org.apache.lucene.index.IndexWriter.maybeMerge
> (IndexWriter.java:1799)
> at org.apache.lucene.index.IndexWriter.maybeMerge
> (IndexWriter.java:1795)
> at org.apache.lucene.index.IndexWriter.maybeMerge
> (IndexWriter.java:1791)
> at org.apache.lucene.index.IndexWriter.addIndexesNoOptimize
> (IndexWriter.java:2231)
>
>
> I thought Lucene should control the number of documents in a segment
> during merge. Is there a different configuration I need to set? Should
> I leave the maxMergeDocs to its default value, Integer#MAX_VALUE?
>
> Thanks,
> -vivek
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org