[jira] Commented: (LUCENE-253) Merging indexes leads to out-of-memory condition

["Kurt De Grave (JIRA)" <ji...@apache.org>]

    [ http://issues.apache.org/jira/browse/LUCENE-253?page=comments#action_12361323 ] 

Kurt De Grave commented on LUCENE-253:
--------------------------------------

To give you an idea how much (or little) memory is needed for indexing in practice: I was able to index 9,999,999 documents without -Xmx.  My documents are about 3-4 KB ascii text. Then, on the 10Mth document, the merge of the ten indexes (a little over 1GB on average) into one, failed with OutOfMemoryError.

Giving it lots of memory doesn't make it any faster on a 1.5.0_05 Sun JVM (one might have expected a very little gain from a less constrained GC), it just gets past large merges. The -Xmx doesn't hurt either; the memory isn't used when it's not needed.


> Merging indexes leads to out-of-memory condition
> ------------------------------------------------
>
>          Key: LUCENE-253
>          URL: http://issues.apache.org/jira/browse/LUCENE-253
>      Project: Lucene - Java
>         Type: Bug
>   Components: Index
>     Versions: 1.4
>  Environment: Operating System: Linux
> Platform: Other
>     Reporter: dan
>     Assignee: Lucene Developers

>
> Version: 1.4 Final
> Platform: Red Hat 9
> Method: IndexWriter.addIndexes
> My experience with merging indexes of any size results in one outcome: 
> java.lang.OutOfMemoryError. This is true even on a 1GB machine where almost all 
> free memory is allocated to the VM.
> Is there a way to reliably merge indexes?
> I've marked this issue as 'major' for the following reason: Without an 
> efficient merge capability, the developer must deploy hundreds of 'little' 
> indexes that will need to be searched in parallel. Unfortunately, this 
> constraint plus java makes an application too slow to be usable in a production 
> environment.
> Thank you for looking into this issue.
> >>>>>>>>> sample code
> Directory dest = FSDirectory.getDirectory( destination, true);
> IndexWriter writer = new IndexWriter( dest, new TermAnalyzer( 
> StopWords.SEARCH_MAP), true);
> writer.addIndexes( sources);
> writer.optimize();
> writer.close();

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org