You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Harald Kirsch <ki...@ebi.ac.uk> on 2004/06/25 17:00:43 UTC

best mergeFactor for merging 100 Indexes

Hi,

after an hour of indexing on a cluster I got 100 Indexes, ca. 25MB
each, 2 indexed fields. I intend now to run code roughly like

   IndexWriter writer = new IndexWriter(destDir, ...);
   writer.addIndexes(my100IndexDirs);
   writer.close()

When I did this a year ago, I know I had tough problems getting around
memory limitations and open file limits at the same time. In the end
it worked with writer.mergeFactor==4000, but I think it was a
specially tweaked kernel on Linux which I don't have anymore.

Since I don't really understand yet how open files, segments, memory
use, indexing time and mergeFacter interact, I would appreciate a good
gues how to combine these indexes.

Which mergeFactor to use?
Use a different strategy then the 3 lines shown above?

  Thanks,
  Harald.

-- 
------------------------------------------------------------------------
Harald Kirsch | kirsch@ebi.ac.uk | +44 (0) 1223/49-2593

---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org


Re: best mergeFactor for merging 100 Indexes

Posted by Otis Gospodnetic <ot...@yahoo.com>.
If this is an option, use compund index format
(writer.setUseCompound(true)).  ulimit -a in some UNIX shells will tell
you the max number of open files allowed.  If you can, increase that
number as high as you can.  Of course, how high you can go also depends
on your RAM.  Finally, don't forget there is a minMergeDocs parameter
you can use to tune things, too.

Otis


--- Harald Kirsch <ki...@ebi.ac.uk> wrote:
> Hi,
> 
> after an hour of indexing on a cluster I got 100 Indexes, ca. 25MB
> each, 2 indexed fields. I intend now to run code roughly like
> 
>    IndexWriter writer = new IndexWriter(destDir, ...);
>    writer.addIndexes(my100IndexDirs);
>    writer.close()
> 
> When I did this a year ago, I know I had tough problems getting
> around
> memory limitations and open file limits at the same time. In the end
> it worked with writer.mergeFactor==4000, but I think it was a
> specially tweaked kernel on Linux which I don't have anymore.
> 
> Since I don't really understand yet how open files, segments, memory
> use, indexing time and mergeFacter interact, I would appreciate a
> good
> gues how to combine these indexes.
> 
> Which mergeFactor to use?
> Use a different strategy then the 3 lines shown above?
> 
>   Thanks,
>   Harald.
> 
> -- 
>
------------------------------------------------------------------------
> Harald Kirsch | kirsch@ebi.ac.uk | +44 (0) 1223/49-2593
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: lucene-user-help@jakarta.apache.org
> 
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org