Merging many indexes

[Ben Halsted <bh...@gmail.com>]

Hi All,

So I've built up over 100 segments + indexes and now when I run:
new IndexMerger(fs, indexesSubDirs.toArray(new File[] {}), index,
tmpDir).merge();

I'm getting the dreaded: "Too many open files" error.
I've checked my system settings for file-max:
$ cat /proc/sys/fs/file-nr
2677 1945 478412

$ cat /proc/sys/fs/file-max
478412

What would be the best way to work around (or fix) this. Merging 10 indexes
at a time and then merging the results down until I get just one index?

What about the dedup process. It seems to be able to manage the 100+ indexes
fine, but if I switch the process and merge the indexes first and then
remove dupes, I think it may speed up the process. Ideas?

Thanks!
Ben

Re: Merging many indexes

[Ben Halsted <bh...@gmail.com>]

On 11/22/05, Doug Cutting <cu...@nutch.org> wrote:
>
> What does 'ulimit -n' print? Look in /etc/security/limits.conf to
> increase the limit.


$ ulimit -n
1024

I'll try increasing the limit in the limits.conf file.

> What would be the best way to work around (or fix) this. Merging 10
> indexes
> > at a time and then merging the results down until I get just one index?
>
> Yes. You can decrease indexer.mergeFactor to make this happen. Perhaps
> we should decrease the default. With the addition of crc files, the
> number of open files is doubled. So 50 indexes with 10 open files each
> yeilds 1000 open files, and the JVM needs more than 24. So I guess the
> default should be decreased to 30 or so.


So this is already handled in the code, sweet =). Mine is currently set to
50

> What about the dedup process. It seems to be able to manage the 100+
> indexes
> > fine, but if I switch the process and merge the indexes first and then
> > remove dupes, I think it may speed up the process. Ideas?
>
> Then you end up with dupes still taking space in your final index, which
> is not optimal for search.


Thanks again! I'll test out the new config and code.

Cheers,
Ben

Re: Merging many indexes

[Doug Cutting <cu...@nutch.org>]

Ben Halsted wrote:
> I'm getting the dreaded: "Too many open files" error.
> I've checked my system settings for file-max:
> $ cat /proc/sys/fs/file-nr
> 2677 1945 478412
> 
> $ cat /proc/sys/fs/file-max
> 478412

What does 'ulimit -n' print?  Look in /etc/security/limits.conf to 
increase the limit.

> What would be the best way to work around (or fix) this. Merging 10 indexes
> at a time and then merging the results down until I get just one index?

Yes.  You can decrease indexer.mergeFactor to make this happen.  Perhaps 
we should decrease the default.  With the addition of crc files, the 
number of open files is doubled.  So 50 indexes with 10 open files each 
yeilds 1000 open files, and the JVM needs more than 24.  So I guess the 
default should be decreased to 30 or so.

> What about the dedup process. It seems to be able to manage the 100+ indexes
> fine, but if I switch the process and merge the indexes first and then
> remove dupes, I think it may speed up the process. Ideas?

Then you end up with dupes still taking space in your final index, which 
is not optimal for search.

Doug

Re: Merging many indexes

[Ben Halsted <bh...@gmail.com>]

Stefan,

Great, I'll modify my code.

Cheers,
Ben

On 11/22/05, Stefan Groschupf <sg...@media-style.com> wrote:
>
> > What would be the best way to work around (or fix) this. Merging 10
> > indexes
> > at a time and then merging the results down until I get just one
> > index?
>
> Yes, this is the way we doing it.
>
>

Re: Merging many indexes

[Stefan Groschupf <sg...@media-style.com>]

> What would be the best way to work around (or fix) this. Merging 10  
> indexes
> at a time and then merging the results down until I get just one  
> index?

Yes, this is the way we doing it.