You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Jarek Zgoda <ja...@redefine.pl> on 2009/04/06 14:58:52 UTC
Too many open files and background merge exceptions
I'm indexing a set of 500000 small documents. I'm adding documents in
batches of 1000. At the beginning I had a setup that optimized the
index each 10000 documents, but quickly I had to optimize after adding
each batch of documents. Unfortunately, I'm still getting the "Too
many open files" IO error on optimize. I went from mergeFactor of 25
down to 10, but I'm still unable to optimize the index.
I have configuration:
<useCompoundFile>false</useCompoundFile>
<ramBufferSizeMB>256</ramBufferSizeMB>
<mergeFactor>2</mergeFactor>
<maxMergeDocs>2147483647</maxMergeDocs>
<maxFieldLength>10000</maxFieldLength>
The machine (2 core AMD64, 4GB RAM) is running Debian Linux, Java is
1.6.0_11 64-Bit, Solr is nightly build (2009-04-02). And no, I can not
change the limit of file descriptors (currently: 1024). What more can
I do?
--
We read Knuth so you don't have to. - Tim Peters
Jarek Zgoda, R&D, Redefine
jarek.zgoda@redefine.pl
Re: Too many open files and background merge exceptions
Posted by Jacob Singh <ja...@gmail.com>.
try ulimit -n50000 or something
On Mon, Apr 6, 2009 at 6:28 PM, Jarek Zgoda <ja...@redefine.pl> wrote:
> I'm indexing a set of 500000 small documents. I'm adding documents in
> batches of 1000. At the beginning I had a setup that optimized the index
> each 10000 documents, but quickly I had to optimize after adding each batch
> of documents. Unfortunately, I'm still getting the "Too many open files" IO
> error on optimize. I went from mergeFactor of 25 down to 10, but I'm still
> unable to optimize the index.
>
> I have configuration:
> <useCompoundFile>false</useCompoundFile>
> <ramBufferSizeMB>256</ramBufferSizeMB>
> <mergeFactor>2</mergeFactor>
> <maxMergeDocs>2147483647</maxMergeDocs>
> <maxFieldLength>10000</maxFieldLength>
>
> The machine (2 core AMD64, 4GB RAM) is running Debian Linux, Java is
> 1.6.0_11 64-Bit, Solr is nightly build (2009-04-02). And no, I can not
> change the limit of file descriptors (currently: 1024). What more can I do?
>
> --
> We read Knuth so you don't have to. - Tim Peters
>
> Jarek Zgoda, R&D, Redefine
> jarek.zgoda@redefine.pl
>
>
--
+1 510 277-0891 (o)
+91 9999 33 7458 (m)
web: http://pajamadesign.com
Skype: pajamadesign
Yahoo: jacobsingh
AIM: jacobsingh
gTalk: jacobsingh@gmail.com
Re: Too many open files and background merge exceptions
Posted by Walter Ferrara <wa...@gmail.com>.
you may try to put true in that useCompoundFile entry; this way indexing
should use far less file descriptors, but it will slow down indexing, see
http://issues.apache.org/jira/browse/LUCENE-888.
Try to see if the reason of lack of descriptors is related only on solr. How
are you using indexing, by using solrj, by posting xmls? Are the files being
opened/parsed on the same machine of solr?
On Mon, Apr 6, 2009 at 2:58 PM, Jarek Zgoda <ja...@redefine.pl> wrote:
> I'm indexing a set of 500000 small documents. I'm adding documents in
> batches of 1000. At the beginning I had a setup that optimized the index
> each 10000 documents, but quickly I had to optimize after adding each batch
> of documents. Unfortunately, I'm still getting the "Too many open files" IO
> error on optimize. I went from mergeFactor of 25 down to 10, but I'm still
> unable to optimize the index.
>
> I have configuration:
> <useCompoundFile>false</useCompoundFile>
> <ramBufferSizeMB>256</ramBufferSizeMB>
> <mergeFactor>2</mergeFactor>
> <maxMergeDocs>2147483647</maxMergeDocs>
> <maxFieldLength>10000</maxFieldLength>
>
> The machine (2 core AMD64, 4GB RAM) is running Debian Linux, Java is
> 1.6.0_11 64-Bit, Solr is nightly build (2009-04-02). And no, I can not
> change the limit of file descriptors (currently: 1024). What more can I do?
>
> --
> We read Knuth so you don't have to. - Tim Peters
>
> Jarek Zgoda, R&D, Redefine
> jarek.zgoda@redefine.pl
>
>