You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Markus Jelsma <ma...@openindex.io> on 2012/02/29 16:32:18 UTC

[SolrCloud] Too many open files - internal server error

Hi,

We're doing some tests with the latest trunk revision on a cluster of five 
high-end machines. There is one collection, five shards and one replica per 
shard on some other node.

We're filling the index from a MapReduce job, 18 processes run concurrently. 
This is plenty when indexing to a single high-end node but with SolrCloud 
things go down pretty soon.

First we get a Too Many Open Files error on all nodes almost at the same time. 
When shutting down the indexer the nodes won't respond anymore except for an 
Internal Server Error.

First the too many open files stack trace:

2012-02-29 15:22:51,067 ERROR [solr.core.SolrCore] - [http-80-6] - : 
java.io.FileNotFoundException: /opt/solr/openindex_b/data/index/_h5_0.tim (Too 
many open files)
        at java.io.RandomAccessFile.open(Native Method)
        at java.io.RandomAccessFile.<init>(RandomAccessFile.java:216)
        at 
org.apache.lucene.store.FSDirectory$FSIndexOutput.<init>(FSDirectory.java:449)
        at 
org.apache.lucene.store.FSDirectory.createOutput(FSDirectory.java:288)
        at 
org.apache.lucene.codecs.BlockTreeTermsWriter.<init>(BlockTreeTermsWriter.java:149)
        at 
org.apache.lucene.codecs.lucene40.Lucene40PostingsFormat.fieldsConsumer(Lucene40PostingsFormat.java:66)
        at 
org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.addField(PerFieldPostingsFormat.java:118)
        at 
org.apache.lucene.index.FreqProxTermsWriterPerField.flush(FreqProxTermsWriterPerField.java:322)
        at 
org.apache.lucene.index.FreqProxTermsWriter.flush(FreqProxTermsWriter.java:92)
        at org.apache.lucene.index.TermsHash.flush(TermsHash.java:117)
        at org.apache.lucene.index.DocInverter.flush(DocInverter.java:53)
        at 
org.apache.lucene.index.DocFieldProcessor.flush(DocFieldProcessor.java:81)
        at 
org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:475)
        at 
org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:422)
        at 
org.apache.lucene.index.DocumentsWriter.postUpdate(DocumentsWriter.java:320)
        at 
org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:389)
        at 
org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1533)
        at 
org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1505)
        at 
org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:168)
        at 
org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:56)
        at 
org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:53)
        at 
org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalAdd(DistributedUpdateProcessor.java:354)
        at 
org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:451)
        at 
org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:258)
        at 
org.apache.solr.update.processor.LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:118)
        at org.apache.solr.handler.XMLLoader.processUpdate(XMLLoader.java:135)
        at org.apache.solr.handler.XMLLoader.load(XMLLoader.java:79)
        at 
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:59)
        at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129)
        at org.apache.solr.core.SolrCore.execute(SolrCore.java:1539)
        at 
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:406)
        at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:255)
        at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
        at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at 
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
        at 
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
        at 
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
        at 
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
        at 
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
        at 
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
        at 
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:859)
        at 
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:602)
        at 
org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
        at java.lang.Thread.run(Thread.java:662



A similar exception sometimes begins with:

%2012-02-29 15:25:36,137 ERROR [solr.update.CommitTracker] - [pool-5-thread-1] 
- : auto commit error...:java.io.FileNotFoundException: /opt/solr/openi
ndex_a/data/index/_j3_0.tim (Too many open files)




Here's the Internal server error stack trace:

%2012-02-29 15:26:51,402 ERROR [solr.core.SolrCore] - [http-80-4] - : 
org.apache.solr.common.SolrException: Internal Server Error

Internal Server Error

request: http://cn005.openindex.io/solr/openindex_b/select
        at 
org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:433)
        at 
org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:251)
        at 
org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:156)
        at 
org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:123)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
        at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662


This one also comes in various flavours such as :


%2012-02-29 15:21:55,292 ERROR [solr.core.SolrCore] - [http-80-15] - : shard 
update error 
http://cn005.openindex.io:80/solr/openindex_b/:org.apache.solr.common.SolrException: 
Internal Server Error



The Linux machines have proper settings for ulimit and friends, 32k open files 
allowed so i suspect there's another limit which i am unaware of. I also 
listed the number of open files while the errors were coming in but it did not 
exceed 11k at any given time.

Any hints or advice? Did i miss something?

Thanks,
Markus

Re: [SolrCloud] Too many open files - internal server error

Posted by Markus Jelsma <ma...@openindex.io>.
Thanks. They are set properly. But i misspelled the tomcat6 username in 
limits.conf :(

On Wednesday 29 February 2012 18:08:55 Yonik Seeley wrote:
> On Wed, Feb 29, 2012 at 10:32 AM, Markus Jelsma
> 
> <ma...@openindex.io> wrote:
> > The Linux machines have proper settings for ulimit and friends, 32k open
> > files allowed
> 
> Maybe you can expand on this point.
> 
> cat /proc/sys/fs/file-max
> cat /proc/sys/fs/nr_open
> 
> Those take precedence over ulimit.  Not sure if there are others...
> 
> -Yonik
> lucenerevolution.com - Lucene/Solr Open Source Search Conference.
> Boston May 7-10

-- 
Markus Jelsma - CTO - Openindex

Re: [SolrCloud] Too many open files - internal server error

Posted by Yonik Seeley <yo...@lucidimagination.com>.
On Wed, Feb 29, 2012 at 10:32 AM, Markus Jelsma
<ma...@openindex.io> wrote:
> The Linux machines have proper settings for ulimit and friends, 32k open files
> allowed

Maybe you can expand on this point.

cat /proc/sys/fs/file-max
cat /proc/sys/fs/nr_open

Those take precedence over ulimit.  Not sure if there are others...

-Yonik
lucenerevolution.com - Lucene/Solr Open Source Search Conference.
Boston May 7-10

Re: [SolrCloud] Too many open files - internal server error

Posted by Carlos Alberto Schneider <ca...@informant.com.br>.
I had this problem sometime ago,
It happened on our homolog machine.

There was 3 solr instances , 1 master 2 slaves, running.
My Solution was: I stoped the slaves, deleted both data folders, runned an
optimize and than started it again.

I tried to raise the OS open file limit first, but i think it was not a
good idea... so i tried this ...


On Wed, Feb 29, 2012 at 2:07 PM, Markus Jelsma
<ma...@openindex.io>wrote:

> I get the correct output for ulimit -n as tomcat6 user. However, i did
> find a




-- 
Carlos Alberto Schneider
Informant -(47) 38010919 - 9904-5517

Re: [SolrCloud] Too many open files - internal server error

Posted by Markus Jelsma <ma...@openindex.io>.
On Wednesday 29 February 2012 17:52:55 Sami Siren wrote:
> On Wed, Feb 29, 2012 at 5:53 PM, Markus Jelsma
> 
> <ma...@openindex.io> wrote:
> > Sami,
> > 
> > As superuser:
> > $ lsof | wc -l
> > 
> > But, just now, i also checked the system handler and it told me:
> > <str name="ulimit">(error executing: ulimit -n)</str>
> 
> That's odd, you should see something like this there:
> 
>     "openFileDescriptorCount":131,
>     "maxFileDescriptorCount":4096,
> 
> Which jvm do you have?

Standard issue SUN Java 6 on Debian. We run that JVM on all machines. But i 
see the same (error executing: ulimit -n) locally with Jetty and Solr trunk 
and Solr 3.5 and on a production server with Solr 3.2 with Tomcat6.

> 
> > This is rather strange, it seems. lsof | wc -l is not higher than 6k
> > right now and ulimit -n is 32k. Is lsof not to be trusted in this case
> > or... something else?
> 
> I am not sure what is going on, are you sure the open file descriptor
> (32k) limit is active for the user running solr?

I get the correct output for ulimit -n as tomcat6 user. However, i did find a 
mistake in /etc/security/limits.conf where i misspelled the tomcat6 user 
(shame). On recent systems only ulimit and sysctl is not enough so spelling 
tomcat6 correctly should fix the open files issue. 

No we only have the issue of (error executing: ulimit -n).

> 
> --
>  Sami Siren

-- 
Markus Jelsma - CTO - Openindex

Re: [SolrCloud] Too many open files - internal server error

Posted by Sami Siren <ss...@gmail.com>.
On Wed, Feb 29, 2012 at 5:53 PM, Markus Jelsma
<ma...@openindex.io> wrote:
> Sami,
>
> As superuser:
> $ lsof | wc -l
>
> But, just now, i also checked the system handler and it told me:
> <str name="ulimit">(error executing: ulimit -n)</str>

That's odd, you should see something like this there:

    "openFileDescriptorCount":131,
    "maxFileDescriptorCount":4096,

Which jvm do you have?

> This is rather strange, it seems. lsof | wc -l is not higher than 6k right now
> and ulimit -n is 32k. Is lsof not to be trusted in this case or... something
> else?

I am not sure what is going on, are you sure the open file descriptor
(32k) limit is active for the user running solr?

--
 Sami Siren

Re: [SolrCloud] Too many open files - internal server error

Posted by Markus Jelsma <ma...@openindex.io>.
Sami,

As superuser:
$ lsof | wc -l

But, just now, i also checked the system handler and it told me:
<str name="ulimit">(error executing: ulimit -n)</str>

This is rather strange, it seems. lsof | wc -l is not higher than 6k right now 
and ulimit -n is 32k. Is lsof not to be trusted in this case or... something 
else? 

Thanks

On Wednesday 29 February 2012 16:44:58 Sami Siren wrote:
> Hi Markus,
> 
> > The Linux machines have proper settings for ulimit and friends, 32k open
> > files allowed so i suspect there's another limit which i am unaware of.
> > I also listed the number of open files while the errors were coming in
> > but it did not exceed 11k at any given time.
> 
> How did you check the number of filedescriptors used? Did you get this
> number from the system info handler
> (http://hotname:8983/solr/admin/system?indent=on&wt=json) or somehow
> differently?
> 
> --
>  Sami Siren

-- 
Markus Jelsma - CTO - Openindex

Re: [SolrCloud] Too many open files - internal server error

Posted by Sami Siren <ss...@gmail.com>.
Hi Markus,

> The Linux machines have proper settings for ulimit and friends, 32k open files
> allowed so i suspect there's another limit which i am unaware of. I also
> listed the number of open files while the errors were coming in but it did not
> exceed 11k at any given time.

How did you check the number of filedescriptors used? Did you get this
number from the system info handler
(http://hotname:8983/solr/admin/system?indent=on&wt=json) or somehow
differently?

--
 Sami Siren