You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Markus Jelsma <ma...@openindex.io> on 2012/02/29 16:32:18 UTC
[SolrCloud] Too many open files - internal server error
Hi,
We're doing some tests with the latest trunk revision on a cluster of five
high-end machines. There is one collection, five shards and one replica per
shard on some other node.
We're filling the index from a MapReduce job, 18 processes run concurrently.
This is plenty when indexing to a single high-end node but with SolrCloud
things go down pretty soon.
First we get a Too Many Open Files error on all nodes almost at the same time.
When shutting down the indexer the nodes won't respond anymore except for an
Internal Server Error.
First the too many open files stack trace:
2012-02-29 15:22:51,067 ERROR [solr.core.SolrCore] - [http-80-6] - :
java.io.FileNotFoundException: /opt/solr/openindex_b/data/index/_h5_0.tim (Too
many open files)
at java.io.RandomAccessFile.open(Native Method)
at java.io.RandomAccessFile.<init>(RandomAccessFile.java:216)
at
org.apache.lucene.store.FSDirectory$FSIndexOutput.<init>(FSDirectory.java:449)
at
org.apache.lucene.store.FSDirectory.createOutput(FSDirectory.java:288)
at
org.apache.lucene.codecs.BlockTreeTermsWriter.<init>(BlockTreeTermsWriter.java:149)
at
org.apache.lucene.codecs.lucene40.Lucene40PostingsFormat.fieldsConsumer(Lucene40PostingsFormat.java:66)
at
org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.addField(PerFieldPostingsFormat.java:118)
at
org.apache.lucene.index.FreqProxTermsWriterPerField.flush(FreqProxTermsWriterPerField.java:322)
at
org.apache.lucene.index.FreqProxTermsWriter.flush(FreqProxTermsWriter.java:92)
at org.apache.lucene.index.TermsHash.flush(TermsHash.java:117)
at org.apache.lucene.index.DocInverter.flush(DocInverter.java:53)
at
org.apache.lucene.index.DocFieldProcessor.flush(DocFieldProcessor.java:81)
at
org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:475)
at
org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:422)
at
org.apache.lucene.index.DocumentsWriter.postUpdate(DocumentsWriter.java:320)
at
org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:389)
at
org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1533)
at
org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1505)
at
org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:168)
at
org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:56)
at
org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:53)
at
org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalAdd(DistributedUpdateProcessor.java:354)
at
org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:451)
at
org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:258)
at
org.apache.solr.update.processor.LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:118)
at org.apache.solr.handler.XMLLoader.processUpdate(XMLLoader.java:135)
at org.apache.solr.handler.XMLLoader.load(XMLLoader.java:79)
at
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:59)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1539)
at
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:406)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:255)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:859)
at
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:602)
at
org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
at java.lang.Thread.run(Thread.java:662
A similar exception sometimes begins with:
%2012-02-29 15:25:36,137 ERROR [solr.update.CommitTracker] - [pool-5-thread-1]
- : auto commit error...:java.io.FileNotFoundException: /opt/solr/openi
ndex_a/data/index/_j3_0.tim (Too many open files)
Here's the Internal server error stack trace:
%2012-02-29 15:26:51,402 ERROR [solr.core.SolrCore] - [http-80-4] - :
org.apache.solr.common.SolrException: Internal Server Error
Internal Server Error
request: http://cn005.openindex.io/solr/openindex_b/select
at
org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:433)
at
org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:251)
at
org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:156)
at
org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:123)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662
This one also comes in various flavours such as :
%2012-02-29 15:21:55,292 ERROR [solr.core.SolrCore] - [http-80-15] - : shard
update error
http://cn005.openindex.io:80/solr/openindex_b/:org.apache.solr.common.SolrException:
Internal Server Error
The Linux machines have proper settings for ulimit and friends, 32k open files
allowed so i suspect there's another limit which i am unaware of. I also
listed the number of open files while the errors were coming in but it did not
exceed 11k at any given time.
Any hints or advice? Did i miss something?
Thanks,
Markus
Re: [SolrCloud] Too many open files - internal server error
Posted by Markus Jelsma <ma...@openindex.io>.
Thanks. They are set properly. But i misspelled the tomcat6 username in
limits.conf :(
On Wednesday 29 February 2012 18:08:55 Yonik Seeley wrote:
> On Wed, Feb 29, 2012 at 10:32 AM, Markus Jelsma
>
> <ma...@openindex.io> wrote:
> > The Linux machines have proper settings for ulimit and friends, 32k open
> > files allowed
>
> Maybe you can expand on this point.
>
> cat /proc/sys/fs/file-max
> cat /proc/sys/fs/nr_open
>
> Those take precedence over ulimit. Not sure if there are others...
>
> -Yonik
> lucenerevolution.com - Lucene/Solr Open Source Search Conference.
> Boston May 7-10
--
Markus Jelsma - CTO - Openindex
Re: [SolrCloud] Too many open files - internal server error
Posted by Yonik Seeley <yo...@lucidimagination.com>.
On Wed, Feb 29, 2012 at 10:32 AM, Markus Jelsma
<ma...@openindex.io> wrote:
> The Linux machines have proper settings for ulimit and friends, 32k open files
> allowed
Maybe you can expand on this point.
cat /proc/sys/fs/file-max
cat /proc/sys/fs/nr_open
Those take precedence over ulimit. Not sure if there are others...
-Yonik
lucenerevolution.com - Lucene/Solr Open Source Search Conference.
Boston May 7-10
Re: [SolrCloud] Too many open files - internal server error
Posted by Carlos Alberto Schneider <ca...@informant.com.br>.
I had this problem sometime ago,
It happened on our homolog machine.
There was 3 solr instances , 1 master 2 slaves, running.
My Solution was: I stoped the slaves, deleted both data folders, runned an
optimize and than started it again.
I tried to raise the OS open file limit first, but i think it was not a
good idea... so i tried this ...
On Wed, Feb 29, 2012 at 2:07 PM, Markus Jelsma
<ma...@openindex.io>wrote:
> I get the correct output for ulimit -n as tomcat6 user. However, i did
> find a
--
Carlos Alberto Schneider
Informant -(47) 38010919 - 9904-5517
Re: [SolrCloud] Too many open files - internal server error
Posted by Markus Jelsma <ma...@openindex.io>.
On Wednesday 29 February 2012 17:52:55 Sami Siren wrote:
> On Wed, Feb 29, 2012 at 5:53 PM, Markus Jelsma
>
> <ma...@openindex.io> wrote:
> > Sami,
> >
> > As superuser:
> > $ lsof | wc -l
> >
> > But, just now, i also checked the system handler and it told me:
> > <str name="ulimit">(error executing: ulimit -n)</str>
>
> That's odd, you should see something like this there:
>
> "openFileDescriptorCount":131,
> "maxFileDescriptorCount":4096,
>
> Which jvm do you have?
Standard issue SUN Java 6 on Debian. We run that JVM on all machines. But i
see the same (error executing: ulimit -n) locally with Jetty and Solr trunk
and Solr 3.5 and on a production server with Solr 3.2 with Tomcat6.
>
> > This is rather strange, it seems. lsof | wc -l is not higher than 6k
> > right now and ulimit -n is 32k. Is lsof not to be trusted in this case
> > or... something else?
>
> I am not sure what is going on, are you sure the open file descriptor
> (32k) limit is active for the user running solr?
I get the correct output for ulimit -n as tomcat6 user. However, i did find a
mistake in /etc/security/limits.conf where i misspelled the tomcat6 user
(shame). On recent systems only ulimit and sysctl is not enough so spelling
tomcat6 correctly should fix the open files issue.
No we only have the issue of (error executing: ulimit -n).
>
> --
> Sami Siren
--
Markus Jelsma - CTO - Openindex
Re: [SolrCloud] Too many open files - internal server error
Posted by Sami Siren <ss...@gmail.com>.
On Wed, Feb 29, 2012 at 5:53 PM, Markus Jelsma
<ma...@openindex.io> wrote:
> Sami,
>
> As superuser:
> $ lsof | wc -l
>
> But, just now, i also checked the system handler and it told me:
> <str name="ulimit">(error executing: ulimit -n)</str>
That's odd, you should see something like this there:
"openFileDescriptorCount":131,
"maxFileDescriptorCount":4096,
Which jvm do you have?
> This is rather strange, it seems. lsof | wc -l is not higher than 6k right now
> and ulimit -n is 32k. Is lsof not to be trusted in this case or... something
> else?
I am not sure what is going on, are you sure the open file descriptor
(32k) limit is active for the user running solr?
--
Sami Siren
Re: [SolrCloud] Too many open files - internal server error
Posted by Markus Jelsma <ma...@openindex.io>.
Sami,
As superuser:
$ lsof | wc -l
But, just now, i also checked the system handler and it told me:
<str name="ulimit">(error executing: ulimit -n)</str>
This is rather strange, it seems. lsof | wc -l is not higher than 6k right now
and ulimit -n is 32k. Is lsof not to be trusted in this case or... something
else?
Thanks
On Wednesday 29 February 2012 16:44:58 Sami Siren wrote:
> Hi Markus,
>
> > The Linux machines have proper settings for ulimit and friends, 32k open
> > files allowed so i suspect there's another limit which i am unaware of.
> > I also listed the number of open files while the errors were coming in
> > but it did not exceed 11k at any given time.
>
> How did you check the number of filedescriptors used? Did you get this
> number from the system info handler
> (http://hotname:8983/solr/admin/system?indent=on&wt=json) or somehow
> differently?
>
> --
> Sami Siren
--
Markus Jelsma - CTO - Openindex
Re: [SolrCloud] Too many open files - internal server error
Posted by Sami Siren <ss...@gmail.com>.
Hi Markus,
> The Linux machines have proper settings for ulimit and friends, 32k open files
> allowed so i suspect there's another limit which i am unaware of. I also
> listed the number of open files while the errors were coming in but it did not
> exceed 11k at any given time.
How did you check the number of filedescriptors used? Did you get this
number from the system info handler
(http://hotname:8983/solr/admin/system?indent=on&wt=json) or somehow
differently?
--
Sami Siren