You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Satya Marivada <sa...@gmail.com> on 2017/05/08 17:40:45 UTC

OutOfMemoryError and Too many open files

Hi,

Started getting below errors/exceptions. I have listed the resolution
inline. Could you please see if I am headed right?

The below error basically says that there are no more threads can be
created as the limit has reached. We have big index and I assume the
threads are being created outside of jvm and could not be because of low
ulimit setting of nproc (4096). It has been increased to 131072. This
number can be found by ulimit -u

java.lang.OutOfMemoryError: unable to create new native thread
        at java.lang.Thread.start0(Native Method)
        at java.lang.Thread.start(Thread.java:714)
        at
java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:950)
        at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1368)
        at
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:214)
        at
java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112)
        at
org.apache.solr.common.cloud.SolrZkClient$3.process(SolrZkClient.java:268)
        at
org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
        at
org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)

The below error basically says that there are no more files can be opened
as the limit has reached. It has been increased to 65536 from 4096. This
number can be found by ulimit -Hn, ulimit -Sn

java.io.IOException: Too many open files
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
        at
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
        at
org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:382)
        at
org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:593)
        at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)

Thanks,
Satya

Re: OutOfMemoryError and Too many open files

Posted by Erick Erickson <er...@gmail.com>.
Solr/Lucene really like having a bunch of files available, so bumping
the ulimit is often the right thing to do.

This assumes you don't have any custom code that is failing to close
searchers and the like.

Best,
Erick

On Mon, May 8, 2017 at 10:40 AM, Satya Marivada
<sa...@gmail.com> wrote:
> Hi,
>
> Started getting below errors/exceptions. I have listed the resolution
> inline. Could you please see if I am headed right?
>
> The below error basically says that there are no more threads can be
> created as the limit has reached. We have big index and I assume the
> threads are being created outside of jvm and could not be because of low
> ulimit setting of nproc (4096). It has been increased to 131072. This
> number can be found by ulimit -u
>
> java.lang.OutOfMemoryError: unable to create new native thread
>         at java.lang.Thread.start0(Native Method)
>         at java.lang.Thread.start(Thread.java:714)
>         at
> java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:950)
>         at
> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1368)
>         at
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:214)
>         at
> java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112)
>         at
> org.apache.solr.common.cloud.SolrZkClient$3.process(SolrZkClient.java:268)
>         at
> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
>         at
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
>
> The below error basically says that there are no more files can be opened
> as the limit has reached. It has been increased to 65536 from 4096. This
> number can be found by ulimit -Hn, ulimit -Sn
>
> java.io.IOException: Too many open files
>         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
>         at
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
>         at
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
>         at
> org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:382)
>         at
> org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:593)
>         at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
>
> Thanks,
> Satya

Re: OutOfMemoryError and Too many open files

Posted by Shawn Heisey <ap...@elyograg.org>.
On 5/8/2017 11:40 AM, Satya Marivada wrote:
> Started getting below errors/exceptions. I have listed the resolution
> inline. Could you please see if I am headed right?
>
> java.lang.OutOfMemoryError: unable to create new native thread
<snip>
> java.io.IOException: Too many open files

I have never had any luck setting these limits with the ulimit command. 
On Linux, I have adjusted these in the /etc/security/limits.conf config
file.  This is what I added to the file:

solr            hard    nproc   61440
solr            soft    nproc   40960

solr            hard    nofile  65535
solr            soft    nofile  49151

A reboot shouldn't be needed to get the change to take effect, though
you may need to log out and back in again before attempting to restart
Solr.  A reboot would be the guaranteed option.

For an OS other than Linux, the method for changing these limits is
probably going to be different.

Thanks,
Shawn