You are viewing a plain text version of this content. The canonical link for it is here.

Posted to solr-user@lucene.apache.org by Srinivas Kashyap <sr...@bamboorose.com.INVALID> on 2020/03/06 13:54:27 UTC

OutOfMemory error solr 8.4.1

Hi All,

I have recently upgraded solr to 8.4.1 and have installed solr as service in linux machine. Once I start my service, it will be up for 15-18hours and suddenly stops without us shutting down. In solr.log I found below error. Can somebody guide me what values should I be increasing in Linux machine?

Earlier, open file limit was not set and now I have increased. Below are my system configuration for solr:

JVM memory: 8GB
RAM: 32GB
Open file descriptor count: 500000

Ulimit -v - unlimited
Ulimit -m - unlimited


ERROR STACK TRACE:

2020-03-06 12:08:03.071 ERROR (qtp1691185247-21) [   x:product] o.a.s.s.HttpSolrCall null:java.lang.RuntimeException: java.lang.OutOfMemoryError: unable to create new native thread
        at org.apache.solr.servlet.HttpSolrCall.sendError(HttpSolrCall.java:752)
        at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:603)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:419)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:351)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1711)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1347)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1678)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1249)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
        at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220)
        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:152)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
        at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
        at org.eclipse.jetty.server.Server.handle(Server.java:505)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:370)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:267)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
        at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
        at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:781)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:917)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.OutOfMemoryError: unable to create new native thread
        at java.lang.Thread.start0(Native Method)
        at java.lang.Thread.start(Thread.java:717)
        at org.apache.solr.handler.dataimport.DataImporter.runAsync(DataImporter.java:466)
        at org.apache.solr.handler.dataimport.DataImportHandler.handleRequestBody(DataImportHandler.java:205)
        at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:211)
        at org.apache.solr.core.SolrCore.execute(SolrCore.java:2596)
        at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:799)
        at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:578)
        ... 36 more




java.lang.OutOfMemoryError: unable to create new native thread
        at java.lang.Thread.start0(Native Method) ~[?:1.8.0_242]
        at java.lang.Thread.start(Thread.java:717) ~[?:1.8.0_242]
        at org.apache.solr.handler.dataimport.DataImporter.runAsync(DataImporter.java:466) ~[?:?]
        at org.apache.solr.handler.dataimport.DataImportHandler.handleRequestBody(DataImportHandler.java:205) ~[?:?]
        at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:211) ~[?:?]
        at org.apache.solr.core.SolrCore.execute(SolrCore.java:2596) ~[?:?]
        at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:799) ~[?:?]
        at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:578) ~[?:?]
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:419) ~[?:?]
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:351) ~[?:?]
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602) ~[jetty-servlet-9.4.19.v20190610.jar:9.4.19.v20190610]
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540) ~[jetty-servlet-9.4.19.v20190610.jar:9.4.19.v20190610]
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146) ~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) ~[jetty-security-9.4.19.v20190610.jar:9.4.19.v20190610]
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) ~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257) ~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1711) ~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255) ~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1347) ~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203) ~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480) ~[jetty-servlet-9.4.19.v20190610.jar:9.4.19.v20190610]
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1678) ~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201) ~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1249) ~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144) ~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
        at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220) ~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:152) ~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) ~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
        at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335) ~[jetty-rewrite-9.4.19.v20190610.jar:9.4.19.v20190610]
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) ~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
        at org.eclipse.jetty.server.Server.handle(Server.java:505) ~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:370) ~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:267) ~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305) ~[jetty-io-9.4.19.v20190610.jar:9.4.19.v20190610]
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103) ~[jetty-io-9.4.19.v20190610.jar:9.4.19.v20190610]
        at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117) ~[jetty-io-9.4.19.v20190610.jar:9.4.19.v20190610]
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333) ~[jetty-util-9.4.19.v20190610.jar:9.4.19.v20190610]
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310) ~[jetty-util-9.4.19.v20190610.jar:9.4.19.v20190610]
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) ~[jetty-util-9.4.19.v20190610.jar:9.4.19.v20190610]
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126) ~[jetty-util-9.4.19.v20190610.jar:9.4.19.v20190610]
        at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366) ~[jetty-util-9.4.19.v20190610.jar:9.4.19.v20190610]
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:781) ~[jetty-util-9.4.19.v20190610.jar:9.4.19.v20190610]
        at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:917) ~[jetty-util-9.4.19.v20190610.jar:9.4.19.v20190610]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_242]




Thanks and Regards,
Srinivas Kashyap

________________________________
DISCLAIMER:
E-mails and attachments from Bamboo Rose, LLC are confidential.
If you are not the intended recipient, please notify the sender immediately by replying to the e-mail, and then delete it without making copies or using it in any way.
No representation is made that this email or any attachments are free of viruses. Virus scanning is recommended and is the responsibility of the recipient.

Disclaimer

The information contained in this communication from the sender is confidential. It is intended solely for use by the recipient and others authorized to receive it. If you are not the recipient, you are hereby notified that any disclosure, copying, distribution or taking action in relation of the contents of this information is strictly prohibited and may be unlawful.

This email has been scanned for viruses and malware, and may have been automatically archived by Mimecast Ltd, an innovator in Software as a Service (SaaS) for business. Providing a safer and more useful place for your human generated data. Specializing in; Security, archiving and compliance. To find out more visit the Mimecast website.

RE: OutOfMemory error solr 8.4.1

Posted by Srinivas Kashyap <sr...@bamboorose.com.INVALID>.

Hi Erick,

Yes you were right, in my custom jar I'm using HttpSolrClient as below:

HttpSolrClient  client = new HttpSolrClient.Builder("http://" + server + ":" + port + "/" + webapp + "/").build();  		
    	 try {
    		 client.request(new QueryRequest(params),coreName);
    	 } catch (SolrServerException e1) {
    		 // TODO Auto-generated catch block
    		 e1.printStackTrace();
    	 } catch (IOException e1) {
    		 // TODO Auto-generated catch block
    		 e1.printStackTrace();
    	 }

And strangely, I'm not closing the connection(client.close()). The same code would work without creating heaps of threads in version 5.2.1. Now after I added finally block and closed the connection, the threads have stopped growing in size.

finally{
    	if(client!=null)
    	{
    		try {
			client.close();
		} catch (IOException e) {
			// TODO Auto-generated catch block
			e.printStackTrace();
		}
    	}
    }

Thanks and Regards,
Srinivas Kashyap

From: Erick Erickson <er...@gmail.com> 
Sent: 09 March 2020 21:13
To: solr-user@lucene.apache.org
Subject: Re: OutOfMemory error solr 8.4.1

I’m 99% certain that something in your custom jar is the culprit, otherwise we’d have seen a _lot_ of these. TIMED_WAITING is usually just a listener thread, but they shouldn’t be generated when SOlr is just sitting there. 

The first thing I’d do is dummy out my custom code or remove it completely and see. If you don’t have this thread explosion, then it’s pretty certainly your custom code.

Best,
Erick

> On Mar 9, 2020, at 01:29, Srinivas Kashyap <sr...@bamboorose.com.invalid> wrote:
> 
> Hi Erick,
> 
> I recompiled my custom code with 8.4.1 jars and placed back my jar in the lib folder. Under Solr admin console/Thread Dump, I'm seeing a lot of below threads which are in TIMED_WAITING stage.
> 
> Connection evictor (999)
>    java.lang.Thread.sleep(Native Method)
>    org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
>    java.lang.Thread.run(Thread.java:748)
> 
> It's been 15 minutes since I restarted the solr, and I believe already 999 threads have started?? And everytime I refresh the console, I'm seeing jump :
> 
> Connection evictor (1106)
> java.lang.Thread.sleep(Native Method)
> org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
> java.lang.Thread.run(Thread.java:748)
> 
> Thanks and Regards,
> Srinivas Kashyap
> 
> -----Original Message-----
> From: Erick Erickson <er...@gmail.com> 
> Sent: 06 March 2020 21:34
> To: solr-user@lucene.apache.org
> Subject: Re: OutOfMemory error solr 8.4.1
> 
> I assume you recompiled the jar file? re-using the same one compiled against 5x is unsupported, nobody will be able to help until you recompile.
> 
> Once you’ve done that, if you still have the problem you need to take a thread dump to see if your custom code is leaking threads, that’s my number one suspect.
> 
> Best,
> Erick
> 
>> On Mar 6, 2020, at 07:36, Srinivas Kashyap <sr...@bamboorose.com.invalid> wrote:
>> 
>> Hi Erick,
>> 
>> We have custom code which are schedulers to run delta imports on our cores and I have added that custom code as a jar and I have placed it on server/solr-webapp/WEB-INF/lib. Basically we are fetching the JNDI datasource configured in the jetty.xml(Oracle) and creating connection object. And after that in the finally block we are closing it too.
>> 
>> Never faced this issue while we were in solr5.2.1 version though. The same jar was placed there too.
>> 
>> Thanks,
>> Srinivas
>> 
>> On 06-Mar-2020 8:55 pm, Erick Erickson <er...@gmail.com> wrote:
>> This one can be a bit tricky. You’re not running out of overall memory, but you are running out of memory to allocate stacks. Which implies that, for some reason, you are creating a zillion threads. Do you have any custom code?
>> 
>> You can take a thread dump and see what your threads are doing, and you don’t need to wait until you see the error. If you take a thread dump my guess is you’ll see the number of threads increase over time. If that’s the case, and if you have no custom code running, we need to see the thread dump.
>> 
>> Best,
>> Erick
>> 
>>>> On Mar 6, 2020, at 05:54, Srinivas Kashyap <sr...@bamboorose.com.invalid> wrote:
>>> 
>>> Hi All,
>>> 
>>> I have recently upgraded solr to 8.4.1 and have installed solr as service in linux machine. Once I start my service, it will be up for 15-18hours and suddenly stops without us shutting down. In solr.log I found below error. Can somebody guide me what values should I be increasing in Linux machine?
>>> 
>>> Earlier, open file limit was not set and now I have increased. Below are my system configuration for solr:
>>> 
>>> JVM memory: 8GB
>>> RAM: 32GB
>>> Open file descriptor count: 500000
>>> 
>>> Ulimit -v - unlimited
>>> Ulimit -m - unlimited
>>> 
>>> 
>>> ERROR STACK TRACE:
>>> 
>>> 2020-03-06 12:08:03.071 ERROR (qtp1691185247-21) [   x:product] o.a.s.s.HttpSolrCall null:java.lang.RuntimeException: java.lang.OutOfMemoryError: unable to create new native thread
>>>      at org.apache.solr.servlet.HttpSolrCall.sendError(HttpSolrCall.java:752)
>>>      at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:603)
>>>      at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:419)
>>>      at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:351)
>>>      at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602)
>>>      at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
>>>      at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
>>>      at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
>>>      at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
>>>      at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
>>>      at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1711)
>>>      at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
>>>      at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1347)
>>>      at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
>>>      at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
>>>      at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1678)
>>>      at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
>>>      at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1249)
>>>      at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
>>>      at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220)
>>>      at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:152)
>>>      at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
>>>      at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
>>>      at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
>>>      at org.eclipse.jetty.server.Server.handle(Server.java:505)
>>>      at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:370)
>>>      at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:267)
>>>      at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
>>>      at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
>>>      at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
>>>      at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
>>>      at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
>>>      at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
>>>      at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
>>>      at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
>>>      at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:781)
>>>      at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:917)
>>>      at java.lang.Thread.run(Thread.java:748)
>>> Caused by: java.lang.OutOfMemoryError: unable to create new native thread
>>>      at java.lang.Thread.start0(Native Method)
>>>      at java.lang.Thread.start(Thread.java:717)
>>>      at org.apache.solr.handler.dataimport.DataImporter.runAsync(DataImporter.java:466)
>>>      at org.apache.solr.handler.dataimport.DataImportHandler.handleRequestBody(DataImportHandler.java:205)
>>>      at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:211)
>>>      at org.apache.solr.core.SolrCore.execute(SolrCore.java:2596)
>>>      at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:799)
>>>      at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:578)
>>>      ... 36 more
>>> 
>>> 
>>> 
>>> 
>>> java.lang.OutOfMemoryError: unable to create new native thread
>>>      at java.lang.Thread.start0(Native Method) ~[?:1.8.0_242]
>>>      at java.lang.Thread.start(Thread.java:717) ~[?:1.8.0_242]
>>>      at org.apache.solr.handler.dataimport.DataImporter.runAsync(DataImporter.java:466) ~[?:?]
>>>      at org.apache.solr.handler.dataimport.DataImportHandler.handleRequestBody(DataImportHandler.java:205) ~[?:?]
>>>      at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:211) ~[?:?]
>>>      at org.apache.solr.core.SolrCore.execute(SolrCore.java:2596) ~[?:?]
>>>      at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:799) ~[?:?]
>>>      at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:578) ~[?:?]
>>>      at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:419) ~[?:?]
>>>      at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:351) ~[?:?]
>>>      at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602) ~[jetty-servlet-9.4.19.v20190610.jar:9.4.19.v20190610]
>>>      at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540) ~[jetty-servlet-9.4.19.v20190610.jar:9.4.19.v20190610]
>>>      at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146) ~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
>>>      at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) ~[jetty-security-9.4.19.v20190610.jar:9.4.19.v20190610]
>>>      at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) ~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
>>>      at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257) ~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
>>>      at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1711) ~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
>>>      at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255) ~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
>>>      at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1347) ~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
>>>      at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203) ~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
>>>      at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480) ~[jetty-servlet-9.4.19.v20190610.jar:9.4.19.v20190610]
>>>      at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1678) ~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
>>>      at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201) ~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
>>>      at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1249) ~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
>>>      at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144) ~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
>>>      at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220) ~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
>>>      at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:152) ~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
>>>      at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) ~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
>>>      at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335) ~[jetty-rewrite-9.4.19.v20190610.jar:9.4.19.v20190610]
>>>      at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) ~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
>>>      at org.eclipse.jetty.server.Server.handle(Server.java:505) ~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
>>>      at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:370) ~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
>>>      at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:267) ~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
>>>      at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305) ~[jetty-io-9.4.19.v20190610.jar:9.4.19.v20190610]
>>>      at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103) ~[jetty-io-9.4.19.v20190610.jar:9.4.19.v20190610]
>>>      at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117) ~[jetty-io-9.4.19.v20190610.jar:9.4.19.v20190610]
>>>      at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333) ~[jetty-util-9.4.19.v20190610.jar:9.4.19.v20190610]
>>>      at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310) ~[jetty-util-9.4.19.v20190610.jar:9.4.19.v20190610]
>>>      at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) ~[jetty-util-9.4.19.v20190610.jar:9.4.19.v20190610]
>>>      at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126) ~[jetty-util-9.4.19.v20190610.jar:9.4.19.v20190610]
>>>      at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366) ~[jetty-util-9.4.19.v20190610.jar:9.4.19.v20190610]
>>>      at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:781) ~[jetty-util-9.4.19.v20190610.jar:9.4.19.v20190610]
>>>      at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:917) ~[jetty-util-9.4.19.v20190610.jar:9.4.19.v20190610]
>>>      at java.lang.Thread.run(Thread.java:748) [?:1.8.0_242]
>>> 
>>> 
>>> 
>>> 
>>> Thanks and Regards,
>>> Srinivas Kashyap
>>> 
>>> ________________________________
>>> DISCLAIMER:
>>> E-mails and attachments from Bamboo Rose, LLC are confidential.
>>> If you are not the intended recipient, please notify the sender immediately by replying to the e-mail, and then delete it without making copies or using it in any way.
>>> No representation is made that this email or any attachments are free of viruses. Virus scanning is recommended and is the responsibility of the recipient.
>>> 
>>> Disclaimer
>>> 
>>> The information contained in this communication from the sender is confidential. It is intended solely for use by the recipient and others authorized to receive it. If you are not the recipient, you are hereby notified that any disclosure, copying, distribution or taking action in relation of the contents of this information is strictly prohibited and may be unlawful.
>>> 
>>> This email has been scanned for viruses and malware, and may have been automatically archived by Mimecast Ltd, an innovator in Software as a Service (SaaS) for business. Providing a safer and more useful place for your human generated data. Specializing in; Security, archiving and compliance. To find out more visit the Mimecast website.
>> 
>

Re: OutOfMemory error solr 8.4.1

Posted by Erick Erickson <er...@gmail.com>.

I’m 99% certain that something in your custom jar is the culprit, otherwise we’d have seen a _lot_ of these. TIMED_WAITING is usually just a listener thread, but they shouldn’t be generated when SOlr is just sitting there. 

The first thing I’d do is dummy out my custom code or remove it completely and see. If you don’t have this thread explosion, then it’s pretty certainly your custom code.

Best,
Erick

> On Mar 9, 2020, at 01:29, Srinivas Kashyap <sr...@bamboorose.com.invalid> wrote:
> 
> Hi Erick,
> 
> I recompiled my custom code with 8.4.1 jars and placed back my jar in the lib folder. Under Solr admin console/Thread Dump, I'm seeing a lot of below threads which are in TIMED_WAITING stage.
> 
> Connection evictor (999)
>    java.lang.Thread.sleep(Native Method)
>    org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
>    java.lang.Thread.run(Thread.java:748)
> 
> It's been 15 minutes since I restarted the solr, and I believe already 999 threads have started?? And everytime I refresh the console, I'm seeing jump :
> 
> Connection evictor (1106)
> java.lang.Thread.sleep(Native Method)
> org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
> java.lang.Thread.run(Thread.java:748)
> 
> Thanks and Regards,
> Srinivas Kashyap
> 
> -----Original Message-----
> From: Erick Erickson <er...@gmail.com> 
> Sent: 06 March 2020 21:34
> To: solr-user@lucene.apache.org
> Subject: Re: OutOfMemory error solr 8.4.1
> 
> I assume you recompiled the jar file? re-using the same one compiled against 5x is unsupported, nobody will be able to help until you recompile.
> 
> Once you’ve done that, if you still have the problem you need to take a thread dump to see if your custom code is leaking threads, that’s my number one suspect.
> 
> Best,
> Erick
> 
>> On Mar 6, 2020, at 07:36, Srinivas Kashyap <sr...@bamboorose.com.invalid> wrote:
>> 
>> Hi Erick,
>> 
>> We have custom code which are schedulers to run delta imports on our cores and I have added that custom code as a jar and I have placed it on server/solr-webapp/WEB-INF/lib. Basically we are fetching the JNDI datasource configured in the jetty.xml(Oracle) and creating connection object. And after that in the finally block we are closing it too.
>> 
>> Never faced this issue while we were in solr5.2.1 version though. The same jar was placed there too.
>> 
>> Thanks,
>> Srinivas
>> 
>> On 06-Mar-2020 8:55 pm, Erick Erickson <er...@gmail.com> wrote:
>> This one can be a bit tricky. You’re not running out of overall memory, but you are running out of memory to allocate stacks. Which implies that, for some reason, you are creating a zillion threads. Do you have any custom code?
>> 
>> You can take a thread dump and see what your threads are doing, and you don’t need to wait until you see the error. If you take a thread dump my guess is you’ll see the number of threads increase over time. If that’s the case, and if you have no custom code running, we need to see the thread dump.
>> 
>> Best,
>> Erick
>> 
>>>> On Mar 6, 2020, at 05:54, Srinivas Kashyap <sr...@bamboorose.com.invalid> wrote:
>>> 
>>> Hi All,
>>> 
>>> I have recently upgraded solr to 8.4.1 and have installed solr as service in linux machine. Once I start my service, it will be up for 15-18hours and suddenly stops without us shutting down. In solr.log I found below error. Can somebody guide me what values should I be increasing in Linux machine?
>>> 
>>> Earlier, open file limit was not set and now I have increased. Below are my system configuration for solr:
>>> 
>>> JVM memory: 8GB
>>> RAM: 32GB
>>> Open file descriptor count: 500000
>>> 
>>> Ulimit -v - unlimited
>>> Ulimit -m - unlimited
>>> 
>>> 
>>> ERROR STACK TRACE:
>>> 
>>> 2020-03-06 12:08:03.071 ERROR (qtp1691185247-21) [   x:product] o.a.s.s.HttpSolrCall null:java.lang.RuntimeException: java.lang.OutOfMemoryError: unable to create new native thread
>>>      at org.apache.solr.servlet.HttpSolrCall.sendError(HttpSolrCall.java:752)
>>>      at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:603)
>>>      at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:419)
>>>      at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:351)
>>>      at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602)
>>>      at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
>>>      at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
>>>      at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
>>>      at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
>>>      at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
>>>      at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1711)
>>>      at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
>>>      at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1347)
>>>      at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
>>>      at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
>>>      at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1678)
>>>      at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
>>>      at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1249)
>>>      at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
>>>      at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220)
>>>      at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:152)
>>>      at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
>>>      at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
>>>      at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
>>>      at org.eclipse.jetty.server.Server.handle(Server.java:505)
>>>      at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:370)
>>>      at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:267)
>>>      at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
>>>      at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
>>>      at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
>>>      at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
>>>      at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
>>>      at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
>>>      at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
>>>      at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
>>>      at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:781)
>>>      at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:917)
>>>      at java.lang.Thread.run(Thread.java:748)
>>> Caused by: java.lang.OutOfMemoryError: unable to create new native thread
>>>      at java.lang.Thread.start0(Native Method)
>>>      at java.lang.Thread.start(Thread.java:717)
>>>      at org.apache.solr.handler.dataimport.DataImporter.runAsync(DataImporter.java:466)
>>>      at org.apache.solr.handler.dataimport.DataImportHandler.handleRequestBody(DataImportHandler.java:205)
>>>      at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:211)
>>>      at org.apache.solr.core.SolrCore.execute(SolrCore.java:2596)
>>>      at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:799)
>>>      at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:578)
>>>      ... 36 more
>>> 
>>> 
>>> 
>>> 
>>> java.lang.OutOfMemoryError: unable to create new native thread
>>>      at java.lang.Thread.start0(Native Method) ~[?:1.8.0_242]
>>>      at java.lang.Thread.start(Thread.java:717) ~[?:1.8.0_242]
>>>      at org.apache.solr.handler.dataimport.DataImporter.runAsync(DataImporter.java:466) ~[?:?]
>>>      at org.apache.solr.handler.dataimport.DataImportHandler.handleRequestBody(DataImportHandler.java:205) ~[?:?]
>>>      at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:211) ~[?:?]
>>>      at org.apache.solr.core.SolrCore.execute(SolrCore.java:2596) ~[?:?]
>>>      at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:799) ~[?:?]
>>>      at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:578) ~[?:?]
>>>      at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:419) ~[?:?]
>>>      at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:351) ~[?:?]
>>>      at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602) ~[jetty-servlet-9.4.19.v20190610.jar:9.4.19.v20190610]
>>>      at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540) ~[jetty-servlet-9.4.19.v20190610.jar:9.4.19.v20190610]
>>>      at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146) ~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
>>>      at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) ~[jetty-security-9.4.19.v20190610.jar:9.4.19.v20190610]
>>>      at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) ~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
>>>      at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257) ~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
>>>      at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1711) ~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
>>>      at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255) ~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
>>>      at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1347) ~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
>>>      at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203) ~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
>>>      at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480) ~[jetty-servlet-9.4.19.v20190610.jar:9.4.19.v20190610]
>>>      at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1678) ~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
>>>      at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201) ~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
>>>      at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1249) ~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
>>>      at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144) ~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
>>>      at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220) ~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
>>>      at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:152) ~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
>>>      at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) ~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
>>>      at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335) ~[jetty-rewrite-9.4.19.v20190610.jar:9.4.19.v20190610]
>>>      at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) ~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
>>>      at org.eclipse.jetty.server.Server.handle(Server.java:505) ~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
>>>      at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:370) ~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
>>>      at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:267) ~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
>>>      at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305) ~[jetty-io-9.4.19.v20190610.jar:9.4.19.v20190610]
>>>      at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103) ~[jetty-io-9.4.19.v20190610.jar:9.4.19.v20190610]
>>>      at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117) ~[jetty-io-9.4.19.v20190610.jar:9.4.19.v20190610]
>>>      at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333) ~[jetty-util-9.4.19.v20190610.jar:9.4.19.v20190610]
>>>      at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310) ~[jetty-util-9.4.19.v20190610.jar:9.4.19.v20190610]
>>>      at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) ~[jetty-util-9.4.19.v20190610.jar:9.4.19.v20190610]
>>>      at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126) ~[jetty-util-9.4.19.v20190610.jar:9.4.19.v20190610]
>>>      at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366) ~[jetty-util-9.4.19.v20190610.jar:9.4.19.v20190610]
>>>      at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:781) ~[jetty-util-9.4.19.v20190610.jar:9.4.19.v20190610]
>>>      at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:917) ~[jetty-util-9.4.19.v20190610.jar:9.4.19.v20190610]
>>>      at java.lang.Thread.run(Thread.java:748) [?:1.8.0_242]
>>> 
>>> 
>>> 
>>> 
>>> Thanks and Regards,
>>> Srinivas Kashyap
>>> 
>>> ________________________________
>>> DISCLAIMER:
>>> E-mails and attachments from Bamboo Rose, LLC are confidential.
>>> If you are not the intended recipient, please notify the sender immediately by replying to the e-mail, and then delete it without making copies or using it in any way.
>>> No representation is made that this email or any attachments are free of viruses. Virus scanning is recommended and is the responsibility of the recipient.
>>> 
>>> Disclaimer
>>> 
>>> The information contained in this communication from the sender is confidential. It is intended solely for use by the recipient and others authorized to receive it. If you are not the recipient, you are hereby notified that any disclosure, copying, distribution or taking action in relation of the contents of this information is strictly prohibited and may be unlawful.
>>> 
>>> This email has been scanned for viruses and malware, and may have been automatically archived by Mimecast Ltd, an innovator in Software as a Service (SaaS) for business. Providing a safer and more useful place for your human generated data. Specializing in; Security, archiving and compliance. To find out more visit the Mimecast website.
>> 
>

RE: OutOfMemory error solr 8.4.1

Posted by Srinivas Kashyap <sr...@bamboorose.com.INVALID>.

Hi Erick,

I recompiled my custom code with 8.4.1 jars and placed back my jar in the lib folder. Under Solr admin console/Thread Dump, I'm seeing a lot of below threads which are in TIMED_WAITING stage.

Connection evictor (999)
	java.lang.Thread.sleep(Native Method)
	org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
	java.lang.Thread.run(Thread.java:748)

It's been 15 minutes since I restarted the solr, and I believe already 999 threads have started?? And everytime I refresh the console, I'm seeing jump :

Connection evictor (1106)
java.lang.Thread.sleep(Native Method)
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
java.lang.Thread.run(Thread.java:748)

Thanks and Regards,
Srinivas Kashyap

-----Original Message-----
From: Erick Erickson <er...@gmail.com> 
Sent: 06 March 2020 21:34
To: solr-user@lucene.apache.org
Subject: Re: OutOfMemory error solr 8.4.1

I assume you recompiled the jar file? re-using the same one compiled against 5x is unsupported, nobody will be able to help until you recompile.

Once you’ve done that, if you still have the problem you need to take a thread dump to see if your custom code is leaking threads, that’s my number one suspect.

Best,
Erick

> On Mar 6, 2020, at 07:36, Srinivas Kashyap <sr...@bamboorose.com.invalid> wrote:
> 
> Hi Erick,
> 
> We have custom code which are schedulers to run delta imports on our cores and I have added that custom code as a jar and I have placed it on server/solr-webapp/WEB-INF/lib. Basically we are fetching the JNDI datasource configured in the jetty.xml(Oracle) and creating connection object. And after that in the finally block we are closing it too.
> 
> Never faced this issue while we were in solr5.2.1 version though. The same jar was placed there too.
> 
> Thanks,
> Srinivas
> 
> On 06-Mar-2020 8:55 pm, Erick Erickson <er...@gmail.com> wrote:
> This one can be a bit tricky. You’re not running out of overall memory, but you are running out of memory to allocate stacks. Which implies that, for some reason, you are creating a zillion threads. Do you have any custom code?
> 
> You can take a thread dump and see what your threads are doing, and you don’t need to wait until you see the error. If you take a thread dump my guess is you’ll see the number of threads increase over time. If that’s the case, and if you have no custom code running, we need to see the thread dump.
> 
> Best,
> Erick
> 
>> On Mar 6, 2020, at 05:54, Srinivas Kashyap <sr...@bamboorose.com.invalid> wrote:
>> 
>> Hi All,
>> 
>> I have recently upgraded solr to 8.4.1 and have installed solr as service in linux machine. Once I start my service, it will be up for 15-18hours and suddenly stops without us shutting down. In solr.log I found below error. Can somebody guide me what values should I be increasing in Linux machine?
>> 
>> Earlier, open file limit was not set and now I have increased. Below are my system configuration for solr:
>> 
>> JVM memory: 8GB
>> RAM: 32GB
>> Open file descriptor count: 500000
>> 
>> Ulimit -v - unlimited
>> Ulimit -m - unlimited
>> 
>> 
>> ERROR STACK TRACE:
>> 
>> 2020-03-06 12:08:03.071 ERROR (qtp1691185247-21) [   x:product] o.a.s.s.HttpSolrCall null:java.lang.RuntimeException: java.lang.OutOfMemoryError: unable to create new native thread
>>       at org.apache.solr.servlet.HttpSolrCall.sendError(HttpSolrCall.java:752)
>>       at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:603)
>>       at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:419)
>>       at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:351)
>>       at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602)
>>       at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
>>       at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
>>       at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
>>       at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
>>       at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
>>       at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1711)
>>       at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
>>       at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1347)
>>       at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
>>       at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
>>       at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1678)
>>       at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
>>       at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1249)
>>       at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
>>       at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220)
>>       at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:152)
>>       at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
>>       at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
>>       at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
>>       at org.eclipse.jetty.server.Server.handle(Server.java:505)
>>       at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:370)
>>       at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:267)
>>       at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
>>       at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
>>       at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
>>       at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
>>       at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
>>       at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
>>       at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
>>       at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
>>       at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:781)
>>       at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:917)
>>       at java.lang.Thread.run(Thread.java:748)
>> Caused by: java.lang.OutOfMemoryError: unable to create new native thread
>>       at java.lang.Thread.start0(Native Method)
>>       at java.lang.Thread.start(Thread.java:717)
>>       at org.apache.solr.handler.dataimport.DataImporter.runAsync(DataImporter.java:466)
>>       at org.apache.solr.handler.dataimport.DataImportHandler.handleRequestBody(DataImportHandler.java:205)
>>       at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:211)
>>       at org.apache.solr.core.SolrCore.execute(SolrCore.java:2596)
>>       at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:799)
>>       at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:578)
>>       ... 36 more
>> 
>> 
>> 
>> 
>> java.lang.OutOfMemoryError: unable to create new native thread
>>       at java.lang.Thread.start0(Native Method) ~[?:1.8.0_242]
>>       at java.lang.Thread.start(Thread.java:717) ~[?:1.8.0_242]
>>       at org.apache.solr.handler.dataimport.DataImporter.runAsync(DataImporter.java:466) ~[?:?]
>>       at org.apache.solr.handler.dataimport.DataImportHandler.handleRequestBody(DataImportHandler.java:205) ~[?:?]
>>       at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:211) ~[?:?]
>>       at org.apache.solr.core.SolrCore.execute(SolrCore.java:2596) ~[?:?]
>>       at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:799) ~[?:?]
>>       at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:578) ~[?:?]
>>       at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:419) ~[?:?]
>>       at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:351) ~[?:?]
>>       at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602) ~[jetty-servlet-9.4.19.v20190610.jar:9.4.19.v20190610]
>>       at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540) ~[jetty-servlet-9.4.19.v20190610.jar:9.4.19.v20190610]
>>       at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146) ~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
>>       at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) ~[jetty-security-9.4.19.v20190610.jar:9.4.19.v20190610]
>>       at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) ~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
>>       at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257) ~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
>>       at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1711) ~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
>>       at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255) ~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
>>       at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1347) ~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
>>       at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203) ~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
>>       at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480) ~[jetty-servlet-9.4.19.v20190610.jar:9.4.19.v20190610]
>>       at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1678) ~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
>>       at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201) ~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
>>       at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1249) ~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
>>       at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144) ~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
>>       at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220) ~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
>>       at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:152) ~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
>>       at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) ~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
>>       at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335) ~[jetty-rewrite-9.4.19.v20190610.jar:9.4.19.v20190610]
>>       at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) ~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
>>       at org.eclipse.jetty.server.Server.handle(Server.java:505) ~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
>>       at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:370) ~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
>>       at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:267) ~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
>>       at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305) ~[jetty-io-9.4.19.v20190610.jar:9.4.19.v20190610]
>>       at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103) ~[jetty-io-9.4.19.v20190610.jar:9.4.19.v20190610]
>>       at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117) ~[jetty-io-9.4.19.v20190610.jar:9.4.19.v20190610]
>>       at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333) ~[jetty-util-9.4.19.v20190610.jar:9.4.19.v20190610]
>>       at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310) ~[jetty-util-9.4.19.v20190610.jar:9.4.19.v20190610]
>>       at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) ~[jetty-util-9.4.19.v20190610.jar:9.4.19.v20190610]
>>       at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126) ~[jetty-util-9.4.19.v20190610.jar:9.4.19.v20190610]
>>       at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366) ~[jetty-util-9.4.19.v20190610.jar:9.4.19.v20190610]
>>       at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:781) ~[jetty-util-9.4.19.v20190610.jar:9.4.19.v20190610]
>>       at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:917) ~[jetty-util-9.4.19.v20190610.jar:9.4.19.v20190610]
>>       at java.lang.Thread.run(Thread.java:748) [?:1.8.0_242]
>> 
>> 
>> 
>> 
>> Thanks and Regards,
>> Srinivas Kashyap
>> 
>> ________________________________
>> DISCLAIMER:
>> E-mails and attachments from Bamboo Rose, LLC are confidential.
>> If you are not the intended recipient, please notify the sender immediately by replying to the e-mail, and then delete it without making copies or using it in any way.
>> No representation is made that this email or any attachments are free of viruses. Virus scanning is recommended and is the responsibility of the recipient.
>> 
>> Disclaimer
>> 
>> The information contained in this communication from the sender is confidential. It is intended solely for use by the recipient and others authorized to receive it. If you are not the recipient, you are hereby notified that any disclosure, copying, distribution or taking action in relation of the contents of this information is strictly prohibited and may be unlawful.
>> 
>> This email has been scanned for viruses and malware, and may have been automatically archived by Mimecast Ltd, an innovator in Software as a Service (SaaS) for business. Providing a safer and more useful place for your human generated data. Specializing in; Security, archiving and compliance. To find out more visit the Mimecast website.
>

Re: OutOfMemory error solr 8.4.1

Posted by Erick Erickson <er...@gmail.com>.

I assume you recompiled the jar file? re-using the same one compiled against 5x is unsupported, nobody will be able to help until you recompile.

Once you’ve done that, if you still have the problem you need to take a thread dump to see if your custom code is leaking threads, that’s my number one suspect.

Best,
Erick

> On Mar 6, 2020, at 07:36, Srinivas Kashyap <sr...@bamboorose.com.invalid> wrote:
> 
> Hi Erick,
> 
> We have custom code which are schedulers to run delta imports on our cores and I have added that custom code as a jar and I have placed it on server/solr-webapp/WEB-INF/lib. Basically we are fetching the JNDI datasource configured in the jetty.xml(Oracle) and creating connection object. And after that in the finally block we are closing it too.
> 
> Never faced this issue while we were in solr5.2.1 version though. The same jar was placed there too.
> 
> Thanks,
> Srinivas
> 
> On 06-Mar-2020 8:55 pm, Erick Erickson <er...@gmail.com> wrote:
> This one can be a bit tricky. You’re not running out of overall memory, but you are running out of memory to allocate stacks. Which implies that, for some reason, you are creating a zillion threads. Do you have any custom code?
> 
> You can take a thread dump and see what your threads are doing, and you don’t need to wait until you see the error. If you take a thread dump my guess is you’ll see the number of threads increase over time. If that’s the case, and if you have no custom code running, we need to see the thread dump.
> 
> Best,
> Erick
> 
>> On Mar 6, 2020, at 05:54, Srinivas Kashyap <sr...@bamboorose.com.invalid> wrote:
>> 
>> Hi All,
>> 
>> I have recently upgraded solr to 8.4.1 and have installed solr as service in linux machine. Once I start my service, it will be up for 15-18hours and suddenly stops without us shutting down. In solr.log I found below error. Can somebody guide me what values should I be increasing in Linux machine?
>> 
>> Earlier, open file limit was not set and now I have increased. Below are my system configuration for solr:
>> 
>> JVM memory: 8GB
>> RAM: 32GB
>> Open file descriptor count: 500000
>> 
>> Ulimit -v - unlimited
>> Ulimit -m - unlimited
>> 
>> 
>> ERROR STACK TRACE:
>> 
>> 2020-03-06 12:08:03.071 ERROR (qtp1691185247-21) [   x:product] o.a.s.s.HttpSolrCall null:java.lang.RuntimeException: java.lang.OutOfMemoryError: unable to create new native thread
>>       at org.apache.solr.servlet.HttpSolrCall.sendError(HttpSolrCall.java:752)
>>       at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:603)
>>       at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:419)
>>       at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:351)
>>       at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602)
>>       at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
>>       at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
>>       at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
>>       at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
>>       at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
>>       at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1711)
>>       at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
>>       at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1347)
>>       at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
>>       at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
>>       at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1678)
>>       at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
>>       at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1249)
>>       at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
>>       at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220)
>>       at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:152)
>>       at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
>>       at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
>>       at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
>>       at org.eclipse.jetty.server.Server.handle(Server.java:505)
>>       at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:370)
>>       at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:267)
>>       at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
>>       at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
>>       at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
>>       at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
>>       at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
>>       at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
>>       at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
>>       at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
>>       at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:781)
>>       at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:917)
>>       at java.lang.Thread.run(Thread.java:748)
>> Caused by: java.lang.OutOfMemoryError: unable to create new native thread
>>       at java.lang.Thread.start0(Native Method)
>>       at java.lang.Thread.start(Thread.java:717)
>>       at org.apache.solr.handler.dataimport.DataImporter.runAsync(DataImporter.java:466)
>>       at org.apache.solr.handler.dataimport.DataImportHandler.handleRequestBody(DataImportHandler.java:205)
>>       at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:211)
>>       at org.apache.solr.core.SolrCore.execute(SolrCore.java:2596)
>>       at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:799)
>>       at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:578)
>>       ... 36 more
>> 
>> 
>> 
>> 
>> java.lang.OutOfMemoryError: unable to create new native thread
>>       at java.lang.Thread.start0(Native Method) ~[?:1.8.0_242]
>>       at java.lang.Thread.start(Thread.java:717) ~[?:1.8.0_242]
>>       at org.apache.solr.handler.dataimport.DataImporter.runAsync(DataImporter.java:466) ~[?:?]
>>       at org.apache.solr.handler.dataimport.DataImportHandler.handleRequestBody(DataImportHandler.java:205) ~[?:?]
>>       at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:211) ~[?:?]
>>       at org.apache.solr.core.SolrCore.execute(SolrCore.java:2596) ~[?:?]
>>       at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:799) ~[?:?]
>>       at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:578) ~[?:?]
>>       at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:419) ~[?:?]
>>       at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:351) ~[?:?]
>>       at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602) ~[jetty-servlet-9.4.19.v20190610.jar:9.4.19.v20190610]
>>       at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540) ~[jetty-servlet-9.4.19.v20190610.jar:9.4.19.v20190610]
>>       at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146) ~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
>>       at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) ~[jetty-security-9.4.19.v20190610.jar:9.4.19.v20190610]
>>       at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) ~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
>>       at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257) ~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
>>       at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1711) ~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
>>       at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255) ~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
>>       at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1347) ~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
>>       at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203) ~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
>>       at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480) ~[jetty-servlet-9.4.19.v20190610.jar:9.4.19.v20190610]
>>       at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1678) ~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
>>       at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201) ~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
>>       at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1249) ~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
>>       at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144) ~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
>>       at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220) ~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
>>       at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:152) ~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
>>       at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) ~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
>>       at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335) ~[jetty-rewrite-9.4.19.v20190610.jar:9.4.19.v20190610]
>>       at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) ~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
>>       at org.eclipse.jetty.server.Server.handle(Server.java:505) ~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
>>       at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:370) ~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
>>       at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:267) ~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
>>       at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305) ~[jetty-io-9.4.19.v20190610.jar:9.4.19.v20190610]
>>       at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103) ~[jetty-io-9.4.19.v20190610.jar:9.4.19.v20190610]
>>       at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117) ~[jetty-io-9.4.19.v20190610.jar:9.4.19.v20190610]
>>       at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333) ~[jetty-util-9.4.19.v20190610.jar:9.4.19.v20190610]
>>       at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310) ~[jetty-util-9.4.19.v20190610.jar:9.4.19.v20190610]
>>       at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) ~[jetty-util-9.4.19.v20190610.jar:9.4.19.v20190610]
>>       at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126) ~[jetty-util-9.4.19.v20190610.jar:9.4.19.v20190610]
>>       at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366) ~[jetty-util-9.4.19.v20190610.jar:9.4.19.v20190610]
>>       at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:781) ~[jetty-util-9.4.19.v20190610.jar:9.4.19.v20190610]
>>       at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:917) ~[jetty-util-9.4.19.v20190610.jar:9.4.19.v20190610]
>>       at java.lang.Thread.run(Thread.java:748) [?:1.8.0_242]
>> 
>> 
>> 
>> 
>> Thanks and Regards,
>> Srinivas Kashyap
>> 
>> ________________________________
>> DISCLAIMER:
>> E-mails and attachments from Bamboo Rose, LLC are confidential.
>> If you are not the intended recipient, please notify the sender immediately by replying to the e-mail, and then delete it without making copies or using it in any way.
>> No representation is made that this email or any attachments are free of viruses. Virus scanning is recommended and is the responsibility of the recipient.
>> 
>> Disclaimer
>> 
>> The information contained in this communication from the sender is confidential. It is intended solely for use by the recipient and others authorized to receive it. If you are not the recipient, you are hereby notified that any disclosure, copying, distribution or taking action in relation of the contents of this information is strictly prohibited and may be unlawful.
>> 
>> This email has been scanned for viruses and malware, and may have been automatically archived by Mimecast Ltd, an innovator in Software as a Service (SaaS) for business. Providing a safer and more useful place for your human generated data. Specializing in; Security, archiving and compliance. To find out more visit the Mimecast website.
>

Re: OutOfMemory error solr 8.4.1

Posted by Srinivas Kashyap <sr...@bamboorose.com.INVALID>.

Hi Erick,

We have custom code which are schedulers to run delta imports on our cores and I have added that custom code as a jar and I have placed it on server/solr-webapp/WEB-INF/lib. Basically we are fetching the JNDI datasource configured in the jetty.xml(Oracle) and creating connection object. And after that in the finally block we are closing it too.

Never faced this issue while we were in solr5.2.1 version though. The same jar was placed there too.

Thanks,
Srinivas

On 06-Mar-2020 8:55 pm, Erick Erickson <er...@gmail.com> wrote:
This one can be a bit tricky. You’re not running out of overall memory, but you are running out of memory to allocate stacks. Which implies that, for some reason, you are creating a zillion threads. Do you have any custom code?

You can take a thread dump and see what your threads are doing, and you don’t need to wait until you see the error. If you take a thread dump my guess is you’ll see the number of threads increase over time. If that’s the case, and if you have no custom code running, we need to see the thread dump.

Best,
Erick

> On Mar 6, 2020, at 05:54, Srinivas Kashyap <sr...@bamboorose.com.invalid> wrote:
>
> Hi All,
>
> I have recently upgraded solr to 8.4.1 and have installed solr as service in linux machine. Once I start my service, it will be up for 15-18hours and suddenly stops without us shutting down. In solr.log I found below error. Can somebody guide me what values should I be increasing in Linux machine?
>
> Earlier, open file limit was not set and now I have increased. Below are my system configuration for solr:
>
> JVM memory: 8GB
> RAM: 32GB
> Open file descriptor count: 500000
>
> Ulimit -v - unlimited
> Ulimit -m - unlimited
>
>
> ERROR STACK TRACE:
>
> 2020-03-06 12:08:03.071 ERROR (qtp1691185247-21) [   x:product] o.a.s.s.HttpSolrCall null:java.lang.RuntimeException: java.lang.OutOfMemoryError: unable to create new native thread
>        at org.apache.solr.servlet.HttpSolrCall.sendError(HttpSolrCall.java:752)
>        at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:603)
>        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:419)
>        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:351)
>        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602)
>        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
>        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
>        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
>        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
>        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
>        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1711)
>        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
>        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1347)
>        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
>        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
>        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1678)
>        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
>        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1249)
>        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
>        at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220)
>        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:152)
>        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
>        at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
>        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
>        at org.eclipse.jetty.server.Server.handle(Server.java:505)
>        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:370)
>        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:267)
>        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
>        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
>        at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
>        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
>        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
>        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
>        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
>        at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
>        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:781)
>        at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:917)
>        at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.OutOfMemoryError: unable to create new native thread
>        at java.lang.Thread.start0(Native Method)
>        at java.lang.Thread.start(Thread.java:717)
>        at org.apache.solr.handler.dataimport.DataImporter.runAsync(DataImporter.java:466)
>        at org.apache.solr.handler.dataimport.DataImportHandler.handleRequestBody(DataImportHandler.java:205)
>        at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:211)
>        at org.apache.solr.core.SolrCore.execute(SolrCore.java:2596)
>        at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:799)
>        at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:578)
>        ... 36 more
>
>
>
>
> java.lang.OutOfMemoryError: unable to create new native thread
>        at java.lang.Thread.start0(Native Method) ~[?:1.8.0_242]
>        at java.lang.Thread.start(Thread.java:717) ~[?:1.8.0_242]
>        at org.apache.solr.handler.dataimport.DataImporter.runAsync(DataImporter.java:466) ~[?:?]
>        at org.apache.solr.handler.dataimport.DataImportHandler.handleRequestBody(DataImportHandler.java:205) ~[?:?]
>        at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:211) ~[?:?]
>        at org.apache.solr.core.SolrCore.execute(SolrCore.java:2596) ~[?:?]
>        at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:799) ~[?:?]
>        at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:578) ~[?:?]
>        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:419) ~[?:?]
>        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:351) ~[?:?]
>        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602) ~[jetty-servlet-9.4.19.v20190610.jar:9.4.19.v20190610]
>        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540) ~[jetty-servlet-9.4.19.v20190610.jar:9.4.19.v20190610]
>        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146) ~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
>        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) ~[jetty-security-9.4.19.v20190610.jar:9.4.19.v20190610]
>        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) ~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
>        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257) ~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
>        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1711) ~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
>        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255) ~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
>        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1347) ~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
>        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203) ~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
>        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480) ~[jetty-servlet-9.4.19.v20190610.jar:9.4.19.v20190610]
>        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1678) ~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
>        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201) ~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
>        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1249) ~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
>        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144) ~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
>        at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220) ~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
>        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:152) ~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
>        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) ~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
>        at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335) ~[jetty-rewrite-9.4.19.v20190610.jar:9.4.19.v20190610]
>        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) ~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
>        at org.eclipse.jetty.server.Server.handle(Server.java:505) ~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
>        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:370) ~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
>        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:267) ~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
>        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305) ~[jetty-io-9.4.19.v20190610.jar:9.4.19.v20190610]
>        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103) ~[jetty-io-9.4.19.v20190610.jar:9.4.19.v20190610]
>        at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117) ~[jetty-io-9.4.19.v20190610.jar:9.4.19.v20190610]
>        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333) ~[jetty-util-9.4.19.v20190610.jar:9.4.19.v20190610]
>        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310) ~[jetty-util-9.4.19.v20190610.jar:9.4.19.v20190610]
>        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) ~[jetty-util-9.4.19.v20190610.jar:9.4.19.v20190610]
>        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126) ~[jetty-util-9.4.19.v20190610.jar:9.4.19.v20190610]
>        at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366) ~[jetty-util-9.4.19.v20190610.jar:9.4.19.v20190610]
>        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:781) ~[jetty-util-9.4.19.v20190610.jar:9.4.19.v20190610]
>        at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:917) ~[jetty-util-9.4.19.v20190610.jar:9.4.19.v20190610]
>        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_242]
>
>
>
>
> Thanks and Regards,
> Srinivas Kashyap
>
> ________________________________
> DISCLAIMER:
> E-mails and attachments from Bamboo Rose, LLC are confidential.
> If you are not the intended recipient, please notify the sender immediately by replying to the e-mail, and then delete it without making copies or using it in any way.
> No representation is made that this email or any attachments are free of viruses. Virus scanning is recommended and is the responsibility of the recipient.
>
> Disclaimer
>
> The information contained in this communication from the sender is confidential. It is intended solely for use by the recipient and others authorized to receive it. If you are not the recipient, you are hereby notified that any disclosure, copying, distribution or taking action in relation of the contents of this information is strictly prohibited and may be unlawful.
>
> This email has been scanned for viruses and malware, and may have been automatically archived by Mimecast Ltd, an innovator in Software as a Service (SaaS) for business. Providing a safer and more useful place for your human generated data. Specializing in; Security, archiving and compliance. To find out more visit the Mimecast website.

Re: OutOfMemory error solr 8.4.1

Posted by Erick Erickson <er...@gmail.com>.

This one can be a bit tricky. You’re not running out of overall memory, but you are running out of memory to allocate stacks. Which implies that, for some reason, you are creating a zillion threads. Do you have any custom code?

You can take a thread dump and see what your threads are doing, and you don’t need to wait until you see the error. If you take a thread dump my guess is you’ll see the number of threads increase over time. If that’s the case, and if you have no custom code running, we need to see the thread dump.

Best,
Erick

> On Mar 6, 2020, at 05:54, Srinivas Kashyap <sr...@bamboorose.com.invalid> wrote:
> 
> Hi All,
> 
> I have recently upgraded solr to 8.4.1 and have installed solr as service in linux machine. Once I start my service, it will be up for 15-18hours and suddenly stops without us shutting down. In solr.log I found below error. Can somebody guide me what values should I be increasing in Linux machine?
> 
> Earlier, open file limit was not set and now I have increased. Below are my system configuration for solr:
> 
> JVM memory: 8GB
> RAM: 32GB
> Open file descriptor count: 500000
> 
> Ulimit -v - unlimited
> Ulimit -m - unlimited
> 
> 
> ERROR STACK TRACE:
> 
> 2020-03-06 12:08:03.071 ERROR (qtp1691185247-21) [   x:product] o.a.s.s.HttpSolrCall null:java.lang.RuntimeException: java.lang.OutOfMemoryError: unable to create new native thread
>        at org.apache.solr.servlet.HttpSolrCall.sendError(HttpSolrCall.java:752)
>        at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:603)
>        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:419)
>        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:351)
>        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602)
>        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
>        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
>        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
>        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
>        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
>        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1711)
>        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
>        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1347)
>        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
>        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
>        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1678)
>        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
>        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1249)
>        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
>        at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220)
>        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:152)
>        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
>        at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
>        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
>        at org.eclipse.jetty.server.Server.handle(Server.java:505)
>        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:370)
>        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:267)
>        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
>        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
>        at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
>        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
>        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
>        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
>        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
>        at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
>        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:781)
>        at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:917)
>        at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.OutOfMemoryError: unable to create new native thread
>        at java.lang.Thread.start0(Native Method)
>        at java.lang.Thread.start(Thread.java:717)
>        at org.apache.solr.handler.dataimport.DataImporter.runAsync(DataImporter.java:466)
>        at org.apache.solr.handler.dataimport.DataImportHandler.handleRequestBody(DataImportHandler.java:205)
>        at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:211)
>        at org.apache.solr.core.SolrCore.execute(SolrCore.java:2596)
>        at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:799)
>        at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:578)
>        ... 36 more
> 
> 
> 
> 
> java.lang.OutOfMemoryError: unable to create new native thread
>        at java.lang.Thread.start0(Native Method) ~[?:1.8.0_242]
>        at java.lang.Thread.start(Thread.java:717) ~[?:1.8.0_242]
>        at org.apache.solr.handler.dataimport.DataImporter.runAsync(DataImporter.java:466) ~[?:?]
>        at org.apache.solr.handler.dataimport.DataImportHandler.handleRequestBody(DataImportHandler.java:205) ~[?:?]
>        at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:211) ~[?:?]
>        at org.apache.solr.core.SolrCore.execute(SolrCore.java:2596) ~[?:?]
>        at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:799) ~[?:?]
>        at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:578) ~[?:?]
>        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:419) ~[?:?]
>        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:351) ~[?:?]
>        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602) ~[jetty-servlet-9.4.19.v20190610.jar:9.4.19.v20190610]
>        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540) ~[jetty-servlet-9.4.19.v20190610.jar:9.4.19.v20190610]
>        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146) ~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
>        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) ~[jetty-security-9.4.19.v20190610.jar:9.4.19.v20190610]
>        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) ~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
>        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257) ~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
>        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1711) ~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
>        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255) ~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
>        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1347) ~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
>        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203) ~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
>        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480) ~[jetty-servlet-9.4.19.v20190610.jar:9.4.19.v20190610]
>        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1678) ~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
>        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201) ~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
>        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1249) ~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
>        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144) ~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
>        at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220) ~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
>        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:152) ~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
>        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) ~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
>        at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335) ~[jetty-rewrite-9.4.19.v20190610.jar:9.4.19.v20190610]
>        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) ~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
>        at org.eclipse.jetty.server.Server.handle(Server.java:505) ~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
>        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:370) ~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
>        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:267) ~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
>        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305) ~[jetty-io-9.4.19.v20190610.jar:9.4.19.v20190610]
>        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103) ~[jetty-io-9.4.19.v20190610.jar:9.4.19.v20190610]
>        at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117) ~[jetty-io-9.4.19.v20190610.jar:9.4.19.v20190610]
>        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333) ~[jetty-util-9.4.19.v20190610.jar:9.4.19.v20190610]
>        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310) ~[jetty-util-9.4.19.v20190610.jar:9.4.19.v20190610]
>        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) ~[jetty-util-9.4.19.v20190610.jar:9.4.19.v20190610]
>        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126) ~[jetty-util-9.4.19.v20190610.jar:9.4.19.v20190610]
>        at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366) ~[jetty-util-9.4.19.v20190610.jar:9.4.19.v20190610]
>        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:781) ~[jetty-util-9.4.19.v20190610.jar:9.4.19.v20190610]
>        at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:917) ~[jetty-util-9.4.19.v20190610.jar:9.4.19.v20190610]
>        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_242]
> 
> 
> 
> 
> Thanks and Regards,
> Srinivas Kashyap
> 
> ________________________________
> DISCLAIMER:
> E-mails and attachments from Bamboo Rose, LLC are confidential.
> If you are not the intended recipient, please notify the sender immediately by replying to the e-mail, and then delete it without making copies or using it in any way.
> No representation is made that this email or any attachments are free of viruses. Virus scanning is recommended and is the responsibility of the recipient.
> 
> Disclaimer
> 
> The information contained in this communication from the sender is confidential. It is intended solely for use by the recipient and others authorized to receive it. If you are not the recipient, you are hereby notified that any disclosure, copying, distribution or taking action in relation of the contents of this information is strictly prohibited and may be unlawful.
> 
> This email has been scanned for viruses and malware, and may have been automatically archived by Mimecast Ltd, an innovator in Software as a Service (SaaS) for business. Providing a safer and more useful place for your human generated data. Specializing in; Security, archiving and compliance. To find out more visit the Mimecast website.