You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@solr.apache.org by Henrik Brautaset Aronsen <he...@gmail.com> on 2022/05/16 07:23:06 UTC
SolrServerException: Max requests queued per destination 3000 exceeded for HttpDestination
Hi.
We keep getting "SolrServerException: Max requests queued per destination
3000 exceeded for HttpDestination" for several different collections and
instances in our SolrCloud dev environment. The instances don't reply to
requests and need to be restarted to function again.
We're not getting those errors in the prod environment even though those
are set up similarly.
Any ideas what could cause this?
We're on Solr 8.11.1 with connTimeout=60000, socketTimeout=60000,
zkClientTimeout=30000, solr.jetty.http.idleTimeout=60000. Full stacktrace
below. Thanks for any help!
Cheers,
Henrik
java.util.concurrent.RejectedExecutionException: Max requests queued per
destination 3000 exceeded for
HttpDestination[http://dev-foo3672.bar.no:12621
]@74097d87,queue=3000,pool=MultiplexConnectionPool@310bfd62
[c=0/4/4,a=0,i=0,q=3000]
at org.eclipse.jetty.client.HttpDestination.send(HttpDestination.java:289)
~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927]
at org.eclipse.jetty.client.HttpDestination.send(HttpDestination.java:262)
~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927]
at org.eclipse.jetty.client.HttpClient.send(HttpClient.java:600)
~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927]
at org.eclipse.jetty.client.HttpRequest.sendAsync(HttpRequest.java:778)
~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927]
at org.eclipse.jetty.client.HttpRequest.send(HttpRequest.java:765)
~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927]
at
org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:409)
~[solr-solrj-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 -
janhoy - 2021-12-14 13:50:57]
at org.apache.solr.client.solrj.SolrServerException: Max requests queued
per destination 3000 exceeded for
HttpDestination[http://dev-foo3672.bar.no:12621
]@74097d87,queue=3000,pool=MultiplexConnectionPool@310bfd62
[c=0/4/4,a=0,i=0,q=3000]
at
org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:439)
~[solr-solrj-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 -
janhoy - 2021-12-14 13:50:57]
at
org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:776)
~[solr-solrj-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 -
janhoy - 2021-12-14 13:50:57]
at
org.apache.solr.client.solrj.impl.ConcurrentUpdateHttp2SolrClient.request(ConcurrentUpdateHttp2SolrClient.java:370)
~[solr-solrj-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 -
janhoy - 2021-12-14 13:50:57]
at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1290)
~[solr-solrj-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 -
janhoy - 2021-12-14 13:50:57]
at
org.apache.solr.update.SolrCmdDistributor.doRequest(SolrCmdDistributor.java:345)
~[solr-core-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 -
janhoy - 2021-12-14 13:50:55]
at
org.apache.solr.update.SolrCmdDistributor.lambda$submit$0(SolrCmdDistributor.java:334)
~[solr-core-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 -
janhoy - 2021-12-14 13:50:55]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
~[?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
at
com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:180)
~[metrics-core-4.1.5.jar:4.1.5]
at
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:218)
~[solr-solrj-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 -
janhoy - 2021-12-14 13:50:57]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
[?:?]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
[?:?]
at java.lang.Thread.run(Thread.java:833) [?:?] Caused by:
java.util.concurrent.RejectedExecutionException: Max requests queued per
destination 3000 exceeded for
HttpDestination[http://dev-foo3672.bar.no:12621
]@74097d87,queue=3000,pool=MultiplexConnectionPool@310bfd62
[c=0/4/4,a=0,i=0,q=3000]
at org.eclipse.jetty.client.HttpDestination.send(HttpDestination.java:289)
~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927]
at org.eclipse.jetty.client.HttpDestination.send(HttpDestination.java:262)
~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927]
at org.eclipse.jetty.client.HttpClient.send(HttpClient.java:600)
~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927]
at org.eclipse.jetty.client.HttpRequest.sendAsync(HttpRequest.java:778)
~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927]
at org.eclipse.jetty.client.HttpRequest.send(HttpRequest.java:765)
~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927]
at
org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:409)
~[solr-solrj-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 -
janhoy - 2021-12-14 13:50:57]
Re: SolrServerException: Max requests queued per destination 3000 exceeded for HttpDestination
Posted by Henrik Brautaset Aronsen <he...@gmail.com>.
Interesting! So a workaround would be to just switch back to http1? That
would totally be an acceptable solution for us. I guess the http1 transport
won't disappear any time soon?
Finding the real root cause would be nice too, of course :)
Henrik
On Wed, May 25, 2022, 18:39 Tomás Fernández Löbbe <to...@gmail.com>
wrote:
> I have seen the same exception raised with search requests too. Not related
> to traffic, since I've seen it in very low query rate cases. Also, not
> related to resource utilization, since I've seen it in very under-utilized
> clusters. An interesting part of the error is this:
>
> > MultiplexConnectionPool@310bfd62
> [c=0/4/4,a=0,i=0,q=3000]
>
> This is the toString() of the MultiplexConnectionPool[1]:
>
> public String toString()
> {
> return String.format("%s@%x[c=%d/%d/%d,a=%d,i=%d,q=%d]",
> getClass().getSimpleName(),
> hashCode(),
> getPendingConnectionCount(),
> getConnectionCount(),
> getMaxConnectionCount(),
> getActiveConnectionCount(),
> getIdleConnectionCount(),
> destination.getQueuedRequestCount());
> }
>
> Meaning, there are 0 pending connections, 4 connections out of a max of 4,
> but then 0 active, and 0 idle. I don't know what state the other
> connections are in. I didn't have time to dive into this further, so I
> ended up switching back to http1. I'm not sure if this is something we are
> doing wrong in Solr or if it's a bug with the Jetty client.
>
> In my case, the client would not recover from this situation until it was
> restarted.
>
> Tomás
>
> [1]
>
> https://github.com/eclipse/jetty.project/blob/d262ef0d62f65f5e0f5f61462617e1a4562ea566/jetty-client/src/main/java/org/eclipse/jetty/client/AbstractConnectionPool.java#L517-L528
>
> On Mon, May 23, 2022 at 1:36 PM Jan Høydahl <ja...@cominvent.com> wrote:
>
> > Hi,
> >
> > I suspect that you have multiple shards, and that UPDATE requests in your
> > cluster expereience errors, causing
> > DistributedUpdateProcessor / ConcurrentUpdateHttp2SolrClient to auto
> retry
> > the request.
> > According to
> >
> https://github.com/apache/solr/blob/main/solr/core/src/java/org/apache/solr/update/processor/DistributedUpdateProcessor.java#L89
> > <
> >
> https://github.com/apache/solr/blob/main/solr/core/src/java/org/apache/solr/update/processor/DistributedUpdateProcessor.java#L89
> >
> >
> > there may be up to 25 retries on forward to leader.
> >
> > I'm just guessing here, that for some reason the indexing operation just
> > gets overloaded (perhaps due to low RAM or other factors)
> > and errors then propell out of control due to retries? 1Gb is not a huge
> > heap for heavy indexing.
> > - How fast are you indexing?
> > - Do you have any kind of indexing back-off/throttling strategy?
> >
> > Perhaps circuit breakers <
> >
> https://solr.apache.org/guide/solr/latest/deployment-guide/circuit-breakers.html
> >
> > and/or rate limiters <
> >
> https://solr.apache.org/guide/solr/latest/deployment-guide/rate-limiters.html
> >
> > could be considered to help throttle update requests? I have not tried
> them
> > myself.. Rate limiters may be a 9.0 only feature.
> >
> > Jan
> >
> >
> > > 23. mai 2022 kl. 13:40 skrev Henrik Brautaset Aronsen <
> > henrik.aronsen@gmail.com>:
> > >
> > > Thanks!
> > >
> > > I'll try doubling the JVM memory and see what happens. The dev
> instances
> > > only had "-Xms1024M -Xmx1024M -Xss256k" set.
> > >
> > > Cheers,
> > > Henrik
> > >
> > > On Mon, May 23, 2022 at 9:56 AM Deepak Goel <de...@gmail.com> wrote:
> > >
> > >> Looks like the memory is getting maxed out. However there must be some
> > way
> > >> to monitor the dev environment and confirm the same (something like a
> > top
> > >> command).
> > >>
> > >> Deepak
> > >> "The greatness of a nation can be judged by the way its animals are
> > >> treated - Mahatma Gandhi"
> > >>
> > >> +91 73500 12833
> > >> deicool@gmail.com
> > >>
> > >> Facebook: https://www.facebook.com/deicool
> > >> LinkedIn: www.linkedin.com/in/deicool
> > >>
> > >> "Plant a Tree, Go Green"
> > >>
> > >> Make In India : http://www.makeinindia.com/home
> > >>
> > >>
> > >> On Mon, May 23, 2022 at 1:20 PM Henrik Brautaset Aronsen <
> > >> henrik.aronsen@gmail.com> wrote:
> > >>
> > >>> Hi Deepak, thanks for reaching out!
> > >>>
> > >>> Prod has 8 servers with 8x Xeon CPU @ 2.00GHz and 64GB RAM
> > >>> Dev has 2 servers with 2x Xeon CPU @ 2.00GHz and 16GB RAM
> > >>>
> > >>> Is there a way to monitor the MultiplexConnectionPool?
> > >>>
> > >>> And is it CPU or RAM that is maxed out, you think?
> > >>>
> > >>> I really appreciate any input :)
> > >>>
> > >>> Cheers,
> > >>> Henrik
> > >>>
> > >>> On Mon, May 16, 2022 at 10:41 AM Deepak Goel <de...@gmail.com>
> > wrote:
> > >>>
> > >>>> Looks like *MultiplexConnectionPool *is getting maxed out.
> > >>>>
> > >>>> Could you please tell us the server configuration of dev and prod?
> > Looks
> > >>>> like the dev environment is getting maxed out.
> > >>>>
> > >>>> Deepak
> > >>>> "The greatness of a nation can be judged by the way its animals are
> > >>>> treated
> > >>>> - Mahatma Gandhi"
> > >>>>
> > >>>> +91 73500 12833
> > >>>> deicool@gmail.com
> > >>>>
> > >>>> Facebook: https://www.facebook.com/deicool
> > >>>> LinkedIn: www.linkedin.com/in/deicool
> > >>>>
> > >>>> "Plant a Tree, Go Green"
> > >>>>
> > >>>> Make In India : http://www.makeinindia.com/home
> > >>>>
> > >>>>
> > >>>> On Mon, May 16, 2022 at 12:53 PM Henrik Brautaset Aronsen <
> > >>>> henrik.aronsen@gmail.com> wrote:
> > >>>>
> > >>>>> Hi.
> > >>>>>
> > >>>>> We keep getting "SolrServerException: Max requests queued per
> > >>>> destination
> > >>>>> 3000 exceeded for HttpDestination" for several different
> collections
> > >>>> and
> > >>>>> instances in our SolrCloud dev environment. The instances don't
> reply
> > >>>> to
> > >>>>> requests and need to be restarted to function again.
> > >>>>>
> > >>>>> We're not getting those errors in the prod environment even though
> > >>>> those
> > >>>>> are set up similarly.
> > >>>>>
> > >>>>> Any ideas what could cause this?
> > >>>>>
> > >>>>> We're on Solr 8.11.1 with connTimeout=60000, socketTimeout=60000,
> > >>>>> zkClientTimeout=30000, solr.jetty.http.idleTimeout=60000. Full
> > >>>> stacktrace
> > >>>>> below. Thanks for any help!
> > >>>>>
> > >>>>> Cheers,
> > >>>>> Henrik
> > >>>>>
> > >>>>> java.util.concurrent.RejectedExecutionException: Max requests
> queued
> > >>>> per
> > >>>>> destination 3000 exceeded for
> > >>>>> HttpDestination[http://dev-foo3672.bar.no:12621
> > >>>>> ]@74097d87,queue=3000,pool=MultiplexConnectionPool@310bfd62
> > >>>>> [c=0/4/4,a=0,i=0,q=3000]
> > >>>>> at
> > >>>>
> > org.eclipse.jetty.client.HttpDestination.send(HttpDestination.java:289)
> > >>>>> ~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927]
> > >>>>> at
> > >>>>
> > org.eclipse.jetty.client.HttpDestination.send(HttpDestination.java:262)
> > >>>>> ~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927]
> > >>>>> at org.eclipse.jetty.client.HttpClient.send(HttpClient.java:600)
> > >>>>> ~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927]
> > >>>>> at
> > >>>> org.eclipse.jetty.client.HttpRequest.sendAsync(HttpRequest.java:778)
> > >>>>> ~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927]
> > >>>>> at org.eclipse.jetty.client.HttpRequest.send(HttpRequest.java:765)
> > >>>>> ~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927]
> > >>>>> at
> > >>>>>
> > >>>>>
> > >>>>
> >
> org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:409)
> > >>>>> ~[solr-solrj-8.11.1.jar:8.11.1
> > >>>> 0b002b11819df70783e83ef36b42ed1223c14b50 -
> > >>>>> janhoy - 2021-12-14 13:50:57]
> > >>>>> at org.apache.solr.client.solrj.SolrServerException: Max requests
> > >>>> queued
> > >>>>> per destination 3000 exceeded for
> > >>>>> HttpDestination[http://dev-foo3672.bar.no:12621
> > >>>>> ]@74097d87,queue=3000,pool=MultiplexConnectionPool@310bfd62
> > >>>>> [c=0/4/4,a=0,i=0,q=3000]
> > >>>>> at
> > >>>>>
> > >>>>>
> > >>>>
> >
> org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:439)
> > >>>>> ~[solr-solrj-8.11.1.jar:8.11.1
> > >>>> 0b002b11819df70783e83ef36b42ed1223c14b50 -
> > >>>>> janhoy - 2021-12-14 13:50:57]
> > >>>>> at
> > >>>>>
> > >>>>>
> > >>>>
> >
> org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:776)
> > >>>>> ~[solr-solrj-8.11.1.jar:8.11.1
> > >>>> 0b002b11819df70783e83ef36b42ed1223c14b50 -
> > >>>>> janhoy - 2021-12-14 13:50:57]
> > >>>>> at
> > >>>>>
> > >>>>>
> > >>>>
> >
> org.apache.solr.client.solrj.impl.ConcurrentUpdateHttp2SolrClient.request(ConcurrentUpdateHttp2SolrClient.java:370)
> > >>>>> ~[solr-solrj-8.11.1.jar:8.11.1
> > >>>> 0b002b11819df70783e83ef36b42ed1223c14b50 -
> > >>>>> janhoy - 2021-12-14 13:50:57]
> > >>>>> at
> > >>>>
> org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1290)
> > >>>>> ~[solr-solrj-8.11.1.jar:8.11.1
> > >>>> 0b002b11819df70783e83ef36b42ed1223c14b50 -
> > >>>>> janhoy - 2021-12-14 13:50:57]
> > >>>>> at
> > >>>>>
> > >>>>>
> > >>>>
> >
> org.apache.solr.update.SolrCmdDistributor.doRequest(SolrCmdDistributor.java:345)
> > >>>>> ~[solr-core-8.11.1.jar:8.11.1
> > 0b002b11819df70783e83ef36b42ed1223c14b50
> > >>>> -
> > >>>>> janhoy - 2021-12-14 13:50:55]
> > >>>>> at
> > >>>>>
> > >>>>>
> > >>>>
> >
> org.apache.solr.update.SolrCmdDistributor.lambda$submit$0(SolrCmdDistributor.java:334)
> > >>>>> ~[solr-core-8.11.1.jar:8.11.1
> > 0b002b11819df70783e83ef36b42ed1223c14b50
> > >>>> -
> > >>>>> janhoy - 2021-12-14 13:50:55]
> > >>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
> > >>>>> at
> > >>>>
> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
> > >>>>> ~[?:?]
> > >>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
> > >>>>> at
> > >>>>>
> > >>>>>
> > >>>>
> >
> com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:180)
> > >>>>> ~[metrics-core-4.1.5.jar:4.1.5]
> > >>>>> at
> > >>>>>
> > >>>>>
> > >>>>
> >
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:218)
> > >>>>> ~[solr-solrj-8.11.1.jar:8.11.1
> > >>>> 0b002b11819df70783e83ef36b42ed1223c14b50 -
> > >>>>> janhoy - 2021-12-14 13:50:57]
> > >>>>> at
> > >>>>>
> > >>>>>
> > >>>>
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
> > >>>>> [?:?]
> > >>>>> at
> > >>>>>
> > >>>>>
> > >>>>
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
> > >>>>> [?:?]
> > >>>>> at java.lang.Thread.run(Thread.java:833) [?:?] Caused by:
> > >>>>> java.util.concurrent.RejectedExecutionException: Max requests
> queued
> > >>>> per
> > >>>>> destination 3000 exceeded for
> > >>>>> HttpDestination[http://dev-foo3672.bar.no:12621
> > >>>>> ]@74097d87,queue=3000,pool=MultiplexConnectionPool@310bfd62
> > >>>>> [c=0/4/4,a=0,i=0,q=3000]
> > >>>>> at
> > >>>>
> > org.eclipse.jetty.client.HttpDestination.send(HttpDestination.java:289)
> > >>>>> ~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927]
> > >>>>> at
> > >>>>
> > org.eclipse.jetty.client.HttpDestination.send(HttpDestination.java:262)
> > >>>>> ~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927]
> > >>>>> at org.eclipse.jetty.client.HttpClient.send(HttpClient.java:600)
> > >>>>> ~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927]
> > >>>>> at
> > >>>> org.eclipse.jetty.client.HttpRequest.sendAsync(HttpRequest.java:778)
> > >>>>> ~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927]
> > >>>>> at org.eclipse.jetty.client.HttpRequest.send(HttpRequest.java:765)
> > >>>>> ~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927]
> > >>>>> at
> > >>>>>
> > >>>>>
> > >>>>
> >
> org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:409)
> > >>>>> ~[solr-solrj-8.11.1.jar:8.11.1
> > >>>> 0b002b11819df70783e83ef36b42ed1223c14b50 -
> > >>>>> janhoy - 2021-12-14 13:50:57]
> > >>>>>
> > >>>>
> > >>>
> >
> >
>
Re: SolrServerException: Max requests queued per destination 3000 exceeded for HttpDestination
Posted by Tomás Fernández Löbbe <to...@gmail.com>.
I have seen the same exception raised with search requests too. Not related
to traffic, since I've seen it in very low query rate cases. Also, not
related to resource utilization, since I've seen it in very under-utilized
clusters. An interesting part of the error is this:
> MultiplexConnectionPool@310bfd62
[c=0/4/4,a=0,i=0,q=3000]
This is the toString() of the MultiplexConnectionPool[1]:
public String toString()
{
return String.format("%s@%x[c=%d/%d/%d,a=%d,i=%d,q=%d]",
getClass().getSimpleName(),
hashCode(),
getPendingConnectionCount(),
getConnectionCount(),
getMaxConnectionCount(),
getActiveConnectionCount(),
getIdleConnectionCount(),
destination.getQueuedRequestCount());
}
Meaning, there are 0 pending connections, 4 connections out of a max of 4,
but then 0 active, and 0 idle. I don't know what state the other
connections are in. I didn't have time to dive into this further, so I
ended up switching back to http1. I'm not sure if this is something we are
doing wrong in Solr or if it's a bug with the Jetty client.
In my case, the client would not recover from this situation until it was
restarted.
Tomás
[1]
https://github.com/eclipse/jetty.project/blob/d262ef0d62f65f5e0f5f61462617e1a4562ea566/jetty-client/src/main/java/org/eclipse/jetty/client/AbstractConnectionPool.java#L517-L528
On Mon, May 23, 2022 at 1:36 PM Jan Høydahl <ja...@cominvent.com> wrote:
> Hi,
>
> I suspect that you have multiple shards, and that UPDATE requests in your
> cluster expereience errors, causing
> DistributedUpdateProcessor / ConcurrentUpdateHttp2SolrClient to auto retry
> the request.
> According to
> https://github.com/apache/solr/blob/main/solr/core/src/java/org/apache/solr/update/processor/DistributedUpdateProcessor.java#L89
> <
> https://github.com/apache/solr/blob/main/solr/core/src/java/org/apache/solr/update/processor/DistributedUpdateProcessor.java#L89>
>
> there may be up to 25 retries on forward to leader.
>
> I'm just guessing here, that for some reason the indexing operation just
> gets overloaded (perhaps due to low RAM or other factors)
> and errors then propell out of control due to retries? 1Gb is not a huge
> heap for heavy indexing.
> - How fast are you indexing?
> - Do you have any kind of indexing back-off/throttling strategy?
>
> Perhaps circuit breakers <
> https://solr.apache.org/guide/solr/latest/deployment-guide/circuit-breakers.html>
> and/or rate limiters <
> https://solr.apache.org/guide/solr/latest/deployment-guide/rate-limiters.html>
> could be considered to help throttle update requests? I have not tried them
> myself.. Rate limiters may be a 9.0 only feature.
>
> Jan
>
>
> > 23. mai 2022 kl. 13:40 skrev Henrik Brautaset Aronsen <
> henrik.aronsen@gmail.com>:
> >
> > Thanks!
> >
> > I'll try doubling the JVM memory and see what happens. The dev instances
> > only had "-Xms1024M -Xmx1024M -Xss256k" set.
> >
> > Cheers,
> > Henrik
> >
> > On Mon, May 23, 2022 at 9:56 AM Deepak Goel <de...@gmail.com> wrote:
> >
> >> Looks like the memory is getting maxed out. However there must be some
> way
> >> to monitor the dev environment and confirm the same (something like a
> top
> >> command).
> >>
> >> Deepak
> >> "The greatness of a nation can be judged by the way its animals are
> >> treated - Mahatma Gandhi"
> >>
> >> +91 73500 12833
> >> deicool@gmail.com
> >>
> >> Facebook: https://www.facebook.com/deicool
> >> LinkedIn: www.linkedin.com/in/deicool
> >>
> >> "Plant a Tree, Go Green"
> >>
> >> Make In India : http://www.makeinindia.com/home
> >>
> >>
> >> On Mon, May 23, 2022 at 1:20 PM Henrik Brautaset Aronsen <
> >> henrik.aronsen@gmail.com> wrote:
> >>
> >>> Hi Deepak, thanks for reaching out!
> >>>
> >>> Prod has 8 servers with 8x Xeon CPU @ 2.00GHz and 64GB RAM
> >>> Dev has 2 servers with 2x Xeon CPU @ 2.00GHz and 16GB RAM
> >>>
> >>> Is there a way to monitor the MultiplexConnectionPool?
> >>>
> >>> And is it CPU or RAM that is maxed out, you think?
> >>>
> >>> I really appreciate any input :)
> >>>
> >>> Cheers,
> >>> Henrik
> >>>
> >>> On Mon, May 16, 2022 at 10:41 AM Deepak Goel <de...@gmail.com>
> wrote:
> >>>
> >>>> Looks like *MultiplexConnectionPool *is getting maxed out.
> >>>>
> >>>> Could you please tell us the server configuration of dev and prod?
> Looks
> >>>> like the dev environment is getting maxed out.
> >>>>
> >>>> Deepak
> >>>> "The greatness of a nation can be judged by the way its animals are
> >>>> treated
> >>>> - Mahatma Gandhi"
> >>>>
> >>>> +91 73500 12833
> >>>> deicool@gmail.com
> >>>>
> >>>> Facebook: https://www.facebook.com/deicool
> >>>> LinkedIn: www.linkedin.com/in/deicool
> >>>>
> >>>> "Plant a Tree, Go Green"
> >>>>
> >>>> Make In India : http://www.makeinindia.com/home
> >>>>
> >>>>
> >>>> On Mon, May 16, 2022 at 12:53 PM Henrik Brautaset Aronsen <
> >>>> henrik.aronsen@gmail.com> wrote:
> >>>>
> >>>>> Hi.
> >>>>>
> >>>>> We keep getting "SolrServerException: Max requests queued per
> >>>> destination
> >>>>> 3000 exceeded for HttpDestination" for several different collections
> >>>> and
> >>>>> instances in our SolrCloud dev environment. The instances don't reply
> >>>> to
> >>>>> requests and need to be restarted to function again.
> >>>>>
> >>>>> We're not getting those errors in the prod environment even though
> >>>> those
> >>>>> are set up similarly.
> >>>>>
> >>>>> Any ideas what could cause this?
> >>>>>
> >>>>> We're on Solr 8.11.1 with connTimeout=60000, socketTimeout=60000,
> >>>>> zkClientTimeout=30000, solr.jetty.http.idleTimeout=60000. Full
> >>>> stacktrace
> >>>>> below. Thanks for any help!
> >>>>>
> >>>>> Cheers,
> >>>>> Henrik
> >>>>>
> >>>>> java.util.concurrent.RejectedExecutionException: Max requests queued
> >>>> per
> >>>>> destination 3000 exceeded for
> >>>>> HttpDestination[http://dev-foo3672.bar.no:12621
> >>>>> ]@74097d87,queue=3000,pool=MultiplexConnectionPool@310bfd62
> >>>>> [c=0/4/4,a=0,i=0,q=3000]
> >>>>> at
> >>>>
> org.eclipse.jetty.client.HttpDestination.send(HttpDestination.java:289)
> >>>>> ~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927]
> >>>>> at
> >>>>
> org.eclipse.jetty.client.HttpDestination.send(HttpDestination.java:262)
> >>>>> ~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927]
> >>>>> at org.eclipse.jetty.client.HttpClient.send(HttpClient.java:600)
> >>>>> ~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927]
> >>>>> at
> >>>> org.eclipse.jetty.client.HttpRequest.sendAsync(HttpRequest.java:778)
> >>>>> ~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927]
> >>>>> at org.eclipse.jetty.client.HttpRequest.send(HttpRequest.java:765)
> >>>>> ~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927]
> >>>>> at
> >>>>>
> >>>>>
> >>>>
> org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:409)
> >>>>> ~[solr-solrj-8.11.1.jar:8.11.1
> >>>> 0b002b11819df70783e83ef36b42ed1223c14b50 -
> >>>>> janhoy - 2021-12-14 13:50:57]
> >>>>> at org.apache.solr.client.solrj.SolrServerException: Max requests
> >>>> queued
> >>>>> per destination 3000 exceeded for
> >>>>> HttpDestination[http://dev-foo3672.bar.no:12621
> >>>>> ]@74097d87,queue=3000,pool=MultiplexConnectionPool@310bfd62
> >>>>> [c=0/4/4,a=0,i=0,q=3000]
> >>>>> at
> >>>>>
> >>>>>
> >>>>
> org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:439)
> >>>>> ~[solr-solrj-8.11.1.jar:8.11.1
> >>>> 0b002b11819df70783e83ef36b42ed1223c14b50 -
> >>>>> janhoy - 2021-12-14 13:50:57]
> >>>>> at
> >>>>>
> >>>>>
> >>>>
> org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:776)
> >>>>> ~[solr-solrj-8.11.1.jar:8.11.1
> >>>> 0b002b11819df70783e83ef36b42ed1223c14b50 -
> >>>>> janhoy - 2021-12-14 13:50:57]
> >>>>> at
> >>>>>
> >>>>>
> >>>>
> org.apache.solr.client.solrj.impl.ConcurrentUpdateHttp2SolrClient.request(ConcurrentUpdateHttp2SolrClient.java:370)
> >>>>> ~[solr-solrj-8.11.1.jar:8.11.1
> >>>> 0b002b11819df70783e83ef36b42ed1223c14b50 -
> >>>>> janhoy - 2021-12-14 13:50:57]
> >>>>> at
> >>>> org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1290)
> >>>>> ~[solr-solrj-8.11.1.jar:8.11.1
> >>>> 0b002b11819df70783e83ef36b42ed1223c14b50 -
> >>>>> janhoy - 2021-12-14 13:50:57]
> >>>>> at
> >>>>>
> >>>>>
> >>>>
> org.apache.solr.update.SolrCmdDistributor.doRequest(SolrCmdDistributor.java:345)
> >>>>> ~[solr-core-8.11.1.jar:8.11.1
> 0b002b11819df70783e83ef36b42ed1223c14b50
> >>>> -
> >>>>> janhoy - 2021-12-14 13:50:55]
> >>>>> at
> >>>>>
> >>>>>
> >>>>
> org.apache.solr.update.SolrCmdDistributor.lambda$submit$0(SolrCmdDistributor.java:334)
> >>>>> ~[solr-core-8.11.1.jar:8.11.1
> 0b002b11819df70783e83ef36b42ed1223c14b50
> >>>> -
> >>>>> janhoy - 2021-12-14 13:50:55]
> >>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
> >>>>> at
> >>>>
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
> >>>>> ~[?:?]
> >>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
> >>>>> at
> >>>>>
> >>>>>
> >>>>
> com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:180)
> >>>>> ~[metrics-core-4.1.5.jar:4.1.5]
> >>>>> at
> >>>>>
> >>>>>
> >>>>
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:218)
> >>>>> ~[solr-solrj-8.11.1.jar:8.11.1
> >>>> 0b002b11819df70783e83ef36b42ed1223c14b50 -
> >>>>> janhoy - 2021-12-14 13:50:57]
> >>>>> at
> >>>>>
> >>>>>
> >>>>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
> >>>>> [?:?]
> >>>>> at
> >>>>>
> >>>>>
> >>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
> >>>>> [?:?]
> >>>>> at java.lang.Thread.run(Thread.java:833) [?:?] Caused by:
> >>>>> java.util.concurrent.RejectedExecutionException: Max requests queued
> >>>> per
> >>>>> destination 3000 exceeded for
> >>>>> HttpDestination[http://dev-foo3672.bar.no:12621
> >>>>> ]@74097d87,queue=3000,pool=MultiplexConnectionPool@310bfd62
> >>>>> [c=0/4/4,a=0,i=0,q=3000]
> >>>>> at
> >>>>
> org.eclipse.jetty.client.HttpDestination.send(HttpDestination.java:289)
> >>>>> ~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927]
> >>>>> at
> >>>>
> org.eclipse.jetty.client.HttpDestination.send(HttpDestination.java:262)
> >>>>> ~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927]
> >>>>> at org.eclipse.jetty.client.HttpClient.send(HttpClient.java:600)
> >>>>> ~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927]
> >>>>> at
> >>>> org.eclipse.jetty.client.HttpRequest.sendAsync(HttpRequest.java:778)
> >>>>> ~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927]
> >>>>> at org.eclipse.jetty.client.HttpRequest.send(HttpRequest.java:765)
> >>>>> ~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927]
> >>>>> at
> >>>>>
> >>>>>
> >>>>
> org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:409)
> >>>>> ~[solr-solrj-8.11.1.jar:8.11.1
> >>>> 0b002b11819df70783e83ef36b42ed1223c14b50 -
> >>>>> janhoy - 2021-12-14 13:50:57]
> >>>>>
> >>>>
> >>>
>
>
Re: SolrServerException: Max requests queued per destination 3000 exceeded for HttpDestination
Posted by Jan Høydahl <ja...@cominvent.com>.
Hi,
I suspect that you have multiple shards, and that UPDATE requests in your cluster expereience errors, causing
DistributedUpdateProcessor / ConcurrentUpdateHttp2SolrClient to auto retry the request.
According to https://github.com/apache/solr/blob/main/solr/core/src/java/org/apache/solr/update/processor/DistributedUpdateProcessor.java#L89 <https://github.com/apache/solr/blob/main/solr/core/src/java/org/apache/solr/update/processor/DistributedUpdateProcessor.java#L89>
there may be up to 25 retries on forward to leader.
I'm just guessing here, that for some reason the indexing operation just gets overloaded (perhaps due to low RAM or other factors)
and errors then propell out of control due to retries? 1Gb is not a huge heap for heavy indexing.
- How fast are you indexing?
- Do you have any kind of indexing back-off/throttling strategy?
Perhaps circuit breakers <https://solr.apache.org/guide/solr/latest/deployment-guide/circuit-breakers.html> and/or rate limiters <https://solr.apache.org/guide/solr/latest/deployment-guide/rate-limiters.html> could be considered to help throttle update requests? I have not tried them myself.. Rate limiters may be a 9.0 only feature.
Jan
> 23. mai 2022 kl. 13:40 skrev Henrik Brautaset Aronsen <he...@gmail.com>:
>
> Thanks!
>
> I'll try doubling the JVM memory and see what happens. The dev instances
> only had "-Xms1024M -Xmx1024M -Xss256k" set.
>
> Cheers,
> Henrik
>
> On Mon, May 23, 2022 at 9:56 AM Deepak Goel <de...@gmail.com> wrote:
>
>> Looks like the memory is getting maxed out. However there must be some way
>> to monitor the dev environment and confirm the same (something like a top
>> command).
>>
>> Deepak
>> "The greatness of a nation can be judged by the way its animals are
>> treated - Mahatma Gandhi"
>>
>> +91 73500 12833
>> deicool@gmail.com
>>
>> Facebook: https://www.facebook.com/deicool
>> LinkedIn: www.linkedin.com/in/deicool
>>
>> "Plant a Tree, Go Green"
>>
>> Make In India : http://www.makeinindia.com/home
>>
>>
>> On Mon, May 23, 2022 at 1:20 PM Henrik Brautaset Aronsen <
>> henrik.aronsen@gmail.com> wrote:
>>
>>> Hi Deepak, thanks for reaching out!
>>>
>>> Prod has 8 servers with 8x Xeon CPU @ 2.00GHz and 64GB RAM
>>> Dev has 2 servers with 2x Xeon CPU @ 2.00GHz and 16GB RAM
>>>
>>> Is there a way to monitor the MultiplexConnectionPool?
>>>
>>> And is it CPU or RAM that is maxed out, you think?
>>>
>>> I really appreciate any input :)
>>>
>>> Cheers,
>>> Henrik
>>>
>>> On Mon, May 16, 2022 at 10:41 AM Deepak Goel <de...@gmail.com> wrote:
>>>
>>>> Looks like *MultiplexConnectionPool *is getting maxed out.
>>>>
>>>> Could you please tell us the server configuration of dev and prod? Looks
>>>> like the dev environment is getting maxed out.
>>>>
>>>> Deepak
>>>> "The greatness of a nation can be judged by the way its animals are
>>>> treated
>>>> - Mahatma Gandhi"
>>>>
>>>> +91 73500 12833
>>>> deicool@gmail.com
>>>>
>>>> Facebook: https://www.facebook.com/deicool
>>>> LinkedIn: www.linkedin.com/in/deicool
>>>>
>>>> "Plant a Tree, Go Green"
>>>>
>>>> Make In India : http://www.makeinindia.com/home
>>>>
>>>>
>>>> On Mon, May 16, 2022 at 12:53 PM Henrik Brautaset Aronsen <
>>>> henrik.aronsen@gmail.com> wrote:
>>>>
>>>>> Hi.
>>>>>
>>>>> We keep getting "SolrServerException: Max requests queued per
>>>> destination
>>>>> 3000 exceeded for HttpDestination" for several different collections
>>>> and
>>>>> instances in our SolrCloud dev environment. The instances don't reply
>>>> to
>>>>> requests and need to be restarted to function again.
>>>>>
>>>>> We're not getting those errors in the prod environment even though
>>>> those
>>>>> are set up similarly.
>>>>>
>>>>> Any ideas what could cause this?
>>>>>
>>>>> We're on Solr 8.11.1 with connTimeout=60000, socketTimeout=60000,
>>>>> zkClientTimeout=30000, solr.jetty.http.idleTimeout=60000. Full
>>>> stacktrace
>>>>> below. Thanks for any help!
>>>>>
>>>>> Cheers,
>>>>> Henrik
>>>>>
>>>>> java.util.concurrent.RejectedExecutionException: Max requests queued
>>>> per
>>>>> destination 3000 exceeded for
>>>>> HttpDestination[http://dev-foo3672.bar.no:12621
>>>>> ]@74097d87,queue=3000,pool=MultiplexConnectionPool@310bfd62
>>>>> [c=0/4/4,a=0,i=0,q=3000]
>>>>> at
>>>> org.eclipse.jetty.client.HttpDestination.send(HttpDestination.java:289)
>>>>> ~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927]
>>>>> at
>>>> org.eclipse.jetty.client.HttpDestination.send(HttpDestination.java:262)
>>>>> ~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927]
>>>>> at org.eclipse.jetty.client.HttpClient.send(HttpClient.java:600)
>>>>> ~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927]
>>>>> at
>>>> org.eclipse.jetty.client.HttpRequest.sendAsync(HttpRequest.java:778)
>>>>> ~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927]
>>>>> at org.eclipse.jetty.client.HttpRequest.send(HttpRequest.java:765)
>>>>> ~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927]
>>>>> at
>>>>>
>>>>>
>>>> org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:409)
>>>>> ~[solr-solrj-8.11.1.jar:8.11.1
>>>> 0b002b11819df70783e83ef36b42ed1223c14b50 -
>>>>> janhoy - 2021-12-14 13:50:57]
>>>>> at org.apache.solr.client.solrj.SolrServerException: Max requests
>>>> queued
>>>>> per destination 3000 exceeded for
>>>>> HttpDestination[http://dev-foo3672.bar.no:12621
>>>>> ]@74097d87,queue=3000,pool=MultiplexConnectionPool@310bfd62
>>>>> [c=0/4/4,a=0,i=0,q=3000]
>>>>> at
>>>>>
>>>>>
>>>> org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:439)
>>>>> ~[solr-solrj-8.11.1.jar:8.11.1
>>>> 0b002b11819df70783e83ef36b42ed1223c14b50 -
>>>>> janhoy - 2021-12-14 13:50:57]
>>>>> at
>>>>>
>>>>>
>>>> org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:776)
>>>>> ~[solr-solrj-8.11.1.jar:8.11.1
>>>> 0b002b11819df70783e83ef36b42ed1223c14b50 -
>>>>> janhoy - 2021-12-14 13:50:57]
>>>>> at
>>>>>
>>>>>
>>>> org.apache.solr.client.solrj.impl.ConcurrentUpdateHttp2SolrClient.request(ConcurrentUpdateHttp2SolrClient.java:370)
>>>>> ~[solr-solrj-8.11.1.jar:8.11.1
>>>> 0b002b11819df70783e83ef36b42ed1223c14b50 -
>>>>> janhoy - 2021-12-14 13:50:57]
>>>>> at
>>>> org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1290)
>>>>> ~[solr-solrj-8.11.1.jar:8.11.1
>>>> 0b002b11819df70783e83ef36b42ed1223c14b50 -
>>>>> janhoy - 2021-12-14 13:50:57]
>>>>> at
>>>>>
>>>>>
>>>> org.apache.solr.update.SolrCmdDistributor.doRequest(SolrCmdDistributor.java:345)
>>>>> ~[solr-core-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50
>>>> -
>>>>> janhoy - 2021-12-14 13:50:55]
>>>>> at
>>>>>
>>>>>
>>>> org.apache.solr.update.SolrCmdDistributor.lambda$submit$0(SolrCmdDistributor.java:334)
>>>>> ~[solr-core-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50
>>>> -
>>>>> janhoy - 2021-12-14 13:50:55]
>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
>>>>> at
>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
>>>>> ~[?:?]
>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
>>>>> at
>>>>>
>>>>>
>>>> com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:180)
>>>>> ~[metrics-core-4.1.5.jar:4.1.5]
>>>>> at
>>>>>
>>>>>
>>>> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:218)
>>>>> ~[solr-solrj-8.11.1.jar:8.11.1
>>>> 0b002b11819df70783e83ef36b42ed1223c14b50 -
>>>>> janhoy - 2021-12-14 13:50:57]
>>>>> at
>>>>>
>>>>>
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
>>>>> [?:?]
>>>>> at
>>>>>
>>>>>
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
>>>>> [?:?]
>>>>> at java.lang.Thread.run(Thread.java:833) [?:?] Caused by:
>>>>> java.util.concurrent.RejectedExecutionException: Max requests queued
>>>> per
>>>>> destination 3000 exceeded for
>>>>> HttpDestination[http://dev-foo3672.bar.no:12621
>>>>> ]@74097d87,queue=3000,pool=MultiplexConnectionPool@310bfd62
>>>>> [c=0/4/4,a=0,i=0,q=3000]
>>>>> at
>>>> org.eclipse.jetty.client.HttpDestination.send(HttpDestination.java:289)
>>>>> ~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927]
>>>>> at
>>>> org.eclipse.jetty.client.HttpDestination.send(HttpDestination.java:262)
>>>>> ~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927]
>>>>> at org.eclipse.jetty.client.HttpClient.send(HttpClient.java:600)
>>>>> ~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927]
>>>>> at
>>>> org.eclipse.jetty.client.HttpRequest.sendAsync(HttpRequest.java:778)
>>>>> ~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927]
>>>>> at org.eclipse.jetty.client.HttpRequest.send(HttpRequest.java:765)
>>>>> ~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927]
>>>>> at
>>>>>
>>>>>
>>>> org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:409)
>>>>> ~[solr-solrj-8.11.1.jar:8.11.1
>>>> 0b002b11819df70783e83ef36b42ed1223c14b50 -
>>>>> janhoy - 2021-12-14 13:50:57]
>>>>>
>>>>
>>>
Re: SolrServerException: Max requests queued per destination 3000 exceeded for HttpDestination
Posted by Henrik Brautaset Aronsen <he...@gmail.com>.
Thanks!
I'll try doubling the JVM memory and see what happens. The dev instances
only had "-Xms1024M -Xmx1024M -Xss256k" set.
Cheers,
Henrik
On Mon, May 23, 2022 at 9:56 AM Deepak Goel <de...@gmail.com> wrote:
> Looks like the memory is getting maxed out. However there must be some way
> to monitor the dev environment and confirm the same (something like a top
> command).
>
> Deepak
> "The greatness of a nation can be judged by the way its animals are
> treated - Mahatma Gandhi"
>
> +91 73500 12833
> deicool@gmail.com
>
> Facebook: https://www.facebook.com/deicool
> LinkedIn: www.linkedin.com/in/deicool
>
> "Plant a Tree, Go Green"
>
> Make In India : http://www.makeinindia.com/home
>
>
> On Mon, May 23, 2022 at 1:20 PM Henrik Brautaset Aronsen <
> henrik.aronsen@gmail.com> wrote:
>
>> Hi Deepak, thanks for reaching out!
>>
>> Prod has 8 servers with 8x Xeon CPU @ 2.00GHz and 64GB RAM
>> Dev has 2 servers with 2x Xeon CPU @ 2.00GHz and 16GB RAM
>>
>> Is there a way to monitor the MultiplexConnectionPool?
>>
>> And is it CPU or RAM that is maxed out, you think?
>>
>> I really appreciate any input :)
>>
>> Cheers,
>> Henrik
>>
>> On Mon, May 16, 2022 at 10:41 AM Deepak Goel <de...@gmail.com> wrote:
>>
>>> Looks like *MultiplexConnectionPool *is getting maxed out.
>>>
>>> Could you please tell us the server configuration of dev and prod? Looks
>>> like the dev environment is getting maxed out.
>>>
>>> Deepak
>>> "The greatness of a nation can be judged by the way its animals are
>>> treated
>>> - Mahatma Gandhi"
>>>
>>> +91 73500 12833
>>> deicool@gmail.com
>>>
>>> Facebook: https://www.facebook.com/deicool
>>> LinkedIn: www.linkedin.com/in/deicool
>>>
>>> "Plant a Tree, Go Green"
>>>
>>> Make In India : http://www.makeinindia.com/home
>>>
>>>
>>> On Mon, May 16, 2022 at 12:53 PM Henrik Brautaset Aronsen <
>>> henrik.aronsen@gmail.com> wrote:
>>>
>>> > Hi.
>>> >
>>> > We keep getting "SolrServerException: Max requests queued per
>>> destination
>>> > 3000 exceeded for HttpDestination" for several different collections
>>> and
>>> > instances in our SolrCloud dev environment. The instances don't reply
>>> to
>>> > requests and need to be restarted to function again.
>>> >
>>> > We're not getting those errors in the prod environment even though
>>> those
>>> > are set up similarly.
>>> >
>>> > Any ideas what could cause this?
>>> >
>>> > We're on Solr 8.11.1 with connTimeout=60000, socketTimeout=60000,
>>> > zkClientTimeout=30000, solr.jetty.http.idleTimeout=60000. Full
>>> stacktrace
>>> > below. Thanks for any help!
>>> >
>>> > Cheers,
>>> > Henrik
>>> >
>>> > java.util.concurrent.RejectedExecutionException: Max requests queued
>>> per
>>> > destination 3000 exceeded for
>>> > HttpDestination[http://dev-foo3672.bar.no:12621
>>> > ]@74097d87,queue=3000,pool=MultiplexConnectionPool@310bfd62
>>> > [c=0/4/4,a=0,i=0,q=3000]
>>> > at
>>> org.eclipse.jetty.client.HttpDestination.send(HttpDestination.java:289)
>>> > ~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927]
>>> > at
>>> org.eclipse.jetty.client.HttpDestination.send(HttpDestination.java:262)
>>> > ~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927]
>>> > at org.eclipse.jetty.client.HttpClient.send(HttpClient.java:600)
>>> > ~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927]
>>> > at
>>> org.eclipse.jetty.client.HttpRequest.sendAsync(HttpRequest.java:778)
>>> > ~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927]
>>> > at org.eclipse.jetty.client.HttpRequest.send(HttpRequest.java:765)
>>> > ~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927]
>>> > at
>>> >
>>> >
>>> org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:409)
>>> > ~[solr-solrj-8.11.1.jar:8.11.1
>>> 0b002b11819df70783e83ef36b42ed1223c14b50 -
>>> > janhoy - 2021-12-14 13:50:57]
>>> > at org.apache.solr.client.solrj.SolrServerException: Max requests
>>> queued
>>> > per destination 3000 exceeded for
>>> > HttpDestination[http://dev-foo3672.bar.no:12621
>>> > ]@74097d87,queue=3000,pool=MultiplexConnectionPool@310bfd62
>>> > [c=0/4/4,a=0,i=0,q=3000]
>>> > at
>>> >
>>> >
>>> org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:439)
>>> > ~[solr-solrj-8.11.1.jar:8.11.1
>>> 0b002b11819df70783e83ef36b42ed1223c14b50 -
>>> > janhoy - 2021-12-14 13:50:57]
>>> > at
>>> >
>>> >
>>> org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:776)
>>> > ~[solr-solrj-8.11.1.jar:8.11.1
>>> 0b002b11819df70783e83ef36b42ed1223c14b50 -
>>> > janhoy - 2021-12-14 13:50:57]
>>> > at
>>> >
>>> >
>>> org.apache.solr.client.solrj.impl.ConcurrentUpdateHttp2SolrClient.request(ConcurrentUpdateHttp2SolrClient.java:370)
>>> > ~[solr-solrj-8.11.1.jar:8.11.1
>>> 0b002b11819df70783e83ef36b42ed1223c14b50 -
>>> > janhoy - 2021-12-14 13:50:57]
>>> > at
>>> org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1290)
>>> > ~[solr-solrj-8.11.1.jar:8.11.1
>>> 0b002b11819df70783e83ef36b42ed1223c14b50 -
>>> > janhoy - 2021-12-14 13:50:57]
>>> > at
>>> >
>>> >
>>> org.apache.solr.update.SolrCmdDistributor.doRequest(SolrCmdDistributor.java:345)
>>> > ~[solr-core-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50
>>> -
>>> > janhoy - 2021-12-14 13:50:55]
>>> > at
>>> >
>>> >
>>> org.apache.solr.update.SolrCmdDistributor.lambda$submit$0(SolrCmdDistributor.java:334)
>>> > ~[solr-core-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50
>>> -
>>> > janhoy - 2021-12-14 13:50:55]
>>> > at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
>>> > at
>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
>>> > ~[?:?]
>>> > at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
>>> > at
>>> >
>>> >
>>> com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:180)
>>> > ~[metrics-core-4.1.5.jar:4.1.5]
>>> > at
>>> >
>>> >
>>> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:218)
>>> > ~[solr-solrj-8.11.1.jar:8.11.1
>>> 0b002b11819df70783e83ef36b42ed1223c14b50 -
>>> > janhoy - 2021-12-14 13:50:57]
>>> > at
>>> >
>>> >
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
>>> > [?:?]
>>> > at
>>> >
>>> >
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
>>> > [?:?]
>>> > at java.lang.Thread.run(Thread.java:833) [?:?] Caused by:
>>> > java.util.concurrent.RejectedExecutionException: Max requests queued
>>> per
>>> > destination 3000 exceeded for
>>> > HttpDestination[http://dev-foo3672.bar.no:12621
>>> > ]@74097d87,queue=3000,pool=MultiplexConnectionPool@310bfd62
>>> > [c=0/4/4,a=0,i=0,q=3000]
>>> > at
>>> org.eclipse.jetty.client.HttpDestination.send(HttpDestination.java:289)
>>> > ~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927]
>>> > at
>>> org.eclipse.jetty.client.HttpDestination.send(HttpDestination.java:262)
>>> > ~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927]
>>> > at org.eclipse.jetty.client.HttpClient.send(HttpClient.java:600)
>>> > ~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927]
>>> > at
>>> org.eclipse.jetty.client.HttpRequest.sendAsync(HttpRequest.java:778)
>>> > ~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927]
>>> > at org.eclipse.jetty.client.HttpRequest.send(HttpRequest.java:765)
>>> > ~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927]
>>> > at
>>> >
>>> >
>>> org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:409)
>>> > ~[solr-solrj-8.11.1.jar:8.11.1
>>> 0b002b11819df70783e83ef36b42ed1223c14b50 -
>>> > janhoy - 2021-12-14 13:50:57]
>>> >
>>>
>>
Re: SolrServerException: Max requests queued per destination 3000 exceeded for HttpDestination
Posted by Deepak Goel <de...@gmail.com>.
Looks like the memory is getting maxed out. However there must be some way
to monitor the dev environment and confirm the same (something like a top
command).
Deepak
"The greatness of a nation can be judged by the way its animals are treated
- Mahatma Gandhi"
+91 73500 12833
deicool@gmail.com
Facebook: https://www.facebook.com/deicool
LinkedIn: www.linkedin.com/in/deicool
"Plant a Tree, Go Green"
Make In India : http://www.makeinindia.com/home
On Mon, May 23, 2022 at 1:20 PM Henrik Brautaset Aronsen <
henrik.aronsen@gmail.com> wrote:
> Hi Deepak, thanks for reaching out!
>
> Prod has 8 servers with 8x Xeon CPU @ 2.00GHz and 64GB RAM
> Dev has 2 servers with 2x Xeon CPU @ 2.00GHz and 16GB RAM
>
> Is there a way to monitor the MultiplexConnectionPool?
>
> And is it CPU or RAM that is maxed out, you think?
>
> I really appreciate any input :)
>
> Cheers,
> Henrik
>
> On Mon, May 16, 2022 at 10:41 AM Deepak Goel <de...@gmail.com> wrote:
>
>> Looks like *MultiplexConnectionPool *is getting maxed out.
>>
>> Could you please tell us the server configuration of dev and prod? Looks
>> like the dev environment is getting maxed out.
>>
>> Deepak
>> "The greatness of a nation can be judged by the way its animals are
>> treated
>> - Mahatma Gandhi"
>>
>> +91 73500 12833
>> deicool@gmail.com
>>
>> Facebook: https://www.facebook.com/deicool
>> LinkedIn: www.linkedin.com/in/deicool
>>
>> "Plant a Tree, Go Green"
>>
>> Make In India : http://www.makeinindia.com/home
>>
>>
>> On Mon, May 16, 2022 at 12:53 PM Henrik Brautaset Aronsen <
>> henrik.aronsen@gmail.com> wrote:
>>
>> > Hi.
>> >
>> > We keep getting "SolrServerException: Max requests queued per
>> destination
>> > 3000 exceeded for HttpDestination" for several different collections and
>> > instances in our SolrCloud dev environment. The instances don't reply to
>> > requests and need to be restarted to function again.
>> >
>> > We're not getting those errors in the prod environment even though those
>> > are set up similarly.
>> >
>> > Any ideas what could cause this?
>> >
>> > We're on Solr 8.11.1 with connTimeout=60000, socketTimeout=60000,
>> > zkClientTimeout=30000, solr.jetty.http.idleTimeout=60000. Full
>> stacktrace
>> > below. Thanks for any help!
>> >
>> > Cheers,
>> > Henrik
>> >
>> > java.util.concurrent.RejectedExecutionException: Max requests queued per
>> > destination 3000 exceeded for
>> > HttpDestination[http://dev-foo3672.bar.no:12621
>> > ]@74097d87,queue=3000,pool=MultiplexConnectionPool@310bfd62
>> > [c=0/4/4,a=0,i=0,q=3000]
>> > at
>> org.eclipse.jetty.client.HttpDestination.send(HttpDestination.java:289)
>> > ~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927]
>> > at
>> org.eclipse.jetty.client.HttpDestination.send(HttpDestination.java:262)
>> > ~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927]
>> > at org.eclipse.jetty.client.HttpClient.send(HttpClient.java:600)
>> > ~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927]
>> > at org.eclipse.jetty.client.HttpRequest.sendAsync(HttpRequest.java:778)
>> > ~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927]
>> > at org.eclipse.jetty.client.HttpRequest.send(HttpRequest.java:765)
>> > ~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927]
>> > at
>> >
>> >
>> org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:409)
>> > ~[solr-solrj-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50
>> -
>> > janhoy - 2021-12-14 13:50:57]
>> > at org.apache.solr.client.solrj.SolrServerException: Max requests
>> queued
>> > per destination 3000 exceeded for
>> > HttpDestination[http://dev-foo3672.bar.no:12621
>> > ]@74097d87,queue=3000,pool=MultiplexConnectionPool@310bfd62
>> > [c=0/4/4,a=0,i=0,q=3000]
>> > at
>> >
>> >
>> org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:439)
>> > ~[solr-solrj-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50
>> -
>> > janhoy - 2021-12-14 13:50:57]
>> > at
>> >
>> >
>> org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:776)
>> > ~[solr-solrj-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50
>> -
>> > janhoy - 2021-12-14 13:50:57]
>> > at
>> >
>> >
>> org.apache.solr.client.solrj.impl.ConcurrentUpdateHttp2SolrClient.request(ConcurrentUpdateHttp2SolrClient.java:370)
>> > ~[solr-solrj-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50
>> -
>> > janhoy - 2021-12-14 13:50:57]
>> > at
>> org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1290)
>> > ~[solr-solrj-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50
>> -
>> > janhoy - 2021-12-14 13:50:57]
>> > at
>> >
>> >
>> org.apache.solr.update.SolrCmdDistributor.doRequest(SolrCmdDistributor.java:345)
>> > ~[solr-core-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 -
>> > janhoy - 2021-12-14 13:50:55]
>> > at
>> >
>> >
>> org.apache.solr.update.SolrCmdDistributor.lambda$submit$0(SolrCmdDistributor.java:334)
>> > ~[solr-core-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 -
>> > janhoy - 2021-12-14 13:50:55]
>> > at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
>> > at
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
>> > ~[?:?]
>> > at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
>> > at
>> >
>> >
>> com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:180)
>> > ~[metrics-core-4.1.5.jar:4.1.5]
>> > at
>> >
>> >
>> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:218)
>> > ~[solr-solrj-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50
>> -
>> > janhoy - 2021-12-14 13:50:57]
>> > at
>> >
>> >
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
>> > [?:?]
>> > at
>> >
>> >
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
>> > [?:?]
>> > at java.lang.Thread.run(Thread.java:833) [?:?] Caused by:
>> > java.util.concurrent.RejectedExecutionException: Max requests queued per
>> > destination 3000 exceeded for
>> > HttpDestination[http://dev-foo3672.bar.no:12621
>> > ]@74097d87,queue=3000,pool=MultiplexConnectionPool@310bfd62
>> > [c=0/4/4,a=0,i=0,q=3000]
>> > at
>> org.eclipse.jetty.client.HttpDestination.send(HttpDestination.java:289)
>> > ~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927]
>> > at
>> org.eclipse.jetty.client.HttpDestination.send(HttpDestination.java:262)
>> > ~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927]
>> > at org.eclipse.jetty.client.HttpClient.send(HttpClient.java:600)
>> > ~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927]
>> > at org.eclipse.jetty.client.HttpRequest.sendAsync(HttpRequest.java:778)
>> > ~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927]
>> > at org.eclipse.jetty.client.HttpRequest.send(HttpRequest.java:765)
>> > ~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927]
>> > at
>> >
>> >
>> org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:409)
>> > ~[solr-solrj-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50
>> -
>> > janhoy - 2021-12-14 13:50:57]
>> >
>>
>
Re: SolrServerException: Max requests queued per destination 3000 exceeded for HttpDestination
Posted by Henrik Brautaset Aronsen <he...@gmail.com>.
Hi Deepak, thanks for reaching out!
Prod has 8 servers with 8x Xeon CPU @ 2.00GHz and 64GB RAM
Dev has 2 servers with 2x Xeon CPU @ 2.00GHz and 16GB RAM
Is there a way to monitor the MultiplexConnectionPool?
And is it CPU or RAM that is maxed out, you think?
I really appreciate any input :)
Cheers,
Henrik
On Mon, May 16, 2022 at 10:41 AM Deepak Goel <de...@gmail.com> wrote:
> Looks like *MultiplexConnectionPool *is getting maxed out.
>
> Could you please tell us the server configuration of dev and prod? Looks
> like the dev environment is getting maxed out.
>
> Deepak
> "The greatness of a nation can be judged by the way its animals are treated
> - Mahatma Gandhi"
>
> +91 73500 12833
> deicool@gmail.com
>
> Facebook: https://www.facebook.com/deicool
> LinkedIn: www.linkedin.com/in/deicool
>
> "Plant a Tree, Go Green"
>
> Make In India : http://www.makeinindia.com/home
>
>
> On Mon, May 16, 2022 at 12:53 PM Henrik Brautaset Aronsen <
> henrik.aronsen@gmail.com> wrote:
>
> > Hi.
> >
> > We keep getting "SolrServerException: Max requests queued per destination
> > 3000 exceeded for HttpDestination" for several different collections and
> > instances in our SolrCloud dev environment. The instances don't reply to
> > requests and need to be restarted to function again.
> >
> > We're not getting those errors in the prod environment even though those
> > are set up similarly.
> >
> > Any ideas what could cause this?
> >
> > We're on Solr 8.11.1 with connTimeout=60000, socketTimeout=60000,
> > zkClientTimeout=30000, solr.jetty.http.idleTimeout=60000. Full stacktrace
> > below. Thanks for any help!
> >
> > Cheers,
> > Henrik
> >
> > java.util.concurrent.RejectedExecutionException: Max requests queued per
> > destination 3000 exceeded for
> > HttpDestination[http://dev-foo3672.bar.no:12621
> > ]@74097d87,queue=3000,pool=MultiplexConnectionPool@310bfd62
> > [c=0/4/4,a=0,i=0,q=3000]
> > at
> org.eclipse.jetty.client.HttpDestination.send(HttpDestination.java:289)
> > ~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927]
> > at
> org.eclipse.jetty.client.HttpDestination.send(HttpDestination.java:262)
> > ~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927]
> > at org.eclipse.jetty.client.HttpClient.send(HttpClient.java:600)
> > ~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927]
> > at org.eclipse.jetty.client.HttpRequest.sendAsync(HttpRequest.java:778)
> > ~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927]
> > at org.eclipse.jetty.client.HttpRequest.send(HttpRequest.java:765)
> > ~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927]
> > at
> >
> >
> org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:409)
> > ~[solr-solrj-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 -
> > janhoy - 2021-12-14 13:50:57]
> > at org.apache.solr.client.solrj.SolrServerException: Max requests queued
> > per destination 3000 exceeded for
> > HttpDestination[http://dev-foo3672.bar.no:12621
> > ]@74097d87,queue=3000,pool=MultiplexConnectionPool@310bfd62
> > [c=0/4/4,a=0,i=0,q=3000]
> > at
> >
> >
> org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:439)
> > ~[solr-solrj-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 -
> > janhoy - 2021-12-14 13:50:57]
> > at
> >
> >
> org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:776)
> > ~[solr-solrj-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 -
> > janhoy - 2021-12-14 13:50:57]
> > at
> >
> >
> org.apache.solr.client.solrj.impl.ConcurrentUpdateHttp2SolrClient.request(ConcurrentUpdateHttp2SolrClient.java:370)
> > ~[solr-solrj-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 -
> > janhoy - 2021-12-14 13:50:57]
> > at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1290)
> > ~[solr-solrj-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 -
> > janhoy - 2021-12-14 13:50:57]
> > at
> >
> >
> org.apache.solr.update.SolrCmdDistributor.doRequest(SolrCmdDistributor.java:345)
> > ~[solr-core-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 -
> > janhoy - 2021-12-14 13:50:55]
> > at
> >
> >
> org.apache.solr.update.SolrCmdDistributor.lambda$submit$0(SolrCmdDistributor.java:334)
> > ~[solr-core-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 -
> > janhoy - 2021-12-14 13:50:55]
> > at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
> > at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
> > ~[?:?]
> > at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
> > at
> >
> >
> com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:180)
> > ~[metrics-core-4.1.5.jar:4.1.5]
> > at
> >
> >
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:218)
> > ~[solr-solrj-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 -
> > janhoy - 2021-12-14 13:50:57]
> > at
> >
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
> > [?:?]
> > at
> >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
> > [?:?]
> > at java.lang.Thread.run(Thread.java:833) [?:?] Caused by:
> > java.util.concurrent.RejectedExecutionException: Max requests queued per
> > destination 3000 exceeded for
> > HttpDestination[http://dev-foo3672.bar.no:12621
> > ]@74097d87,queue=3000,pool=MultiplexConnectionPool@310bfd62
> > [c=0/4/4,a=0,i=0,q=3000]
> > at
> org.eclipse.jetty.client.HttpDestination.send(HttpDestination.java:289)
> > ~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927]
> > at
> org.eclipse.jetty.client.HttpDestination.send(HttpDestination.java:262)
> > ~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927]
> > at org.eclipse.jetty.client.HttpClient.send(HttpClient.java:600)
> > ~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927]
> > at org.eclipse.jetty.client.HttpRequest.sendAsync(HttpRequest.java:778)
> > ~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927]
> > at org.eclipse.jetty.client.HttpRequest.send(HttpRequest.java:765)
> > ~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927]
> > at
> >
> >
> org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:409)
> > ~[solr-solrj-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 -
> > janhoy - 2021-12-14 13:50:57]
> >
>
Re: SolrServerException: Max requests queued per destination 3000 exceeded for HttpDestination
Posted by Deepak Goel <de...@gmail.com>.
Looks like *MultiplexConnectionPool *is getting maxed out.
Could you please tell us the server configuration of dev and prod? Looks
like the dev environment is getting maxed out.
Deepak
"The greatness of a nation can be judged by the way its animals are treated
- Mahatma Gandhi"
+91 73500 12833
deicool@gmail.com
Facebook: https://www.facebook.com/deicool
LinkedIn: www.linkedin.com/in/deicool
"Plant a Tree, Go Green"
Make In India : http://www.makeinindia.com/home
On Mon, May 16, 2022 at 12:53 PM Henrik Brautaset Aronsen <
henrik.aronsen@gmail.com> wrote:
> Hi.
>
> We keep getting "SolrServerException: Max requests queued per destination
> 3000 exceeded for HttpDestination" for several different collections and
> instances in our SolrCloud dev environment. The instances don't reply to
> requests and need to be restarted to function again.
>
> We're not getting those errors in the prod environment even though those
> are set up similarly.
>
> Any ideas what could cause this?
>
> We're on Solr 8.11.1 with connTimeout=60000, socketTimeout=60000,
> zkClientTimeout=30000, solr.jetty.http.idleTimeout=60000. Full stacktrace
> below. Thanks for any help!
>
> Cheers,
> Henrik
>
> java.util.concurrent.RejectedExecutionException: Max requests queued per
> destination 3000 exceeded for
> HttpDestination[http://dev-foo3672.bar.no:12621
> ]@74097d87,queue=3000,pool=MultiplexConnectionPool@310bfd62
> [c=0/4/4,a=0,i=0,q=3000]
> at org.eclipse.jetty.client.HttpDestination.send(HttpDestination.java:289)
> ~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927]
> at org.eclipse.jetty.client.HttpDestination.send(HttpDestination.java:262)
> ~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927]
> at org.eclipse.jetty.client.HttpClient.send(HttpClient.java:600)
> ~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927]
> at org.eclipse.jetty.client.HttpRequest.sendAsync(HttpRequest.java:778)
> ~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927]
> at org.eclipse.jetty.client.HttpRequest.send(HttpRequest.java:765)
> ~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927]
> at
>
> org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:409)
> ~[solr-solrj-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 -
> janhoy - 2021-12-14 13:50:57]
> at org.apache.solr.client.solrj.SolrServerException: Max requests queued
> per destination 3000 exceeded for
> HttpDestination[http://dev-foo3672.bar.no:12621
> ]@74097d87,queue=3000,pool=MultiplexConnectionPool@310bfd62
> [c=0/4/4,a=0,i=0,q=3000]
> at
>
> org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:439)
> ~[solr-solrj-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 -
> janhoy - 2021-12-14 13:50:57]
> at
>
> org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:776)
> ~[solr-solrj-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 -
> janhoy - 2021-12-14 13:50:57]
> at
>
> org.apache.solr.client.solrj.impl.ConcurrentUpdateHttp2SolrClient.request(ConcurrentUpdateHttp2SolrClient.java:370)
> ~[solr-solrj-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 -
> janhoy - 2021-12-14 13:50:57]
> at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1290)
> ~[solr-solrj-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 -
> janhoy - 2021-12-14 13:50:57]
> at
>
> org.apache.solr.update.SolrCmdDistributor.doRequest(SolrCmdDistributor.java:345)
> ~[solr-core-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 -
> janhoy - 2021-12-14 13:50:55]
> at
>
> org.apache.solr.update.SolrCmdDistributor.lambda$submit$0(SolrCmdDistributor.java:334)
> ~[solr-core-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 -
> janhoy - 2021-12-14 13:50:55]
> at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
> ~[?:?]
> at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
> at
>
> com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:180)
> ~[metrics-core-4.1.5.jar:4.1.5]
> at
>
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:218)
> ~[solr-solrj-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 -
> janhoy - 2021-12-14 13:50:57]
> at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
> [?:?]
> at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
> [?:?]
> at java.lang.Thread.run(Thread.java:833) [?:?] Caused by:
> java.util.concurrent.RejectedExecutionException: Max requests queued per
> destination 3000 exceeded for
> HttpDestination[http://dev-foo3672.bar.no:12621
> ]@74097d87,queue=3000,pool=MultiplexConnectionPool@310bfd62
> [c=0/4/4,a=0,i=0,q=3000]
> at org.eclipse.jetty.client.HttpDestination.send(HttpDestination.java:289)
> ~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927]
> at org.eclipse.jetty.client.HttpDestination.send(HttpDestination.java:262)
> ~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927]
> at org.eclipse.jetty.client.HttpClient.send(HttpClient.java:600)
> ~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927]
> at org.eclipse.jetty.client.HttpRequest.sendAsync(HttpRequest.java:778)
> ~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927]
> at org.eclipse.jetty.client.HttpRequest.send(HttpRequest.java:765)
> ~[jetty-client-9.4.44.v20210927.jar:9.4.44.v20210927]
> at
>
> org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:409)
> ~[solr-solrj-8.11.1.jar:8.11.1 0b002b11819df70783e83ef36b42ed1223c14b50 -
> janhoy - 2021-12-14 13:50:57]
>