You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Markus Jelsma <ma...@openindex.io> on 2017/07/18 11:10:07 UTC
RE: SolrJ 6.6.0 Connection pool shutdown now with stack trace
The problem was never resolved but Shawn asked for the stack trace, here it is:
org.apache.solr.client.solrj.SolrServerException: java.lang.IllegalStateException: Connection pool shut down
at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:485)
at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:388)
at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1383)
at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:1134)
at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:1073)
at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:160)
at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:942)
at org.apache.solr.client.solrj.SolrClient.getById(SolrClient.java:1173)
at org.apache.solr.client.solrj.SolrClient.getById(SolrClient.java:1090)
at org.apache.solr.client.solrj.SolrClient.getById(SolrClient.java:1110)
..internal method calling getById()..
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalStateException: Connection pool shut down
at org.apache.http.util.Asserts.check(Asserts.java:34)
at org.apache.http.pool.AbstractConnPool.lease(AbstractConnPool.java:184)
at org.apache.http.pool.AbstractConnPool.lease(AbstractConnPool.java:217)
at org.apache.http.impl.conn.PoolingClientConnectionManager.requestConnection(PoolingClientConnectionManager.java:184)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:415)
at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:515)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:279)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:268)
at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:447)
... 24 more
So, to summarize, we have a program checking presence of documents in Solr using getById() and we don't want this exception to bubble up, we want SolrJ to restore the connection pool just as CloudSolrClient would move on to another node if one went down in the mean time.
Is this possible? How?
Many thanks,
Markus
-----Original message-----
> From:Markus Jelsma <ma...@openindex.io>
> Sent: Thursday 29th June 2017 16:38
> To: solr-user@lucene.apache.org
> Subject: RE: SolrJ 6.6.0 Connection pool shutdown
>
> Thanks. I probably should have mentioned there is no firewall limiting connections between those hosts. Actually, the processes run on the same hosts as the Solr cluster is running on.
>
> Thanks,
> Markus
>
>
>
> -----Original message-----
> > From:Alexandre Rafalovitch <ar...@gmail.com>
> > Sent: Thursday 29th June 2017 15:38
> > To: solr-user <so...@lucene.apache.org>
> > Subject: Re: SolrJ 6.6.0 Connection pool shutdown
> >
> > One thing to check is whether there is a firewall between the client
> > and the server. They - sometimes - cut the silent connections in the
> > _middle_ (at the firewall). The usual solution is keepAlive request of
> > some kind or not using the connection pool.
> >
> > One way to check is with network tracer like Wireshark and checking
> > whether the actual hardware at the other end of the connection is a
> > normal server or some sort of unexpected hardware piece of equipment
> > (firewall). Yes, that's using the hammer to swat a fly :-)
> >
> > Regards,
> > Alex.
> > ----
> > http://www.solr-start.com/ - Resources for Solr users, new and experienced
> >
> >
> > On 29 June 2017 at 08:21, Markus Jelsma <ma...@openindex.io> wrote:
> > > Hi,
> > >
> > > Everything is 6.6.0. I could include a stack trace (i don't print them in my program), but that would only be the the trace from getById() to CloudSolrClient.requestWithRetryOnStaleState() and little deeper, that what you're looking for?
> > >
> > > We haven't called close() in that particular part of the program.
> > >
> > > Method requestWithRetryOnStaleState has some retry logic built-in but doesn't seem to work for the exception i got.
> > >
> > > I'll let it print the stack trace and get back if it happens again.
> > >
> > > Thanks,
> > > Markus
> > >
> > > -----Original message-----
> > >> From:Shawn Heisey <ap...@elyograg.org>
> > >> Sent: Tuesday 27th June 2017 23:02
> > >> To: solr-user@lucene.apache.org
> > >> Subject: Re: SolrJ 6.6.0 Connection pool shutdown
> > >>
> > >> On 6/27/2017 6:50 AM, Markus Jelsma wrote:
> > >> > We have a proces checking presence of many documents in a collection, just a simple client.getById(id). It sometimes begins throwing lots of these exceptions in a row:
> > >> >
> > >> > org.apache.solr.client.solrj.SolrServerException: java.lang.IllegalStateException: Connection pool shut down
> > >> >
> > >> > Then, as suddenly as it appeared, it's gone again a no longer a problem. I would expect SolrJ not to throw this but to wait until it the connection pool, or whatever mechanism is there, to recover.
> > >> >
> > >> > Did i miss a magic parameter for SolrJ?\
> > >>
> > >> That error message will be much longer than what you've provided here.
> > >> It will have a java stacktrace that's typically a dozen or so lines
> > >> long. There may also be one or more "Caused by" sections after the
> > >> stacktrace, each with a stacktrace of its own. Can you share the full
> > >> error message? Is the server also running 6.6.0, or a different version?
> > >>
> > >> It would also be helpful if you can share the SolrJ code you've written,
> > >> cleanly redacted to remove anything sensitive.
> > >>
> > >> That particular message ("Connection pool shut down") sounds like it
> > >> probably came from HttpClient, which SolrJ uses ... and I would expect
> > >> that to only happen if you close/shutdown the HttpClient or the
> > >> SolrClient. After closing, a client can't be used any more. Normally
> > >> the only time you should close a client is right before exiting the
> > >> program, although if the program's about to exit, it's generally
> > >> unnecessary, so in my opinion for *most* usages, closing the client
> > >> likely never needs to happen.
> > >>
> > >> Thanks,
> > >> Shawn
> > >>
> > >>
> >
>
Re: SolrJ 6.6.0 Connection pool shutdown now with stack trace
Posted by Susheel Kumar <su...@gmail.com>.
Then most likely its due to closing of connection as mentioned above though
you said it's not happening in that part of your code. To rule out
firewall possibility, you can test in some other/local env. Also how many
requests/client/connections happening concurrently.
Thanks,
Susheel
On Tue, Jul 18, 2017 at 8:43 AM, Markus Jelsma <ma...@openindex.io>
wrote:
> Hello Susheel,
>
> No, nothing at all. I've check all six nodes, they are clean.
>
> Thanks,
> Markus
>
>
>
> -----Original message-----
> > From:Susheel Kumar <su...@gmail.com>
> > Sent: Tuesday 18th July 2017 14:30
> > To: solr-user@lucene.apache.org
> > Subject: Re: SolrJ 6.6.0 Connection pool shutdown now with stack trace
> >
> > Do you see any errors etc. in solr.log during this time?
> >
> > On Tue, Jul 18, 2017 at 7:10 AM, Markus Jelsma <
> markus.jelsma@openindex.io>
> > wrote:
> >
> > > The problem was never resolved but Shawn asked for the stack trace,
> here
> > > it is:
> > >
> > > org.apache.solr.client.solrj.SolrServerException: java.lang.
> IllegalStateException:
> > > Connection pool shut down
> > > at org.apache.solr.client.solrj.impl.LBHttpSolrClient.
> > > doRequest(LBHttpSolrClient.java:485)
> > > at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(
> > > LBHttpSolrClient.java:388)
> > > at org.apache.solr.client.solrj.impl.CloudSolrClient.
> > > sendRequest(CloudSolrClient.java:1383)
> > > at org.apache.solr.client.solrj.impl.CloudSolrClient.
> > > requestWithRetryOnStaleState(CloudSolrClient.java:1134)
> > > at org.apache.solr.client.solrj.impl.CloudSolrClient.request(
> > > CloudSolrClient.java:1073)
> > > at org.apache.solr.client.solrj.SolrRequest.process(
> SolrRequest.java:160)
> > > at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:942)
> > > at org.apache.solr.client.solrj.SolrClient.getById(SolrClient.
> java:1173)
> > > at org.apache.solr.client.solrj.SolrClient.getById(SolrClient.
> java:1090)
> > > at org.apache.solr.client.solrj.SolrClient.getById(SolrClient.
> java:1110)
> > > ..internal method calling getById()..
> > > at java.lang.Thread.run(Thread.java:748)
> > > Caused by: java.lang.IllegalStateException: Connection pool shut down
> > > at org.apache.http.util.Asserts.check(Asserts.java:34)
> > > at org.apache.http.pool.AbstractConnPool.lease(
> AbstractConnPool.java:184)
> > > at org.apache.http.pool.AbstractConnPool.lease(
> AbstractConnPool.java:217)
> > > at org.apache.http.impl.conn.PoolingClientConnectionManager
> > > .requestConnection(PoolingClientConnectionManager.java:184)
> > > at org.apache.http.impl.client.DefaultRequestDirector.execute(
> > > DefaultRequestDirector.java:415)
> > > at org.apache.http.impl.client.AbstractHttpClient.doExecute(
> > > AbstractHttpClient.java:882)
> > > at org.apache.http.impl.client.CloseableHttpClient.execute(
> > > CloseableHttpClient.java:82)
> > > at org.apache.http.impl.client.CloseableHttpClient.execute(
> > > CloseableHttpClient.java:55)
> > > at org.apache.solr.client.solrj.impl.HttpSolrClient.
> > > executeMethod(HttpSolrClient.java:515)
> > > at org.apache.solr.client.solrj.impl.HttpSolrClient.request(
> > > HttpSolrClient.java:279)
> > > at org.apache.solr.client.solrj.impl.HttpSolrClient.request(
> > > HttpSolrClient.java:268)
> > > at org.apache.solr.client.solrj.impl.LBHttpSolrClient.
> > > doRequest(LBHttpSolrClient.java:447)
> > > ... 24 more
> > >
> > > So, to summarize, we have a program checking presence of documents in
> Solr
> > > using getById() and we don't want this exception to bubble up, we want
> > > SolrJ to restore the connection pool just as CloudSolrClient would
> move on
> > > to another node if one went down in the mean time.
> > >
> > > Is this possible? How?
> > >
> > > Many thanks,
> > > Markus
> > >
> > > -----Original message-----
> > > > From:Markus Jelsma <ma...@openindex.io>
> > > > Sent: Thursday 29th June 2017 16:38
> > > > To: solr-user@lucene.apache.org
> > > > Subject: RE: SolrJ 6.6.0 Connection pool shutdown
> > > >
> > > > Thanks. I probably should have mentioned there is no firewall
> limiting
> > > connections between those hosts. Actually, the processes run on the
> same
> > > hosts as the Solr cluster is running on.
> > > >
> > > > Thanks,
> > > > Markus
> > > >
> > > >
> > > >
> > > > -----Original message-----
> > > > > From:Alexandre Rafalovitch <ar...@gmail.com>
> > > > > Sent: Thursday 29th June 2017 15:38
> > > > > To: solr-user <so...@lucene.apache.org>
> > > > > Subject: Re: SolrJ 6.6.0 Connection pool shutdown
> > > > >
> > > > > One thing to check is whether there is a firewall between the
> client
> > > > > and the server. They - sometimes - cut the silent connections in
> the
> > > > > _middle_ (at the firewall). The usual solution is keepAlive
> request of
> > > > > some kind or not using the connection pool.
> > > > >
> > > > > One way to check is with network tracer like Wireshark and checking
> > > > > whether the actual hardware at the other end of the connection is a
> > > > > normal server or some sort of unexpected hardware piece of
> equipment
> > > > > (firewall). Yes, that's using the hammer to swat a fly :-)
> > > > >
> > > > > Regards,
> > > > > Alex.
> > > > > ----
> > > > > http://www.solr-start.com/ - Resources for Solr users, new and
> > > experienced
> > > > >
> > > > >
> > > > > On 29 June 2017 at 08:21, Markus Jelsma <
> markus.jelsma@openindex.io>
> > > wrote:
> > > > > > Hi,
> > > > > >
> > > > > > Everything is 6.6.0. I could include a stack trace (i don't print
> > > them in my program), but that would only be the the trace from
> getById() to
> > > CloudSolrClient.requestWithRetryOnStaleState() and little deeper, that
> > > what you're looking for?
> > > > > >
> > > > > > We haven't called close() in that particular part of the program.
> > > > > >
> > > > > > Method requestWithRetryOnStaleState has some retry logic built-in
> > > but doesn't seem to work for the exception i got.
> > > > > >
> > > > > > I'll let it print the stack trace and get back if it happens
> again.
> > > > > >
> > > > > > Thanks,
> > > > > > Markus
> > > > > >
> > > > > > -----Original message-----
> > > > > >> From:Shawn Heisey <ap...@elyograg.org>
> > > > > >> Sent: Tuesday 27th June 2017 23:02
> > > > > >> To: solr-user@lucene.apache.org
> > > > > >> Subject: Re: SolrJ 6.6.0 Connection pool shutdown
> > > > > >>
> > > > > >> On 6/27/2017 6:50 AM, Markus Jelsma wrote:
> > > > > >> > We have a proces checking presence of many documents in a
> > > collection, just a simple client.getById(id). It sometimes begins
> throwing
> > > lots of these exceptions in a row:
> > > > > >> >
> > > > > >> > org.apache.solr.client.solrj.SolrServerException: java.lang.
> IllegalStateException:
> > > Connection pool shut down
> > > > > >> >
> > > > > >> > Then, as suddenly as it appeared, it's gone again a no longer
> a
> > > problem. I would expect SolrJ not to throw this but to wait until it
> the
> > > connection pool, or whatever mechanism is there, to recover.
> > > > > >> >
> > > > > >> > Did i miss a magic parameter for SolrJ?\
> > > > > >>
> > > > > >> That error message will be much longer than what you've provided
> > > here.
> > > > > >> It will have a java stacktrace that's typically a dozen or so
> lines
> > > > > >> long. There may also be one or more "Caused by" sections after
> the
> > > > > >> stacktrace, each with a stacktrace of its own. Can you share
> the
> > > full
> > > > > >> error message? Is the server also running 6.6.0, or a different
> > > version?
> > > > > >>
> > > > > >> It would also be helpful if you can share the SolrJ code you've
> > > written,
> > > > > >> cleanly redacted to remove anything sensitive.
> > > > > >>
> > > > > >> That particular message ("Connection pool shut down") sounds
> like it
> > > > > >> probably came from HttpClient, which SolrJ uses ... and I would
> > > expect
> > > > > >> that to only happen if you close/shutdown the HttpClient or the
> > > > > >> SolrClient. After closing, a client can't be used any more.
> > > Normally
> > > > > >> the only time you should close a client is right before exiting
> the
> > > > > >> program, although if the program's about to exit, it's generally
> > > > > >> unnecessary, so in my opinion for *most* usages, closing the
> client
> > > > > >> likely never needs to happen.
> > > > > >>
> > > > > >> Thanks,
> > > > > >> Shawn
> > > > > >>
> > > > > >>
> > > > >
> > > >
> > >
> >
>
Re: SolrJ 6.6.0 Connection pool shutdown now with stack trace
Posted by Susheel Kumar <su...@gmail.com>.
Do you see any errors etc. in solr.log during this time?
On Tue, Jul 18, 2017 at 7:10 AM, Markus Jelsma <ma...@openindex.io>
wrote:
> The problem was never resolved but Shawn asked for the stack trace, here
> it is:
>
> org.apache.solr.client.solrj.SolrServerException: java.lang.IllegalStateException:
> Connection pool shut down
> at org.apache.solr.client.solrj.impl.LBHttpSolrClient.
> doRequest(LBHttpSolrClient.java:485)
> at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(
> LBHttpSolrClient.java:388)
> at org.apache.solr.client.solrj.impl.CloudSolrClient.
> sendRequest(CloudSolrClient.java:1383)
> at org.apache.solr.client.solrj.impl.CloudSolrClient.
> requestWithRetryOnStaleState(CloudSolrClient.java:1134)
> at org.apache.solr.client.solrj.impl.CloudSolrClient.request(
> CloudSolrClient.java:1073)
> at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:160)
> at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:942)
> at org.apache.solr.client.solrj.SolrClient.getById(SolrClient.java:1173)
> at org.apache.solr.client.solrj.SolrClient.getById(SolrClient.java:1090)
> at org.apache.solr.client.solrj.SolrClient.getById(SolrClient.java:1110)
> ..internal method calling getById()..
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.IllegalStateException: Connection pool shut down
> at org.apache.http.util.Asserts.check(Asserts.java:34)
> at org.apache.http.pool.AbstractConnPool.lease(AbstractConnPool.java:184)
> at org.apache.http.pool.AbstractConnPool.lease(AbstractConnPool.java:217)
> at org.apache.http.impl.conn.PoolingClientConnectionManager
> .requestConnection(PoolingClientConnectionManager.java:184)
> at org.apache.http.impl.client.DefaultRequestDirector.execute(
> DefaultRequestDirector.java:415)
> at org.apache.http.impl.client.AbstractHttpClient.doExecute(
> AbstractHttpClient.java:882)
> at org.apache.http.impl.client.CloseableHttpClient.execute(
> CloseableHttpClient.java:82)
> at org.apache.http.impl.client.CloseableHttpClient.execute(
> CloseableHttpClient.java:55)
> at org.apache.solr.client.solrj.impl.HttpSolrClient.
> executeMethod(HttpSolrClient.java:515)
> at org.apache.solr.client.solrj.impl.HttpSolrClient.request(
> HttpSolrClient.java:279)
> at org.apache.solr.client.solrj.impl.HttpSolrClient.request(
> HttpSolrClient.java:268)
> at org.apache.solr.client.solrj.impl.LBHttpSolrClient.
> doRequest(LBHttpSolrClient.java:447)
> ... 24 more
>
> So, to summarize, we have a program checking presence of documents in Solr
> using getById() and we don't want this exception to bubble up, we want
> SolrJ to restore the connection pool just as CloudSolrClient would move on
> to another node if one went down in the mean time.
>
> Is this possible? How?
>
> Many thanks,
> Markus
>
> -----Original message-----
> > From:Markus Jelsma <ma...@openindex.io>
> > Sent: Thursday 29th June 2017 16:38
> > To: solr-user@lucene.apache.org
> > Subject: RE: SolrJ 6.6.0 Connection pool shutdown
> >
> > Thanks. I probably should have mentioned there is no firewall limiting
> connections between those hosts. Actually, the processes run on the same
> hosts as the Solr cluster is running on.
> >
> > Thanks,
> > Markus
> >
> >
> >
> > -----Original message-----
> > > From:Alexandre Rafalovitch <ar...@gmail.com>
> > > Sent: Thursday 29th June 2017 15:38
> > > To: solr-user <so...@lucene.apache.org>
> > > Subject: Re: SolrJ 6.6.0 Connection pool shutdown
> > >
> > > One thing to check is whether there is a firewall between the client
> > > and the server. They - sometimes - cut the silent connections in the
> > > _middle_ (at the firewall). The usual solution is keepAlive request of
> > > some kind or not using the connection pool.
> > >
> > > One way to check is with network tracer like Wireshark and checking
> > > whether the actual hardware at the other end of the connection is a
> > > normal server or some sort of unexpected hardware piece of equipment
> > > (firewall). Yes, that's using the hammer to swat a fly :-)
> > >
> > > Regards,
> > > Alex.
> > > ----
> > > http://www.solr-start.com/ - Resources for Solr users, new and
> experienced
> > >
> > >
> > > On 29 June 2017 at 08:21, Markus Jelsma <ma...@openindex.io>
> wrote:
> > > > Hi,
> > > >
> > > > Everything is 6.6.0. I could include a stack trace (i don't print
> them in my program), but that would only be the the trace from getById() to
> CloudSolrClient.requestWithRetryOnStaleState() and little deeper, that
> what you're looking for?
> > > >
> > > > We haven't called close() in that particular part of the program.
> > > >
> > > > Method requestWithRetryOnStaleState has some retry logic built-in
> but doesn't seem to work for the exception i got.
> > > >
> > > > I'll let it print the stack trace and get back if it happens again.
> > > >
> > > > Thanks,
> > > > Markus
> > > >
> > > > -----Original message-----
> > > >> From:Shawn Heisey <ap...@elyograg.org>
> > > >> Sent: Tuesday 27th June 2017 23:02
> > > >> To: solr-user@lucene.apache.org
> > > >> Subject: Re: SolrJ 6.6.0 Connection pool shutdown
> > > >>
> > > >> On 6/27/2017 6:50 AM, Markus Jelsma wrote:
> > > >> > We have a proces checking presence of many documents in a
> collection, just a simple client.getById(id). It sometimes begins throwing
> lots of these exceptions in a row:
> > > >> >
> > > >> > org.apache.solr.client.solrj.SolrServerException: java.lang.IllegalStateException:
> Connection pool shut down
> > > >> >
> > > >> > Then, as suddenly as it appeared, it's gone again a no longer a
> problem. I would expect SolrJ not to throw this but to wait until it the
> connection pool, or whatever mechanism is there, to recover.
> > > >> >
> > > >> > Did i miss a magic parameter for SolrJ?\
> > > >>
> > > >> That error message will be much longer than what you've provided
> here.
> > > >> It will have a java stacktrace that's typically a dozen or so lines
> > > >> long. There may also be one or more "Caused by" sections after the
> > > >> stacktrace, each with a stacktrace of its own. Can you share the
> full
> > > >> error message? Is the server also running 6.6.0, or a different
> version?
> > > >>
> > > >> It would also be helpful if you can share the SolrJ code you've
> written,
> > > >> cleanly redacted to remove anything sensitive.
> > > >>
> > > >> That particular message ("Connection pool shut down") sounds like it
> > > >> probably came from HttpClient, which SolrJ uses ... and I would
> expect
> > > >> that to only happen if you close/shutdown the HttpClient or the
> > > >> SolrClient. After closing, a client can't be used any more.
> Normally
> > > >> the only time you should close a client is right before exiting the
> > > >> program, although if the program's about to exit, it's generally
> > > >> unnecessary, so in my opinion for *most* usages, closing the client
> > > >> likely never needs to happen.
> > > >>
> > > >> Thanks,
> > > >> Shawn
> > > >>
> > > >>
> > >
> >
>
RE: SolrJ 6.6.0 Connection pool shutdown now with stack trace
Posted by Markus Jelsma <ma...@openindex.io>.
Hello Susheel,
No, nothing at all. I've check all six nodes, they are clean.
Thanks,
Markus
-----Original message-----
> From:Susheel Kumar <su...@gmail.com>
> Sent: Tuesday 18th July 2017 14:30
> To: solr-user@lucene.apache.org
> Subject: Re: SolrJ 6.6.0 Connection pool shutdown now with stack trace
>
> Do you see any errors etc. in solr.log during this time?
>
> On Tue, Jul 18, 2017 at 7:10 AM, Markus Jelsma <ma...@openindex.io>
> wrote:
>
> > The problem was never resolved but Shawn asked for the stack trace, here
> > it is:
> >
> > org.apache.solr.client.solrj.SolrServerException: java.lang.IllegalStateException:
> > Connection pool shut down
> > at org.apache.solr.client.solrj.impl.LBHttpSolrClient.
> > doRequest(LBHttpSolrClient.java:485)
> > at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(
> > LBHttpSolrClient.java:388)
> > at org.apache.solr.client.solrj.impl.CloudSolrClient.
> > sendRequest(CloudSolrClient.java:1383)
> > at org.apache.solr.client.solrj.impl.CloudSolrClient.
> > requestWithRetryOnStaleState(CloudSolrClient.java:1134)
> > at org.apache.solr.client.solrj.impl.CloudSolrClient.request(
> > CloudSolrClient.java:1073)
> > at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:160)
> > at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:942)
> > at org.apache.solr.client.solrj.SolrClient.getById(SolrClient.java:1173)
> > at org.apache.solr.client.solrj.SolrClient.getById(SolrClient.java:1090)
> > at org.apache.solr.client.solrj.SolrClient.getById(SolrClient.java:1110)
> > ..internal method calling getById()..
> > at java.lang.Thread.run(Thread.java:748)
> > Caused by: java.lang.IllegalStateException: Connection pool shut down
> > at org.apache.http.util.Asserts.check(Asserts.java:34)
> > at org.apache.http.pool.AbstractConnPool.lease(AbstractConnPool.java:184)
> > at org.apache.http.pool.AbstractConnPool.lease(AbstractConnPool.java:217)
> > at org.apache.http.impl.conn.PoolingClientConnectionManager
> > .requestConnection(PoolingClientConnectionManager.java:184)
> > at org.apache.http.impl.client.DefaultRequestDirector.execute(
> > DefaultRequestDirector.java:415)
> > at org.apache.http.impl.client.AbstractHttpClient.doExecute(
> > AbstractHttpClient.java:882)
> > at org.apache.http.impl.client.CloseableHttpClient.execute(
> > CloseableHttpClient.java:82)
> > at org.apache.http.impl.client.CloseableHttpClient.execute(
> > CloseableHttpClient.java:55)
> > at org.apache.solr.client.solrj.impl.HttpSolrClient.
> > executeMethod(HttpSolrClient.java:515)
> > at org.apache.solr.client.solrj.impl.HttpSolrClient.request(
> > HttpSolrClient.java:279)
> > at org.apache.solr.client.solrj.impl.HttpSolrClient.request(
> > HttpSolrClient.java:268)
> > at org.apache.solr.client.solrj.impl.LBHttpSolrClient.
> > doRequest(LBHttpSolrClient.java:447)
> > ... 24 more
> >
> > So, to summarize, we have a program checking presence of documents in Solr
> > using getById() and we don't want this exception to bubble up, we want
> > SolrJ to restore the connection pool just as CloudSolrClient would move on
> > to another node if one went down in the mean time.
> >
> > Is this possible? How?
> >
> > Many thanks,
> > Markus
> >
> > -----Original message-----
> > > From:Markus Jelsma <ma...@openindex.io>
> > > Sent: Thursday 29th June 2017 16:38
> > > To: solr-user@lucene.apache.org
> > > Subject: RE: SolrJ 6.6.0 Connection pool shutdown
> > >
> > > Thanks. I probably should have mentioned there is no firewall limiting
> > connections between those hosts. Actually, the processes run on the same
> > hosts as the Solr cluster is running on.
> > >
> > > Thanks,
> > > Markus
> > >
> > >
> > >
> > > -----Original message-----
> > > > From:Alexandre Rafalovitch <ar...@gmail.com>
> > > > Sent: Thursday 29th June 2017 15:38
> > > > To: solr-user <so...@lucene.apache.org>
> > > > Subject: Re: SolrJ 6.6.0 Connection pool shutdown
> > > >
> > > > One thing to check is whether there is a firewall between the client
> > > > and the server. They - sometimes - cut the silent connections in the
> > > > _middle_ (at the firewall). The usual solution is keepAlive request of
> > > > some kind or not using the connection pool.
> > > >
> > > > One way to check is with network tracer like Wireshark and checking
> > > > whether the actual hardware at the other end of the connection is a
> > > > normal server or some sort of unexpected hardware piece of equipment
> > > > (firewall). Yes, that's using the hammer to swat a fly :-)
> > > >
> > > > Regards,
> > > > Alex.
> > > > ----
> > > > http://www.solr-start.com/ - Resources for Solr users, new and
> > experienced
> > > >
> > > >
> > > > On 29 June 2017 at 08:21, Markus Jelsma <ma...@openindex.io>
> > wrote:
> > > > > Hi,
> > > > >
> > > > > Everything is 6.6.0. I could include a stack trace (i don't print
> > them in my program), but that would only be the the trace from getById() to
> > CloudSolrClient.requestWithRetryOnStaleState() and little deeper, that
> > what you're looking for?
> > > > >
> > > > > We haven't called close() in that particular part of the program.
> > > > >
> > > > > Method requestWithRetryOnStaleState has some retry logic built-in
> > but doesn't seem to work for the exception i got.
> > > > >
> > > > > I'll let it print the stack trace and get back if it happens again.
> > > > >
> > > > > Thanks,
> > > > > Markus
> > > > >
> > > > > -----Original message-----
> > > > >> From:Shawn Heisey <ap...@elyograg.org>
> > > > >> Sent: Tuesday 27th June 2017 23:02
> > > > >> To: solr-user@lucene.apache.org
> > > > >> Subject: Re: SolrJ 6.6.0 Connection pool shutdown
> > > > >>
> > > > >> On 6/27/2017 6:50 AM, Markus Jelsma wrote:
> > > > >> > We have a proces checking presence of many documents in a
> > collection, just a simple client.getById(id). It sometimes begins throwing
> > lots of these exceptions in a row:
> > > > >> >
> > > > >> > org.apache.solr.client.solrj.SolrServerException: java.lang.IllegalStateException:
> > Connection pool shut down
> > > > >> >
> > > > >> > Then, as suddenly as it appeared, it's gone again a no longer a
> > problem. I would expect SolrJ not to throw this but to wait until it the
> > connection pool, or whatever mechanism is there, to recover.
> > > > >> >
> > > > >> > Did i miss a magic parameter for SolrJ?\
> > > > >>
> > > > >> That error message will be much longer than what you've provided
> > here.
> > > > >> It will have a java stacktrace that's typically a dozen or so lines
> > > > >> long. There may also be one or more "Caused by" sections after the
> > > > >> stacktrace, each with a stacktrace of its own. Can you share the
> > full
> > > > >> error message? Is the server also running 6.6.0, or a different
> > version?
> > > > >>
> > > > >> It would also be helpful if you can share the SolrJ code you've
> > written,
> > > > >> cleanly redacted to remove anything sensitive.
> > > > >>
> > > > >> That particular message ("Connection pool shut down") sounds like it
> > > > >> probably came from HttpClient, which SolrJ uses ... and I would
> > expect
> > > > >> that to only happen if you close/shutdown the HttpClient or the
> > > > >> SolrClient. After closing, a client can't be used any more.
> > Normally
> > > > >> the only time you should close a client is right before exiting the
> > > > >> program, although if the program's about to exit, it's generally
> > > > >> unnecessary, so in my opinion for *most* usages, closing the client
> > > > >> likely never needs to happen.
> > > > >>
> > > > >> Thanks,
> > > > >> Shawn
> > > > >>
> > > > >>
> > > >
> > >
> >
>
Re: SolrJ 6.6.0 Connection pool shutdown now with stack trace
Posted by Shawn Heisey <ap...@elyograg.org>.
On 7/18/2017 5:10 AM, Markus Jelsma wrote:
> The problem was never resolved but Shawn asked for the stack trace, here it is:
<snip>
> Caused by: java.lang.IllegalStateException: Connection pool shut down
> at org.apache.http.util.Asserts.check(Asserts.java:34)
As I suspected, it is the connection pool inside HttpClient that is shut
down (closed).
Earlier today before I came into the office, I asked the HttpClient user
list whether this could ever happen for a reason other than an explicit
close/shutdown. They looked at the code and found that the exception
only is thrown if the "isShutDown" boolean flag is true, and the only
place that ever gets set to true is when an explicit shutdown is called
on the connection pool.
When a solr client is built without an external HttpClient, calling
close() on the solr client will shut down the internal HttpClient. If
an external HttpClient is used, the user code would need to shut it down
for this to happen. Recent versions of SolrJ are using
CloseableHttpClient, which will shut down the connection pool if close()
is called.
It's looking like this error has happened because the HttpClient object
inside the solr client has been shut down explicitly, which might have
happened because one of the outer layers had close() called.
Thanks,
Shawn