You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@solr.apache.org by "Clemens WYSS (Helbling Technik)" <Cl...@helbling.ch> on 2022/05/30 06:30:28 UTC
SolrJ connection handling, best practice(s)?
We have a non clustered single (8.x) Solr server setup and use SolrJ to connect our tomcat to the Solr server. Under high load (e.g. last black friday) we face a bottleneck due to our webapp-search-requests. Calls to solr take seconds, dangling http-connecitons, ...
Firing the same ( or even way more ) search requests directly onto our Solr server showed, as expected, no problem. Therefore my implication is that SolrJ/HttpSolrClient/HttpClient is not setup as expected.
Our search API is setup alike:
private final Map<String, SolrClient> coreConnections; // one SolrClient per core
...
// create a new SolrClient for a core if not existant
HttpSolrClient.Builder builder = new HttpSolrClient.Builder( coreUrl );
solrClient = builder.build();
coreConnections.put( key, solrClient );
As you can see
* we keep one SolrClient-connection per core in a map and make use of these connections
* making use of HttpSolrClient ( Http2SolrClient is denoted "experimental" and has cause some problem )
* no special tweaking of underneath HttpClient
Any reason for this minimal setup not being able to stem many many concurrent requests? What is the proposed SolrJ based solr client setup (for a single Solr server)?
Thx
- Clemens
Re: SolrJ connection handling, best practice(s)?
Posted by Christopher Schultz <ch...@christopherschultz.net>.
Jan and Shawn,
On 5/30/22 10:01, Jan Høydahl wrote:
> To me it looks like it defaults to 10000 max connections per route <https://github.com/apache/lucene-solr/blob/branch_8_11/solr/solrj/src/java/org/apache/solr/client/solrj/impl/HttpClientUtil.java#L298> in 8.11?
> And you can override that if you provide your custom HttpClient using Builder.withHttpClient(httpClient) <https://github.com/apache/lucene-solr/blob/branch_8_11/solr/solrj/src/java/org/apache/solr/client/solrj/impl/SolrClientBuilder.java#L36>.
>
> PS: It may be possible to use latest 8.11.1 SolrJ in your app even if Solr server side is an earlier 8.x version.
I'm using SolrJClient 7.3.0 and all of that is also possible (e.g. use
your own custom HttpClient). We use it to configure TLS and HTTP
authentication whenever necessary.
We just do it like this:
So an upgrade to 8.x isn't strictly necessary. For the Http2 stuff,
maybe it would be better to upgrade the client to get any refactorings
or bug fixes in that component that have been added since previous releases.
solrClient = new HttpSolrClient.Builder(url)
.withHttpClient(getHttpClient())
.build();
Our getHttpClient method builds its own ClientConnectionManager with
whatever limits we want on it.
-chris
>> 30. mai 2022 kl. 15:06 skrev Shawn Heisey <ap...@elyograg.org>:
>>
>> On 5/30/2022 12:30 AM, Clemens WYSS (Helbling Technik) wrote:
>>> As you can see
>>> * we keep one SolrClient-connection per core in a map and make use of these connections
>>> * making use of HttpSolrClient ( Http2SolrClient is denoted "experimental" and has cause some problem )
>>> * no special tweaking of underneath HttpClient
>>>
>>> Any reason for this minimal setup not being able to stem many many concurrent requests? What is the proposed SolrJ based solr client setup (for a single Solr server)?
>>
>> Vincenzo is correct. Used properly, the solr clients are thread-safe and designed to handle concurrent connections.
>>
>> You actually only need one HttpSolrClient per Solr instance (host/port combination), not one for every core. When you make a request to Solr via a SolrClient, you can tell it what core/collection should receive the request. As long as the number of cores isn't enormous, having a client per core is not likely to cause issues. Just something for you to think about the next time you are ready to do a major code revision.
>>
>> What version of SolrJ are you using in your code? For a long time, SolrJ defaulted to an HttpClient that only allowed two threads per destination. Then that was greatly increased. Now from what I can decipher in current 8.x code using the Builder, we may be back to a default of two connections per destination, which is NOT a good thing when concurrency is expected.
>>
>> Looking at HttpSolrClientBuilder, I do not see a way to specify a custom http client. Anyone know how that can be done?
>>
>> Thanks,
>> Shawn
>>
>
>
Re: SolrJ connection handling, best practice(s)?
Posted by Jan Høydahl <ja...@cominvent.com>.
To me it looks like it defaults to 10000 max connections per route <https://github.com/apache/lucene-solr/blob/branch_8_11/solr/solrj/src/java/org/apache/solr/client/solrj/impl/HttpClientUtil.java#L298> in 8.11?
And you can override that if you provide your custom HttpClient using Builder.withHttpClient(httpClient) <https://github.com/apache/lucene-solr/blob/branch_8_11/solr/solrj/src/java/org/apache/solr/client/solrj/impl/SolrClientBuilder.java#L36>.
PS: It may be possible to use latest 8.11.1 SolrJ in your app even if Solr server side is an earlier 8.x version.
Jan
> 30. mai 2022 kl. 15:06 skrev Shawn Heisey <ap...@elyograg.org>:
>
> On 5/30/2022 12:30 AM, Clemens WYSS (Helbling Technik) wrote:
>> As you can see
>> * we keep one SolrClient-connection per core in a map and make use of these connections
>> * making use of HttpSolrClient ( Http2SolrClient is denoted "experimental" and has cause some problem )
>> * no special tweaking of underneath HttpClient
>>
>> Any reason for this minimal setup not being able to stem many many concurrent requests? What is the proposed SolrJ based solr client setup (for a single Solr server)?
>
> Vincenzo is correct. Used properly, the solr clients are thread-safe and designed to handle concurrent connections.
>
> You actually only need one HttpSolrClient per Solr instance (host/port combination), not one for every core. When you make a request to Solr via a SolrClient, you can tell it what core/collection should receive the request. As long as the number of cores isn't enormous, having a client per core is not likely to cause issues. Just something for you to think about the next time you are ready to do a major code revision.
>
> What version of SolrJ are you using in your code? For a long time, SolrJ defaulted to an HttpClient that only allowed two threads per destination. Then that was greatly increased. Now from what I can decipher in current 8.x code using the Builder, we may be back to a default of two connections per destination, which is NOT a good thing when concurrency is expected.
>
> Looking at HttpSolrClientBuilder, I do not see a way to specify a custom http client. Anyone know how that can be done?
>
> Thanks,
> Shawn
>
Re: SolrJ connection handling, best practice(s)?
Posted by Christopher Schultz <ch...@christopherschultz.net>.
Shawn,
On 5/31/22 12:27, Shawn Heisey wrote:
> On 5/31/2022 9:36 AM, Christopher Schultz wrote:
>> Thread pool or connection pool? Usually, you want your application
>> threads to be waiting on that data, not handing execution off to
>> another thread, only to wait for it to complete. Any asynchronous
>> environment should be providing its own threads. I would find it very
>> surprising if the Solr client provided a thread pool internally.
>
> HttpSolrClient uses Apache httpclient, version 4.5. Http2SolrClient uses
> the Jetty http client.
Aha that's a pretty important change. I stated elsewhere in this thread
that I was manually-specifying the (Apache) HttpClient when constructing
SolrJClient instances; apologies if I confused things if Jetty's
HttpClient and the Http2SolrClient work differently.
> The Apache httpclient defaults to only allowing two simultaneous
> connections. At one point, HttpSolrClient was improved to build a
> client that allows more connections, but it looks like that got lost in
> the conversion to the Builder paradigm, so now we are back to only
> having two by default.
>
> This code is in the method that actually creates Http2SolrClient:
>
> ThreadPoolExecutor httpClientExecutor = new
> ExecutorUtil.MDCAwareThreadPoolExecutor(32,
> 256, 60, TimeUnit.SECONDS, queue, new
> SolrNamedThreadFactory("h2sc"));
>
> And then that executor is used in the actual httpClient creation.
Humph. Okay. :)
> I'm looking at the createHttpClient method in Http2SolrClient.java in
> branch_8_11 from the lucene-solr repo.
As we start looking at upgrading I'm sure I'll become much more familiar
with the New Stuff :)
Thanks,
-chris
Re: SolrJ connection handling, best practice(s)?
Posted by Shawn Heisey <el...@elyograg.org>.
On 5/31/2022 9:36 AM, Christopher Schultz wrote:
> Thread pool or connection pool? Usually, you want your application
> threads to be waiting on that data, not handing execution off to
> another thread, only to wait for it to complete. Any asynchronous
> environment should be providing its own threads. I would find it very
> surprising if the Solr client provided a thread pool internally.
HttpSolrClient uses Apache httpclient, version 4.5. Http2SolrClient uses
the Jetty http client.
The Apache httpclient defaults to only allowing two simultaneous
connections. At one point, HttpSolrClient was improved to build a
client that allows more connections, but it looks like that got lost in
the conversion to the Builder paradigm, so now we are back to only
having two by default.
This code is in the method that actually creates Http2SolrClient:
ThreadPoolExecutor httpClientExecutor = new
ExecutorUtil.MDCAwareThreadPoolExecutor(32,
256, 60, TimeUnit.SECONDS, queue, new
SolrNamedThreadFactory("h2sc"));
And then that executor is used in the actual httpClient creation.
I'm looking at the createHttpClient method in Http2SolrClient.java in
branch_8_11 from the lucene-solr repo.
Thanks,
Shawn
Re: SolrJ connection handling, best practice(s)?
Posted by Christopher Schultz <ch...@christopherschultz.net>.
Shawn,
On 5/30/22 10:05, Shawn Heisey wrote:
> On 5/30/2022 7:06 AM, Shawn Heisey wrote:
>> What version of SolrJ are you using in your code? For a long time,
>> SolrJ defaulted to an HttpClient that only allowed two threads per
>> destination. Then that was greatly increased. Now from what I can
>> decipher in current 8.x code using the Builder, we may be back to a
>> default of two connections per destination, which is NOT a good thing
>> when concurrency is expected.
>
> From what I can decipher, Http2SolrClient may default to a max thread
> pool size of 256.
Thread pool or connection pool? Usually, you want your application
threads to be waiting on that data, not handing execution off to another
thread, only to wait for it to complete. Any asynchronous environment
should be providing its own threads. I would find it very surprising if
the Solr client provided a thread pool internally.
> That might be a low number for HIGHLY concurrent
> systems, but it should be a lot better than HttpSolrClient. And I see
> that others have recommended using it. It also looks like the Builder
> for that client lets you easily set the maximum number of connections
> per host. From what I can tell, if you allow it to use http/2 it will
> not honor max connections per host, but I do not think that's a problem,
> as http/2 multiplexes many requests over a single connection. A
> well-optimized http/2 setup really only needs one TCP connection for all
> concurrent activity.
Yes. An h2 server can limit the number of simultaneous requests per
connection, but theoretically its quite high.
> I do think that the http2 client builder should allow setting the max
> size of the thread pool. But maybe its defaults will serve you well
> until that happens.
Still seems weird to me.
-chris
AW: SolrJ connection handling, best practice(s)?
Posted by "Clemens WYSS (Helbling Technik)" <Cl...@helbling.ch>.
Thanks for all your advices! We will track this further down by either switching to 8.11/9 and/or switch to Http2SolrClient or (as a first try) apply Jan's approach of injecting a "preconfigured httpClient".
Final solution will be 9.0 using Http2SolrClient (with, if necessary, additional connections 😉)
Regards
-Clemens
-----Ursprüngliche Nachricht-----
Von: Shawn Heisey <ap...@elyograg.org>
Gesendet: Montag, 30. Mai 2022 16:06
An: users@solr.apache.org
Betreff: Re: SolrJ connection handling, best practice(s)?
[Sie erhalten nicht oft eine E-Mail von apache@elyograg.org. Weitere Informationen dazu, warum dies wichtig ist, finden Sie unter https://aka.ms/LearnAboutSenderIdentification.]
On 5/30/2022 7:06 AM, Shawn Heisey wrote:
> What version of SolrJ are you using in your code? For a long time,
> SolrJ defaulted to an HttpClient that only allowed two threads per
> destination. Then that was greatly increased. Now from what I can
> decipher in current 8.x code using the Builder, we may be back to a
> default of two connections per destination, which is NOT a good thing
> when concurrency is expected.
From what I can decipher, Http2SolrClient may default to a max thread pool size of 256. That might be a low number for HIGHLY concurrent systems, but it should be a lot better than HttpSolrClient. And I see that others have recommended using it. It also looks like the Builder for that client lets you easily set the maximum number of connections per host. From what I can tell, if you allow it to use http/2 it will not honor max connections per host, but I do not think that's a problem, as http/2 multiplexes many requests over a single connection. A well-optimized http/2 setup really only needs one TCP connection for all concurrent activity.
I do think that the http2 client builder should allow setting the max size of the thread pool. But maybe its defaults will serve you well until that happens.
Thanks,
Shawn
Re: SolrJ connection handling, best practice(s)?
Posted by Shawn Heisey <ap...@elyograg.org>.
On 5/30/2022 7:06 AM, Shawn Heisey wrote:
> What version of SolrJ are you using in your code? For a long time,
> SolrJ defaulted to an HttpClient that only allowed two threads per
> destination. Then that was greatly increased. Now from what I can
> decipher in current 8.x code using the Builder, we may be back to a
> default of two connections per destination, which is NOT a good thing
> when concurrency is expected.
From what I can decipher, Http2SolrClient may default to a max thread
pool size of 256. That might be a low number for HIGHLY concurrent
systems, but it should be a lot better than HttpSolrClient. And I see
that others have recommended using it. It also looks like the Builder
for that client lets you easily set the maximum number of connections
per host. From what I can tell, if you allow it to use http/2 it will
not honor max connections per host, but I do not think that's a problem,
as http/2 multiplexes many requests over a single connection. A
well-optimized http/2 setup really only needs one TCP connection for all
concurrent activity.
I do think that the http2 client builder should allow setting the max
size of the thread pool. But maybe its defaults will serve you well
until that happens.
Thanks,
Shawn
Re: SolrJ connection handling, best practice(s)?
Posted by Shawn Heisey <ap...@elyograg.org>.
On 5/30/2022 12:30 AM, Clemens WYSS (Helbling Technik) wrote:
> As you can see
> * we keep one SolrClient-connection per core in a map and make use of these connections
> * making use of HttpSolrClient ( Http2SolrClient is denoted "experimental" and has cause some problem )
> * no special tweaking of underneath HttpClient
>
> Any reason for this minimal setup not being able to stem many many concurrent requests? What is the proposed SolrJ based solr client setup (for a single Solr server)?
Vincenzo is correct. Used properly, the solr clients are thread-safe
and designed to handle concurrent connections.
You actually only need one HttpSolrClient per Solr instance (host/port
combination), not one for every core. When you make a request to Solr
via a SolrClient, you can tell it what core/collection should receive
the request. As long as the number of cores isn't enormous, having a
client per core is not likely to cause issues. Just something for you
to think about the next time you are ready to do a major code revision.
What version of SolrJ are you using in your code? For a long time,
SolrJ defaulted to an HttpClient that only allowed two threads per
destination. Then that was greatly increased. Now from what I can
decipher in current 8.x code using the Builder, we may be back to a
default of two connections per destination, which is NOT a good thing
when concurrency is expected.
Looking at HttpSolrClientBuilder, I do not see a way to specify a custom
http client. Anyone know how that can be done?
Thanks,
Shawn
Re: SolrJ connection handling, best practice(s)?
Posted by Deepak Goel <de...@gmail.com>.
Could you please share your Tomcat web server configuration?
Deepak
"The greatness of a nation can be judged by the way its animals are treated
- Mahatma Gandhi"
+91 73500 12833
deicool@gmail.com
Facebook: https://www.facebook.com/deicool
LinkedIn: www.linkedin.com/in/deicool
"Plant a Tree, Go Green"
Make In India : http://www.makeinindia.com/home
On Mon, May 30, 2022 at 4:56 PM Clemens WYSS (Helbling Technik) <
Clemens.Wyss@helbling.ch> wrote:
> > Are the JVM parameters good enough?
> Non search-related http requests run smoothly as long as not all AJPs are
> blocked waiting for Solr-resonses 😉
>
> -----Ursprüngliche Nachricht-----
> Von: Deepak Goel <de...@gmail.com>
> Gesendet: Montag, 30. Mai 2022 13:16
> An: users@solr.apache.org
> Betreff: Re: SolrJ connection handling, best practice(s)?
>
> Could you please share your Tomcat web server configuration? Are the JVM
> parameters good enough?
>
> Deepak
> "The greatness of a nation can be judged by the way its animals are treated
> - Mahatma Gandhi"
>
> +91 73500 12833
> deicool@gmail.com
>
> Facebook:
> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.facebook.com%2Fdeicool&data=05%7C01%7CClemens.Wyss%40helbling.ch%7Ccaf45aa7c3294dbc7b3708da422dd127%7Cbcbab93b0f0d4747b4a4bc57e6753bb4%7C0%7C0%7C637895061793187418%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=DMh0j1aYYos3wMw9Dcgfh46yHIU5HYjklrmX4TSFPt0%3D&reserved=0
> LinkedIn:
> https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.linkedin.com%2Fin%2Fdeicool&data=05%7C01%7CClemens.Wyss%40helbling.ch%7Ccaf45aa7c3294dbc7b3708da422dd127%7Cbcbab93b0f0d4747b4a4bc57e6753bb4%7C0%7C0%7C637895061793187418%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=xoALkzrXnivcwVxZdBx3pTXCc%2BJShGEkeSSRvlj%2BU3M%3D&reserved=0
>
> "Plant a Tree, Go Green"
>
> Make In India :
> https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.makeinindia.com%2Fhome&data=05%7C01%7CClemens.Wyss%40helbling.ch%7Ccaf45aa7c3294dbc7b3708da422dd127%7Cbcbab93b0f0d4747b4a4bc57e6753bb4%7C0%7C0%7C637895061793187418%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=xLgg5Q8i85NNCe7M2sC7Bd9y8C1MhuGubxt2jAaOg0U%3D&reserved=0
>
>
> On Mon, May 30, 2022 at 4:41 PM Clemens WYSS (Helbling Technik) <
> Clemens.Wyss@helbling.ch> wrote:
>
> > Ciao Vincenzo,
> > > Why do you think that this problem is caused by the wrong usage of
> > >the clients?
> > * during the "peak" the solr server itself was not really "under
> pressure"
> > * stressing the solr server with k6 showed that the Solr-Server(setup)
> > can handle the amount of expected concurrent requests
> >
> > So to me it's either within tomcat, i.e. the HttpClient "connection
> > handling" (e.g. MaxConnections, MaxConnectionsPerHost, ...), or the
> > "tomcat host server setup" (limit of outgoing connections, ephemeral
> > ports, ulimit?)?
> >
> > https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fhc.a
> > pache.org%2Fhttpclient-legacy%2Fperformance.html&data=05%7C01%7CCl
> > emens.Wyss%40helbling.ch%7Ccaf45aa7c3294dbc7b3708da422dd127%7Cbcbab93b
> > 0f0d4747b4a4bc57e6753bb4%7C0%7C0%7C637895061793187418%7CUnknown%7CTWFp
> > bGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn
> > 0%3D%7C3000%7C%7C%7C&sdata=LAbOw8pr0D7Ut2kVm%2F%2FZMjXIrcQMS3toLZ5
> > 4tTHNKcI%3D&reserved=0 Does HttpSolrClient make use of
> > MultiThreadedHttpConnectionManager?
> >
> > https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fhc.a
> > pache.org%2Fhttpclient-legacy%2Fthreading.html&data=05%7C01%7CClem
> > ens.Wyss%40helbling.ch%7Ccaf45aa7c3294dbc7b3708da422dd127%7Cbcbab93b0f
> > 0d4747b4a4bc57e6753bb4%7C0%7C0%7C637895061793187418%7CUnknown%7CTWFpbG
> > Zsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%
> > 3D%7C3000%7C%7C%7C&sdata=iTNWeMzrY2jHzUS6ZXCvXBl9zanMTJ4jLpRXuDBEz
> > LY%3D&reserved=0 Other things HttpSolrClient may need to consider
> >
> > Thx
> > - Clemens
> > -----Ursprüngliche Nachricht-----
> > Von: Vincenzo D'Amore <v....@gmail.com>
> > Gesendet: Montag, 30. Mai 2022 09:12
> > An: users@solr.apache.org
> > Betreff: Re: SolrJ connection handling, best practice(s)?
> >
> > Hi Clemens, Solrj clients are thread-safe singletons, from my
> > experience the strategy you are using works well even under high load,
> > when calls to Solr take seconds to answer usually it happens because
> > of the number of the queries, the kind of queries (grouping or too
> > many facets) or even for garbage collection problems.
> > Why do you think that this problem is caused by the wrong usage of the
> > clients?
> >
> > On Mon, May 30, 2022 at 8:30 AM Clemens WYSS (Helbling Technik) <
> > Clemens.Wyss@helbling.ch> wrote:
> >
> > > We have a non clustered single (8.x) Solr server setup and use SolrJ
> > > to connect our tomcat to the Solr server. Under high load (e.g. last
> > > black
> > > friday) we face a bottleneck due to our webapp-search-requests.
> > > Calls to solr take seconds, dangling http-connecitons, ...
> > > Firing the same ( or even way more ) search requests directly onto
> > > our Solr server showed, as expected, no problem. Therefore my
> > > implication is that SolrJ/HttpSolrClient/HttpClient is not setup as
> expected.
> > > Our search API is setup alike:
> > >
> > > private final Map<String, SolrClient> coreConnections; // one
> > > SolrClient per core ...
> > >
> > > // create a new SolrClient for a core if not existant
> > > HttpSolrClient.Builder builder = new HttpSolrClient.Builder( coreUrl
> > > ); solrClient = builder.build(); coreConnections.put( key,
> > > solrClient );
> > >
> > > As you can see
> > > * we keep one SolrClient-connection per core in a map and make use
> > > of these connections
> > > * making use of HttpSolrClient ( Http2SolrClient is denoted
> > "experimental"
> > > and has cause some problem )
> > > * no special tweaking of underneath HttpClient
> > >
> > > Any reason for this minimal setup not being able to stem many many
> > > concurrent requests? What is the proposed SolrJ based solr client
> > > setup (for a single Solr server)?
> > >
> > > Thx
> > > - Clemens
> > >
> >
> >
> > --
> > Vincenzo D'Amore
> >
>
AW: SolrJ connection handling, best practice(s)?
Posted by "Clemens WYSS (Helbling Technik)" <Cl...@helbling.ch>.
> Are the JVM parameters good enough?
Non search-related http requests run smoothly as long as not all AJPs are blocked waiting for Solr-resonses 😉
-----Ursprüngliche Nachricht-----
Von: Deepak Goel <de...@gmail.com>
Gesendet: Montag, 30. Mai 2022 13:16
An: users@solr.apache.org
Betreff: Re: SolrJ connection handling, best practice(s)?
Could you please share your Tomcat web server configuration? Are the JVM parameters good enough?
Deepak
"The greatness of a nation can be judged by the way its animals are treated
- Mahatma Gandhi"
+91 73500 12833
deicool@gmail.com
Facebook: https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.facebook.com%2Fdeicool&data=05%7C01%7CClemens.Wyss%40helbling.ch%7Ccaf45aa7c3294dbc7b3708da422dd127%7Cbcbab93b0f0d4747b4a4bc57e6753bb4%7C0%7C0%7C637895061793187418%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=DMh0j1aYYos3wMw9Dcgfh46yHIU5HYjklrmX4TSFPt0%3D&reserved=0
LinkedIn: https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.linkedin.com%2Fin%2Fdeicool&data=05%7C01%7CClemens.Wyss%40helbling.ch%7Ccaf45aa7c3294dbc7b3708da422dd127%7Cbcbab93b0f0d4747b4a4bc57e6753bb4%7C0%7C0%7C637895061793187418%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=xoALkzrXnivcwVxZdBx3pTXCc%2BJShGEkeSSRvlj%2BU3M%3D&reserved=0
"Plant a Tree, Go Green"
Make In India : https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.makeinindia.com%2Fhome&data=05%7C01%7CClemens.Wyss%40helbling.ch%7Ccaf45aa7c3294dbc7b3708da422dd127%7Cbcbab93b0f0d4747b4a4bc57e6753bb4%7C0%7C0%7C637895061793187418%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=xLgg5Q8i85NNCe7M2sC7Bd9y8C1MhuGubxt2jAaOg0U%3D&reserved=0
On Mon, May 30, 2022 at 4:41 PM Clemens WYSS (Helbling Technik) < Clemens.Wyss@helbling.ch> wrote:
> Ciao Vincenzo,
> > Why do you think that this problem is caused by the wrong usage of
> >the clients?
> * during the "peak" the solr server itself was not really "under pressure"
> * stressing the solr server with k6 showed that the Solr-Server(setup)
> can handle the amount of expected concurrent requests
>
> So to me it's either within tomcat, i.e. the HttpClient "connection
> handling" (e.g. MaxConnections, MaxConnectionsPerHost, ...), or the
> "tomcat host server setup" (limit of outgoing connections, ephemeral
> ports, ulimit?)?
>
> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fhc.a
> pache.org%2Fhttpclient-legacy%2Fperformance.html&data=05%7C01%7CCl
> emens.Wyss%40helbling.ch%7Ccaf45aa7c3294dbc7b3708da422dd127%7Cbcbab93b
> 0f0d4747b4a4bc57e6753bb4%7C0%7C0%7C637895061793187418%7CUnknown%7CTWFp
> bGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn
> 0%3D%7C3000%7C%7C%7C&sdata=LAbOw8pr0D7Ut2kVm%2F%2FZMjXIrcQMS3toLZ5
> 4tTHNKcI%3D&reserved=0 Does HttpSolrClient make use of
> MultiThreadedHttpConnectionManager?
>
> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fhc.a
> pache.org%2Fhttpclient-legacy%2Fthreading.html&data=05%7C01%7CClem
> ens.Wyss%40helbling.ch%7Ccaf45aa7c3294dbc7b3708da422dd127%7Cbcbab93b0f
> 0d4747b4a4bc57e6753bb4%7C0%7C0%7C637895061793187418%7CUnknown%7CTWFpbG
> Zsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%
> 3D%7C3000%7C%7C%7C&sdata=iTNWeMzrY2jHzUS6ZXCvXBl9zanMTJ4jLpRXuDBEz
> LY%3D&reserved=0 Other things HttpSolrClient may need to consider
>
> Thx
> - Clemens
> -----Ursprüngliche Nachricht-----
> Von: Vincenzo D'Amore <v....@gmail.com>
> Gesendet: Montag, 30. Mai 2022 09:12
> An: users@solr.apache.org
> Betreff: Re: SolrJ connection handling, best practice(s)?
>
> Hi Clemens, Solrj clients are thread-safe singletons, from my
> experience the strategy you are using works well even under high load,
> when calls to Solr take seconds to answer usually it happens because
> of the number of the queries, the kind of queries (grouping or too
> many facets) or even for garbage collection problems.
> Why do you think that this problem is caused by the wrong usage of the
> clients?
>
> On Mon, May 30, 2022 at 8:30 AM Clemens WYSS (Helbling Technik) <
> Clemens.Wyss@helbling.ch> wrote:
>
> > We have a non clustered single (8.x) Solr server setup and use SolrJ
> > to connect our tomcat to the Solr server. Under high load (e.g. last
> > black
> > friday) we face a bottleneck due to our webapp-search-requests.
> > Calls to solr take seconds, dangling http-connecitons, ...
> > Firing the same ( or even way more ) search requests directly onto
> > our Solr server showed, as expected, no problem. Therefore my
> > implication is that SolrJ/HttpSolrClient/HttpClient is not setup as expected.
> > Our search API is setup alike:
> >
> > private final Map<String, SolrClient> coreConnections; // one
> > SolrClient per core ...
> >
> > // create a new SolrClient for a core if not existant
> > HttpSolrClient.Builder builder = new HttpSolrClient.Builder( coreUrl
> > ); solrClient = builder.build(); coreConnections.put( key,
> > solrClient );
> >
> > As you can see
> > * we keep one SolrClient-connection per core in a map and make use
> > of these connections
> > * making use of HttpSolrClient ( Http2SolrClient is denoted
> "experimental"
> > and has cause some problem )
> > * no special tweaking of underneath HttpClient
> >
> > Any reason for this minimal setup not being able to stem many many
> > concurrent requests? What is the proposed SolrJ based solr client
> > setup (for a single Solr server)?
> >
> > Thx
> > - Clemens
> >
>
>
> --
> Vincenzo D'Amore
>
Re: SolrJ connection handling, best practice(s)?
Posted by Deepak Goel <de...@gmail.com>.
Could you please share your Tomcat web server configuration? Are the JVM
parameters good enough?
Deepak
"The greatness of a nation can be judged by the way its animals are treated
- Mahatma Gandhi"
+91 73500 12833
deicool@gmail.com
Facebook: https://www.facebook.com/deicool
LinkedIn: www.linkedin.com/in/deicool
"Plant a Tree, Go Green"
Make In India : http://www.makeinindia.com/home
On Mon, May 30, 2022 at 4:41 PM Clemens WYSS (Helbling Technik) <
Clemens.Wyss@helbling.ch> wrote:
> Ciao Vincenzo,
> > Why do you think that this problem is caused
> >by the wrong usage of the clients?
> * during the "peak" the solr server itself was not really "under pressure"
> * stressing the solr server with k6 showed that the Solr-Server(setup) can
> handle the amount of expected concurrent requests
>
> So to me it's either within tomcat, i.e. the HttpClient "connection
> handling" (e.g. MaxConnections, MaxConnectionsPerHost, ...), or the "tomcat
> host server setup" (limit of outgoing connections, ephemeral ports,
> ulimit?)?
>
> https://hc.apache.org/httpclient-legacy/performance.html
> Does HttpSolrClient make use of MultiThreadedHttpConnectionManager?
>
> https://hc.apache.org/httpclient-legacy/threading.html
> Other things HttpSolrClient may need to consider
>
> Thx
> - Clemens
> -----Ursprüngliche Nachricht-----
> Von: Vincenzo D'Amore <v....@gmail.com>
> Gesendet: Montag, 30. Mai 2022 09:12
> An: users@solr.apache.org
> Betreff: Re: SolrJ connection handling, best practice(s)?
>
> Hi Clemens, Solrj clients are thread-safe singletons, from my experience
> the strategy you are using works well even under high load, when calls to
> Solr take seconds to answer usually it happens because of the number of the
> queries, the kind of queries (grouping or too many facets) or even for
> garbage collection problems.
> Why do you think that this problem is caused by the wrong usage of the
> clients?
>
> On Mon, May 30, 2022 at 8:30 AM Clemens WYSS (Helbling Technik) <
> Clemens.Wyss@helbling.ch> wrote:
>
> > We have a non clustered single (8.x) Solr server setup and use SolrJ
> > to connect our tomcat to the Solr server. Under high load (e.g. last
> > black
> > friday) we face a bottleneck due to our webapp-search-requests. Calls
> > to solr take seconds, dangling http-connecitons, ...
> > Firing the same ( or even way more ) search requests directly onto our
> > Solr server showed, as expected, no problem. Therefore my implication
> > is that SolrJ/HttpSolrClient/HttpClient is not setup as expected.
> > Our search API is setup alike:
> >
> > private final Map<String, SolrClient> coreConnections; // one
> > SolrClient per core ...
> >
> > // create a new SolrClient for a core if not existant
> > HttpSolrClient.Builder builder = new HttpSolrClient.Builder( coreUrl
> > ); solrClient = builder.build(); coreConnections.put( key, solrClient
> > );
> >
> > As you can see
> > * we keep one SolrClient-connection per core in a map and make use of
> > these connections
> > * making use of HttpSolrClient ( Http2SolrClient is denoted
> "experimental"
> > and has cause some problem )
> > * no special tweaking of underneath HttpClient
> >
> > Any reason for this minimal setup not being able to stem many many
> > concurrent requests? What is the proposed SolrJ based solr client
> > setup (for a single Solr server)?
> >
> > Thx
> > - Clemens
> >
>
>
> --
> Vincenzo D'Amore
>
AW: SolrJ connection handling, best practice(s)?
Posted by "Clemens WYSS (Helbling Technik)" <Cl...@helbling.ch>.
Ciao Vincenzo,
> Why do you think that this problem is caused
>by the wrong usage of the clients?
* during the "peak" the solr server itself was not really "under pressure"
* stressing the solr server with k6 showed that the Solr-Server(setup) can handle the amount of expected concurrent requests
So to me it's either within tomcat, i.e. the HttpClient "connection handling" (e.g. MaxConnections, MaxConnectionsPerHost, ...), or the "tomcat host server setup" (limit of outgoing connections, ephemeral ports, ulimit?)?
https://hc.apache.org/httpclient-legacy/performance.html
Does HttpSolrClient make use of MultiThreadedHttpConnectionManager?
https://hc.apache.org/httpclient-legacy/threading.html
Other things HttpSolrClient may need to consider
Thx
- Clemens
-----Ursprüngliche Nachricht-----
Von: Vincenzo D'Amore <v....@gmail.com>
Gesendet: Montag, 30. Mai 2022 09:12
An: users@solr.apache.org
Betreff: Re: SolrJ connection handling, best practice(s)?
Hi Clemens, Solrj clients are thread-safe singletons, from my experience the strategy you are using works well even under high load, when calls to Solr take seconds to answer usually it happens because of the number of the queries, the kind of queries (grouping or too many facets) or even for garbage collection problems.
Why do you think that this problem is caused by the wrong usage of the clients?
On Mon, May 30, 2022 at 8:30 AM Clemens WYSS (Helbling Technik) < Clemens.Wyss@helbling.ch> wrote:
> We have a non clustered single (8.x) Solr server setup and use SolrJ
> to connect our tomcat to the Solr server. Under high load (e.g. last
> black
> friday) we face a bottleneck due to our webapp-search-requests. Calls
> to solr take seconds, dangling http-connecitons, ...
> Firing the same ( or even way more ) search requests directly onto our
> Solr server showed, as expected, no problem. Therefore my implication
> is that SolrJ/HttpSolrClient/HttpClient is not setup as expected.
> Our search API is setup alike:
>
> private final Map<String, SolrClient> coreConnections; // one
> SolrClient per core ...
>
> // create a new SolrClient for a core if not existant
> HttpSolrClient.Builder builder = new HttpSolrClient.Builder( coreUrl
> ); solrClient = builder.build(); coreConnections.put( key, solrClient
> );
>
> As you can see
> * we keep one SolrClient-connection per core in a map and make use of
> these connections
> * making use of HttpSolrClient ( Http2SolrClient is denoted "experimental"
> and has cause some problem )
> * no special tweaking of underneath HttpClient
>
> Any reason for this minimal setup not being able to stem many many
> concurrent requests? What is the proposed SolrJ based solr client
> setup (for a single Solr server)?
>
> Thx
> - Clemens
>
--
Vincenzo D'Amore
Re: SolrJ connection handling, best practice(s)?
Posted by Vincenzo D'Amore <v....@gmail.com>.
Hi Clemens, Solrj clients are thread-safe singletons, from my experience
the strategy you are using works well even under high load, when calls to
Solr take seconds to answer usually it happens because of the number of the
queries, the kind of queries (grouping or too many facets) or even for
garbage collection problems.
Why do you think that this problem is caused by the wrong usage of the
clients?
On Mon, May 30, 2022 at 8:30 AM Clemens WYSS (Helbling Technik) <
Clemens.Wyss@helbling.ch> wrote:
> We have a non clustered single (8.x) Solr server setup and use SolrJ to
> connect our tomcat to the Solr server. Under high load (e.g. last black
> friday) we face a bottleneck due to our webapp-search-requests. Calls to
> solr take seconds, dangling http-connecitons, ...
> Firing the same ( or even way more ) search requests directly onto our
> Solr server showed, as expected, no problem. Therefore my implication is
> that SolrJ/HttpSolrClient/HttpClient is not setup as expected.
> Our search API is setup alike:
>
> private final Map<String, SolrClient> coreConnections; // one SolrClient
> per core
> ...
>
> // create a new SolrClient for a core if not existant
> HttpSolrClient.Builder builder = new HttpSolrClient.Builder( coreUrl );
> solrClient = builder.build();
> coreConnections.put( key, solrClient );
>
> As you can see
> * we keep one SolrClient-connection per core in a map and make use of
> these connections
> * making use of HttpSolrClient ( Http2SolrClient is denoted "experimental"
> and has cause some problem )
> * no special tweaking of underneath HttpClient
>
> Any reason for this minimal setup not being able to stem many many
> concurrent requests? What is the proposed SolrJ based solr client setup
> (for a single Solr server)?
>
> Thx
> - Clemens
>
--
Vincenzo D'Amore
AW: SolrJ connection handling, best practice(s)?
Posted by "Clemens WYSS (Helbling Technik)" <Cl...@helbling.ch>.
Hej Jan,
> later 8.x versions, Http2SolrClient is no longer experimental
I have read that. As of now HttpSolrClient is denoted deprecated and the suggested HttpSolr2Client is experimental 😉
> hard to diagnose
indeed
> profile your Tomcat webapp
agree. Nevertheless I wanted to ask upfront if there are any "obvious" things to consider before building up a profiling environment. Even when the profiling env is setup we should ideally have parameters we can tweak to see their impact
- Clemens
-----Ursprüngliche Nachricht-----
Von: Jan Høydahl <ja...@cominvent.com>
Gesendet: Montag, 30. Mai 2022 11:56
An: users@solr.apache.org
Betreff: Re: SolrJ connection handling, best practice(s)?
Hi,
On later 8.x versions, Http2SolrClient is no longer experimental and would be the preferred one since it uses a HTTP2 connections.
Such high-load bottlencks may be hard to diagnose, so you'd likely need to profile your Tomcat webapp and look for signs of what is happening, such as timeouts, threadpool starvation etc.
Jan
> 30. mai 2022 kl. 08:30 skrev Clemens WYSS (Helbling Technik) <Cl...@helbling.ch>:
>
> We have a non clustered single (8.x) Solr server setup and use SolrJ to connect our tomcat to the Solr server. Under high load (e.g. last black friday) we face a bottleneck due to our webapp-search-requests. Calls to solr take seconds, dangling http-connecitons, ...
> Firing the same ( or even way more ) search requests directly onto our Solr server showed, as expected, no problem. Therefore my implication is that SolrJ/HttpSolrClient/HttpClient is not setup as expected.
> Our search API is setup alike:
>
> private final Map<String, SolrClient> coreConnections; // one
> SolrClient per core ...
>
> // create a new SolrClient for a core if not existant
> HttpSolrClient.Builder builder = new HttpSolrClient.Builder( coreUrl
> ); solrClient = builder.build(); coreConnections.put( key, solrClient
> );
>
> As you can see
> * we keep one SolrClient-connection per core in a map and make use of
> these connections
> * making use of HttpSolrClient ( Http2SolrClient is denoted
> "experimental" and has cause some problem )
> * no special tweaking of underneath HttpClient
>
> Any reason for this minimal setup not being able to stem many many concurrent requests? What is the proposed SolrJ based solr client setup (for a single Solr server)?
>
> Thx
> - Clemens
Re: SolrJ connection handling, best practice(s)?
Posted by Jan Høydahl <ja...@cominvent.com>.
Hi,
On later 8.x versions, Http2SolrClient is no longer experimental and would be the preferred one since it uses a HTTP2 connections.
Such high-load bottlencks may be hard to diagnose, so you'd likely need to profile your Tomcat webapp and look for signs of what is happening, such as timeouts, threadpool starvation etc.
Jan
> 30. mai 2022 kl. 08:30 skrev Clemens WYSS (Helbling Technik) <Cl...@helbling.ch>:
>
> We have a non clustered single (8.x) Solr server setup and use SolrJ to connect our tomcat to the Solr server. Under high load (e.g. last black friday) we face a bottleneck due to our webapp-search-requests. Calls to solr take seconds, dangling http-connecitons, ...
> Firing the same ( or even way more ) search requests directly onto our Solr server showed, as expected, no problem. Therefore my implication is that SolrJ/HttpSolrClient/HttpClient is not setup as expected.
> Our search API is setup alike:
>
> private final Map<String, SolrClient> coreConnections; // one SolrClient per core
> ...
>
> // create a new SolrClient for a core if not existant
> HttpSolrClient.Builder builder = new HttpSolrClient.Builder( coreUrl );
> solrClient = builder.build();
> coreConnections.put( key, solrClient );
>
> As you can see
> * we keep one SolrClient-connection per core in a map and make use of these connections
> * making use of HttpSolrClient ( Http2SolrClient is denoted "experimental" and has cause some problem )
> * no special tweaking of underneath HttpClient
>
> Any reason for this minimal setup not being able to stem many many concurrent requests? What is the proposed SolrJ based solr client setup (for a single Solr server)?
>
> Thx
> - Clemens