You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@solr.apache.org by "Jude Muriithi (BLOOMBERG/ 919 3RD A)" <jm...@bloomberg.net> on 2025/02/10 19:19:16 UTC
Identifying performance issues in Solr 9.7/9.8
Greetings all,
In the process of following up on this thread in the users mailing list: https://lists.apache.org/thread/96ndn3n999dxpm71xjcwr4c6bjpd4sdy, we ran the solr-benchmark project on various local development builds of Solr. For context, this benchmark (https://github.com/pavel-chumakou/solr-benchmark) repeats a set of 20 queries over an index of selected Wikipedia articles. Running Solr locally, we saw the following results (all from a cold start):
- Solr 9.8.0: Search time, ms: 742 473 390 375 381 307 380 327 297 334 304 386 342 315 286 308 311 285 306 430
- Solr 9.7.0: Search time, ms: 767 479 398 331 314 308 289 296 336 339 324 325 286 288 294 288 307 299 399 295
- Solr 9.6.1: Search time, ms: 328 59 7 8 6 4 8 6 2 2 1 2 1 2 2 2 1 1 2 2
After observing this discrepancy, we used the JFR to profile our local builds of Solr while running the benchmark. This led us to the following discovery:
- Build of upstream commit b8410234: Search time, ms: 1026 741 704 674 624 604 841 719 630 614 642 610 629 599 626 592 617 616 626 603
- Build of the previous upstream commit (08a2bdd1): Search time, ms: 385 81 39 13 3 7 6 8 6 5 4 27 75 2 3 4 2 1 1 2
Without drawing any conclusions, our observations indicate that some change related to the above commit is causing major performance degradation. This would align with the various reports of performance degradation following the release of Solr 9.7.
All benchmarks above were run on an M3 Macbook Pro with Temurin Java 21.0.5. To execute each benchmark, we ran the following commands:
git clone https://github.com/pavel-chumakou/solr-benchmark.git
cd solr-benchmark
unzip wikitest.zip
cd apache/solr
git checkout releases/solr/9.x.x
./gradlew clean dev
cd solr/packaging/build/dev
cp -r ~/solr-benchmark/wikitest/ ./server/solr/wikitest/
bin/solr start
cd ~/solr-benchmark
java RunSolrQuery.java
Re: Identifying performance issues in Solr 9.7/9.8
Posted by Mark Miller <ma...@gmail.com>.
When we looked at the issue (I’m not looking at the code), it didn’t look
like a lack of CPU problem - it looked like, by default you would get one
thread, I suppose trying to be the same as you’d get previously by default.
Except, you’d now get Lucene’s parallel segment search code path with that
one thread instead of the standard code path. And that was slow. To get the
old behavior, you’d have to use -1 as you suggest.
And it does seem like a bug to me, and I also think the default should have
been -1 (or any fix that would still get you the standard Lucene code path
if you haven’t turned on parallel segment search)
Re: Identifying performance issues in Solr 9.7/9.8
Posted by David Smiley <ds...@gmail.com>.
Hi Jude,
I love the data-driven approach here!
Let’s be explicit about the change that brought this about: it’s multiThreaded=true https://issues.apache.org/jira/browse/SOLR-13350
Can you try setting https://solr.apache.org/guide/solr/latest/configuration-guide/configuring-solr-xml.html#indexSearcherExecutorThreads to -1 ?
multiThreaded is supposed to perform well if there’s ample CPU. For users/benchmarks with lots of concurrent load, I don’t expect this option to perform well.
It’s really a shame to see that the performance is impacted without even setting multiThreaded=true ! I think should disable this so it’s fully opt-in.
~ David
> On Feb 10, 2025, at 2:19 PM, Jude Muriithi (BLOOMBERG/ 919 3RD A) <jm...@bloomberg.net> wrote:
>
> Greetings all,
>
> In the process of following up on this thread in the users mailing list: https://lists.apache.org/thread/96ndn3n999dxpm71xjcwr4c6bjpd4sdy, we ran the solr-benchmark project on various local development builds of Solr. For context, this benchmark (https://github.com/pavel-chumakou/solr-benchmark) repeats a set of 20 queries over an index of selected Wikipedia articles. Running Solr locally, we saw the following results (all from a cold start):
>
> - Solr 9.8.0: Search time, ms: 742 473 390 375 381 307 380 327 297 334 304 386 342 315 286 308 311 285 306 430
> - Solr 9.7.0: Search time, ms: 767 479 398 331 314 308 289 296 336 339 324 325 286 288 294 288 307 299 399 295
> - Solr 9.6.1: Search time, ms: 328 59 7 8 6 4 8 6 2 2 1 2 1 2 2 2 1 1 2 2
>
> After observing this discrepancy, we used the JFR to profile our local builds of Solr while running the benchmark. This led us to the following discovery:
>
> - Build of upstream commit b8410234: Search time, ms: 1026 741 704 674 624 604 841 719 630 614 642 610 629 599 626 592 617 616 626 603
> - Build of the previous upstream commit (08a2bdd1): Search time, ms: 385 81 39 13 3 7 6 8 6 5 4 27 75 2 3 4 2 1 1 2
>
> Without drawing any conclusions, our observations indicate that some change related to the above commit is causing major performance degradation. This would align with the various reports of performance degradation following the release of Solr 9.7.
>
> All benchmarks above were run on an M3 Macbook Pro with Temurin Java 21.0.5. To execute each benchmark, we ran the following commands:
>
> git clone https://github.com/pavel-chumakou/solr-benchmark.git
> cd solr-benchmark
> unzip wikitest.zip
>
> cd apache/solr
> git checkout releases/solr/9.x.x
> ./gradlew clean dev
> cd solr/packaging/build/dev
> cp -r ~/solr-benchmark/wikitest/ ./server/solr/wikitest/
> bin/solr start
>
> cd ~/solr-benchmark
> java RunSolrQuery.java
Re: Identifying performance issues in Solr 9.7/9.8
Posted by "Luke Kot-Zaniewski (BLOOMBERG/ 919 3RD A)" <lk...@bloomberg.net>.
Joining the chorus of people looking at this benchmark.
I debated on replying on the original thread but see that this one is most up-to-date.
I can reproduce some of the behavior mentioned in this thread and the earlier one on the subject.
Most notable observations:
1. Disabling the Executor via indexSearcherExecutorThreads=-1 brings back "good" performance of 9.6
2. Enabling the Executor (default threads set to available cores) with multiThreaded=true increases latency of the benchmark by at least 4X. It's worth noting that the benchmark sends these queries one at a time so the CPU is under-utilized and any theoretical benefits of parallelization are not realized.
3. Enabling the executor (default threads set to available cores) with multiThreaded=false also increases latency of the benchmark.
Thanks David for starting the discussion on issue #3 specifically in the JIRA: https://issues.apache.org/jira/browse/SOLR-17665. I've added my own comments to this as well as some JFR recordings which I hope might be helpful in getting to the bottom of what is going on.
From: dev@solr.apache.org At: 02/11/25 15:35:47 UTC-5:00To: dev@solr.apache.org
Subject: Re: Identifying performance issues in Solr 9.7/9.8
Changing the solr.xml seems to solve the issue:
- Solr 9.8.0 (indexSearcherExecutorThreads=-1): Search time, ms: 137 5 9 10 3 3
6 2 2 2 3 0 1 2 1 1 2 2 1 5
From: dev@solr.apache.org At: 02/11/25 13:52:12 UTC-5:00To: dev@solr.apache.org
Subject: Re: Identifying performance issues in Solr 9.7/9.8
I filed a JIRA: https://issues.apache.org/jira/browse/SOLR-17665
Re: Identifying performance issues in Solr 9.7/9.8
Posted by "Jude Muriithi (BLOOMBERG/ 919 3RD A)" <jm...@bloomberg.net>.
Changing the solr.xml seems to solve the issue:
- Solr 9.8.0 (indexSearcherExecutorThreads=-1): Search time, ms: 137 5 9 10 3 3 6 2 2 2 3 0 1 2 1 1 2 2 1 5
From: dev@solr.apache.org At: 02/11/25 13:52:12 UTC-5:00To: dev@solr.apache.org
Subject: Re: Identifying performance issues in Solr 9.7/9.8
I filed a JIRA: https://issues.apache.org/jira/browse/SOLR-17665
Re: Identifying performance issues in Solr 9.7/9.8
Posted by David Smiley <ds...@gmail.com>.
I filed a JIRA: https://issues.apache.org/jira/browse/SOLR-17665
Re: Identifying performance issues in Solr 9.7/9.8
Posted by David Smiley <ds...@gmail.com>.
I filed a JIRA: https://issues.apache.org/jira/browse/SOLR-17665