You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@solr.apache.org by Sidharth Negi <si...@gmail.com> on 2022/08/26 08:55:38 UTC
Solr 6 vs Solr 8 considerable performance gap
Hi Solr Community,
We observed a very counterintuitive phenomenon.
We set up Solr 6 and Solr 8 on two identical AWS instances (16 cores, 128
GB of which Solr was given Xmx=50GB) and indexed the same data on them and
tested under the same load of traffic. The schema and solrconfig.xml are
exactly identical - the schema file is just renamed as managed-schema in
Solr 8. None of the two machines are indexing data or taking replication
and both have about equal number of segments (42 and 45 segments for Solr 6
and Solr 8 respectively)
What's surprising is that Solr 6.6.1 CPU usage is considerably lower than
Solr 8.11.2. Just look at the screenshot attached. The blue line is Solr
8.11.2 while the orange one is Solr 6.6.1. Note that the Solr 8 CPU usage
is considerably higher with identical traffic.
Any leads for possible reasons for this because the data suggests that it's
a Solr version issue . I'm attaching solrconfig and schema for reference.
[image: Screenshot 2022-08-26 at 2.03.57 PM.png]
Re: Solr 6 vs Solr 8 considerable performance gap
Posted by Sidharth Negi <si...@gmail.com>.
Errata: I meant 80k requests per minute and NOT 80k per second.
On Sat, Aug 27, 2022 at 1:48 AM Sidharth Negi <si...@gmail.com>
wrote:
> Interesting to note that when I ran the experiment with Solr 9, the CPU
> usage was about the same as Solr 6.
>
> On Fri, Aug 26, 2022 at 7:02 PM Shawn Heisey <ap...@elyograg.org.invalid>
> wrote:
>
>> On 8/26/22 02:55, Sidharth Negi wrote:
>> > We set up Solr 6 and Solr 8 on two identical AWS instances (16 cores,
>> > 128 GB of which Solr was given Xmx=50GB) and indexed the same data on
>> > them and tested under the same load of traffic. The schema and
>> > solrconfig.xml are exactly identical - the schema file is just renamed
>> > as managed-schema in Solr 8. None of the two machines are indexing
>> > data or taking replication and both have about equal number of
>> > segments (42 and 45 segments for Solr 6 and Solr 8 respectively)
>>
>> Are you really sure that the heap needs to be that big? It really is
>> huge, and due to the way that Java works, anything 32GB or larger
>> requires 64-bit pointers. So a heap size of 31GB actually has more
>> memory available than a heap size of 32GB. At 50 GB, you have likely
>> passed the break-even point. But unless you're dealing with hundreds of
>> millions of documents, it is very unlikely that you need a heap that big.
>
>
> I agree - the number of documents we are dealing with is ~30 million so
> most of the heap is unused (over 30 GB).
>
>
>
>
>> > What's surprising is that Solr 6.6.1 CPU usage is considerably lower
>> > than Solr 8.11.2. Just look at the screenshot attached. The blue line
>> > is Solr 8.11.2 while the orange one is Solr 6.6.1. Note that the Solr
>> > 8 CPU usage is considerably higher with identical traffic.
>>
>> You have higher CPU usage, but does Solr 8 actually perform worse than
>> Solr 6? What do other metrics show, like CPU iowait percentage?
>
>
> I don't think Solr 8 performs any "worse" in terms of query times taken
> for a query as such - it's just that the CPU usage is linearly increasing
> with traffic and the screenshot is for 30% traffic. Hence for full scale
> traffic, Solr 6 will win out as that will need a lesser number of machines
> since we want to keep CPU usage well under 70% on a production instance
> even though the query times are about the same.
>
>
>
>
>> You've talked about segment counts, but haven't talked about index
>> size. Is the total disk space consumed by the index about the same on
>> both?
>
>
> The disk space taken by the index of both Solr versions was about ~35 GB
> and the number of docs ~30 million in both.
>
>
>> I can think of two differences between 6 and 8 that are fundamental:
>> First: 6 uses CMS for garbage collection and 8 uses G1. G1 has better
>> overall performance because more of its work can function in parallel
>> with the application, and I can imagine that it uses a little bit more
>> of resources like memory and CPU. Second: 6 uses log4j 1 and 8 uses
>> log4j 2. The later logging library is much faster because it takes
>> advantage of threads, which could increase the overall CPU usage.
>> Whether that would cause a significant impact depends mostly on how busy
>> the server is and whether the logging configuration has been changed.
>> With default settings, at least one log message is created for almost
>> every request that Solr receives.
>>
>
> Let me run an experiment using the same GC settings on both to see if that
> works. Is there anything else we can do to narrow down the reason for sure?
> All slaves combined will have to serve over 80k requests per second once we
> set the number of slaves such that the CPU usage of all remains well below
> 70% at peaks.
>
>
>> There have also been a lot of advancements in other areas, and those
>> probably contribute. Higher CPU usage does not automatically mean that
>> performance is worse. Sometimes applications actually perform better
>> when using more CPU.
>>
>
> I agree - higher CPU usage is not directly meaning worse performance but
> as mentioned above - for us that would translate into more infra and hence
> added cost.
>
>
>> Thanks,
>> Shawn
>>
>>
Re: Solr 6 vs Solr 8 considerable performance gap
Posted by Sidharth Negi <si...@gmail.com>.
Update: Solr 6 and Solr 8 both parsed the same input query differently. Is
this an expected difference given that the config and schema are the same.
I suspect this difference could also be a reason for the difference in CPU
performance.
How can I tweak Solr 8 to parse the given query as *BoostQuery* instead of
*FunctionScoreQuery *to confirm this?
*Solr 6 Debug Information:*
"querystring": "aliases:\"love\"^700 OR tedge1:love^900 OR
title_variation_edge:\"love\"^900 OR titleNgram:\"love\"^10 OR
artistEdge:\"love\"^250 OR keyword:\"love\"^10 OR title:\"love\"^1000 OR
(entity_edge:love^800)","parsedquery":
"BoostedQuery(boost(+((aliases:love)^700.0
(tedge1:love)^900.0 (title_variation_edge:love)^900.0
(titleNgram:love)^10.0 (artistEdge:love)^250.0 (keyword:love)^10.0
(title:love)^1000.0 (entity_edge:love)^800.0) ((keywords:\"700 900 (10 ten)
250 (10 ten) 1000\"~2)^25.0 | (aliases:\"700 900 (10 ten) 250 (10 ten)
1000\"~2)^50.0 | (title:\"700 900 (10 ten) 250 (10 ten)
1000\"~2)^60.0),product(sum(const(1),double(n_c_pop)),product(const(1.0E-4),sum(const(1),sum(product(const(0.01),long(search_click)),product(const(1),long(search_clickone)),product(const(10),long(search_clicktwo)),product(const(100),long(search_clickthree))))))))",
*Solr 8 Debug Information:*"querystring": "aliases:\"love\"^700 OR
tedge:love^900 OR title_variation_edge:\"love\"^900 OR
titleNgram:\"love\"^10 OR artistEdge:\"love\"^250 OR keyword:\"love\"^10 OR
title:\"love\"^1000 OR (entity_edge:love^800)","parsedquery": "
FunctionScoreQuery(FunctionScoreQuery(+((aliases:love)^700.0
(tedge:love)^900.0 (title_variation_edge:love)^900.0 (titleNgram:love)^10.0
(artistEdge:love)^250.0 (keyword:love)^10.0 (title:love)^1000.0
(entity_edge:love)^800.0), scored by
boost(product(sum(const(1),double(n_c_pop)),product(const(1.0E-4),sum(const(1),sum(product(const(0.01),long(search_click)),product(const(1),long(search_clickone)),product(const(10),long(search_clicktwo)),product(const(100),long(search_clickthree)))))))))"
,
*Solr 6 Query Params:*
"responseHeader": {"status": 0,"QTime": 366,"params": {"q":
"aliases:\"love\"^700
OR tedge:love^900 OR title_variation_edge:\"love\"^900 OR
titleNgram:\"love\"^10 OR artistEdge:\"love\"^250 OR keyword:\"love\"^10 OR
title:\"love\"^1000 OR (entity_edge:love^800)","debug": "true","fl":
"doc_id,recording_id,title,type,status,keywords,aliases,score,search_click,thumbnail,year,album,artist,primaryArtist,language,original_title,is_hellotune,isCurated,explicitType"
,"start": "0","boost":
"product(sum(1,n_c_pop),product(0.0001,sum(1,sum(product(0.01,search_click),product(1,search_clickone),product(10,search_clicktwo),product(100,search_clickthree)))))"
,"fq": ["status:(PUBLISH_PUBLISHED LIVE ENRICHED)","((type:song type:artist
type:playlist type:podcast) OR (type:album AND ((creationTime:[* TO
NOW/DAY-31DAYS ] AND size:[2 TO *]) OR creationTime:[NOW/DAY-31DAYS TO *
])))"],"rows": "40","wt": "json","debug.explain.structured": "true"}
*Solr 8 Query Params:*
"responseHeader": {"status": 0,"QTime": 40,"params": {"q":
"aliases:\"love\"^700
OR tedge:love^900 OR title_variation_edge:\"love\"^900 OR
titleNgram:\"love\"^10 OR artistEdge:\"love\"^250 OR keyword:\"love\"^10 OR
title:\"love\"^1000 OR (entity_edge:love^800)","debug": "true","fl":
"doc_id,recording_id,title,type,status,keywords,aliases,score,search_click,thumbnail,year,album,artist,primaryArtist,language,original_title,is_hellotune,isCurated,explicitType"
,"start": "0","boost":
"product(sum(1,n_c_pop),product(0.0001,sum(1,sum(product(0.01,search_click),product(1,search_clickone),product(10,search_clicktwo),product(100,search_clickthree)))))"
,"fq": ["status:(PUBLISH_PUBLISHED LIVE ENRICHED)","((type:song type:artist
type:playlist type:podcast) OR (type:album AND ((creationTime:[* TO
NOW/DAY-31DAYS ] AND size:[2 TO *]) OR creationTime:[NOW/DAY-31DAYS TO *
])))"],"rows": "40","wt": "json","debug.explain.structured": "true"}}
On Sat, Aug 27, 2022 at 2:38 AM Shawn Heisey <el...@elyograg.org.invalid>
wrote:
> On 8/26/22 14:18, Sidharth Negi wrote:
> > The disk space taken by the index of both Solr versions was about ~35 GB
> > and the number of docs ~30 million in both.
>
> Unless that system is handling insanely complex queries that chew up
> lots of memory, I would not expect it to need more than about 8GB of
> heap with that index size, and quite possibly even less.
>
> 48GB total system memory would probably work if the server is not
> handling anything other than that Solr index.
>
> If you share solr GC logs that cover a day or more of heavy usage
> (several days being better), I might be able to determine a more precise
> heap size recommendation. Unfortunately there is no generic way to
> calculate heap requirements based on data statistics. GC logs provide
> the best info, otherwise it requires experimentation. My guess of 8GB
> is just that -- a guess. It might be wrong, either too high or too low.
>
> > Let me run an experiment using the same GC settings on both to see if
> that
> > works. Is there anything else we can do to narrow down the reason for
> sure?
> > All slaves combined will have to serve over 80k requests per second once
> we
> > set the number of slaves such that the CPU usage of all remains well
> below
> > 70% at peaks.
>
> Very high query rate, but it sounds like you're going to size
> appropriately to deal with it.
>
> > Interesting to note that when I ran the experiment with Solr 9, the CPU
> > usage was about the same as Solr 6.
>
> That is interesting. Solr 9 also uses G1 and log4j2, so those are
> probably not the culprit. Deploying the latest release would be our
> recommendation, because only extremely bad bugs, mostly security issues,
> are going to be fixed in new 8.x releases. It is very unlikely that a
> new 8.11.x version will address this issue.
>
> Thanks,
> Shawn
>
>
Re: Solr 6 vs Solr 8 considerable performance gap
Posted by Shawn Heisey <el...@elyograg.org.INVALID>.
On 8/26/22 14:18, Sidharth Negi wrote:
> The disk space taken by the index of both Solr versions was about ~35 GB
> and the number of docs ~30 million in both.
Unless that system is handling insanely complex queries that chew up
lots of memory, I would not expect it to need more than about 8GB of
heap with that index size, and quite possibly even less.
48GB total system memory would probably work if the server is not
handling anything other than that Solr index.
If you share solr GC logs that cover a day or more of heavy usage
(several days being better), I might be able to determine a more precise
heap size recommendation. Unfortunately there is no generic way to
calculate heap requirements based on data statistics. GC logs provide
the best info, otherwise it requires experimentation. My guess of 8GB
is just that -- a guess. It might be wrong, either too high or too low.
> Let me run an experiment using the same GC settings on both to see if that
> works. Is there anything else we can do to narrow down the reason for sure?
> All slaves combined will have to serve over 80k requests per second once we
> set the number of slaves such that the CPU usage of all remains well below
> 70% at peaks.
Very high query rate, but it sounds like you're going to size
appropriately to deal with it.
> Interesting to note that when I ran the experiment with Solr 9, the CPU
> usage was about the same as Solr 6.
That is interesting. Solr 9 also uses G1 and log4j2, so those are
probably not the culprit. Deploying the latest release would be our
recommendation, because only extremely bad bugs, mostly security issues,
are going to be fixed in new 8.x releases. It is very unlikely that a
new 8.11.x version will address this issue.
Thanks,
Shawn
Re: Solr 6 vs Solr 8 considerable performance gap
Posted by Sidharth Negi <si...@gmail.com>.
Interesting to note that when I ran the experiment with Solr 9, the CPU
usage was about the same as Solr 6.
On Fri, Aug 26, 2022 at 7:02 PM Shawn Heisey <ap...@elyograg.org.invalid>
wrote:
> On 8/26/22 02:55, Sidharth Negi wrote:
> > We set up Solr 6 and Solr 8 on two identical AWS instances (16 cores,
> > 128 GB of which Solr was given Xmx=50GB) and indexed the same data on
> > them and tested under the same load of traffic. The schema and
> > solrconfig.xml are exactly identical - the schema file is just renamed
> > as managed-schema in Solr 8. None of the two machines are indexing
> > data or taking replication and both have about equal number of
> > segments (42 and 45 segments for Solr 6 and Solr 8 respectively)
>
> Are you really sure that the heap needs to be that big? It really is
> huge, and due to the way that Java works, anything 32GB or larger
> requires 64-bit pointers. So a heap size of 31GB actually has more
> memory available than a heap size of 32GB. At 50 GB, you have likely
> passed the break-even point. But unless you're dealing with hundreds of
> millions of documents, it is very unlikely that you need a heap that big.
I agree - the number of documents we are dealing with is ~30 million so
most of the heap is unused (over 30 GB).
> > What's surprising is that Solr 6.6.1 CPU usage is considerably lower
> > than Solr 8.11.2. Just look at the screenshot attached. The blue line
> > is Solr 8.11.2 while the orange one is Solr 6.6.1. Note that the Solr
> > 8 CPU usage is considerably higher with identical traffic.
>
> You have higher CPU usage, but does Solr 8 actually perform worse than
> Solr 6? What do other metrics show, like CPU iowait percentage?
I don't think Solr 8 performs any "worse" in terms of query times taken for
a query as such - it's just that the CPU usage is linearly increasing with
traffic and the screenshot is for 30% traffic. Hence for full scale
traffic, Solr 6 will win out as that will need a lesser number of machines
since we want to keep CPU usage well under 70% on a production instance
even though the query times are about the same.
> You've talked about segment counts, but haven't talked about index
> size. Is the total disk space consumed by the index about the same on
> both?
The disk space taken by the index of both Solr versions was about ~35 GB
and the number of docs ~30 million in both.
> I can think of two differences between 6 and 8 that are fundamental:
> First: 6 uses CMS for garbage collection and 8 uses G1. G1 has better
> overall performance because more of its work can function in parallel
> with the application, and I can imagine that it uses a little bit more
> of resources like memory and CPU. Second: 6 uses log4j 1 and 8 uses
> log4j 2. The later logging library is much faster because it takes
> advantage of threads, which could increase the overall CPU usage.
> Whether that would cause a significant impact depends mostly on how busy
> the server is and whether the logging configuration has been changed.
> With default settings, at least one log message is created for almost
> every request that Solr receives.
>
Let me run an experiment using the same GC settings on both to see if that
works. Is there anything else we can do to narrow down the reason for sure?
All slaves combined will have to serve over 80k requests per second once we
set the number of slaves such that the CPU usage of all remains well below
70% at peaks.
> There have also been a lot of advancements in other areas, and those
> probably contribute. Higher CPU usage does not automatically mean that
> performance is worse. Sometimes applications actually perform better
> when using more CPU.
>
I agree - higher CPU usage is not directly meaning worse performance but as
mentioned above - for us that would translate into more infra and hence
added cost.
> Thanks,
> Shawn
>
>
Re: Solr 6 vs Solr 8 considerable performance gap
Posted by Deepak Goel <de...@gmail.com>.
Are the response times the same for the 2 machines? Or is Solr8 faster than
Solr6?
Deepak
"The greatness of a nation can be judged by the way its animals are treated
- Mahatma Gandhi"
+91 73500 12833
deicool@gmail.com
Facebook: https://www.facebook.com/deicool
LinkedIn: www.linkedin.com/in/deicool
"Plant a Tree, Go Green"
Make In India : http://www.makeinindia.com/home
On Fri, Aug 26, 2022 at 7:02 PM Shawn Heisey <ap...@elyograg.org.invalid>
wrote:
> On 8/26/22 02:55, Sidharth Negi wrote:
> > We set up Solr 6 and Solr 8 on two identical AWS instances (16 cores,
> > 128 GB of which Solr was given Xmx=50GB) and indexed the same data on
> > them and tested under the same load of traffic. The schema and
> > solrconfig.xml are exactly identical - the schema file is just renamed
> > as managed-schema in Solr 8. None of the two machines are indexing
> > data or taking replication and both have about equal number of
> > segments (42 and 45 segments for Solr 6 and Solr 8 respectively)
>
> Are you really sure that the heap needs to be that big? It really is
> huge, and due to the way that Java works, anything 32GB or larger
> requires 64-bit pointers. So a heap size of 31GB actually has more
> memory available than a heap size of 32GB. At 50 GB, you have likely
> passed the break-even point. But unless you're dealing with hundreds of
> millions of documents, it is very unlikely that you need a heap that big.
>
> > What's surprising is that Solr 6.6.1 CPU usage is considerably lower
> > than Solr 8.11.2. Just look at the screenshot attached. The blue line
> > is Solr 8.11.2 while the orange one is Solr 6.6.1. Note that the Solr
> > 8 CPU usage is considerably higher with identical traffic.
>
> You have higher CPU usage, but does Solr 8 actually perform worse than
> Solr 6? What do other metrics show, like CPU iowait percentage?
>
> You've talked about segment counts, but haven't talked about index
> size. Is the total disk space consumed by the index about the same on
> both?
>
> I can think of two differences between 6 and 8 that are fundamental:
> First: 6 uses CMS for garbage collection and 8 uses G1. G1 has better
> overall performance because more of its work can function in parallel
> with the application, and I can imagine that it uses a little bit more
> of resources like memory and CPU. Second: 6 uses log4j 1 and 8 uses
> log4j 2. The later logging library is much faster because it takes
> advantage of threads, which could increase the overall CPU usage.
> Whether that would cause a significant impact depends mostly on how busy
> the server is and whether the logging configuration has been changed.
> With default settings, at least one log message is created for almost
> every request that Solr receives.
>
> There have also been a lot of advancements in other areas, and those
> probably contribute. Higher CPU usage does not automatically mean that
> performance is worse. Sometimes applications actually perform better
> when using more CPU.
>
> Thanks,
> Shawn
>
>
Re: Solr 6 vs Solr 8 considerable performance gap
Posted by Shawn Heisey <ap...@elyograg.org.INVALID>.
On 8/26/22 02:55, Sidharth Negi wrote:
> We set up Solr 6 and Solr 8 on two identical AWS instances (16 cores,
> 128 GB of which Solr was given Xmx=50GB) and indexed the same data on
> them and tested under the same load of traffic. The schema and
> solrconfig.xml are exactly identical - the schema file is just renamed
> as managed-schema in Solr 8. None of the two machines are indexing
> data or taking replication and both have about equal number of
> segments (42 and 45 segments for Solr 6 and Solr 8 respectively)
Are you really sure that the heap needs to be that big? It really is
huge, and due to the way that Java works, anything 32GB or larger
requires 64-bit pointers. So a heap size of 31GB actually has more
memory available than a heap size of 32GB. At 50 GB, you have likely
passed the break-even point. But unless you're dealing with hundreds of
millions of documents, it is very unlikely that you need a heap that big.
> What's surprising is that Solr 6.6.1 CPU usage is considerably lower
> than Solr 8.11.2. Just look at the screenshot attached. The blue line
> is Solr 8.11.2 while the orange one is Solr 6.6.1. Note that the Solr
> 8 CPU usage is considerably higher with identical traffic.
You have higher CPU usage, but does Solr 8 actually perform worse than
Solr 6? What do other metrics show, like CPU iowait percentage?
You've talked about segment counts, but haven't talked about index
size. Is the total disk space consumed by the index about the same on both?
I can think of two differences between 6 and 8 that are fundamental:
First: 6 uses CMS for garbage collection and 8 uses G1. G1 has better
overall performance because more of its work can function in parallel
with the application, and I can imagine that it uses a little bit more
of resources like memory and CPU. Second: 6 uses log4j 1 and 8 uses
log4j 2. The later logging library is much faster because it takes
advantage of threads, which could increase the overall CPU usage.
Whether that would cause a significant impact depends mostly on how busy
the server is and whether the logging configuration has been changed.
With default settings, at least one log message is created for almost
every request that Solr receives.
There have also been a lot of advancements in other areas, and those
probably contribute. Higher CPU usage does not automatically mean that
performance is worse. Sometimes applications actually perform better
when using more CPU.
Thanks,
Shawn