You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@solr.apache.org by "Kevin Risden (Jira)" <ji...@apache.org> on 2022/11/17 18:35:00 UTC
[jira] [Commented] (SOLR-16555) SolrIndexSearcher - FilterCache intersections/andNot should not clone bitsets repeatedly
[ https://issues.apache.org/jira/browse/SOLR-16555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17635499#comment-17635499 ]
Kevin Risden commented on SOLR-16555:
-------------------------------------
So I've spent some time trying to "fix" this and it seems like there are tons of defensive copies to avoid altering the bitset since it is/could be cached. However, this seems to go too far in this case. We should be able to make say one defensive copy and then just modify that by "and"ing the other filter queries to it. This should result in SIGNIFICANTLY fewer memory allocations and be faster since you could just set the bits on the bitset that should/shouldn't be enabled.
The downside is the way the code is written today is you get a copy back everywhere. Its hard to modify just this section right now.
> SolrIndexSearcher - FilterCache intersections/andNot should not clone bitsets repeatedly
> ----------------------------------------------------------------------------------------
>
> Key: SOLR-16555
> URL: https://issues.apache.org/jira/browse/SOLR-16555
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: query
> Reporter: Kevin Risden
> Priority: Major
> Labels: performance
> Attachments: Screenshot 2022-11-16 at 14.52.37.png, Screenshot 2022-11-16 at 14.53.23.png, Screenshot 2022-11-16 at 14.53.35.png, Screenshot 2022-11-17 at 13.03.21.png, Screenshot 2022-11-17 at 13.25.57.png, Screenshot 2022-11-17 at 13.28.06.png
>
>
> SolrIndexSearcher takes the bitset from the result and tries to combine it with all the cached filter queries. Currently this duplicates the bitset multiple times based on the number of filter queries. It looks like this isn't necessary and instead could just operate on the bitset itself or a single mutable copy of the bitset.
> Lines 1219 to 1225
> https://github.com/apache/solr/blob/main/solr/core/src/java/org/apache/solr/search/SolrIndexSearcher.java#L1219
> ----
> I've been using async profiler (https://github.com/jvm-profiling-tools/async-profiler) to look at some performance stuff with Solr for a client. Originally I looked at CPU in the profile and found that I could also capture and look at memory allocations during the same run. This led me to finding this crazy amount of memory allocation over a short period of time.
> Async profiler is being run with the following parameters which captures cpu, memory, and lock information for a 300 second period on some pid.
> {code:java}
> /opt/async-profiler/profiler.sh -a -d 300 -o jfr -e cpu,alloc,lock -f /tmp/profile.jfr PID_GOES_HERE
> {code}
> The resulting JFR for this is ~100-200MB usually and so not going to attach it here since it has some client specific methods in some calls in it.
> However screenshots of the findings from loading the jfr in both IntelliJ and Java Mission Control you can see some of the findings:
> !Screenshot 2022-11-17 at 13.28.06.png|width=750!
> The memory allocated from SolrIndexSearcher#getProcessedFilter is ~60% of total memory allocated during the 5 minute profile period.
> !Screenshot 2022-11-16 at 14.52.37.png|width=750!
> This shows that in 5 minutes ~1TB (yes thats TB=terabyte or 1000GB) of memory allocations for SolrIndexSearcher#getProcessedFilter
> !Screenshot 2022-11-16 at 14.53.23.png|width=750!
> ~680GB was allocated from BitDocSet#intersection
> !Screenshot 2022-11-16 at 14.53.35.png|width=750!
> ~315GB was allocated from BitDocSet#andNot
> Based on CPU profiling, it is amazing to me but G1 garbage collector is keeping up. Each of these objects are very short lived.
> This was during some load testing and able to give some query types in question:
> * ~30 queries/second
> * ~5 fq parameters per query
> * so ~9000 queries in 5 minutes with ~45000 fq clauses
> * 10GB heap for the Solr instance with 128GB ram on the node and index size completely fits in memory.
> * this is one shard on the node for testing and ~23 million documents in the shard - optimized so no deletes.
> * This was tested with Solr 8.8, but as far as I can tell the code has not changed for the main branch significantly.
> Based on my rough calculations, that is ~24MB of heap per filter query clause (1.06TB/45000) or ~120MB of heap per query (assuming 5 fq per query).
> ----
> I loaded the same JFR up in Java Mission Control to see if there were other insights and found TLAB memory allocation details.
> !Screenshot 2022-11-17 at 13.25.57.png|width=750!
> Since most of these are large allocations, Java mission control is very helpful in saying that there are a large number of allocations outside of TLAB (~80%) and that you probably shouldn't do that.
> !Screenshot 2022-11-17 at 13.03.21.png|width=750!
> From what I understand about allocations outside of TLAB is that they should in theory be a small portion of allocations, but they aren't...
> * https://shipilev.net/jvm/anatomy-quarks/4-tlab-allocation/
> * https://www.opsian.com/blog/jvm-tlabs-important-multicore/
> * https://stackoverflow.com/questions/26351243/allocations-in-new-tlab-vs-allocations-outside-tlab
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org