You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@solr.apache.org by "ASF subversion and git services (Jira)" <ji...@apache.org> on 2025/02/22 17:58:00 UTC
[jira] [Commented] (SOLR-17670) Fix unnecessary memory allocation caused by a large reRankDocs param
[ https://issues.apache.org/jira/browse/SOLR-17670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17929377#comment-17929377 ]
ASF subversion and git services commented on SOLR-17670:
--------------------------------------------------------
Commit 76c09a35dba42913a6bcb281b52b00f87564624a in solr's branch refs/heads/main from jiabao.gao
[ https://gitbox.apache.org/repos/asf?p=solr.git;h=76c09a35dba ]
SOLR-17670: Fix unnecessary memory allocation caused by a large reRankDocs param (#3181)
> Fix unnecessary memory allocation caused by a large reRankDocs param
> --------------------------------------------------------------------
>
> Key: SOLR-17670
> URL: https://issues.apache.org/jira/browse/SOLR-17670
> Project: Solr
> Issue Type: Bug
> Reporter: JiaBaoGao
> Priority: Major
> Labels: pull-request-available
> Time Spent: 1h
> Remaining Estimate: 0h
>
> The reRank function has a reRankDocs parameter that specifies the number of documents to re-rank. I've observed that increasing this parameter to test its performance impact causes queries to become progressively slower. Even when the parameter value exceeds the total number of documents in the index, further increases continue to slow down the query, which is counterintuitive.
>
> Therefore, I investigated the code:
>
> For a query containing re-ranking, such as:
> {code:java}
> {
> "start": "0",
> "rows": 10,
> "fl": "ID,score",
> "q": "*:*",
> "rq": "{!rerank reRankQuery='{!func} 100' reRankDocs=1000000000 reRankWeight=2}"
> } {code}
>
> The current execution logic is as follows:
> 1. Perform normal retrieval using the q parameter.
> 2. Re-score all documents retrieved in the q phase using the rq parameter.
>
> During the retrieval in phase 1 (using q), a TopScoreDocCollector is created. Underneath, this creates a PriorityQueue which contains an Object[]. The length of this Object[] continuously increases with reRankDocs without any limit.
>
> On my local test cluster with limited JVM memory, this can even trigger an OOM, causing the Solr node to crash. I can also reproduce the OOM situation using the SolrCloudTestCase unit test.
>
> I think limiting the length of the Object[] array using searcher.getIndexReader().maxDoc() at ReRankCollector would resolve this issue. This way, when reRankDocs exceeds maxDoc, memory allocation will not continue to increase indefinitely.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org