[jira] [Commented] (LUCENE-10103) QueryCache not estimating query size properly

["ASF subversion and git services (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/LUCENE-10103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17428204#comment-17428204 ] 

ASF subversion and git services commented on LUCENE-10103:
----------------------------------------------------------

Commit 6a41bc6310e2dd2a92b9531141de877eb1b132fd in lucene's branch refs/heads/main from Patrick Zhai
[ https://gitbox.apache.org/repos/asf?p=lucene.git;h=6a41bc6 ]

LUCENE-10103 Make QueryCache respect Accountable queries (#346)



> QueryCache not estimating query size properly
> ---------------------------------------------
>
>                 Key: LUCENE-10103
>                 URL: https://issues.apache.org/jira/browse/LUCENE-10103
>             Project: Lucene - Core
>          Issue Type: Improvement
>          Components: core/search
>            Reporter: Haoyu Zhai
>            Priority: Minor
>         Attachments: query_cache_error_demo.patch
>
>          Time Spent: 20m
>  Remaining Estimate: 0h
>
> QueryCache seems estimating the cached query size using a [constant|https://github.com/apache/lucene/blob/main/lucene/core/src/java/org/apache/lucene/search/LRUQueryCache.java#L302], it will cause OOM error in some extreme cases where queries cached will use far more memories than assumed. (The default QueryCache tries to use [only 5% of heap|https://github.com/apache/lucene/blob/main/lucene/core/src/java/org/apache/lucene/search/IndexSearcher.java#L89]) One example of such memory-eating query is AutomatonQuery, it will each carry a [RunAutomaton|https://github.com/apache/lucene/blob/main/lucene/core/src/java/org/apache/lucene/util/automaton/RunAutomaton.java#L42] , which consumes a good amount of memory in exchange for the speed.
> On the other hand, we actually have a good implementation of {{Accountable}} interface for AutomatonQuery (though it will become a bit more complicated later since this query will eventually be rewritten to something else), so maybe QueryCache could use those estimation directly (using an {{instanceof}} check)? Or moreover we could make all {{Query}} implement {Accountable}}, and maybe the default implementation could just be returning the current constant we're using, and only override the method of the potential troublesome queries?



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org