You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Adrien Grand (JIRA)" <ji...@apache.org> on 2014/12/05 20:20:12 UTC

[jira] [Updated] (LUCENE-6096) Improve FilterCache.ramBytesUsed

     [ https://issues.apache.org/jira/browse/LUCENE-6096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Adrien Grand updated LUCENE-6096:
---------------------------------
    Attachment: LUCENE-6096.patch

Here is a patch:
 - it uses a more sensible memory usage for filters that don't implement Accountable (which is the memory usage for a QueryWrapperFilter around a TermQuery, see test)
 - it tries to take the memory usage of the hash tables into account
 - filters are deduplicated, this actually did not require many changes since filters were already stored in a top-level LinkedHashMap for LRU evictions.

> Improve FilterCache.ramBytesUsed
> --------------------------------
>
>                 Key: LUCENE-6096
>                 URL: https://issues.apache.org/jira/browse/LUCENE-6096
>             Project: Lucene - Core
>          Issue Type: Improvement
>            Reporter: Adrien Grand
>            Assignee: Adrien Grand
>            Priority: Minor
>         Attachments: LUCENE-6096.patch
>
>
> I worked on some tests to improve ram usage estimation of LRUFilterCache and found some issues:
> 1. It currently only takes into account filters and DocIdSets, not the internal overhead of the various hash tables and linked lists that are being maintained. If the filter cache mostly stores sparse doc id sets (or even worse DocIdSet.EMPTY instances that require 0 bytes since it's a singleton) then the memory usage can be significantly under-estimated.
> 2. It treats all filters as if they were singletons. The filter cache is essentially a Map<SegmentCoreReaders, Map<Filter, DocIdSet>> and it treats filters as if all filters that are equal are the same instance when it comes to memory usage. But this is not the case actually since the map on the second level uses the first-seen filter as a key, which might be different on different segments because of merging.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org