[jira] [Updated] (SOLR-9764) Design a memory efficient DocSet if a query returns all docs

["Yonik Seeley (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SOLR-9764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Yonik Seeley updated SOLR-9764:
-------------------------------
    Attachment: SOLR-9764.patch

tldr; the attached patch should make all queries that end up matching all documents use the same DocSet instead of caching different sets.

Notes on changes from the previous patch:
- I'm not sure what version this patch was made for, but it won't compile on either trunk or 6x.
{code}
[javac] /opt/code/lusolr/trunk/solr/core/src/java/org/apache/solr/query/SolrRangeQuery.java:157: error: incompatible types: DocSet cannot be converted to BitDocSet
[javac]     BitDocSet liveDocs = searcher.getLiveDocs();
{code}
- Some code currently explicitly relies on BitDocSet from liveDocs (hence the compilation error above)

- hopefully the following change is just an optimization, and not meant to ensure that the same amount of 0 padding is used for each bitset?
  it's buggy in the general case (size is cardinality, not capacity)
{code}
   protected FixedBitSet getBits() {
-    FixedBitSet bits = new FixedBitSet(64);
+    FixedBitSet bits = new FixedBitSet(size());
{code}

- if MatchAllDocs is used as a lucene filter, it can cause a lot of unnessesary memory use... in fact it would end up creating a new bit set 
  for each use.  This is one instance of a more generic problem... the BaseDocSet implementations are often very inefficient and should be
  overridden by any DocSet meant for use in any common case.  Much of the code was also written for best performance with the knowledge that
  there were only 2 common sets (small and large)... that code will need to be revisited / re-reviewed when adding a 3rd into the mix.

- Given the current problems with MatchAllDocs, I've backed out that part of the patch for now.
  As detailed above, this is more a problem of the fragility of the current code base than with your class.
  It's probably best handled in a separate issue, and perhaps in a more general way that can handle more cases (like most docs matching or segments with deleted docs),
  or if the robustness of the DocSet hierarchy can be improved, we could even add multiple new implementations (Roaring, MatchMost, MatchAll, etc)

- The setting of liveDocs was not thread-safe (unsafe object publishing)

- added size() to DocSetCollector in favor of the more specific isMatchLiveDocs

- The check for liveDocs was only done in createDocSetGeneric, meaning many instances wouldn't be detected (term query, range query on string,
  non-fq parameters like base queries, etc). I created some DocSetUtil methods to handle these cases and called them
  from most of the appropriate places.

- just a draft patch, but if people agree, we can add tests for sharing/deduplication of liveDocs (which includes the MatchAllDocs case) and commit.



> Design a memory efficient DocSet if a query returns all docs
> ------------------------------------------------------------
>
>                 Key: SOLR-9764
>                 URL: https://issues.apache.org/jira/browse/SOLR-9764
>             Project: Solr
>          Issue Type: Improvement
>      Security Level: Public(Default Security Level. Issues are Public) 
>            Reporter: Michael Sun
>         Attachments: SOLR_9764_no_cloneMe.patch, SOLR-9764.patch, SOLR-9764.patch, SOLR-9764.patch, SOLR-9764.patch, SOLR-9764.patch, SOLR-9764.patch, SOLR-9764.patch
>
>
> In some use cases, particularly use cases with time series data, using collection alias and partitioning data into multiple small collections using timestamp, a filter query can match all documents in a collection. Currently BitDocSet is used which contains a large array of long integers with every bits set to 1. After querying, the resulted DocSet saved in filter cache is large and becomes one of the main memory consumers in these use cases.
> For example. suppose a Solr setup has 14 collections for data in last 14 days, each collection with one day of data. A filter query for last one week data would result in at least six DocSet in filter cache which matches all documents in six collections respectively.   
> This is to design a new DocSet that is memory efficient for such a use case.  The new DocSet removes the large array, reduces memory usage and GC pressure without losing advantage of large filter cache.
> In particular, for use cases when using time series data, collection alias and partition data into multiple small collections using timestamp, the gain can be large.
> For further optimization, it may be helpful to design a DocSet with run length encoding. Thanks [~mmokhtar] for suggestion. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org