You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Yonik Seeley (JIRA)" <ji...@apache.org> on 2017/05/11 15:06:04 UTC

[jira] [Updated] (SOLR-10634) Move calculation of some aggregations to collection phase

     [ https://issues.apache.org/jira/browse/SOLR-10634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Yonik Seeley updated SOLR-10634:
--------------------------------
    Attachment: SOLR-10634.patch

Here's a first-cut patch.  Existing tests pass, but no specific tests yet to even see if it's being exercised.  I don't think that limit:-1 is exercised much at all either.

> Move calculation of some aggregations to collection phase
> ---------------------------------------------------------
>
>                 Key: SOLR-10634
>                 URL: https://issues.apache.org/jira/browse/SOLR-10634
>             Project: Solr
>          Issue Type: Improvement
>      Security Level: Public(Default Security Level. Issues are Public) 
>          Components: Facet Module
>            Reporter: Yonik Seeley
>         Attachments: SOLR-10634.patch
>
>
> From http://markmail.org/message/pwgnt7iqxkzcnckh
> {quote}
> The current code is more optimized for finding the top K buckets from
> a total of N.
> When one asks to return the top 10 buckets when there are potentially
> millions of buckets, it makes sense to defer calculating other metrics
> for those buckets until we know which ones they are.  After we
> identify the top 10 buckets, we calculate the domain for that bucket
> and use that to calculate the remaining metrics.
> The current method is obviously much slower when one is requesting
> *all* buckets.  We might as well just calculate all metrics in the
> first pass rather than trying to defer them.
> {quote}
> So we should move aggregations from the second pass to the first pass under the following conditions:
> - no limit (or a high limit compared to the number of potential buckets?)
> - no sub-facets (or anything else) that will need the domain calculated anyway
> - aggregation is not really memory intensive per-slot (i.e. moving some calculations from per-bucket in the second phase, to all-buckets-in-parallel in the first phase could be really bad for peak memory usage)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org