[jira] [Updated] (SOLR-6349) LocalParams for enabling/disabling individual stats

["Hoss Man (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SOLR-6349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hoss Man updated SOLR-6349:
---------------------------
    Attachment: make-data-and-queries.pl
                SOLR-6349.patch

Well, after wrapping up SOLR-7171, i came back to this patch and started making the following improvements...

{panel:title=changes in this patch}
* TestDistributedSearch
** beefed up inspection of shard responses now that SOLR-7171 is fixed
* StatsValuesFactory
** introduce new final booleans to track the stats we want to encourage JIT optimization
** fix some inconsistencies in when/if various stats are returned to clients depending on values seen (notably with Dates before the epoch)
* StatsComponentTest
** testFieldStatisticsResultsDateFieldAlwaysMissing
*** the set of stats that we've returned for Dates in the past has been inconsistent depending on if there were "non-missing" docs ... updated this test to expect everything
{panel}

...after making these changes, i went to re-run that mini-benchmark i wrote before, and in looking at it realized i made a serious conceptual mistake...

{quote}
* user currently asks for stats on fields, only cares about 4of8 of them
...
...the sequence of stat field requests are identicle between the 2 bash files, but in one URLs include localparams to only compute min/max/mean/stddev for the field. 
{quote}

...the problem being that because these are distributed requests, the new style requests still require that sum, count & sumOfSquares be computed on every shard (in addition to the min & max) ... the *final* responses to the query client are smaller, but we're still computing virtually the same amount of math calculations and each shard is returning virtually the same amount of data.  (the only calculation "skipped" is the "missing" stat which is actually irrelevant in this test because every doc has a value in every field)

So i did a quick tweak to make-data-and-queries.pl so that the only stats requested are "min" & "mean" -- so the shards only have to compute min, sum, count.  With that change, and the new patch, the numbers look much better...

{noformat}
        pre-patch        with patch       with patch 
Run #   old style        old style        new style
        (all stats)      (all stats)      (2 stats, 3 deps)
  1      135.7 sec        134.4 sec        109.9 sec
  2      130.1 sec        132.4 sec        109.0 sec
  3      132.3 sec        132.8 sec        108.2 sec
total    398.1 sec        399.6 sec        327.1 sec
                        00.3 % slower    17.8 % faster
{noformat}

...so these numbers are a lot more promising.

this also makes me want to run some more perf tests, on more permutations of stats - in particular i want to check the non-cloud mode, make sure we haven't slowed that down.


> LocalParams for enabling/disabling individual stats
> ---------------------------------------------------
>
>                 Key: SOLR-6349
>                 URL: https://issues.apache.org/jira/browse/SOLR-6349
>             Project: Solr
>          Issue Type: Sub-task
>            Reporter: Hoss Man
>         Attachments: SOLR-6349-tflobbe.patch, SOLR-6349-tflobbe.patch, SOLR-6349-tflobbe.patch, SOLR-6349-xu.patch, SOLR-6349-xu.patch, SOLR-6349-xu.patch, SOLR-6349-xu.patch, SOLR-6349.patch, SOLR-6349.patch, SOLR-6349.patch, SOLR-6349.patch, SOLR-6349.patch, SOLR-6349.patch, SOLR-6349___bad_idea_broken.patch, make-data-and-queries.pl, make-data-and-queries.pl
>
>
> Stats component currently computes all stats (except for one) every time because they are relatively cheap, and in some cases dependent on eachother for distrib computation -- but if we start layering stats on other things it becomes unnecessarily expensive to compute all the stats when they just want the "sum" (and it will definitely become excessively verbose in the responses).  
> The plan here is to use local params to make this configurable.  All of the existing stat options could be modeled as a simple boolean param, but future params (like percentiles) might take in a more complex param value...
> Example:
> {noformat}
> stats.field={!min=true max=true percentiles='99,99.999'}price
> stats.field={!mean=true}weight
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org