[jira] [Comment Edited] (LUCENE-5476) Facet sampling

["Gilad Barkai (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/LUCENE-5476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13917158#comment-13917158 ] 

Gilad Barkai edited comment on LUCENE-5476 at 3/1/14 7:02 PM:
--------------------------------------------------------------

Great effort!

I wish to through in another part - the description of this issue is about sampling, but the implementation is about *random* sampling.
This is not always the case, nor it is very fast (indeed, calling 1M times Random.nextInt would be measurable by itself IMHO).
A different sample could be (pseudo-code)
{code}
int acceptedModulu = (int)(1/sampleRatio);

int next() {
  do {
    nextDoc = inner.next();
  } while (nextDoc != NO_MORE_DOCX && nextDoc % acceptedModulu != 0) ;

  return nextDoc;
}
{code}

This should be faster as a sampler, and perhaps saves us from creating a new {{DocIdSet}}.

One last thing - if I did the math right - the sample crafted by the code in the patch would be twice as large as the user may expect.
For a sample ratio of 0.1, the random.nextInt() would be called with 10, so the avg. "jump" is actually 5 - and every 5th document in the original set (again, in avg) would be selected, and not every 10th in avg. I think the random.nextInt should be called with twice the size it is called now (e.g 20, making the avg random selection 10).


was (Author: gilad):
Great effort!

I wish to through in another part - the description of this issue is about sampling, but the implementation is about *random* sampling.
This is not always the case, nor it is very fast (indeed, calling 1M times Random.nextInt would be measurable by itself IMHO).
A different sample could be
{{code}}
int acceptedModulu = (int)(1/sampleRatio);

int next() {
  do {
    nextDoc = inner.next();
  } while (nextDoc != NO_MORE_DOCX && nextDoc % acceptedModulu != 0) ;

  return nextDoc;
}
{{code}}

This should be faster as a sampler, and perhaps saves us from creating a new {DocIdSet}.

One last thing - if I did the math right - the sample crafted by the code in the patch would be twice as large as the user may expect.
For a sample ratio of 0.1, the random.nextInt() would be called with 10, so the avg. "jump" is actually 5 - and every 5th document in the original set (again, in avg) would be selected, and not every 10th in avg. I think the random.nextInt should be called with twice the size it is called now (e.g 20, making the avg random selection 10).

> Facet sampling
> --------------
>
>                 Key: LUCENE-5476
>                 URL: https://issues.apache.org/jira/browse/LUCENE-5476
>             Project: Lucene - Core
>          Issue Type: Improvement
>            Reporter: Rob Audenaerde
>         Attachments: LUCENE-5476.patch, SamplingFacetsCollector.java
>
>
> With LUCENE-5339 facet sampling disappeared. 
> When trying to display facet counts on large datasets (>10M documents) counting facets is rather expensive, as all the hits are collected and processed. 
> Sampling greatly reduced this and thus provided a nice speedup. Could it be brought back?



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org