[GitHub] incubator-ponymail issue #334: Bug: stats.lua ignores maxresults if it is > ...

[sebbASF <gi...@git.apache.org>]

GitHub user sebbASF opened an issue:

    https://github.com/apache/incubator-ponymail/issues/334

    Bug: stats.lua ignores maxresults if it is > 10000

    If maxresults is < 10000, then at most maxresults will be returned, because the query uses the size=maxresults parameter.
    
    If maxresults is > 10000 then stats.lua has to use scan/scroll to fetch the results.
    
    Now when using scrolling, the size parameter is the max number of results per batch and if using scan/scroll it is the maximum results per shard per batch. It does not control the total responses; the scrolling only stops when no more results are available.
    
    So when using scrolling the maxresults limit is ignored.


----

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-ponymail issue #334: Bug: stats.lua ignores maxresults if it is > ...

[sebbASF <gi...@git.apache.org>]

Github user sebbASF commented on the issue:

    https://github.com/apache/incubator-ponymail/issues/334
  
    There does not seem to be an efficient way to fix this without using search_after, which requires ES 5.x.
    
    For most efficient scrolling, the query must use _doc order (*). This means that the first N results won't in general be the same as the corresponding query using the desired sort order.
    
    For versions before ES 5.x, it looks like the only options are:
    - use the 'wrong' sort order for the scroll, probably causing performance issues
    - stop scrolling early; this will likely skew the results.
    - ignore the problem
    
    (*) this is not a problem when scrolling is used as intended - i.e. to process all matching documents.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---