You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Steve Rowe (JIRA)" <ji...@apache.org> on 2013/12/06 20:47:39 UTC

[jira] [Comment Edited] (SOLR-5463) Provide cursor/token based "searchAfter" support that works with arbitrary sorting (ie: "deep paging")

    [ https://issues.apache.org/jira/browse/SOLR-5463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13841598#comment-13841598 ] 

Steve Rowe edited comment on SOLR-5463 at 12/6/13 7:46 PM:
-----------------------------------------------------------

bq. However.... when manually using this with the example configs in order to sanity check that that was going to feel right as an API, i discovered problems where the queryResultCache was coming into play and never getting past "page" 3. I felt stupid for not thinking to test this earlier, and updated the test configs to include queryResultCache, but i can't reproduce with a failure ... still not sure why.

All the new tests pass for me with the latest patch.

I'm never getting past "page" 2 (rather than 3) - {{searchAfter=<the page 2 totem>}} returns the same totem, along with the same page 2 results. 

Maybe not unexpected, but when I commented out {{<<queryResultWindowSize>20</queryResultWindowSize>}} and set {{<queryResultMaxDocsCached>}} to a number smaller than the {{rows}} param, I can get past page 2 - success getting as far as page 5, in fact.

Seems like the queryResultCache isn't paying attention to all query params?  (I don't know the code myself.) From {{solrconfig.xml}} - a literal reading is that only the {{q}}, {{sort}}, {{start}} and {{rows}} params are taken into consideration:

{code:xml}
    <!-- Query Result Cache
         
         Caches results of searches - ordered lists of document ids
         (DocList) based on a query, a sort, and the range of documents requested.  
      -->
{code}


was (Author: steve_rowe):
bq. However.... when manually using this with the example configs in order to sanity check that that was going to feel right as an API, i discovered problems where the queryResultCache was coming into play and never getting past "page" 3. I felt stupid for not thinking to test this earlier, and updated the test configs to include queryResultCache, but i can't reproduce with a failure ... still not sure why.

All the new tests pass for me with the latest patch.

I'm never getting past "page" 2 (rather than 3) - {{searchAfter=<the page 2 totem>}} returns the same totem, along with the same page 2 results. 

Maybe not unexpected, but when I commented out {{<<queryResultWindowSize>20</queryResultWindowSize>}} and set {{<queryResultMaxDocsCached>}} to a number smaller than the {{rows}} param, I can get past page 2 - success getting as far as page 5, in fact.

Seems like the queryResultCache isn't paying attention to all query params?  (I don't know the code myself.) From {{solrconfig.xml}} - a literal reading is that only the {{q}}, {{sort}}, {{start}} and {{rows}} params are taken into consideration:

{code:xml}
    <!-- Query Result Cache
         
         Caches results of searches - ordered lists of document ids
         (DocList) based on a query, a sort, and the range of documents requested.  
      -->
{xml}

> Provide cursor/token based "searchAfter" support that works with arbitrary sorting (ie: "deep paging")
> ------------------------------------------------------------------------------------------------------
>
>                 Key: SOLR-5463
>                 URL: https://issues.apache.org/jira/browse/SOLR-5463
>             Project: Solr
>          Issue Type: New Feature
>            Reporter: Hoss Man
>            Assignee: Hoss Man
>         Attachments: SOLR-5463__straw_man.patch, SOLR-5463__straw_man.patch, SOLR-5463__straw_man.patch, SOLR-5463__straw_man.patch, SOLR-5463__straw_man.patch, SOLR-5463__straw_man.patch
>
>
> I'd like to revist a solution to the problem of "deep paging" in Solr, leveraging an HTTP based API similar to how IndexSearcher.searchAfter works at the lucene level: require the clients to provide back a token indicating the sort values of the last document seen on the previous "page".  This is similar to the "cursor" model I've seen in several other REST APIs that support "pagnation" over a large sets of results (notable the twitter API and it's "since_id" param) except that we'll want something that works with arbitrary multi-level sort critera that can be either ascending or descending.
> SOLR-1726 laid some initial ground work here and was commited quite a while ago, but the key bit of argument parsing to leverage it was commented out due to some problems (see comments in that issue).  It's also somewhat out of date at this point: at the time it was commited, IndexSearcher only supported searchAfter for simple scores, not arbitrary field sorts; and the params added in SOLR-1726 suffer from this limitation as well.
> ---
> I think it would make sense to start fresh with a new issue with a focus on ensuring that we have deep paging which:
> * supports arbitrary field sorts in addition to sorting by score
> * works in distributed mode



--
This message was sent by Atlassian JIRA
(v6.1#6144)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org