[jira] [Updated] (SOLR-10231) Cursor value always different for last page with sorting by a date based function using NOW

["Hoss Man (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SOLR-10231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hoss Man updated SOLR-10231:
----------------------------
    Summary: Cursor value always different for last page with sorting by a date based function using NOW  (was: Cursor value always different for last page with sorting by function)

this isn't a general problem with sorting by function, the problem is specific to sorting by a date based function that involves the {{NOW}} constant.

 The problem is that every time this function is computed for a document, the value can change -- so when requests asks for everything with a cursor value "after" the computed value of the last doc on the previous request, you're getting overlap with some existing documents -- and ultimately the cursor never ends, because the "last" doc constantly computes a sort value that comes "after" the sort value it computed the "last" time the request was made.

(what's happening is essentially the same as what you would see if, between every request for the "next" page of the cursor using "sort=counter+asc", someone did an atomic update on the doc to {{inc counter}} ... but in this case the counter increase is just happening because time elapses)

----

The best work around I can suggest would be to to include a fixed value for the {{NOW}} param in any requests involving sorting by date math -- that way the computed sort values will be consistent across all the subsequent requests.

(Perhaps the NOW value should also be encoded into the cursor values so this happens automatically under the covers? ... not sure if that's a good idea in general, would need to think about it more)


> Cursor value always different for last page with sorting by a date based function using NOW
> -------------------------------------------------------------------------------------------
>
>                 Key: SOLR-10231
>                 URL: https://issues.apache.org/jira/browse/SOLR-10231
>             Project: Solr
>          Issue Type: Bug
>      Security Level: Public(Default Security Level. Issues are Public) 
>          Components: SearchComponents - other
>    Affects Versions: 4.10.2
>            Reporter: Dmitry Kan
>
> Cursor based results fetching is a deal breaker for search performance.
> It works extremely well when paging using sort by field(s).
> Example, that works (Id is unique field in the schema):
> Query:
> {code}
> http://solr-host:8983/solr/documents/select?q=*:*&fq=DocumentId:76581059&cursorMark=AoIGAAAAAC5TU1ItNzY1ODEwNTktMQ==&fl=DocumentId&sort=UserId+asc%2CId+desc&rows=1
> {code}
> Response:
> {code}
> <response>
> <lst name="responseHeader">
> <int name="status">0</int>
> <int name="QTime">4</int>
> <lst name="params">
> <str name="q">*:*</str>
> <str name="fl">DocumentId</str>
> <str name="cursorMark">AoIGAAAAAC5TU1ItNzY1ODEwNTktMQ==</str>
> <str name="fq">DocumentId:76581059</str>
> <str name="sort">UserId asc,Id desc</str>
> <str name="rows">1</str>
> </lst>
> </lst>
> <result name="response" numFound="1" start="0"/>
> <str name="nextCursorMark">AoIGAAAAAC5TU1ItNzY1ODEwNTktMQ==</str>
> </response>
> {code}
> nextCursorMark equals to cursorMark and so we know this is last page.
> However, sorting by function behaves differently:
> Query:
> {code}
> http://solr-host:8983/solr/documents/select?rows=1&q=*:*&fq=DocumentId:76581059&cursorMark=AoIFQf9yCCAAAAAuU1NSLTc2NTgxMDU5LTE=&fl=DocumentId&sort=min(ms(NOW,DynamicDateField_1),ms(NOW,DynamicDateField_12),ms(NOW,DynamicDateField_3),ms(NOW,DynamicDateField_5))%20asc,Id%20desc
> {code}
> Response:
> {code}
> <response>
> <lst name="responseHeader">
> <int name="status">0</int>
> <int name="QTime">6</int>
> <lst name="params">
> <str name="q">*:*</str>
> <str name="fl">DocumentId</str>
> <str name="cursorMark">AoIFQf9yCCAAAAAuU1NSLTc2NTgxMDU5LTE=</str>
> <str name="fq">DocumentId:76581059</str>
> <str name="sort">
> min(ms(NOW,DynamicDateField_1),ms(NOW,DynamicDateField_12),ms(NOW,DynamicDateField_3),ms(NOW,DynamicDateField_5)) asc,Id desc
> </str>
> <str name="rows">1</str>
> </lst>
> </lst>
> <result name="response" numFound="1" start="0">
> <doc>
> <str name="DocumentId">76581059</str>
> </doc>
> </result>
> <str name="nextCursorMark">AoIFQf9yFyAAAAAuU1NSLTc2NTgxMDU5LTE=</str>
> </response>
> {code}
> nextCursorMark does not equal to cursorMark, which suggests there are more results. Which is not true (numFound=1). And so the client goes into infinite loop.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org