You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Bryan Bende <bb...@gmail.com> on 2014/05/19 22:05:55 UTC
Issue paging when sorting on a Date field
Using Solr 4.6.1 and in my schema I have a date field storing the time a
document was added to Solr.
I have a utility program which:
- queries for all of the documents in the previous day sorted by create date
- pages through the results keeping track of the unique document ids
- compare the total number of unique doc ids to the numFound to see if it
they match
I've noticed that if I use a page size larger than the number of documents
for the given day (aka get everything in one query), then everything works
as expected (results sorted correctly, unique doc ids size == numFound).
However, when I use a smaller page say, say 10 rows per page, I randomly
see cases where the last document of a page will be duplicated as the first
document of the next page, even though the "start" and "rows" parameters
increased correctly. So I might see something like numFound=100 but unique
doc ids is 97, and then I see three occurrences where the last doc id on a
page was also the first on the next page.
It is not consistent between tests, the number of occurrences changes and
the locations of the occurrences can change as well. The larger the result
set, and smaller the page size, the more frequent the occurrences are.
The only thing I have noticed is that if I change the sorting of the
initial query to use a non-date field, then this doesn't happen anymore.
Are there any know issues/limitations sorting/paging on a date field ?
The only mention I can find is this thread:
http://mail-archives.apache.org/mod_mbox/lucene-solr-user/200909.mbox/%3C57912A0644B6AB4381816DE07CB1C38D02A00F5C@S2NA1EXCLUSTER.na1.ad.group%3E
Re: Issue paging when sorting on a Date field
Posted by Shawn Heisey <so...@elyograg.org>.
On 5/19/2014 2:05 PM, Bryan Bende wrote:
> Using Solr 4.6.1 and in my schema I have a date field storing the time a
> document was added to Solr.
>
> I have a utility program which:
> - queries for all of the documents in the previous day sorted by create date
> - pages through the results keeping track of the unique document ids
> - compare the total number of unique doc ids to the numFound to see if it
> they match
>
> I've noticed that if I use a page size larger than the number of documents
> for the given day (aka get everything in one query), then everything works
> as expected (results sorted correctly, unique doc ids size == numFound).
>
> However, when I use a smaller page say, say 10 rows per page, I randomly
> see cases where the last document of a page will be duplicated as the first
> document of the next page, even though the "start" and "rows" parameters
> increased correctly. So I might see something like numFound=100 but unique
> doc ids is 97, and then I see three occurrences where the last doc id on a
> page was also the first on the next page.
This *sounds* like a situation where you have a sharded index that has
the same uniqueKey value in more than one shard. This situation will
cause Solr to behave in a way that looks completely unpredictable.
There is no way for Solr to deal with this problem in a way that would
not consume large amounts of real time, CPU time, and RAM ... so Solr
does not do anything for dealing with this problem other than removing
duplicates from the actual results returned -- which is actually how the
discrepancies occur.
If you are absolutely sure that you are not running into the duplicate
document problem I described, then I am not sure what's going on. It
might be related to the sort, and if that's true, adding a second sort
parameter using your uniqueKey field might be a solution.
Thanks,
Shawn
Re: Issue paging when sorting on a Date field
Posted by Chris Hostetter <ho...@fucit.org>.
: So I think when I was paging through the results, if the query for page N
: was handled by replica1 and page N+1 handled by replica2, and the page
: boundary happened to be where the reversed rows were, this would produce
: the behavior I was seeing where the last row from the previous page was
: also the first row from the next page.
Right, this can actually happen even in a single solr node.
when 2 docs have identical sort values, the final ordering is
non-deterministic -- they usually come back in "index" order (the order
that they appear in the segments on disk) but that's not garunteed. In
particular if you have concurrent index updates that cause segment merges
the order of documents can change (even if those updates don't directly
affect the docs being reutrned).
If you want to ensure that docs with equal sort values are returned in a
consistent order across pagination (in either single or multi-node setups)
you have to have a "tie breaker" sort of some kind -- the uniquekey can be
useul here.
-Hoss
http://www.lucidworks.com/
Re: Issue paging when sorting on a Date field
Posted by Bryan Bende <bb...@gmail.com>.
This is using the solr.TrieDateField, it is the field type "date" from the
example schema in solr 4.6.1:
<fieldType name="date" class="solr.TrieDateField" precisionStep="0"
positionIncrementGap="0" />
After further testing I was only able to reproduce this in a sharded &
replicated environment (numShards=3, replicationFactor=2) and I think I
have narrowed down the issue, and at this point it may be expected
behavior...
I took a query like q=create_date:[2014-05-19T00:00:00Z TO
2014-05-19T23:59:59Z]&sort=create_date DESC&start=0&rows=10000 which should
get all the documents for yesterday sorted by create date, and then added
distrib=false and ran it against shard1_replica1 and shard1_replica2. Then
I diff'd the files and it showed 5 occurrences where two consecutive rows
in one replica were reversed in the other replica, and in all 5 cases the
flipped flopped rows had the exact same create_date value, which happened
to only go down to the minute.
As an example:
shard1_replica1:
...
docX, 2014-05-19T20:15:00Z
docY, 2014-05-19T20:15:00Z
...
shard1_replica2:
...
docY, 2014-05-19T20:15:00Z
docX, 2014-05-19T20:15:00Z
...
So I think when I was paging through the results, if the query for page N
was handled by replica1 and page N+1 handled by replica2, and the page
boundary happened to be where the reversed rows were, this would produce
the behavior I was seeing where the last row from the previous page was
also the first row from the next page.
I guess the obvious solution is to ensure the date field is always more
granular than minutes, or add another field to the sort order to
consistently break ties.
On Mon, May 19, 2014 at 4:19 PM, Chris Hostetter
<ho...@fucit.org>wrote:
>
> : Using Solr 4.6.1 and in my schema I have a date field storing the time a
> : document was added to Solr.
>
> what *exactly* does your schema look like? are you using "solr.DateField"
> or "solr.TrieDateField" ? what field options do you have specified?
>
> : I have a utility program which:
> : - queries for all of the documents in the previous day sorted by create
> date
> : - pages through the results keeping track of the unique document ids
> : - compare the total number of unique doc ids to the numFound to see if it
> : they match
>
> what *exactly* do your queries look like? show us some examples please
> (URL & results). Are you using distributed searching across multiple
> nodes, or a single node? do you have concurrent updates going on during
> your test?
>
> : It is not consistent between tests, the number of occurrences changes and
> : the locations of the occurrences can change as well. The larger the
> result
> : set, and smaller the page size, the more frequent the occurrences are.
>
> if you bring up a test instance of Solr using your current configs, can
> you reproduce (even occasionally) with some synthetic data you can share
> with us? If so please provide your full configs & sample data (ie: create
> a Jira & attach all the neccessary files i na ZIP)
>
>
> -Hoss
> http://www.lucidworks.com/
>
Re: Issue paging when sorting on a Date field
Posted by Chris Hostetter <ho...@fucit.org>.
: Using Solr 4.6.1 and in my schema I have a date field storing the time a
: document was added to Solr.
what *exactly* does your schema look like? are you using "solr.DateField"
or "solr.TrieDateField" ? what field options do you have specified?
: I have a utility program which:
: - queries for all of the documents in the previous day sorted by create date
: - pages through the results keeping track of the unique document ids
: - compare the total number of unique doc ids to the numFound to see if it
: they match
what *exactly* do your queries look like? show us some examples please
(URL & results). Are you using distributed searching across multiple
nodes, or a single node? do you have concurrent updates going on during
your test?
: It is not consistent between tests, the number of occurrences changes and
: the locations of the occurrences can change as well. The larger the result
: set, and smaller the page size, the more frequent the occurrences are.
if you bring up a test instance of Solr using your current configs, can
you reproduce (even occasionally) with some synthetic data you can share
with us? If so please provide your full configs & sample data (ie: create
a Jira & attach all the neccessary files i na ZIP)
-Hoss
http://www.lucidworks.com/