Pagination issue

[Halim Chaibi <ha...@gmail.com>]

Hi,
We are using solr cloud 4.10.4, with 4 shards, one replica for each shard.
The id field definition is a unique composite key id (key_id!document_id)
<field name="id" type="string" indexed="true" stored="true" required="true"
/>

We are having an issue with the pagination:
Some documents found with a request, let's says, of 100 documents starting
from 200(rows=100&start=200)
Are still found with the same request with rows=100&start=300.
The only difference with the same document found with the two request is in
the timestamp, it's very slight difference a few ms.

Please, can you help us to figure out what is happening ?
thank you



--
View this message in context: http://lucene.472066.n3.nabble.com/Pagination-issue-tp4308226.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Pagination issue

[Halim Chaibi <ha...@gmail.com>]

Yes I am sure the uniqueKeyId are unique

Here are exactly my two requests:

http://SOLR_IP:8201/solr/Staging/select?q=*:*&rows=100&start=800&fl=id,position_id,job_id&sort=position_posted_date_id%20desc

http://SOLR_IP:8201/solr/Staging/select?q=*:*&rows=100&start=900&fl=id,position_id,job_id&sort=position_posted_date_id%20desc

I looked to the docId I am getting through the two requests using
q=id:suspect_id&distrib=false, I can find it only in one shard and in it's
replica (with different timestamp)

I also noticed that I am having two slightly different result through the
exactly two same request.

Thank you for your help




--
View this message in context: http://lucene.472066.n3.nabble.com/Pagination-issue-tp4308226p4308297.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Pagination issue

[Erick Erickson <er...@gmail.com>]

Got it. The problem is tiebreakers. Consider two documents with the
same date. Whenever two documents are tied through all the sort
criteria you specify then the _internal_ Lucene document ID is used as
a tiebreaker.

The kicker is that doc1 and doc2 don't have the same internal IDs on
replica1 and replica2. In fact there is no guarantee that doc1 even
has the same relationship to doc2 on the two replicas. That is, if
internal_id_doc1 < internal_id_doc2 on replica 1 its quite possible
that internal_id_doc1 > internal_id_doc2 on replica2.

This has to do with segment merging and the fact that the segments are
all merged locally, there's no coordination between replicas.

So adding the secondary sort by ID is the usual way to handle this, so
you're on the right track.

Best,
Erick

On Fri, Dec 2, 2016 at 7:56 AM, Halim Chaibi <ha...@gmail.com> wrote:
> I think it's related to the sort clause: sort=position_posted_date_id, this
> is happening only when documents have the same position_posted_date_id, I am
> not sure what would be the default sort in this case, but depending on which
> shards is handling the request, I am getting same result but sorted
> differently.
>
> When adding a second sort by id like this
> sort=position_posted_date_id%20desc,id%20desc, the issue is no longer
> happening.
>
> Thanks
>
>
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/Pagination-issue-tp4308226p4308310.html
> Sent from the Solr - User mailing list archive at Nabble.com.

Re: Pagination issue

[Halim Chaibi <ha...@gmail.com>]

I think it's related to the sort clause: sort=position_posted_date_id, this
is happening only when documents have the same position_posted_date_id, I am
not sure what would be the default sort in this case, but depending on which
shards is handling the request, I am getting same result but sorted
differently.

When adding a second sort by id like this
sort=position_posted_date_id%20desc,id%20desc, the issue is no longer
happening.

Thanks



--
View this message in context: http://lucene.472066.n3.nabble.com/Pagination-issue-tp4308226p4308310.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Pagination issue

[Erick Erickson <er...@gmail.com>]

Are you absolutely and totally sure that the <uniqueKey> s are unique
(id by default) when you say that the only difference is the date? It
sounds like you're somehow getting the same doc indexed with two (or
more) different IDs.

You can look at this by identifying some suspect IDs and issuing queries like
solr/collection1_shard1_replica1?q=id:suspect_id&distrib=false

against a single replica for each shard. The doc shoule be found on
one and only one shard (any replica).

It's important that you add &distrib=false and address a specific
replica when you do this.

Best,
Erick



On Thu, Dec 1, 2016 at 11:42 AM, Halim Chaibi <ha...@gmail.com> wrote:
> Hi,
> We are using solr cloud 4.10.4, with 4 shards, one replica for each shard.
> The id field definition is a unique composite key id (key_id!document_id)
> <field name="id" type="string" indexed="true" stored="true" required="true"
> />
>
> We are having an issue with the pagination:
> Some documents found with a request, let's says, of 100 documents starting
> from 200(rows=100&start=200)
> Are still found with the same request with rows=100&start=300.
> The only difference with the same document found with the two request is in
> the timestamp, it's very slight difference a few ms.
>
> Please, can you help us to figure out what is happening ?
> thank you
>
>
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/Pagination-issue-tp4308226.html
> Sent from the Solr - User mailing list archive at Nabble.com.