You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Rohit <ro...@in-rev.com> on 2012/08/08 14:10:23 UTC
numFound changes on changing start and rows
Hi,
We are using Solr3.6 and 2 shards, we are noticing that when we fire a query
with start as 0 and rows X the total numFound and the total numFound changes
when we fire the same exact query with start as y and rows X.
For example.
First time
query=abc&start=0&rows=4000
numFound- 56000
Second time
query=abc&start=4000&rows=4000
numFound- 55998
What can cause this?
Regards,
Rohit
RE: numFound changes on changing start and rows
Posted by Jie Sun <js...@yahoo.com>.
ok when my head is cooled down, I remember this old school issue... that I
have been dealing with it myself.
so I do not expect this can be straighten out or fixed in anyways.
basically when you have to sorted results sets you need to merge, and
paginate through, it is never an easy job (if all is possible) to figure out
what is exactly the number if you only require a portion of the results
being returned.
for example if 1 set has 40,000 rows returned, the other set has 50,000
returned, and you want the start=440 and rows=20 (paginate on UI), the
typical algorithm will be sort both sets and return the near portion of both
sets, toss away the duplicates in that range (20 rows), so even you
calcualte with the duplicates prior to that start point, you have no way to
tell how many duplicates after that point, so you really do not know for
fact the exact / accurate numFound, unless you require return the whole
thing. and that is why when I give a huge rows number, it will give me the
accurate count each time. However, solr shard query will throw 500 server
error if the returned set is around 50k, which is reasonable.
So find work around in the context is the only solution. Check with google
search pattern, may get some fuzzy idea :-)
thanks
jie
--
View this message in context: http://lucene.472066.n3.nabble.com/numFound-changes-on-changing-start-and-rows-tp3999752p4061633.html
Sent from the Solr - User mailing list archive at Nabble.com.
RE: numFound changes on changing start and rows
Posted by Jie Sun <js...@yahoo.com>.
any update on this?
will this be addressed/fixed?
in our system, our UI will allow user to paginate through search results.
As my in deep test find out, if the rows=0, the results size is consistently
the total sum of the documents on all shards regardless there is any
duplicates; if the rows is a number larger than the supposedly returned the
merge document number, the result numFound is accurate and consistent,
however, if the rows is with a number smaller than the supposedly merge
results size, it will be non-deterministic.
unfortunately, in our system, it is not easy to work around this problem. we
have to issue and query whenever use click on Next button, and the rows is
20 in our case and in most of the cases it is smaller than the merged
results size, so we get a different number each time.
If we do rows=0 up in front, it wont work either, since we want the accurate
number and others may have indexed new documents at the same time.
Especially when user hit the last page, sometimes we see the numFound off by
hundreds, this wont work.
please advice.
thanks
Jie
--
View this message in context: http://lucene.472066.n3.nabble.com/numFound-changes-on-changing-start-and-rows-tp3999752p4061628.html
Sent from the Solr - User mailing list archive at Nabble.com.
RE: numFound changes on changing start and rows
Posted by Rohit <ro...@in-rev.com>.
I can cross check our shards once again, but I am sure this is not the case.
Regards,
Rohit
Mobile: +91-9901768202
-----Original Message-----
From: Chris Hostetter [mailto:hossman_lucene@fucit.org]
Sent: 08 August 2012 21:04
To: solr-user@lucene.apache.org
Subject: Re: numFound changes on changing start and rows
: We are using Solr3.6 and 2 shards, we are noticing that when we fire a
query
: with start as 0 and rows X the total numFound and the total numFound
changes
: when we fire the same exact query with start as y and rows X.
The only situation where i've ever heard of this happening is when multiple
shards have documents with identical uniqueKeys...
http://mail-archives.apache.org/mod_mbox/lucene-solr-user/201206.mbox/%3CCAP
oDz8S4Z-jnyptFXdv7VJdWntY0Lx_=Nzhvq0qTCFqyx7MFMw@mail.gmail.com%3E
http://mail-archives.apache.org/mod_mbox/lucene-solr-user/201206.mbox/%3Calp
ine.DEB.2.00.1206191429520.19329@bester%3E
http://mail-archives.apache.org/mod_mbox/lucene-solr-user/201206.mbox/%3CCAP
oDz8S59kzUdCAZwHRquzUhM=C90ReyCNe3Au00xsc=wh0mHQ@mail.gmail.com%3E
As noted in the docs..
http://wiki.apache.org/solr/DistributedSearch?#Distributed_Searching_Limitat
ions
"The unique key field must be unique across all shards. If docs with
duplicate unique keys are encountered, Solr will make an attempt to return
valid results, but the behavior may be non-deterministic. "
-Hoss
Re: numFound changes on changing start and rows
Posted by Michael Della Bitta <mi...@appinions.com>.
Our documents are keyed with UUIDs, and we shard chronologically. The
write events are issued as part of a SQS queue that only allows one
reader to see the message. I think it's pretty unlikely that we have
more than one document with the same uniquekey.
I can actually prove this if it will help the discussion, since I just
dumped 4 of our shards to JSON, but it's over 117 million docs, so
I'll wait until someone asks. :)
Michael Della Bitta
------------------------------------------------
Appinions | 18 East 41st St., Suite 1806 | New York, NY 10017
www.appinions.com
Where Influence Isn’t a Game
On Wed, Aug 8, 2012 at 11:33 AM, Chris Hostetter
<ho...@fucit.org> wrote:
>
> : We are using Solr3.6 and 2 shards, we are noticing that when we fire a query
> : with start as 0 and rows X the total numFound and the total numFound changes
> : when we fire the same exact query with start as y and rows X.
>
> The only situation where i've ever heard of this happening is when
> multiple
> shards have documents with identical uniqueKeys...
>
> http://mail-archives.apache.org/mod_mbox/lucene-solr-user/201206.mbox/%3CCAPoDz8S4Z-jnyptFXdv7VJdWntY0Lx_=Nzhvq0qTCFqyx7MFMw@mail.gmail.com%3E
> http://mail-archives.apache.org/mod_mbox/lucene-solr-user/201206.mbox/%3Calpine.DEB.2.00.1206191429520.19329@bester%3E
> http://mail-archives.apache.org/mod_mbox/lucene-solr-user/201206.mbox/%3CCAPoDz8S59kzUdCAZwHRquzUhM=C90ReyCNe3Au00xsc=wh0mHQ@mail.gmail.com%3E
>
> As noted in the docs..
>
> http://wiki.apache.org/solr/DistributedSearch?#Distributed_Searching_Limitations
>
> "The unique key field must be unique across all shards. If docs with
> duplicate unique keys are encountered, Solr will make an attempt to return
> valid results, but the behavior may be non-deterministic. "
>
>
>
>
> -Hoss
Re: numFound changes on changing start and rows
Posted by Chris Hostetter <ho...@fucit.org>.
: We are using Solr3.6 and 2 shards, we are noticing that when we fire a query
: with start as 0 and rows X the total numFound and the total numFound changes
: when we fire the same exact query with start as y and rows X.
The only situation where i've ever heard of this happening is when
multiple
shards have documents with identical uniqueKeys...
http://mail-archives.apache.org/mod_mbox/lucene-solr-user/201206.mbox/%3CCAPoDz8S4Z-jnyptFXdv7VJdWntY0Lx_=Nzhvq0qTCFqyx7MFMw@mail.gmail.com%3E
http://mail-archives.apache.org/mod_mbox/lucene-solr-user/201206.mbox/%3Calpine.DEB.2.00.1206191429520.19329@bester%3E
http://mail-archives.apache.org/mod_mbox/lucene-solr-user/201206.mbox/%3CCAPoDz8S59kzUdCAZwHRquzUhM=C90ReyCNe3Au00xsc=wh0mHQ@mail.gmail.com%3E
As noted in the docs..
http://wiki.apache.org/solr/DistributedSearch?#Distributed_Searching_Limitations
"The unique key field must be unique across all shards. If docs with
duplicate unique keys are encountered, Solr will make an attempt to return
valid results, but the behavior may be non-deterministic. "
-Hoss
Re: numFound changes on changing start and rows
Posted by Michael Della Bitta <mi...@appinions.com>.
Sorry, in my time range example, I forgot to mention that you can
repeatedly execute the 8 hour query and receive no results, even after
the 7 hour query retrieves them.
Kind of an important detail to not forget. :)
Michael Della Bitta
------------------------------------------------
Appinions | 18 East 41st St., Suite 1806 | New York, NY 10017
www.appinions.com
Where Influence Isn’t a Game
On Wed, Aug 8, 2012 at 8:42 AM, Michael Della Bitta
<mi...@appinions.com> wrote:
> We've noticed some pretty non-deterministic behavior with sharded
> setups as well.
>
> One thing we've noticed is that a query server can hang on to the set
> of document ids that correspond to a given query even if caching is
> off, which results in some weird behavior, such as a query like:
>
> timestamp:[NOW TO NOW-8HOUR]
>
> Will return no results, but:
>
> timestamp:[NOW TO NOW-7HOUR]
>
> ...will, IF the former query was executed prior to a replication that
> brought in documents that match both queries.
>
> We've also noticed numFound changing during paging through query
> results, as you mention.
>
> One of our use cases is more of a reporting function and it depends on
> there being more deterministic behavior than this, so in the shifting
> numFound case, we've written code to detect a shift and restart the
> query from the beginning.
>
> In the case of cached documentIds not revealing fresher information,
> I'm worried that we're going to have to move to querying each shard in
> turn, which may mean we get left out of using SolrCloud. We haven't
> tried to evaluate it yet, however.
>
> Michael Della Bitta
>
> ------------------------------------------------
> Appinions | 18 East 41st St., Suite 1806 | New York, NY 10017
> www.appinions.com
> Where Influence Isn’t a Game
>
>
> On Wed, Aug 8, 2012 at 8:10 AM, Rohit <ro...@in-rev.com> wrote:
>> Hi,
>>
>>
>>
>> We are using Solr3.6 and 2 shards, we are noticing that when we fire a query
>> with start as 0 and rows X the total numFound and the total numFound changes
>> when we fire the same exact query with start as y and rows X.
>>
>>
>>
>> For example.
>>
>>
>>
>> First time
>>
>> query=abc&start=0&rows=4000
>>
>> numFound- 56000
>>
>>
>>
>> Second time
>>
>> query=abc&start=4000&rows=4000
>>
>> numFound- 55998
>>
>>
>>
>> What can cause this?
>>
>>
>>
>>
>>
>>
>>
>> Regards,
>>
>> Rohit
>>
>>
>>
Re: numFound changes on changing start and rows
Posted by Michael Della Bitta <mi...@appinions.com>.
We've noticed some pretty non-deterministic behavior with sharded
setups as well.
One thing we've noticed is that a query server can hang on to the set
of document ids that correspond to a given query even if caching is
off, which results in some weird behavior, such as a query like:
timestamp:[NOW TO NOW-8HOUR]
Will return no results, but:
timestamp:[NOW TO NOW-7HOUR]
...will, IF the former query was executed prior to a replication that
brought in documents that match both queries.
We've also noticed numFound changing during paging through query
results, as you mention.
One of our use cases is more of a reporting function and it depends on
there being more deterministic behavior than this, so in the shifting
numFound case, we've written code to detect a shift and restart the
query from the beginning.
In the case of cached documentIds not revealing fresher information,
I'm worried that we're going to have to move to querying each shard in
turn, which may mean we get left out of using SolrCloud. We haven't
tried to evaluate it yet, however.
Michael Della Bitta
------------------------------------------------
Appinions | 18 East 41st St., Suite 1806 | New York, NY 10017
www.appinions.com
Where Influence Isn’t a Game
On Wed, Aug 8, 2012 at 8:10 AM, Rohit <ro...@in-rev.com> wrote:
> Hi,
>
>
>
> We are using Solr3.6 and 2 shards, we are noticing that when we fire a query
> with start as 0 and rows X the total numFound and the total numFound changes
> when we fire the same exact query with start as y and rows X.
>
>
>
> For example.
>
>
>
> First time
>
> query=abc&start=0&rows=4000
>
> numFound- 56000
>
>
>
> Second time
>
> query=abc&start=4000&rows=4000
>
> numFound- 55998
>
>
>
> What can cause this?
>
>
>
>
>
>
>
> Regards,
>
> Rohit
>
>
>