You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by devb <pa...@behaviormatrix.com> on 2013/02/11 20:09:47 UTC
Solr Cloud: Duplicate records while retrieving documents
We are running a six node SOLR cloud which 3 shards and 3 replications. The
version of solr cloud is 4.0.0.2012.08.06.22.50.47. We use Python PySolr
client to interact with Solr. Documents that we add to solr have a unique id
and it can never have duplicates.
Our use case is to query the index for a give searchterm and pull all
documents that matches the query. Usually our query hits over 40K documents.
While we iterate through all 40K+ documents, after few iteration, we see the
same documents ids repeated over and over, and at the end we see some 20-33%
of the records are duplicates.
In the below code snippet after some iterations, we see a difference in the
length of idslist and idsset. Any insight into how to troubleshoot this
issue is greatly appreciated.
from pysolr import Solr
solr= Solr('http://solrhost/solr/#/collection1')
if __name__ == '__main__':
idslist = list()
idsset = set()
query = 'snow'
skip = 0
limit= 500
i = 0
while True:
response = solr.search(q=query, rows=limit, start=skip,
shards='host1:7575/solr,host2:7575/solr,host3:7575/solr', fl="id,source")
if skip == 0:
hits = response.hits
line = "Solr Hits Count: (%s)\n" % (hits)
print line
if len(response.docs) == 0:
break
for result in response:
idslist.append(result['id'])
idsset.add(result['id'])
if i % 500 == 0:
print len(idslist), len(idsset)
i+=1
skip += limit
--
View this message in context: http://lucene.472066.n3.nabble.com/Solr-Cloud-Duplicate-records-while-retrieving-documents-tp4039776.html
Sent from the Solr - User mailing list archive at Nabble.com.
RE: Solr Cloud: Duplicate records while retrieving documents
Posted by devb <pa...@behaviormatrix.com>.
Thanks for your detailed response. We will try your suggestions and let you know how it goes.
Paddy
From: Shawn Heisey-4 [via Lucene] [mailto:ml-node+s472066n4039838h93@n3.nabble.com]
Sent: Monday, February 11, 2013 6:21 PM
To: Paddy Krishnamoorthy
Subject: Re: Solr Cloud: Duplicate records while retrieving documents
On 2/11/2013 12:09 PM, devb wrote:
> We are running a six node SOLR cloud which 3 shards and 3 replications. The
> version of solr cloud is 4.0.0.2012.08.06.22.50.47. We use Python PySolr
> client to interact with Solr. Documents that we add to solr have a unique id
> and it can never have duplicates.
> Our use case is to query the index for a give searchterm and pull all
> documents that matches the query. Usually our query hits over 40K documents.
> While we iterate through all 40K+ documents, after few iteration, we see the
> same documents ids repeated over and over, and at the end we see some 20-33%
> of the records are duplicates.
> In the below code snippet after some iterations, we see a difference in the
> length of idslist and idsset. Any insight into how to troubleshoot this
> issue is greatly appreciated.
For discussion purposes Let's first assume that there are no bugs in
Solr. I don't think we can make that assumption, of course.
General note 1: Your Solr URL in your code has a # in it. The URLs with
# in them are Admin UI URLs. If that's working, I'm amazed... I would
take that part of the URL out so that you are pointing at:
<a href="http://host:port/solr/collection1">http://host:port/solr/collection1
General note 2: Paging through that many results with a distributed
query (known as deep paging) is SLOW.
http://solr.pl/en/2011/07/18/deep-paging-problem/
The first thing I'd do is ask Solr to sort your results. I can see from
some google searches that pysolr has sort capability. Once you pick the
sort field, I'd probably do the sort ascending, not descending. The
default "sort" is relevance.
The next thing to check is whether or not you are updating your index
during the time that you are attempting to pull 40,000 documents. If
you are, that could completely explain what you are seeing. If you are
only adding documents when you update, then you may be able to set a
sort parameter that will cause new documents to be at the end of the
results, so pagination won't get messed up. If you are deleting
documents, then you won't be able to make this work, you'll have to stop
your index updates while you pull that many results.
After all that, if the problem persists and you are absolutely sure that
you don't have duplicate document X on two different shards, then you
might be running into a bug.
Thanks,
Shawn
________________________________
If you reply to this email, your message will be added to the discussion below:
http://lucene.472066.n3.nabble.com/Solr-Cloud-Duplicate-records-while-retrieving-documents-tp4039776p4039838.html
To unsubscribe from Solr Cloud: Duplicate records while retrieving documents, click here<http://lucene.472066.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=4039776&code=cGFkZHlrQGJlaGF2aW9ybWF0cml4LmNvbXw0MDM5Nzc2fDcxODU5MTAwMQ==>.
NAML<http://lucene.472066.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
--
View this message in context: http://lucene.472066.n3.nabble.com/Solr-Cloud-Duplicate-records-while-retrieving-documents-tp4039776p4039993.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solr Cloud: Duplicate records while retrieving documents
Posted by Shawn Heisey <so...@elyograg.org>.
On 2/11/2013 12:09 PM, devb wrote:
> We are running a six node SOLR cloud which 3 shards and 3 replications. The
> version of solr cloud is 4.0.0.2012.08.06.22.50.47. We use Python PySolr
> client to interact with Solr. Documents that we add to solr have a unique id
> and it can never have duplicates.
> Our use case is to query the index for a give searchterm and pull all
> documents that matches the query. Usually our query hits over 40K documents.
> While we iterate through all 40K+ documents, after few iteration, we see the
> same documents ids repeated over and over, and at the end we see some 20-33%
> of the records are duplicates.
> In the below code snippet after some iterations, we see a difference in the
> length of idslist and idsset. Any insight into how to troubleshoot this
> issue is greatly appreciated.
For discussion purposes Let's first assume that there are no bugs in
Solr. I don't think we can make that assumption, of course.
General note 1: Your Solr URL in your code has a # in it. The URLs with
# in them are Admin UI URLs. If that's working, I'm amazed... I would
take that part of the URL out so that you are pointing at:
http://host:port/solr/collection1
General note 2: Paging through that many results with a distributed
query (known as deep paging) is SLOW.
http://solr.pl/en/2011/07/18/deep-paging-problem/
The first thing I'd do is ask Solr to sort your results. I can see from
some google searches that pysolr has sort capability. Once you pick the
sort field, I'd probably do the sort ascending, not descending. The
default "sort" is relevance.
The next thing to check is whether or not you are updating your index
during the time that you are attempting to pull 40,000 documents. If
you are, that could completely explain what you are seeing. If you are
only adding documents when you update, then you may be able to set a
sort parameter that will cause new documents to be at the end of the
results, so pagination won't get messed up. If you are deleting
documents, then you won't be able to make this work, you'll have to stop
your index updates while you pull that many results.
After all that, if the problem persists and you are absolutely sure that
you don't have duplicate document X on two different shards, then you
might be running into a bug.
Thanks,
Shawn