You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by "Buttler, David" <bu...@llnl.gov> on 2012/11/16 19:34:16 UTC
inconsistent number of results returned in solr cloud
Hi all,
I buried an issue in my last post, so let me pop it up.
I have a cluster with 10 collections on it. The first collection I loaded works perfectly. But every subsequent collection returns an inconsistent number of results for each query. The queries can be simply *:*, or more complex facet queries. If I go to individual cores and issue the query, with distrib=false, I get a consistent number of results. I am wondering if there is some delay in returning results from my shards, and the queried node just times out and displays the number of results that it has received so far. If there is such a timeout, it must be very small, as my QTime is around 11 ms.
Dave
Re: inconsistent number of results returned in solr cloud
Posted by Erick Erickson <er...@gmail.com>.
Just glad it's resolved....
Erick
On Thu, Nov 29, 2012 at 7:46 PM, Buttler, David <bu...@llnl.gov> wrote:
> Sorry, yes, I had been using the BETA version. I have deleted all of
> that, replaced the jars with the released versions (reduced my core count),
> and now I have consistent results.
> I guess I missed that JIRA ticket, sorry for the false alarm.
> Dave
>
>
> -----Original Message-----
> From: Erick Erickson [mailto:erickerickson@gmail.com]
> Sent: Friday, November 23, 2012 4:25 AM
> To: solr-user@lucene.apache.org
> Subject: Re: inconsistent number of results returned in solr cloud
>
> Dave:
>
> I should have asked this first. What version of Solr are you using? I Not
> sure whether it was fixed in BETA or not (certainly is in the 4.0 GA
> release). There was a problem with adding a doclist via solrj, here's one
> related JIRA, although it wasn't the main fix:
> https://issues.apache.org/jira/browse/SOLR-3001. I suspect that's the
> "known problem" Mark mentioned.
>
> Because what you're seeing _sure_ sounds similar....
>
> Best
> Erick
>
>
> On Mon, Nov 19, 2012 at 12:49 PM, Buttler, David <bu...@llnl.gov>
> wrote:
>
> > Answers inline below
> >
> > -----Original Message-----
> > From: Erick Erickson [mailto:erickerickson@gmail.com]
> > Sent: Saturday, November 17, 2012 6:40 AM
> > To: solr-user@lucene.apache.org
> > Subject: Re: inconsistent number of results returned in solr cloud
> >
> > Hmmm, first an aside. If by "commit after every batch of documents "
> > you mean after every call to server.add(doclist), there's no real need
> > to do that unless you're striving for really low latency. the usual
> > recommendation is to use commitWithin when adding and commit only at
> > the very end of the run. This shouldn't actually be germane to your
> > issue, just an FYI.
> >
> > DB> Good point. The code for committing docs to solr is fairly old.
> > DB> I
> > will update it since I don't have a latency requirement.
> >
> > So you're saying that the inconsistency is permanent? By that I mean
> > it keeps coming back inconsistently for minutes/hours/days?
> >
> > DB> Yes, it is permanent. I have collections that have been up for
> > DB> weeks,
> > and are still returning inconsistent results, and I haven't been
> > adding any additional documents.
> > DB> Related to this, I seem to have a discrepancy between the number
> > DB> of
> > documents I think I am sending to solr, and the number of documents it
> > is reporting. I have tried reducing the number of shards for one of
> > my small collections, so I deleted all references to this collections,
> > and reloaded it. I think I have 260 documents submitted (counted from a
> hadoop job).
> > Solr returns a count of ~430 (it varies), and the first returned
> > document is not consistent.
> >
> > I guess if I were trying to test this I'd need to know how you added
> > subsequent collections. In particular what you did re: zookeeper as
> > you added each collection.
> >
> > DB> These are my steps
> > DB> 1. Create the collection via the HTTP API: http://
> > <host>:<port>/solr/admin/collections?action=CREATE&name=<collection>&n
> > umShards=6&%20collection.configName=<collection>
> > DB> 2. Relaunch one of my JVM processes, bootstrapping the collection:
> > DB> java -Xmx16g -Dcollection.configName=<collection>
> > DB> -Djetty.port=<port>
> > -DzkHost=<zkhost> -Dsolr.solr.home=<solr home> -DnumShards=6
> > -Dbootstrap_confdir=conf -jar start.jar
> > DB> load data
> >
> > DB> Let me know if something is unclear. I can run through the
> > DB> process
> > again and document it more carefully.
> > DB>
> > DB> Thanks for looking at it,
> > DB> Dave
> >
> > Best
> > Erick
> >
> >
> > On Fri, Nov 16, 2012 at 2:55 PM, Buttler, David <bu...@llnl.gov>
> wrote:
> >
> > > My typical way of adding documents is through SolrJ, where I commit
> > > after every batch of documents (where the batch size is
> > > configurable)
> > >
> > > I have now tried committing several times, from the command line
> > > (curl) with and without openSearcher=true. It does not affect
> anything.
> > >
> > > Dave
> > >
> > > -----Original Message-----
> > > From: Mark Miller [mailto:markrmiller@gmail.com]
> > > Sent: Friday, November 16, 2012 11:04 AM
> > > To: solr-user@lucene.apache.org
> > > Subject: Re: inconsistent number of results returned in solr cloud
> > >
> > > How did you do the final commit? Can you try a lone commit (with
> > > openSearcher=true) and see if that affects things?
> > >
> > > Trying to determine if this is a known issue or not.
> > >
> > > - Mark
> > >
> > > On Nov 16, 2012, at 1:34 PM, "Buttler, David" <bu...@llnl.gov>
> wrote:
> > >
> > > > Hi all,
> > > > I buried an issue in my last post, so let me pop it up.
> > > >
> > > > I have a cluster with 10 collections on it. The first collection
> > > > I
> > > loaded works perfectly. But every subsequent collection returns an
> > > inconsistent number of results for each query. The queries can be
> > > simply *:*, or more complex facet queries. If I go to individual
> > > cores and
> > issue
> > > the query, with distrib=false, I get a consistent number of results.
> > > I
> > am
> > > wondering if there is some delay in returning results from my
> > > shards, and the queried node just times out and displays the number
> > > of results that
> > it
> > > has received so far. If there is such a timeout, it must be very
> > > small,
> > as
> > > my QTime is around 11 ms.
> > > >
> > > > Dave
> > >
> > >
> >
>
RE: inconsistent number of results returned in solr cloud
Posted by "Buttler, David" <bu...@llnl.gov>.
Sorry, yes, I had been using the BETA version. I have deleted all of that, replaced the jars with the released versions (reduced my core count), and now I have consistent results.
I guess I missed that JIRA ticket, sorry for the false alarm.
Dave
-----Original Message-----
From: Erick Erickson [mailto:erickerickson@gmail.com]
Sent: Friday, November 23, 2012 4:25 AM
To: solr-user@lucene.apache.org
Subject: Re: inconsistent number of results returned in solr cloud
Dave:
I should have asked this first. What version of Solr are you using? I Not sure whether it was fixed in BETA or not (certainly is in the 4.0 GA release). There was a problem with adding a doclist via solrj, here's one related JIRA, although it wasn't the main fix:
https://issues.apache.org/jira/browse/SOLR-3001. I suspect that's the "known problem" Mark mentioned.
Because what you're seeing _sure_ sounds similar....
Best
Erick
On Mon, Nov 19, 2012 at 12:49 PM, Buttler, David <bu...@llnl.gov> wrote:
> Answers inline below
>
> -----Original Message-----
> From: Erick Erickson [mailto:erickerickson@gmail.com]
> Sent: Saturday, November 17, 2012 6:40 AM
> To: solr-user@lucene.apache.org
> Subject: Re: inconsistent number of results returned in solr cloud
>
> Hmmm, first an aside. If by "commit after every batch of documents "
> you mean after every call to server.add(doclist), there's no real need
> to do that unless you're striving for really low latency. the usual
> recommendation is to use commitWithin when adding and commit only at
> the very end of the run. This shouldn't actually be germane to your
> issue, just an FYI.
>
> DB> Good point. The code for committing docs to solr is fairly old.
> DB> I
> will update it since I don't have a latency requirement.
>
> So you're saying that the inconsistency is permanent? By that I mean
> it keeps coming back inconsistently for minutes/hours/days?
>
> DB> Yes, it is permanent. I have collections that have been up for
> DB> weeks,
> and are still returning inconsistent results, and I haven't been
> adding any additional documents.
> DB> Related to this, I seem to have a discrepancy between the number
> DB> of
> documents I think I am sending to solr, and the number of documents it
> is reporting. I have tried reducing the number of shards for one of
> my small collections, so I deleted all references to this collections,
> and reloaded it. I think I have 260 documents submitted (counted from a hadoop job).
> Solr returns a count of ~430 (it varies), and the first returned
> document is not consistent.
>
> I guess if I were trying to test this I'd need to know how you added
> subsequent collections. In particular what you did re: zookeeper as
> you added each collection.
>
> DB> These are my steps
> DB> 1. Create the collection via the HTTP API: http://
> <host>:<port>/solr/admin/collections?action=CREATE&name=<collection>&n
> umShards=6&%20collection.configName=<collection>
> DB> 2. Relaunch one of my JVM processes, bootstrapping the collection:
> DB> java -Xmx16g -Dcollection.configName=<collection>
> DB> -Djetty.port=<port>
> -DzkHost=<zkhost> -Dsolr.solr.home=<solr home> -DnumShards=6
> -Dbootstrap_confdir=conf -jar start.jar
> DB> load data
>
> DB> Let me know if something is unclear. I can run through the
> DB> process
> again and document it more carefully.
> DB>
> DB> Thanks for looking at it,
> DB> Dave
>
> Best
> Erick
>
>
> On Fri, Nov 16, 2012 at 2:55 PM, Buttler, David <bu...@llnl.gov> wrote:
>
> > My typical way of adding documents is through SolrJ, where I commit
> > after every batch of documents (where the batch size is
> > configurable)
> >
> > I have now tried committing several times, from the command line
> > (curl) with and without openSearcher=true. It does not affect anything.
> >
> > Dave
> >
> > -----Original Message-----
> > From: Mark Miller [mailto:markrmiller@gmail.com]
> > Sent: Friday, November 16, 2012 11:04 AM
> > To: solr-user@lucene.apache.org
> > Subject: Re: inconsistent number of results returned in solr cloud
> >
> > How did you do the final commit? Can you try a lone commit (with
> > openSearcher=true) and see if that affects things?
> >
> > Trying to determine if this is a known issue or not.
> >
> > - Mark
> >
> > On Nov 16, 2012, at 1:34 PM, "Buttler, David" <bu...@llnl.gov> wrote:
> >
> > > Hi all,
> > > I buried an issue in my last post, so let me pop it up.
> > >
> > > I have a cluster with 10 collections on it. The first collection
> > > I
> > loaded works perfectly. But every subsequent collection returns an
> > inconsistent number of results for each query. The queries can be
> > simply *:*, or more complex facet queries. If I go to individual
> > cores and
> issue
> > the query, with distrib=false, I get a consistent number of results.
> > I
> am
> > wondering if there is some delay in returning results from my
> > shards, and the queried node just times out and displays the number
> > of results that
> it
> > has received so far. If there is such a timeout, it must be very
> > small,
> as
> > my QTime is around 11 ms.
> > >
> > > Dave
> >
> >
>
Re: inconsistent number of results returned in solr cloud
Posted by Erick Erickson <er...@gmail.com>.
Dave:
I should have asked this first. What version of Solr are you using? I Not
sure whether it was fixed in BETA or not (certainly is in the 4.0 GA
release). There was a problem with adding a doclist via solrj, here's one
related JIRA, although it wasn't the main fix:
https://issues.apache.org/jira/browse/SOLR-3001. I suspect that's the
"known problem" Mark mentioned.
Because what you're seeing _sure_ sounds similar....
Best
Erick
On Mon, Nov 19, 2012 at 12:49 PM, Buttler, David <bu...@llnl.gov> wrote:
> Answers inline below
>
> -----Original Message-----
> From: Erick Erickson [mailto:erickerickson@gmail.com]
> Sent: Saturday, November 17, 2012 6:40 AM
> To: solr-user@lucene.apache.org
> Subject: Re: inconsistent number of results returned in solr cloud
>
> Hmmm, first an aside. If by "commit after every batch of documents " you
> mean after every call to server.add(doclist), there's no real need to do
> that unless you're striving for really low latency. the usual
> recommendation is to use commitWithin when adding and commit only at the
> very end of the run. This shouldn't actually be germane to your issue, just
> an FYI.
>
> DB> Good point. The code for committing docs to solr is fairly old. I
> will update it since I don't have a latency requirement.
>
> So you're saying that the inconsistency is permanent? By that I mean it
> keeps coming back inconsistently for minutes/hours/days?
>
> DB> Yes, it is permanent. I have collections that have been up for weeks,
> and are still returning inconsistent results, and I haven't been adding any
> additional documents.
> DB> Related to this, I seem to have a discrepancy between the number of
> documents I think I am sending to solr, and the number of documents it is
> reporting. I have tried reducing the number of shards for one of my small
> collections, so I deleted all references to this collections, and reloaded
> it. I think I have 260 documents submitted (counted from a hadoop job).
> Solr returns a count of ~430 (it varies), and the first returned document
> is not consistent.
>
> I guess if I were trying to test this I'd need to know how you added
> subsequent collections. In particular what you did re: zookeeper as you
> added each collection.
>
> DB> These are my steps
> DB> 1. Create the collection via the HTTP API: http://
> <host>:<port>/solr/admin/collections?action=CREATE&name=<collection>&numShards=6&%20collection.configName=<collection>
> DB> 2. Relaunch one of my JVM processes, bootstrapping the collection:
> DB> java -Xmx16g -Dcollection.configName=<collection> -Djetty.port=<port>
> -DzkHost=<zkhost> -Dsolr.solr.home=<solr home> -DnumShards=6
> -Dbootstrap_confdir=conf -jar start.jar
> DB> load data
>
> DB> Let me know if something is unclear. I can run through the process
> again and document it more carefully.
> DB>
> DB> Thanks for looking at it,
> DB> Dave
>
> Best
> Erick
>
>
> On Fri, Nov 16, 2012 at 2:55 PM, Buttler, David <bu...@llnl.gov> wrote:
>
> > My typical way of adding documents is through SolrJ, where I commit after
> > every batch of documents (where the batch size is configurable)
> >
> > I have now tried committing several times, from the command line (curl)
> > with and without openSearcher=true. It does not affect anything.
> >
> > Dave
> >
> > -----Original Message-----
> > From: Mark Miller [mailto:markrmiller@gmail.com]
> > Sent: Friday, November 16, 2012 11:04 AM
> > To: solr-user@lucene.apache.org
> > Subject: Re: inconsistent number of results returned in solr cloud
> >
> > How did you do the final commit? Can you try a lone commit (with
> > openSearcher=true) and see if that affects things?
> >
> > Trying to determine if this is a known issue or not.
> >
> > - Mark
> >
> > On Nov 16, 2012, at 1:34 PM, "Buttler, David" <bu...@llnl.gov> wrote:
> >
> > > Hi all,
> > > I buried an issue in my last post, so let me pop it up.
> > >
> > > I have a cluster with 10 collections on it. The first collection I
> > loaded works perfectly. But every subsequent collection returns an
> > inconsistent number of results for each query. The queries can be simply
> > *:*, or more complex facet queries. If I go to individual cores and
> issue
> > the query, with distrib=false, I get a consistent number of results. I
> am
> > wondering if there is some delay in returning results from my shards, and
> > the queried node just times out and displays the number of results that
> it
> > has received so far. If there is such a timeout, it must be very small,
> as
> > my QTime is around 11 ms.
> > >
> > > Dave
> >
> >
>
RE: inconsistent number of results returned in solr cloud
Posted by "Buttler, David" <bu...@llnl.gov>.
Answers inline below
-----Original Message-----
From: Erick Erickson [mailto:erickerickson@gmail.com]
Sent: Saturday, November 17, 2012 6:40 AM
To: solr-user@lucene.apache.org
Subject: Re: inconsistent number of results returned in solr cloud
Hmmm, first an aside. If by "commit after every batch of documents " you
mean after every call to server.add(doclist), there's no real need to do
that unless you're striving for really low latency. the usual
recommendation is to use commitWithin when adding and commit only at the
very end of the run. This shouldn't actually be germane to your issue, just
an FYI.
DB> Good point. The code for committing docs to solr is fairly old. I will update it since I don't have a latency requirement.
So you're saying that the inconsistency is permanent? By that I mean it
keeps coming back inconsistently for minutes/hours/days?
DB> Yes, it is permanent. I have collections that have been up for weeks, and are still returning inconsistent results, and I haven't been adding any additional documents.
DB> Related to this, I seem to have a discrepancy between the number of documents I think I am sending to solr, and the number of documents it is reporting. I have tried reducing the number of shards for one of my small collections, so I deleted all references to this collections, and reloaded it. I think I have 260 documents submitted (counted from a hadoop job). Solr returns a count of ~430 (it varies), and the first returned document is not consistent.
I guess if I were trying to test this I'd need to know how you added
subsequent collections. In particular what you did re: zookeeper as you
added each collection.
DB> These are my steps
DB> 1. Create the collection via the HTTP API: http://<host>:<port>/solr/admin/collections?action=CREATE&name=<collection>&numShards=6&%20collection.configName=<collection>
DB> 2. Relaunch one of my JVM processes, bootstrapping the collection:
DB> java -Xmx16g -Dcollection.configName=<collection> -Djetty.port=<port> -DzkHost=<zkhost> -Dsolr.solr.home=<solr home> -DnumShards=6 -Dbootstrap_confdir=conf -jar start.jar
DB> load data
DB> Let me know if something is unclear. I can run through the process again and document it more carefully.
DB>
DB> Thanks for looking at it,
DB> Dave
Best
Erick
On Fri, Nov 16, 2012 at 2:55 PM, Buttler, David <bu...@llnl.gov> wrote:
> My typical way of adding documents is through SolrJ, where I commit after
> every batch of documents (where the batch size is configurable)
>
> I have now tried committing several times, from the command line (curl)
> with and without openSearcher=true. It does not affect anything.
>
> Dave
>
> -----Original Message-----
> From: Mark Miller [mailto:markrmiller@gmail.com]
> Sent: Friday, November 16, 2012 11:04 AM
> To: solr-user@lucene.apache.org
> Subject: Re: inconsistent number of results returned in solr cloud
>
> How did you do the final commit? Can you try a lone commit (with
> openSearcher=true) and see if that affects things?
>
> Trying to determine if this is a known issue or not.
>
> - Mark
>
> On Nov 16, 2012, at 1:34 PM, "Buttler, David" <bu...@llnl.gov> wrote:
>
> > Hi all,
> > I buried an issue in my last post, so let me pop it up.
> >
> > I have a cluster with 10 collections on it. The first collection I
> loaded works perfectly. But every subsequent collection returns an
> inconsistent number of results for each query. The queries can be simply
> *:*, or more complex facet queries. If I go to individual cores and issue
> the query, with distrib=false, I get a consistent number of results. I am
> wondering if there is some delay in returning results from my shards, and
> the queried node just times out and displays the number of results that it
> has received so far. If there is such a timeout, it must be very small, as
> my QTime is around 11 ms.
> >
> > Dave
>
>
Re: inconsistent number of results returned in solr cloud
Posted by Erick Erickson <er...@gmail.com>.
Hmmm, first an aside. If by "commit after every batch of documents " you
mean after every call to server.add(doclist), there's no real need to do
that unless you're striving for really low latency. the usual
recommendation is to use commitWithin when adding and commit only at the
very end of the run. This shouldn't actually be germane to your issue, just
an FYI.
So you're saying that the inconsistency is permanent? By that I mean it
keeps coming back inconsistently for minutes/hours/days?
I guess if I were trying to test this I'd need to know how you added
subsequent collections. In particular what you did re: zookeeper as you
added each collection.
Best
Erick
On Fri, Nov 16, 2012 at 2:55 PM, Buttler, David <bu...@llnl.gov> wrote:
> My typical way of adding documents is through SolrJ, where I commit after
> every batch of documents (where the batch size is configurable)
>
> I have now tried committing several times, from the command line (curl)
> with and without openSearcher=true. It does not affect anything.
>
> Dave
>
> -----Original Message-----
> From: Mark Miller [mailto:markrmiller@gmail.com]
> Sent: Friday, November 16, 2012 11:04 AM
> To: solr-user@lucene.apache.org
> Subject: Re: inconsistent number of results returned in solr cloud
>
> How did you do the final commit? Can you try a lone commit (with
> openSearcher=true) and see if that affects things?
>
> Trying to determine if this is a known issue or not.
>
> - Mark
>
> On Nov 16, 2012, at 1:34 PM, "Buttler, David" <bu...@llnl.gov> wrote:
>
> > Hi all,
> > I buried an issue in my last post, so let me pop it up.
> >
> > I have a cluster with 10 collections on it. The first collection I
> loaded works perfectly. But every subsequent collection returns an
> inconsistent number of results for each query. The queries can be simply
> *:*, or more complex facet queries. If I go to individual cores and issue
> the query, with distrib=false, I get a consistent number of results. I am
> wondering if there is some delay in returning results from my shards, and
> the queried node just times out and displays the number of results that it
> has received so far. If there is such a timeout, it must be very small, as
> my QTime is around 11 ms.
> >
> > Dave
>
>
RE: inconsistent number of results returned in solr cloud
Posted by "Buttler, David" <bu...@llnl.gov>.
My typical way of adding documents is through SolrJ, where I commit after every batch of documents (where the batch size is configurable)
I have now tried committing several times, from the command line (curl) with and without openSearcher=true. It does not affect anything.
Dave
-----Original Message-----
From: Mark Miller [mailto:markrmiller@gmail.com]
Sent: Friday, November 16, 2012 11:04 AM
To: solr-user@lucene.apache.org
Subject: Re: inconsistent number of results returned in solr cloud
How did you do the final commit? Can you try a lone commit (with openSearcher=true) and see if that affects things?
Trying to determine if this is a known issue or not.
- Mark
On Nov 16, 2012, at 1:34 PM, "Buttler, David" <bu...@llnl.gov> wrote:
> Hi all,
> I buried an issue in my last post, so let me pop it up.
>
> I have a cluster with 10 collections on it. The first collection I loaded works perfectly. But every subsequent collection returns an inconsistent number of results for each query. The queries can be simply *:*, or more complex facet queries. If I go to individual cores and issue the query, with distrib=false, I get a consistent number of results. I am wondering if there is some delay in returning results from my shards, and the queried node just times out and displays the number of results that it has received so far. If there is such a timeout, it must be very small, as my QTime is around 11 ms.
>
> Dave
Re: inconsistent number of results returned in solr cloud
Posted by Mark Miller <ma...@gmail.com>.
How did you do the final commit? Can you try a lone commit (with openSearcher=true) and see if that affects things?
Trying to determine if this is a known issue or not.
- Mark
On Nov 16, 2012, at 1:34 PM, "Buttler, David" <bu...@llnl.gov> wrote:
> Hi all,
> I buried an issue in my last post, so let me pop it up.
>
> I have a cluster with 10 collections on it. The first collection I loaded works perfectly. But every subsequent collection returns an inconsistent number of results for each query. The queries can be simply *:*, or more complex facet queries. If I go to individual cores and issue the query, with distrib=false, I get a consistent number of results. I am wondering if there is some delay in returning results from my shards, and the queried node just times out and displays the number of results that it has received so far. If there is such a timeout, it must be very small, as my QTime is around 11 ms.
>
> Dave