You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Tim Vaillancourt <ti...@elementspace.com> on 2013/12/04 23:17:42 UTC
Inconsistent numFound in SC when querying core directly
Hey guys,
I'm looking into a strange issue on an unhealthy 4.3.1 SolrCloud with
3-node external Zookeeper and 1 collection (2 shards, 2 replicas).
Currently we are noticing inconsistent results from the SolrCloud when
performing the same simple /select query many times to our collection.
Almost every other query the numFound count (and the returned data)
jumps between two very different values.
Initially I suspected a replica in a shard of the collection was
inconsistent (and every other request hit that node) and started
performing the same /select query direct to the individual cores of the
SolrCloud collection on each instance, only to notice the same problem -
the count jumps between two very different values!
I may be incorrect here, but I assumed when querying a single core of a
SolrCloud collection, the SolrCloud routing is bypassed and I am talking
directly to a plain/non-SolrCloud core.
As you can see here, the count for 1 core of my SolrCloud collection
fluctuates wildly, and is only receiving updates and no deletes to
explain the jumps:
"solrcloud [tvaillancourt@prodapp solr_cloud]$ curl -s
'http://backend:8983/solr/app_shard2_replica2/select?q=*:*&wt=json&rows=0&indent=true'|grep
numFound
"response":{"numFound":123596839,"start":0,"maxScore":1.0,"docs":[]
solrcloud [tvaillancourt@prodapp solr_cloud]$ curl -s
'http://backend:8983/solr/app_shard2_replica2/select?q=*:*&wt=json&rows=0&indent=true'|grep
numFound
"response":{"numFound":84739144,"start":0,"maxScore":1.0,"docs":[]
solrcloud [tvaillancourt@prodapp solr_cloud]$ curl -s
'http://backend:8983/solr/app_shard2_replica2/select?q=*:*&wt=json&rows=0&indent=true'|grep
numFound
"response":{"numFound":123596839,"start":0,"maxScore":1.0,"docs":[]
solrcloud [tvaillancourt@prodapp solr_cloud]$ curl -s
'http://backend:8983/solr/app_shard2_replica2/select?q=*:*&wt=json&rows=0&indent=true'|grep
numFound
"response":{"numFound":84771358,"start":0,"maxScore":1.0,"docs":[]"
Could anyone help me understand why the same /select query direct to a
single core would return inconsistent, flapping results if there are no
deletes issued in my app to cause such jumps? Am I incorrect in my
assumption that I am querying the core "directly"?
An interesting observation is when I do an /admin/cores call to see the
docCount of the core's index, it does not fluctuate, only the query result.
That was hard to explain, hopefully someone has some insight! :)
Thanks!
Tim
Re: Inconsistent numFound in SC when querying core directly
Posted by Tim Vaillancourt <ti...@elementspace.com>.
Thanks Markus,
I'm not sure if I'm encountering the same issue. This JIRA mentions 10s
of docs difference, I'm seeing differences in the multi-millions of
docs, and even more strangely it very predictably flaps between a 123M
value and an 87M value, a 30M+ doc difference.
Secondly, I'm not comparing values from 2 instances (Leader to Replica),
I'm currently performing the same curl call to the same core directly
and am seeing flapping results each time I perform the query, so this is
currently happening within a single instance/core unless I am
misunderstanding how to directly query a core.
Cheers,
Tim
On 04/12/13 02:46 PM, Markus Jelsma wrote:
> https://issues.apache.org/jira/browse/SOLR-4260
>
> Join the club Tim! Can you upgrade to trunk or incorporate the latest patches of related issues? You can fix it by trashing the bad node's data, although without multiple clusters it may be difficult to decide which node is bad.
>
> We use the latest commits now (since tuesday) and are still waiting for it to happen again.
>
> -----Original message-----
>> From:Tim Vaillancourt<ti...@elementspace.com>
>> Sent: Wednesday 4th December 2013 23:38
>> To: solr-user@lucene.apache.org
>> Subject: Re: Inconsistent numFound in SC when querying core directly
>>
>> To add two more pieces of data:
>>
>> 1) This occurs with real, conditional queries as well (eg:
>> "q=key:timvaillancourt"), not just the "q=*:*" I provided in my email.
>> 2) I've noticed when I bring a node of the SolrCloud down it is
>> remaining "state: active" in my /clusterstate.json - something is really
>> wrong with this cloud! Would a Zookeeper issue explain my varied results
>> when querying a core directly?
>>
>> Thanks again!
>>
>> Tim
>>
>> On 04/12/13 02:17 PM, Tim Vaillancourt wrote:
>>> Hey guys,
>>>
>>> I'm looking into a strange issue on an unhealthy 4.3.1 SolrCloud with
>>> 3-node external Zookeeper and 1 collection (2 shards, 2 replicas).
>>>
>>> Currently we are noticing inconsistent results from the SolrCloud when
>>> performing the same simple /select query many times to our collection.
>>> Almost every other query the numFound count (and the returned data)
>>> jumps between two very different values.
>>>
>>> Initially I suspected a replica in a shard of the collection was
>>> inconsistent (and every other request hit that node) and started
>>> performing the same /select query direct to the individual cores of
>>> the SolrCloud collection on each instance, only to notice the same
>>> problem - the count jumps between two very different values!
>>>
>>> I may be incorrect here, but I assumed when querying a single core of
>>> a SolrCloud collection, the SolrCloud routing is bypassed and I am
>>> talking directly to a plain/non-SolrCloud core.
>>>
>>> As you can see here, the count for 1 core of my SolrCloud collection
>>> fluctuates wildly, and is only receiving updates and no deletes to
>>> explain the jumps:
>>>
>>> "solrcloud [tvaillancourt@prodapp solr_cloud]$ curl -s
>>> 'http://backend:8983/solr/app_shard2_replica2/select?q=*:*&wt=json&rows=0&indent=true'|grep
>>> numFound
>>> "response":{"numFound":123596839,"start":0,"maxScore":1.0,"docs":[]
>>>
>>> solrcloud [tvaillancourt@prodapp solr_cloud]$ curl -s
>>> 'http://backend:8983/solr/app_shard2_replica2/select?q=*:*&wt=json&rows=0&indent=true'|grep
>>> numFound
>>> "response":{"numFound":84739144,"start":0,"maxScore":1.0,"docs":[]
>>>
>>> solrcloud [tvaillancourt@prodapp solr_cloud]$ curl -s
>>> 'http://backend:8983/solr/app_shard2_replica2/select?q=*:*&wt=json&rows=0&indent=true'|grep
>>> numFound
>>> "response":{"numFound":123596839,"start":0,"maxScore":1.0,"docs":[]
>>>
>>> solrcloud [tvaillancourt@prodapp solr_cloud]$ curl -s
>>> 'http://backend:8983/solr/app_shard2_replica2/select?q=*:*&wt=json&rows=0&indent=true'|grep
>>> numFound
>>> "response":{"numFound":84771358,"start":0,"maxScore":1.0,"docs":[]"
>>>
>>>
>>> Could anyone help me understand why the same /select query direct to a
>>> single core would return inconsistent, flapping results if there are
>>> no deletes issued in my app to cause such jumps? Am I incorrect in my
>>> assumption that I am querying the core "directly"?
>>>
>>> An interesting observation is when I do an /admin/cores call to see
>>> the docCount of the core's index, it does not fluctuate, only the
>>> query result.
>>>
>>> That was hard to explain, hopefully someone has some insight! :)
>>>
>>> Thanks!
>>>
>>> Tim
RE: Inconsistent numFound in SC when querying core directly
Posted by Markus Jelsma <ma...@openindex.io>.
https://issues.apache.org/jira/browse/SOLR-4260
Join the club Tim! Can you upgrade to trunk or incorporate the latest patches of related issues? You can fix it by trashing the bad node's data, although without multiple clusters it may be difficult to decide which node is bad.
We use the latest commits now (since tuesday) and are still waiting for it to happen again.
-----Original message-----
> From:Tim Vaillancourt <ti...@elementspace.com>
> Sent: Wednesday 4th December 2013 23:38
> To: solr-user@lucene.apache.org
> Subject: Re: Inconsistent numFound in SC when querying core directly
>
> To add two more pieces of data:
>
> 1) This occurs with real, conditional queries as well (eg:
> "q=key:timvaillancourt"), not just the "q=*:*" I provided in my email.
> 2) I've noticed when I bring a node of the SolrCloud down it is
> remaining "state: active" in my /clusterstate.json - something is really
> wrong with this cloud! Would a Zookeeper issue explain my varied results
> when querying a core directly?
>
> Thanks again!
>
> Tim
>
> On 04/12/13 02:17 PM, Tim Vaillancourt wrote:
> > Hey guys,
> >
> > I'm looking into a strange issue on an unhealthy 4.3.1 SolrCloud with
> > 3-node external Zookeeper and 1 collection (2 shards, 2 replicas).
> >
> > Currently we are noticing inconsistent results from the SolrCloud when
> > performing the same simple /select query many times to our collection.
> > Almost every other query the numFound count (and the returned data)
> > jumps between two very different values.
> >
> > Initially I suspected a replica in a shard of the collection was
> > inconsistent (and every other request hit that node) and started
> > performing the same /select query direct to the individual cores of
> > the SolrCloud collection on each instance, only to notice the same
> > problem - the count jumps between two very different values!
> >
> > I may be incorrect here, but I assumed when querying a single core of
> > a SolrCloud collection, the SolrCloud routing is bypassed and I am
> > talking directly to a plain/non-SolrCloud core.
> >
> > As you can see here, the count for 1 core of my SolrCloud collection
> > fluctuates wildly, and is only receiving updates and no deletes to
> > explain the jumps:
> >
> > "solrcloud [tvaillancourt@prodapp solr_cloud]$ curl -s
> > 'http://backend:8983/solr/app_shard2_replica2/select?q=*:*&wt=json&rows=0&indent=true'|grep
> > numFound
> > "response":{"numFound":123596839,"start":0,"maxScore":1.0,"docs":[]
> >
> > solrcloud [tvaillancourt@prodapp solr_cloud]$ curl -s
> > 'http://backend:8983/solr/app_shard2_replica2/select?q=*:*&wt=json&rows=0&indent=true'|grep
> > numFound
> > "response":{"numFound":84739144,"start":0,"maxScore":1.0,"docs":[]
> >
> > solrcloud [tvaillancourt@prodapp solr_cloud]$ curl -s
> > 'http://backend:8983/solr/app_shard2_replica2/select?q=*:*&wt=json&rows=0&indent=true'|grep
> > numFound
> > "response":{"numFound":123596839,"start":0,"maxScore":1.0,"docs":[]
> >
> > solrcloud [tvaillancourt@prodapp solr_cloud]$ curl -s
> > 'http://backend:8983/solr/app_shard2_replica2/select?q=*:*&wt=json&rows=0&indent=true'|grep
> > numFound
> > "response":{"numFound":84771358,"start":0,"maxScore":1.0,"docs":[]"
> >
> >
> > Could anyone help me understand why the same /select query direct to a
> > single core would return inconsistent, flapping results if there are
> > no deletes issued in my app to cause such jumps? Am I incorrect in my
> > assumption that I am querying the core "directly"?
> >
> > An interesting observation is when I do an /admin/cores call to see
> > the docCount of the core's index, it does not fluctuate, only the
> > query result.
> >
> > That was hard to explain, hopefully someone has some insight! :)
> >
> > Thanks!
> >
> > Tim
>
Re: Inconsistent numFound in SC when querying core directly
Posted by Erick Erickson <er...@gmail.com>.
bq: If I changed the routing strategy back to composite (which it should be). is
it ok?
I sincerely doubt it. The docs have already been routed to the wrong
place (actually, I'm not sure how it worked at all). You can't get
them redistributed simply by changing the definition in ZooKeeper,
they're _already_ in the wrong place.
I'd tear down the corrupted data center and rebuild the collection.
Here "tear down" is delete all the affected collections and start over
again.
On the plus side, if you can get a window during which you are _not_
indexing you can copy the indexes from one of your good data centers
to the new one. Do it like this:
- Stop indexing.
- Set up the new collection in the corrupted data center. It's
important that it have _exactly_ the same number of shards ad the DC
you're going to transfer _from_. Also, make it leader only, i.e.
exactly 1 replica per shard.
- copy the indexes over from the good data center to the corresponding
shards. Here "corresponding" means that the source and destination
have the same hash range, which you can see from the state.json (or
clusterstate.json if you're on an earlier format). NOTE: there are two
ways to do this:
-- Just do file copies, scp, hand carry CDs, whatever. Solr should be
offline in the target data center.
-- use the replication API to issue a "fetchindex" command. This works
even in cloud mode, all the target Solr instance needs is access to a
URL it can pull from. Solr of course needs to be running in this case.
- Bring up Solr on the target data center and verify it's working.
- Use the Collections API to ADDREPLICA on the target system until you
build out the collection with the numbers of replicas you want.
- Start indexing to the target data center.
The bits about shutting off indexing is a safety measure, it
guarantees that the indexes are consistent. If you can't shut indexing
down during the transfer, you'll need to index docs to the
newly-rebuilt cluster in some manner that guarantees the two DCs will
have the same docs eventually.
Best,
Erick
On Tue, Mar 14, 2017 at 3:26 PM, vbindal <vi...@gmail.com> wrote:
> I think I dint explain properly.
>
> I have 3 data centers each with its own SOLR cloud.
>
> My original strategy was composite routing but when one data center went
> down and we brought it back, somehow the routing strategy on this changed to
> implicit (Other 2 DC still have composit and they are working absolutely
> fine).
>
> This might be the reason for the data corruption on that DS because the
> routing strategy got changed.
>
> If I changed the routing strategy back to composite (which it should be). is
> it ok? Do I need to do anything more than simply changing the strategy in
> the clusterState.json?
>
>
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/Inconsistent-numFound-in-SC-when-querying-core-directly-tp4105009p4325001.html
> Sent from the Solr - User mailing list archive at Nabble.com.
Re: Inconsistent numFound in SC when querying core directly
Posted by vbindal <vi...@gmail.com>.
I think I dint explain properly.
I have 3 data centers each with its own SOLR cloud.
My original strategy was composite routing but when one data center went
down and we brought it back, somehow the routing strategy on this changed to
implicit (Other 2 DC still have composit and they are working absolutely
fine).
This might be the reason for the data corruption on that DS because the
routing strategy got changed.
If I changed the routing strategy back to composite (which it should be). is
it ok? Do I need to do anything more than simply changing the strategy in
the clusterState.json?
--
View this message in context: http://lucene.472066.n3.nabble.com/Inconsistent-numFound-in-SC-when-querying-core-directly-tp4105009p4325001.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Inconsistent numFound in SC when querying core directly
Posted by Erick Erickson <er...@gmail.com>.
That would make the problem even worse. If you created the collection
with implicit routing, there are no hash ranges for each shard.
CompositeId requires hash ranges to be defined for each shard. Don't
even try.
Best,
Erick
On Tue, Mar 14, 2017 at 11:13 AM, vbindal <vi...@gmail.com> wrote:
> Compared it against the other 2 datacenters and they both have `compositeId
> `.
>
> This started happening after 1 of our zookeeper died due to hardware issue
> and we had to setup a new zookeeper machine. update the config in all the
> solr machine and restart the cloud. My guess is something went wrong and
> `implicit` router got created.
>
> Can I simply change the `clusterstate.json` to take care of this?
>
>
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/Inconsistent-numFound-in-SC-when-querying-core-directly-tp4105009p4324950.html
> Sent from the Solr - User mailing list archive at Nabble.com.
Re: Inconsistent numFound in SC when querying core directly
Posted by vbindal <vi...@gmail.com>.
Compared it against the other 2 datacenters and they both have `compositeId
`.
This started happening after 1 of our zookeeper died due to hardware issue
and we had to setup a new zookeeper machine. update the config in all the
solr machine and restart the cloud. My guess is something went wrong and
`implicit` router got created.
Can I simply change the `clusterstate.json` to take care of this?
--
View this message in context: http://lucene.472066.n3.nabble.com/Inconsistent-numFound-in-SC-when-querying-core-directly-tp4105009p4324950.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Inconsistent numFound in SC when querying core directly
Posted by Erick Erickson <er...@gmail.com>.
The default router has always been compositeId, but when you created
your collection you may have created it with implicit. Looking at the
clusterstate.json and/or state.json in the individual collection
should show you (admin UI>>cloud>>tree).
But we need to be very clear about what a "duplicate" document is.
Solr routes/replaces documents based on whatever you've defined as
<uniqueKey> in your schema file (assuming compositeID routing). When
you say you get dups when you re-index, it sounds like you are somehow
using different <uniqueKey>s for what you consider the "same"
document.
bq: Also, This started happening after 1 of our zookeeper died due to hardware
issue and we had to setup a new zookeeper machine. update the config in all
the solr machine and restart the cloud.
Hmmm. "update the config in all the solr machine...". You should not
have to do this in SolrCloud. All the configs are stored in Zookeeper
and loaded from ZK when the Solr instance starts. What it's starting
to sound like is that you've somehow mixed up SorlCloud and older
"stand-alone" concepts and "somehow" your restoration process messed
up your configs.
So if the issue isn't that you're somehow using different
<unkiqueKey>'s, I'd recommend just starting over with a new collection
since you can re-index from scratch.
Best,
Erick
On Tue, Mar 14, 2017 at 10:35 AM, vbindal <vi...@gmail.com> wrote:
> Hi Shawn,
>
> We are on 4.10.0 version. Is that the default router in this version? Also,
> we dont see all the documents duplicated, only some of them. I have a
> indexer job to index data in SOLR. After I delete all the records and run
> this job, the count is correct but when I run the job again, we start seeing
> higher count and duplicate records (random records) in shards.
>
> Also, This started happening after 1 of our zookeeper died due to hardware
> issue and we had to setup a new zookeeper machine. update the config in all
> the solr machine and restart the cloud.
>
>
>
>
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/Inconsistent-numFound-in-SC-when-querying-core-directly-tp4105009p4324937.html
> Sent from the Solr - User mailing list archive at Nabble.com.
Re: Inconsistent numFound in SC when querying core directly
Posted by vbindal <vi...@gmail.com>.
Hi Shawn,
We are on 4.10.0 version. Is that the default router in this version? Also,
we dont see all the documents duplicated, only some of them. I have a
indexer job to index data in SOLR. After I delete all the records and run
this job, the count is correct but when I run the job again, we start seeing
higher count and duplicate records (random records) in shards.
Also, This started happening after 1 of our zookeeper died due to hardware
issue and we had to setup a new zookeeper machine. update the config in all
the solr machine and restart the cloud.
--
View this message in context: http://lucene.472066.n3.nabble.com/Inconsistent-numFound-in-SC-when-querying-core-directly-tp4105009p4324937.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Inconsistent numFound in SC when querying core directly
Posted by Shawn Heisey <ap...@elyograg.org>.
On 3/13/2017 3:16 AM, vbindal wrote:
> I am facing the same issue where my query *:* returns inconsistent number (almost 3) time the actual number in millions.
>
> When I try disturb=false on every machine, the results are correct. but
> without `disturb=false` results are incorrect.
This most likely means that you've got duplicate documents in different
shards of your cloud. This most commonly happens when the router is
"implicit" which is easier to understand if you imagine that "implicit"
is "manual" -- which would be a far better name for it.
In a later message you mention duplicate documents and 3 shards. It
sounds like you have indexed all your documents to all three shards, and
are probably using the implicit router.
The implicit router basically means that there *IS* no shard routing --
documents would be either indexed by the shard that received them, or
directed to the shard indicated by explicit routing parameters. With
three shards and "distrib=false" requests to one shard, you should see
about one-third of your total document count, not the full document count.
The router that you typically want if you don't want to be concerned
about how documents are routed to different shards is the compositeId
router. This *should* be the default for a multi-shard collection in
any recent version, but the first one or two releases of SolrCloud might
have created the wrong type.
Thanks,
Shawn
Re: Inconsistent numFound in SC when querying core directly
Posted by vbindal <vi...@gmail.com>.
Hi,
I am facing the same issue where my query *:* returns inconsistent number
(almost 3 ) time the actual number in millions.
When I try disturb=false on every machine, the results are correct. but
without `disturb=false` results are incorrect.
Can you guys suggest something?
--
View this message in context: http://lucene.472066.n3.nabble.com/Inconsistent-numFound-in-SC-when-querying-core-directly-tp4105009p4324561.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Inconsistent numFound in SC when querying core directly
Posted by Tim Vaillancourt <ti...@elementspace.com>.
I spoke too soon, my plan for fixing this didn't quite work.
I've moved this issue into a new thread/topic: "No /clusterstate.json
updates on Solrcloud 4.3.1 Cores API UNLOAD/CREATE".
Thanks all for the help on this one!
Tim
On 5 December 2013 11:37, Tim Vaillancourt <ti...@elementspace.com> wrote:
> Very good point. I've seen this issue occur once before when I was playing
> with 4.3.1 and don't remember it happening since 4.5.0+, so that is good
> news - we are just behind.
>
> For anyone that is curious, on my earlier mention that
> Zookeeper/clusterstate.json was not taking updates: this was NOT correct.
> Zookeeper has no issues taking set/creates to clusterstate.json (or any
> znode), just this one node seemed to stay stuck as "state: active" while it
> was very inconsistent for reasons unknown, potentially just bugs.
>
> The good news is this will be resolved today with a create/destroy of the
> bad replica.
>
> Thanks all!
>
> Tim
>
>
> On 4 December 2013 16:50, Mark Miller <ma...@gmail.com> wrote:
>
>> Keep in mind, there have been a *lot* of bug fixes since 4.3.1.
>>
>> - Mark
>>
>> On Dec 4, 2013, at 7:07 PM, Tim Vaillancourt <ti...@elementspace.com>
>> wrote:
>>
>> > Hey all,
>> >
>> > Now that I am getting correct results with "distrib=false", I've
>> identified that 1 of my nodes has just 1/3rd of the total data set and
>> totally explains the flapping in results. The fix for this is obvious
>> (rebuild replica) but the cause is less obvious.
>> >
>> > There is definately more than one issue going on with this SolrCloud
>> (but 1 down thanks to Chris' suggestion!), so I'm guessing the fact that
>> /clusterstate.json doesn't seem to get updated when nodes are brought
>> down/up is the reason why this replica remained in the distributed request
>> chain without recovering/re-replicating from leader.
>> >
>> > I imagine my Zookeeper ensemble is having some problems unrelated to
>> Solr that is the real root cause.
>> >
>> > Thanks!
>> >
>> > Tim
>> >
>> > On 04/12/13 03:00 PM, Tim Vaillancourt wrote:
>> >> Chris, this is extremely helpful and it's silly I didn't think of this
>> sooner! Thanks a lot, this makes the situation make much more sense.
>> >>
>> >> I will gather some proper data with your suggestion and get back to
>> the thread shortly.
>> >>
>> >> Thanks!!
>> >>
>> >> Tim
>> >>
>> >> On 04/12/13 02:57 PM, Chris Hostetter wrote:
>> >>> :
>> >>> : I may be incorrect here, but I assumed when querying a single core
>> of a
>> >>> : SolrCloud collection, the SolrCloud routing is bypassed and I am
>> talking
>> >>> : directly to a plain/non-SolrCloud core.
>> >>>
>> >>> No ... every query received from a client by solr is handled by a
>> single
>> >>> core -- if that core knows it's part of a SolrCloud collection then it
>> >>> will do a distributed search across a random replica from each shard
>> in
>> >>> that collection.
>> >>>
>> >>> If you want to bypass the distribute search logic, you have to say so
>> >>> explicitly...
>> >>>
>> >>> To ask an arbitrary replica to only search itself add "distrib=false"
>> to
>> >>> the request.
>> >>>
>> >>> Alternatively: you can ask that only certain shard names (or certain
>> >>> explicit replicas) be included in a distribute request..
>> >>>
>> >>> https://cwiki.apache.org/confluence/display/solr/Distributed+Requests
>> >>>
>> >>>
>> >>>
>> >>> -Hoss
>> >>> http://www.lucidworks.com/
>>
>>
>
Re: Inconsistent numFound in SC when querying core directly
Posted by Tim Vaillancourt <ti...@elementspace.com>.
Very good point. I've seen this issue occur once before when I was playing
with 4.3.1 and don't remember it happening since 4.5.0+, so that is good
news - we are just behind.
For anyone that is curious, on my earlier mention that
Zookeeper/clusterstate.json was not taking updates: this was NOT correct.
Zookeeper has no issues taking set/creates to clusterstate.json (or any
znode), just this one node seemed to stay stuck as "state: active" while it
was very inconsistent for reasons unknown, potentially just bugs.
The good news is this will be resolved today with a create/destroy of the
bad replica.
Thanks all!
Tim
On 4 December 2013 16:50, Mark Miller <ma...@gmail.com> wrote:
> Keep in mind, there have been a *lot* of bug fixes since 4.3.1.
>
> - Mark
>
> On Dec 4, 2013, at 7:07 PM, Tim Vaillancourt <ti...@elementspace.com> wrote:
>
> > Hey all,
> >
> > Now that I am getting correct results with "distrib=false", I've
> identified that 1 of my nodes has just 1/3rd of the total data set and
> totally explains the flapping in results. The fix for this is obvious
> (rebuild replica) but the cause is less obvious.
> >
> > There is definately more than one issue going on with this SolrCloud
> (but 1 down thanks to Chris' suggestion!), so I'm guessing the fact that
> /clusterstate.json doesn't seem to get updated when nodes are brought
> down/up is the reason why this replica remained in the distributed request
> chain without recovering/re-replicating from leader.
> >
> > I imagine my Zookeeper ensemble is having some problems unrelated to
> Solr that is the real root cause.
> >
> > Thanks!
> >
> > Tim
> >
> > On 04/12/13 03:00 PM, Tim Vaillancourt wrote:
> >> Chris, this is extremely helpful and it's silly I didn't think of this
> sooner! Thanks a lot, this makes the situation make much more sense.
> >>
> >> I will gather some proper data with your suggestion and get back to the
> thread shortly.
> >>
> >> Thanks!!
> >>
> >> Tim
> >>
> >> On 04/12/13 02:57 PM, Chris Hostetter wrote:
> >>> :
> >>> : I may be incorrect here, but I assumed when querying a single core
> of a
> >>> : SolrCloud collection, the SolrCloud routing is bypassed and I am
> talking
> >>> : directly to a plain/non-SolrCloud core.
> >>>
> >>> No ... every query received from a client by solr is handled by a
> single
> >>> core -- if that core knows it's part of a SolrCloud collection then it
> >>> will do a distributed search across a random replica from each shard in
> >>> that collection.
> >>>
> >>> If you want to bypass the distribute search logic, you have to say so
> >>> explicitly...
> >>>
> >>> To ask an arbitrary replica to only search itself add "distrib=false"
> to
> >>> the request.
> >>>
> >>> Alternatively: you can ask that only certain shard names (or certain
> >>> explicit replicas) be included in a distribute request..
> >>>
> >>> https://cwiki.apache.org/confluence/display/solr/Distributed+Requests
> >>>
> >>>
> >>>
> >>> -Hoss
> >>> http://www.lucidworks.com/
>
>
Re: Inconsistent numFound in SC when querying core directly
Posted by Mark Miller <ma...@gmail.com>.
Keep in mind, there have been a *lot* of bug fixes since 4.3.1.
- Mark
On Dec 4, 2013, at 7:07 PM, Tim Vaillancourt <ti...@elementspace.com> wrote:
> Hey all,
>
> Now that I am getting correct results with "distrib=false", I've identified that 1 of my nodes has just 1/3rd of the total data set and totally explains the flapping in results. The fix for this is obvious (rebuild replica) but the cause is less obvious.
>
> There is definately more than one issue going on with this SolrCloud (but 1 down thanks to Chris' suggestion!), so I'm guessing the fact that /clusterstate.json doesn't seem to get updated when nodes are brought down/up is the reason why this replica remained in the distributed request chain without recovering/re-replicating from leader.
>
> I imagine my Zookeeper ensemble is having some problems unrelated to Solr that is the real root cause.
>
> Thanks!
>
> Tim
>
> On 04/12/13 03:00 PM, Tim Vaillancourt wrote:
>> Chris, this is extremely helpful and it's silly I didn't think of this sooner! Thanks a lot, this makes the situation make much more sense.
>>
>> I will gather some proper data with your suggestion and get back to the thread shortly.
>>
>> Thanks!!
>>
>> Tim
>>
>> On 04/12/13 02:57 PM, Chris Hostetter wrote:
>>> :
>>> : I may be incorrect here, but I assumed when querying a single core of a
>>> : SolrCloud collection, the SolrCloud routing is bypassed and I am talking
>>> : directly to a plain/non-SolrCloud core.
>>>
>>> No ... every query received from a client by solr is handled by a single
>>> core -- if that core knows it's part of a SolrCloud collection then it
>>> will do a distributed search across a random replica from each shard in
>>> that collection.
>>>
>>> If you want to bypass the distribute search logic, you have to say so
>>> explicitly...
>>>
>>> To ask an arbitrary replica to only search itself add "distrib=false" to
>>> the request.
>>>
>>> Alternatively: you can ask that only certain shard names (or certain
>>> explicit replicas) be included in a distribute request..
>>>
>>> https://cwiki.apache.org/confluence/display/solr/Distributed+Requests
>>>
>>>
>>>
>>> -Hoss
>>> http://www.lucidworks.com/
Re: Inconsistent numFound in SC when querying core directly
Posted by Tim Vaillancourt <ti...@elementspace.com>.
Hey all,
Now that I am getting correct results with "distrib=false", I've
identified that 1 of my nodes has just 1/3rd of the total data set and
totally explains the flapping in results. The fix for this is obvious
(rebuild replica) but the cause is less obvious.
There is definately more than one issue going on with this SolrCloud
(but 1 down thanks to Chris' suggestion!), so I'm guessing the fact that
/clusterstate.json doesn't seem to get updated when nodes are brought
down/up is the reason why this replica remained in the distributed
request chain without recovering/re-replicating from leader.
I imagine my Zookeeper ensemble is having some problems unrelated to
Solr that is the real root cause.
Thanks!
Tim
On 04/12/13 03:00 PM, Tim Vaillancourt wrote:
> Chris, this is extremely helpful and it's silly I didn't think of this
> sooner! Thanks a lot, this makes the situation make much more sense.
>
> I will gather some proper data with your suggestion and get back to
> the thread shortly.
>
> Thanks!!
>
> Tim
>
> On 04/12/13 02:57 PM, Chris Hostetter wrote:
>> :
>> : I may be incorrect here, but I assumed when querying a single core
>> of a
>> : SolrCloud collection, the SolrCloud routing is bypassed and I am
>> talking
>> : directly to a plain/non-SolrCloud core.
>>
>> No ... every query received from a client by solr is handled by a single
>> core -- if that core knows it's part of a SolrCloud collection then it
>> will do a distributed search across a random replica from each shard in
>> that collection.
>>
>> If you want to bypass the distribute search logic, you have to say so
>> explicitly...
>>
>> To ask an arbitrary replica to only search itself add "distrib=false" to
>> the request.
>>
>> Alternatively: you can ask that only certain shard names (or certain
>> explicit replicas) be included in a distribute request..
>>
>> https://cwiki.apache.org/confluence/display/solr/Distributed+Requests
>>
>>
>>
>> -Hoss
>> http://www.lucidworks.com/
Re: Inconsistent numFound in SC when querying core directly
Posted by Tim Vaillancourt <ti...@elementspace.com>.
Chris, this is extremely helpful and it's silly I didn't think of this
sooner! Thanks a lot, this makes the situation make much more sense.
I will gather some proper data with your suggestion and get back to the
thread shortly.
Thanks!!
Tim
On 04/12/13 02:57 PM, Chris Hostetter wrote:
> :
> : I may be incorrect here, but I assumed when querying a single core of a
> : SolrCloud collection, the SolrCloud routing is bypassed and I am talking
> : directly to a plain/non-SolrCloud core.
>
> No ... every query received from a client by solr is handled by a single
> core -- if that core knows it's part of a SolrCloud collection then it
> will do a distributed search across a random replica from each shard in
> that collection.
>
> If you want to bypass the distribute search logic, you have to say so
> explicitly...
>
> To ask an arbitrary replica to only search itself add "distrib=false" to
> the request.
>
> Alternatively: you can ask that only certain shard names (or certain
> explicit replicas) be included in a distribute request..
>
> https://cwiki.apache.org/confluence/display/solr/Distributed+Requests
>
>
>
> -Hoss
> http://www.lucidworks.com/
Re: Inconsistent numFound in SC when querying core directly
Posted by Chris Hostetter <ho...@fucit.org>.
:
: I may be incorrect here, but I assumed when querying a single core of a
: SolrCloud collection, the SolrCloud routing is bypassed and I am talking
: directly to a plain/non-SolrCloud core.
No ... every query received from a client by solr is handled by a single
core -- if that core knows it's part of a SolrCloud collection then it
will do a distributed search across a random replica from each shard in
that collection.
If you want to bypass the distribute search logic, you have to say so
explicitly...
To ask an arbitrary replica to only search itself add "distrib=false" to
the request.
Alternatively: you can ask that only certain shard names (or certain
explicit replicas) be included in a distribute request..
https://cwiki.apache.org/confluence/display/solr/Distributed+Requests
-Hoss
http://www.lucidworks.com/
Re: Inconsistent numFound in SC when querying core directly
Posted by Tim Vaillancourt <ti...@elementspace.com>.
To add two more pieces of data:
1) This occurs with real, conditional queries as well (eg:
"q=key:timvaillancourt"), not just the "q=*:*" I provided in my email.
2) I've noticed when I bring a node of the SolrCloud down it is
remaining "state: active" in my /clusterstate.json - something is really
wrong with this cloud! Would a Zookeeper issue explain my varied results
when querying a core directly?
Thanks again!
Tim
On 04/12/13 02:17 PM, Tim Vaillancourt wrote:
> Hey guys,
>
> I'm looking into a strange issue on an unhealthy 4.3.1 SolrCloud with
> 3-node external Zookeeper and 1 collection (2 shards, 2 replicas).
>
> Currently we are noticing inconsistent results from the SolrCloud when
> performing the same simple /select query many times to our collection.
> Almost every other query the numFound count (and the returned data)
> jumps between two very different values.
>
> Initially I suspected a replica in a shard of the collection was
> inconsistent (and every other request hit that node) and started
> performing the same /select query direct to the individual cores of
> the SolrCloud collection on each instance, only to notice the same
> problem - the count jumps between two very different values!
>
> I may be incorrect here, but I assumed when querying a single core of
> a SolrCloud collection, the SolrCloud routing is bypassed and I am
> talking directly to a plain/non-SolrCloud core.
>
> As you can see here, the count for 1 core of my SolrCloud collection
> fluctuates wildly, and is only receiving updates and no deletes to
> explain the jumps:
>
> "solrcloud [tvaillancourt@prodapp solr_cloud]$ curl -s
> 'http://backend:8983/solr/app_shard2_replica2/select?q=*:*&wt=json&rows=0&indent=true'|grep
> numFound
> "response":{"numFound":123596839,"start":0,"maxScore":1.0,"docs":[]
>
> solrcloud [tvaillancourt@prodapp solr_cloud]$ curl -s
> 'http://backend:8983/solr/app_shard2_replica2/select?q=*:*&wt=json&rows=0&indent=true'|grep
> numFound
> "response":{"numFound":84739144,"start":0,"maxScore":1.0,"docs":[]
>
> solrcloud [tvaillancourt@prodapp solr_cloud]$ curl -s
> 'http://backend:8983/solr/app_shard2_replica2/select?q=*:*&wt=json&rows=0&indent=true'|grep
> numFound
> "response":{"numFound":123596839,"start":0,"maxScore":1.0,"docs":[]
>
> solrcloud [tvaillancourt@prodapp solr_cloud]$ curl -s
> 'http://backend:8983/solr/app_shard2_replica2/select?q=*:*&wt=json&rows=0&indent=true'|grep
> numFound
> "response":{"numFound":84771358,"start":0,"maxScore":1.0,"docs":[]"
>
>
> Could anyone help me understand why the same /select query direct to a
> single core would return inconsistent, flapping results if there are
> no deletes issued in my app to cause such jumps? Am I incorrect in my
> assumption that I am querying the core "directly"?
>
> An interesting observation is when I do an /admin/cores call to see
> the docCount of the core's index, it does not fluctuate, only the
> query result.
>
> That was hard to explain, hopefully someone has some insight! :)
>
> Thanks!
>
> Tim