Re: Inconsistent numFound in SC when querying core directly

[vbindal <vi...@gmail.com>]

Hi,

I am facing the same issue where my query *:* returns inconsistent number
(almost 3 ) time the actual number in millions. 

When I try disturb=false on every machine, the results are correct. but
without `disturb=false` results are incorrect. 

Can you guys suggest something?



--
View this message in context: http://lucene.472066.n3.nabble.com/Inconsistent-numFound-in-SC-when-querying-core-directly-tp4105009p4324561.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Inconsistent numFound in SC when querying core directly

[Erick Erickson <er...@gmail.com>]

bq: If I changed the routing strategy back to composite (which it should be). is
it ok?

I sincerely doubt it. The docs have already been routed to the wrong
place (actually, I'm not sure how it worked at all). You can't get
them redistributed simply by changing the definition in ZooKeeper,
they're _already_ in the wrong place.

I'd tear down the corrupted data center and rebuild the collection.
Here "tear down" is delete all the affected collections and start over
again.

On the plus side, if you can get a window during which you are _not_
indexing you can copy the indexes from one of your good data centers
to the new one. Do it like this:


- Stop indexing.

- Set up the new collection in the corrupted data center. It's
important that it have _exactly_ the same number of shards ad the DC
you're going to transfer _from_. Also, make it leader only, i.e.
exactly 1 replica per shard.

- copy the indexes over from the good data center to the corresponding
shards. Here "corresponding" means that the source and destination
have the same hash range, which you can see from the state.json (or
clusterstate.json if you're on an earlier format). NOTE: there are two
ways to do this:
-- Just do file copies, scp, hand carry CDs, whatever. Solr should be
offline in the target data center.
-- use the replication API to issue a "fetchindex" command. This works
even in cloud mode, all the target Solr instance needs is access to a
URL it can pull from. Solr of course needs to be running in this case.

- Bring up Solr on the target data center and verify it's working.

- Use the Collections API to ADDREPLICA on the target system until you
build out the collection with the numbers of replicas you want.

- Start indexing to the target data center.

The bits about shutting off indexing is a safety measure, it
guarantees that the indexes are consistent. If you can't shut indexing
down during the transfer, you'll need to index docs to the
newly-rebuilt cluster in some manner that guarantees the two DCs will
have the same docs eventually.

Best,
Erick



On Tue, Mar 14, 2017 at 3:26 PM, vbindal <vi...@gmail.com> wrote:
> I think I dint explain properly.
>
> I have 3 data centers each with its own SOLR cloud.
>
> My original strategy was composite routing but when one data center went
> down and we brought it back, somehow the routing strategy on this changed to
> implicit (Other 2 DC still have composit and they are working absolutely
> fine).
>
> This might be the reason for the data corruption on that DS because the
> routing strategy got changed.
>
> If I changed the routing strategy back to composite (which it should be). is
> it ok? Do I need to do anything more than simply changing the strategy in
> the clusterState.json?
>
>
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/Inconsistent-numFound-in-SC-when-querying-core-directly-tp4105009p4325001.html
> Sent from the Solr - User mailing list archive at Nabble.com.

Re: Inconsistent numFound in SC when querying core directly

[vbindal <vi...@gmail.com>]

I think I dint explain properly.

I have 3 data centers each with its own SOLR cloud. 

My original strategy was composite routing but when one data center went
down and we brought it back, somehow the routing strategy on this changed to
implicit (Other 2 DC still have composit and they are working absolutely
fine).

This might be the reason for the data corruption on that DS because the
routing strategy got changed.

If I changed the routing strategy back to composite (which it should be). is
it ok? Do I need to do anything more than simply changing the strategy in
the clusterState.json? 



--
View this message in context: http://lucene.472066.n3.nabble.com/Inconsistent-numFound-in-SC-when-querying-core-directly-tp4105009p4325001.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Inconsistent numFound in SC when querying core directly

[Erick Erickson <er...@gmail.com>]

That would make the problem even worse. If you created the collection
with implicit routing, there are no hash ranges for each shard.
CompositeId requires hash ranges to be defined for each shard. Don't
even try.

Best,
Erick

On Tue, Mar 14, 2017 at 11:13 AM, vbindal <vi...@gmail.com> wrote:
> Compared it against the other 2 datacenters and they both have `compositeId
> `.
>
> This started happening after 1 of our zookeeper died due to hardware issue
> and we had to setup a new zookeeper machine. update the config in all the
> solr machine and restart the cloud. My guess is something went wrong and
> `implicit` router got created.
>
> Can I simply change the `clusterstate.json` to take care of this?
>
>
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/Inconsistent-numFound-in-SC-when-querying-core-directly-tp4105009p4324950.html
> Sent from the Solr - User mailing list archive at Nabble.com.

Re: Inconsistent numFound in SC when querying core directly

[vbindal <vi...@gmail.com>]

Compared it against the other 2 datacenters and they both have `compositeId
`.

This started happening after 1 of our zookeeper died due to hardware issue
and we had to setup a new zookeeper machine. update the config in all the
solr machine and restart the cloud. My guess is something went wrong and
`implicit` router got created.

Can I simply change the `clusterstate.json` to take care of this?



--
View this message in context: http://lucene.472066.n3.nabble.com/Inconsistent-numFound-in-SC-when-querying-core-directly-tp4105009p4324950.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Inconsistent numFound in SC when querying core directly

[Erick Erickson <er...@gmail.com>]

The default router has always been compositeId, but when you created
your collection you may have created it with implicit. Looking at the
clusterstate.json and/or state.json in the individual collection
should show you (admin UI>>cloud>>tree).

But we need to be very clear about what a "duplicate" document is.
Solr routes/replaces documents based on whatever you've defined as
<uniqueKey> in your schema file (assuming compositeID routing). When
you say you get dups when you re-index, it sounds like you are somehow
using different <uniqueKey>s for what you consider the "same"
document.

bq: Also, This started happening after 1 of our zookeeper died due to hardware
issue and we had to setup a new zookeeper machine. update the config in all
the solr machine and restart the cloud.

Hmmm. "update the config in all the solr machine...". You should not
have to do this in SolrCloud. All the configs are stored in Zookeeper
and loaded from ZK when the Solr instance starts. What it's starting
to sound like is that you've somehow mixed up SorlCloud and older
"stand-alone" concepts and "somehow" your restoration process messed
up your configs.

So if the issue isn't that you're somehow using different
<unkiqueKey>'s, I'd recommend just starting over with a new collection
since you can re-index from scratch.

Best,
Erick

On Tue, Mar 14, 2017 at 10:35 AM, vbindal <vi...@gmail.com> wrote:
> Hi Shawn,
>
> We are on 4.10.0 version. Is that the default router in this version? Also,
> we dont see all the documents duplicated, only some of them. I have a
> indexer job to index data in SOLR. After I delete all the records and run
> this job, the count is correct but when I run the job again, we start seeing
> higher count and duplicate records (random records) in shards.
>
> Also, This started happening after 1 of our zookeeper died due to hardware
> issue and we had to setup a new zookeeper machine. update the config in all
> the solr machine and restart the cloud.
>
>
>
>
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/Inconsistent-numFound-in-SC-when-querying-core-directly-tp4105009p4324937.html
> Sent from the Solr - User mailing list archive at Nabble.com.

Re: Inconsistent numFound in SC when querying core directly

[vbindal <vi...@gmail.com>]

Hi Shawn,

We are on 4.10.0 version. Is that the default router in this version? Also,
we dont see all the documents duplicated, only some of them. I have a
indexer job to index data in SOLR. After I delete all the records and run
this job, the count is correct but when I run the job again, we start seeing
higher count and duplicate records (random records) in shards. 

Also, This started happening after 1 of our zookeeper died due to hardware
issue and we had to setup a new zookeeper machine. update the config in all
the solr machine and restart the cloud. 





--
View this message in context: http://lucene.472066.n3.nabble.com/Inconsistent-numFound-in-SC-when-querying-core-directly-tp4105009p4324937.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Inconsistent numFound in SC when querying core directly

[Shawn Heisey <ap...@elyograg.org>]

On 3/13/2017 3:16 AM, vbindal wrote:
> I am facing the same issue where my query *:* returns inconsistent number (almost 3) time the actual number in millions.
>
> When I try disturb=false on every machine, the results are correct. but
> without `disturb=false` results are incorrect.

This most likely means that you've got duplicate documents in different
shards of your cloud.  This most commonly happens when the router is
"implicit" which is easier to understand if you imagine that "implicit"
is "manual" -- which would be a far better name for it.

In a later message you mention duplicate documents and 3 shards.  It
sounds like you have indexed all your documents to all three shards, and
are probably using the implicit router.

The implicit router basically means that there *IS* no shard routing --
documents would be either indexed by the shard that received them, or
directed to the shard indicated by explicit routing parameters.  With
three shards and "distrib=false" requests to one shard, you should see
about one-third of your total document count, not the full document count.

The router that you typically want if you don't want to be concerned
about how documents are routed to different shards is the compositeId
router.  This *should* be the default for a multi-shard collection in
any recent version, but the first one or two releases of SolrCloud might
have created the wrong type.

Thanks,
Shawn