Delete replica on down node, after start down node, the deleted replica comes back.

[Jerome Yang <je...@pivotal.io>]

Hi all,

I run into a strange behavior.
Both on solr6.1 and solr5.3.

For example, there are 4 nodes in cloud mode, one of them is stopped.
Then I delete a replica on the down node.
After that I start the down node.
The deleted replica comes back.

Is this a normal behavior?

Same situation.
4 nodes, 1 node is down.
And I delete a collection.
After start the down node.
Replicas in the down node of that collection come back again.
And I can not use collection api DELETE to delete it.
It says that collection is not exist.
But if I use CREATE action to create a same name collection, it says
collection is already exist.
The only way is to make things right is to clean it manually from zookeeper
and data directory.

How to prevent this happen?

Regards,
Jerome

Re: Delete replica on down node, after start down node, the deleted replica comes back.

[Erick Erickson <er...@gmail.com>]

Right, when you restart the downed node, all the
structure is still on disk, i.e. the index is there, the
core.properties file is there etc. I'm assuming you
use the collections DELETEREPLICA command.

Now when Solr starts up on that node, it uses
"core discovery" to find all the "core.properties" files
and reads the information there which includes
the collection and replica that that core belongs to
and registers itself in Zookeeper, thus the node
"comes back".

To get it to be gone permanently either
1> use DELETEREPLICA when the node is running
or
2> nuke the entire directory on the downed machine.
Actually, just renaming core.properties to something
else would do.

Here's a bit about core.properties....
https://cwiki.apache.org/confluence/display/solr/Defining+core.properties

Best,
Erick

On Tue, Aug 16, 2016 at 12:53 AM, Jerome Yang <je...@pivotal.io> wrote:
> Hi all,
>
> I run into a strange behavior.
> Both on solr6.1 and solr5.3.
>
> For example, there are 4 nodes in cloud mode, one of them is stopped.
> Then I delete a replica on the down node.
> After that I start the down node.
> The deleted replica comes back.
>
> Is this a normal behavior?
>
> Same situation.
> 4 nodes, 1 node is down.
> And I delete a collection.
> After start the down node.
> Replicas in the down node of that collection come back again.
> And I can not use collection api DELETE to delete it.
> It says that collection is not exist.
> But if I use CREATE action to create a same name collection, it says
> collection is already exist.
> The only way is to make things right is to clean it manually from zookeeper
> and data directory.
>
> How to prevent this happen?
>
> Regards,
> Jerome