Restoring a REPLICA from existing replica

[Yuri Klayman <yk...@varonis.com>]

Hey,

I have a scenario where we run a Solr cluster in Azure, with Solr running in a VM and the data on a managed disk. One scenario that we want to handle and are considering different solutions for, is that one of the nodes\VMs will be down due to a network issue or a crash, but the disk will still be up.

Is it possible for us, to create a new VM, and use  MOVEREPLICA/ADDREPLICA/RESTORE on the existing data to create the replacement replicas ?
I guess I'm looking for a valid, safe way to update the collection state in ZK to point to the new node, does something like this exist ?

I assume the risk is that some of the indexes were corrupted, but the vast majority shouldn't have been, and it would save me copying TBs of data.
Probably we will also need something that shuts down the failed node after it disconnects in case it comes back up online.


Regards,
Yuri

________________________________
This email and any attachments thereto may contain private, confidential, and privileged material for the sole use of the intended recipient. Any review, copying, or distribution of this email (or any attachments thereto) by others is strictly prohibited. If you are not the intended recipient, please contact the sender immediately and permanently delete the original and any copies of this email and any attachments thereto.

Re: Restoring a REPLICA from existing replica

[Erick Erickson <er...@gmail.com>]

Well, the first question is whether you have other replicas, i.e. at
least one healthy leader for every shard. In that case, the simplest
thing to do would be ADDREPLICA/DELETEREPLICA

The ADDREPLICA will copy data from the leader to the new replica and
you should be fine.

If your data is on a shared FS and you _don't_ have a healthy leader,
MOVEREPLICA might work, but as you say this assumes the data is fine.

But indexes are just files. If you can ADDREPLICA, then shut down the
Solr node you just added the replica on and then copy the index over
(be careful not to copy the core.properties file, just the indexDIr
below) and bring Solr up. I'd run "CheckIndex' on it first though. You
may have to FORCELEADER if it's the only copy, but wait for 5 minutes
or so to see if it's elected leader, there's a fairly lengthy delay
before Solr will adopt a replica as leader in this situation.

Best,
Erick

On Sun, Jan 13, 2019 at 9:53 AM Yuri Klayman <yk...@varonis.com> wrote:
>
> Hey,
>
> I have a scenario where we run a Solr cluster in Azure, with Solr running in a VM and the data on a managed disk. One scenario that we want to handle and are considering different solutions for, is that one of the nodes\VMs will be down due to a network issue or a crash, but the disk will still be up.
>
> Is it possible for us, to create a new VM, and use  MOVEREPLICA/ADDREPLICA/RESTORE on the existing data to create the replacement replicas ?
> I guess I'm looking for a valid, safe way to update the collection state in ZK to point to the new node, does something like this exist ?
>
> I assume the risk is that some of the indexes were corrupted, but the vast majority shouldn't have been, and it would save me copying TBs of data.
> Probably we will also need something that shuts down the failed node after it disconnects in case it comes back up online.
>
>
> Regards,
> Yuri
>
> ________________________________
> This email and any attachments thereto may contain private, confidential, and privileged material for the sole use of the intended recipient. Any review, copying, or distribution of this email (or any attachments thereto) by others is strictly prohibited. If you are not the intended recipient, please contact the sender immediately and permanently delete the original and any copies of this email and any attachments thereto.