Zookeeper migration

[Dwane Hall <dw...@hotmail.com>]

Hey Solr community,


I’m wondering if anyone has ever managed a zookeeper migration while running SolrCloud or if they have any advice on the process (not a zookeeper upgrade but a new physical instance migration)? I could not seem to find any endpoints in the collections or coreadmin api’s that catered for this scenario.


Initially I was hoping I could do all of the required zookeeper preparation (znode creation, clusterprops …) and start my existing Solr instance pointing at the new zookeeper but of course the new instance is unaware of the cluster state (state.json).  In fact when trailing this in a Development environment it was quite destructive operation as data from my SOLR_HOME (/var/solr/data) was physically deleted after I connected Solr to the new zookeeper instance! I’m uncertain if this is the expected behaviour or not but it’s certainly something for people to be aware of!


After some investigation and testing my thoughts are I’ll need to complete the following:


-         Stop the existing Solr instance so no updates are occurring

-         Backup the data

-         Create the znode on the new zookeeper instance

-         Update/upload the appropriate zookeeper managed files to the new zookeeper instance (security.json, clusterprops.json, solr.xml etc.)

-         Start Solr using ZK_HOST equal to the new zookeeper instance and znode (possibly use new Solr nodes here and not the existing ones)

-         Replicate the collection creation process on the new zookeeper instance

-         Physically copy the data from the old Solr nodes to the new Solr nodes and carefully map each replica and shard to the new location (which will have new replica names)

-         Start the new Solr instance

-         Clean up the old instance


So in summary has anybody completed a similar migration, can offer any advice, or are they aware of an easy way to transfer state between zookeeper instances to avoid the migration process I’ve outlined above?


Many thanks,


Dwane


Environment (SolrCloud)

Existing Zk: 3.4.2 (Bare metal)

New Zk: 3.4.2 (Docker)

Existing Solr: 7.7.2 (Docker)