[jira] [Created] (SOLR-16506) Flag exception during startup if replica node name does not match zk info

["Patson Luk (Jira)" <ji...@apache.org>]

Patson Luk created SOLR-16506:
---------------------------------

             Summary: Flag exception during startup if replica node name does not match zk info
                 Key: SOLR-16506
                 URL: https://issues.apache.org/jira/browse/SOLR-16506
             Project: Solr
          Issue Type: Improvement
      Security Level: Public (Default Security Level. Issues are Public)
          Components: SolrCloud
    Affects Versions: 9.1
            Reporter: Patson Luk


## Description

We have a scenario which 2 nodes (n1, n2) have under the data folder (`solr_data`) the same core name, both folders have `core.properties` but ONLY n1 has the data folder. And in the state.json for such collection, such core/replica has  `node_name` and `base_url` pointing at n1. 

Therefore n1 is the real node hosting the replica, we are not quite sure how we got to such state - could be from some migration failure. We call the replica on n2 the "ghost replica".

Now if we restart n2, it will actually took over such replica and even deletes the data from n1:
1. `CoreContainer#load`, calls `CorePropertiesLocator` which finds all the cores hosted on this node by walking through the solr data directory. It finds the ghost core and creates a `CoreDescriptor` for it
2. `CoreContainer#createFromDescriptor` is invoked to create a `SolrCore` out of the`CoreDescriptor`
3. `ZkController#preRegister` is called for such `CoreDescription`, which at would publish the replica state as `DOWN`, take note that usually `isPublishAsDownOnStartup` should return false, [but in this case it returns `true`|https://github.com/apache/solr/blob/main/solr/core/src/java/org/apache/solr/cloud/ZkController.java#L1988] as `replica.getNodeName().equals(getNodeName())` is `false`
4.  During `ZkController#publish`, it will [publish the state.json|https://github.com/apache/solr/blob/main/solr/core/src/java/org/apache/solr/cloud/ZkController.java#L1771] with incorrect `base_url` and `node_name` (n2)
5. Once the state.json is updated with the incorrect values, it triggers`UnloadCoreOnDeleteWatcher`, which [unload/delete|https://github.com/apache/solr/blob/b8ca0ce23e2ebe1b33c85b71fc61ab9cf8411a35/solr/core/src/java/org/apache/solr/cloud/ZkController.java#L2865] the core. It will also later publish `DELETECORE` to remove such core from zk

## Solution
It seems rather risky to update the state.json and publish such replica as down if such core does exist in the state.json but with different node name. 

Instead in `ZkController`, method `preRegister` -> `checkStateInZk`, we should interrupt the core loading if current node name is different from zookeeper state.json's value. Such that it should not attempt to publish DOWN to such replica and update the state.json, which possibly is the wrong node name





--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org