[jira] [Created] (HDFS-16792) Add -newEditsOnly option to name node initializeSharedEdits command to format unformatted journal nodes for master replacement

["Ashutosh Gupta (Jira)" <ji...@apache.org>]

Ashutosh Gupta created HDFS-16792:
-------------------------------------

             Summary: Add -newEditsOnly option to name node initializeSharedEdits command to format unformatted journal nodes for master replacement
                 Key: HDFS-16792
                 URL: https://issues.apache.org/jira/browse/HDFS-16792
             Project: Hadoop HDFS
          Issue Type: New Feature
          Components: journal-node
    Affects Versions: 3.3.4, 3.3.3
            Reporter: Ashutosh Gupta
            Assignee: Ashutosh Gupta


Add -newEditsOnly option to name node initializeSharedEdits command to format unformatted journal nodes for master replacement.

 

The current hadoop has limitations on formatting journal nodes. First, it requires all journal nodes to be present when formatting name node. After that, new journal nodes could not be formatted unless we reformat all of them and destroy all HDFS edit logs.

 

$ hdfs namenode -initializeSharedEdits -newEditsOnly

There are two cases here:

1) the replaced master instance has name node running on it, then the above command will only be run once on the existing name node.

2) the replaced master instance does not have a name node running on it, then the above command will be run twice. But the above command is idempotent as long as the same command will not be issued at the same time, which could be guaranteed by instance controller when reconfiguring master instances. The reason is that all hadoop name node commands such as format, bootstrap, and initializeSharedEdits are not designed to be thread safe and they rely on an external mechanism, which is a manual work for hadoop admin for on-premise clusters, to make sure that they are not executed concurrently. We still keep the same behavior of the initializeSharedEdits command to minimize the code/logic changes.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org