You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@bigtop.apache.org by GitBox <gi...@apache.org> on 2022/06/14 09:47:33 UTC

[GitHub] [bigtop] guyuqi opened a new pull request, #913: BIGTOP-3701: Remove deprecated Hadoop fs-image after uninstalling HDF namenode

guyuqi opened a new pull request, #913:
URL: https://github.com/apache/bigtop/pull/913

   
   <!--
     Thanks for sending a pull request!
       1. If this is your first time, please read our contributor guidelines: https://cwiki.apache.org/confluence/display/BIGTOP/How+to+Contribute
       2. Make sure your PR title starts with JIRA issue id, e.g., 'BIGTOP-3638: Your PR title ...'.
   -->
   
   ### Description of PR
   When uninstalled Hadoop-2.10 and reinstalled Hadoop-3.2, it would fail to format namenode and The user have to restart NameNode with the "-rollingUpgrade started" option if a rolling upgrade is already started; or restart NameNode with the "-upgrade" option to start a new upgrade. 
   It would confuse some new BIgtop Hadoop users. 
   
   It's better to remove deprecated Hadoop fs-image file after uninstalling HDFS-namenode when uninstalling package.
   
   ### How was this patch tested?
   Build Hadoop by Bigtop and install/remove Hadoop package.
   ```
   e.g. :
   rpm -e hadoop-hdfs-3.2.3-1.el7.x86_64.rpm
   or
   rpm -qa | grep hadoop | sudo xargs yum -y remove
   ```
   
   
   ### For code changes:
   
   - [ ] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'BIGTOP-3638. Your PR title ...')?
   - [ ] Make sure that newly added files do not have any licensing issues. When in doubt refer to https://www.apache.org/licenses/


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [bigtop] guyuqi commented on pull request #913: BIGTOP-3701: Remove deprecated Hadoop fs-image after uninstalling HDF namenode

Posted by GitBox <gi...@apache.org>.
guyuqi commented on PR #913:
URL: https://github.com/apache/bigtop/pull/913#issuecomment-1156060230

   @iwasakims Thanks for your comments.
   Just as you mentioned,  users could overwrite the existing fsimage by `hdfs namenode -format -force` or `-upgrade`, but not all new comers would know how to handle it when the issues occurred like: 
   ```
   Failed to start namenode.
   java.io.IOException:
   File system image contains an old layout version -63.
   An upgrade to version -65 is required.
   .....
   ...
   ```
   
   Hdfs is the distributed file system to make data safe.  `Data replication `and `secondarynamenode` would help recover one node from the other nodes of the same cluster if fsimage and edit files were deleted.   (Is it really dangerous?)
   From my perspective, if users remove(`$1 = 0` here, not upgrade) the Hadoop packages from one node, it seems there is no necessary to retain the deprecated data. 
   
   On the other hand, it's convennient for users to deployment different Hadoop verisons by Mpack (or other automatic deployment tools) if we removed deprecated Hadoop fs-image after RPM/deb uninstallation.
   
   Is it the institutionalized acceptance that we don't touch `/hadoop/hdfs` even if there would be a obstacle to new comers?
   
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [bigtop] guyuqi commented on pull request #913: BIGTOP-3701: Remove deprecated Hadoop fs-image after uninstalling HDF namenode

Posted by GitBox <gi...@apache.org>.
guyuqi commented on PR #913:
URL: https://github.com/apache/bigtop/pull/913#issuecomment-1156170072

   OK, it seems to be well-accepted that rpm should not touch files which are not defined by rpm itself.  Appreciate your comments.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [bigtop] guyuqi closed pull request #913: BIGTOP-3701: Remove deprecated Hadoop fs-image after uninstalling HDF namenode

Posted by GitBox <gi...@apache.org>.
guyuqi closed pull request #913: BIGTOP-3701: Remove deprecated Hadoop fs-image after uninstalling HDF namenode
URL: https://github.com/apache/bigtop/pull/913


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [bigtop] iwasakims commented on pull request #913: BIGTOP-3701: Remove deprecated Hadoop fs-image after uninstalling HDF namenode

Posted by GitBox <gi...@apache.org>.
iwasakims commented on PR #913:
URL: https://github.com/apache/bigtop/pull/913#issuecomment-1156079870

   The paths for containing HDFS data are not defined by rpm/deb. Users arbitrarily specify them (by `dfs.namenode.name.dir` and `dfs.datanode.data.dir` in hdfs-site.xml). If the `/hadoop/hdfs` is defined in Puppet manifests, it shoud be cared by them. Modifying .spec to remove them does not make sense at all.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [bigtop] iwasakims commented on pull request #913: BIGTOP-3701: Remove deprecated Hadoop fs-image after uninstalling HDF namenode

Posted by GitBox <gi...@apache.org>.
iwasakims commented on PR #913:
URL: https://github.com/apache/bigtop/pull/913#issuecomment-1155894654

   -1. Users should do this dangerous operation on their own. We usually expect user-generated data to be preserved even after uninstalling (rpm/deb) package. If users want to format HDFS overwriting existing fsimage, `hdfs namenode -format -force` can be used on their responsibility.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org