You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ambari.apache.org by "amarnath reddy pappu (Jira)" <ji...@apache.org> on 2019/12/02 22:43:00 UTC
[jira] [Updated] (AMBARI-25438) Ambari starts NN with incorrect
flag during upgrade
[ https://issues.apache.org/jira/browse/AMBARI-25438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
amarnath reddy pappu updated AMBARI-25438:
------------------------------------------
Description:
Steps to reproduce:
# Install Ambari 2.6.x and HDP 2.6.x
# enable NN HA
# Upgrade Ambari 2.7.x
# Now Start HDP upgrade to 3.x and after HDFS upgrade is done pause the Upgrade
# Now stop both NNs are then attempt to start one from Ambari UI - then Ambari would attempt to start the NN with "–upgrade flag"
# If so it fails to come up with below exception.
{noformat}
2019-12-02 22:05:58,627 ERROR namenode.NameNode (NameNode.java:main(1715)) - Failed to start namenode. org.apache.hadoop.hdfs.server.common.InconsistentFSStateException: Directory /hadoop/hdfs/namenode is in an inconsistent state: previous fs state should not exist during upgrade. Finalize or rollback first. at org.apache.hadoop.hdfs.server.namenode.FSImage.checkUpgrade(FSImage.java:411) at org.apache.hadoop.hdfs.server.namenode.FSImage.checkUpgrade(FSImage.java:418) at org.apache.hadoop.hdfs.server.namenode.FSImage.doUpgrade(FSImage.java:438) at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:310) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1090) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:714) at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:632) at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:694) at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:937) at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:910) at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1643) at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1710) 2019-12-02 22:05:58,629 INFO util.ExitUtil (ExitUtil.java:terminate(210)) - Exiting with status 1: org.apache.hadoop.hdfs.server.common.InconsistentFSStateException: Directory /hadoop/hdfs/namenode is in an inconsistent state: previous fs state should not exist during upgrade. Finalize or rollback first.
{noformat}
Ambari should check if NN upgrade is already done - if it is done already then it should not check start "upgrade" flag.
was:
Steps to reproduce:
# Install Ambari 2.6.x and HDP 2.6.x
# enable NN HA
# Upgrade Ambari 2.7.x
# Now Start HDP upgrade to 3.x and after HDFS upgrade is done if you attempt to re-start Namenode outside of Upgrade flow ( Pause Upgrade) then Ambari would attempt to start the NN with "–upgrade flag"
# If so it fails to come up with below exception.
{noformat}
2019-12-02 22:05:58,627 ERROR namenode.NameNode (NameNode.java:main(1715)) - Failed to start namenode. org.apache.hadoop.hdfs.server.common.InconsistentFSStateException: Directory /hadoop/hdfs/namenode is in an inconsistent state: previous fs state should not exist during upgrade. Finalize or rollback first. at org.apache.hadoop.hdfs.server.namenode.FSImage.checkUpgrade(FSImage.java:411) at org.apache.hadoop.hdfs.server.namenode.FSImage.checkUpgrade(FSImage.java:418) at org.apache.hadoop.hdfs.server.namenode.FSImage.doUpgrade(FSImage.java:438) at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:310) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1090) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:714) at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:632) at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:694) at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:937) at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:910) at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1643) at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1710) 2019-12-02 22:05:58,629 INFO util.ExitUtil (ExitUtil.java:terminate(210)) - Exiting with status 1: org.apache.hadoop.hdfs.server.common.InconsistentFSStateException: Directory /hadoop/hdfs/namenode is in an inconsistent state: previous fs state should not exist during upgrade. Finalize or rollback first.
{noformat}
Ambari should check if NN upgrade is already done - if it is done already then it should not check start "upgrade" flag.
> Ambari starts NN with incorrect flag during upgrade
> ---------------------------------------------------
>
> Key: AMBARI-25438
> URL: https://issues.apache.org/jira/browse/AMBARI-25438
> Project: Ambari
> Issue Type: Bug
> Components: ambari-server
> Affects Versions: 2.7.3
> Reporter: amarnath reddy pappu
> Priority: Major
>
> Steps to reproduce:
> # Install Ambari 2.6.x and HDP 2.6.x
> # enable NN HA
> # Upgrade Ambari 2.7.x
> # Now Start HDP upgrade to 3.x and after HDFS upgrade is done pause the Upgrade
> # Now stop both NNs are then attempt to start one from Ambari UI - then Ambari would attempt to start the NN with "–upgrade flag"
> # If so it fails to come up with below exception.
> {noformat}
> 2019-12-02 22:05:58,627 ERROR namenode.NameNode (NameNode.java:main(1715)) - Failed to start namenode. org.apache.hadoop.hdfs.server.common.InconsistentFSStateException: Directory /hadoop/hdfs/namenode is in an inconsistent state: previous fs state should not exist during upgrade. Finalize or rollback first. at org.apache.hadoop.hdfs.server.namenode.FSImage.checkUpgrade(FSImage.java:411) at org.apache.hadoop.hdfs.server.namenode.FSImage.checkUpgrade(FSImage.java:418) at org.apache.hadoop.hdfs.server.namenode.FSImage.doUpgrade(FSImage.java:438) at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:310) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1090) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:714) at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:632) at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:694) at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:937) at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:910) at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1643) at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1710) 2019-12-02 22:05:58,629 INFO util.ExitUtil (ExitUtil.java:terminate(210)) - Exiting with status 1: org.apache.hadoop.hdfs.server.common.InconsistentFSStateException: Directory /hadoop/hdfs/namenode is in an inconsistent state: previous fs state should not exist during upgrade. Finalize or rollback first.
> {noformat}
> Ambari should check if NN upgrade is already done - if it is done already then it should not check start "upgrade" flag.
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)