You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-dev@hadoop.apache.org by Azuryy Yu <az...@gmail.com> on 2013/11/21 05:18:43 UTC
HDFS upgrade problem of fsImage
Hi Dear,
I have a small test cluster with hadoop-2.0x, and HA configuraded, but I
want to upgrade to hadoop-2.2.
I dont't want to stop cluster during upgrade, so my steps are:
1) on standby NN: hadoop-dameon.sh stop namenode
2) remove HA configuration in the conf
3) hadoop-daemon.sh start namenode -upgrade -clusterID test-cluster
but Exception in the NN log, so how to upgrade and don't stop the whole
cluster.
Thanks.
org.apache.hadoop.hdfs.server.common.InconsistentFSStateException:
Directory /hdfs/name is in an inconsistent state: previous fs state should
not exist during upgrade. Finalize or rollback first.
at
org.apache.hadoop.hdfs.server.namenode.FSImage.doUpgrade(FSImage.java:323)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:248)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:858)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:620)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:445)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:494)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:692)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:677)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1279)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1345)
Re: HDFS upgrade problem of fsImage
Posted by Azuryy Yu <az...@gmail.com>.
Thanks Steve,
I'd like to contribute for this JIRA. so do you mean HDFSv2 doesn't support
rolling upgrade currently?
On Thu, Nov 21, 2013 at 6:16 PM, Steve Loughran <st...@hortonworks.com>wrote:
> Given you are asking questions on hdfs-dev rather than hadoop-user, the
> normal home of user-level questions, i'd like to extend to you the
> opportunity to help the HDFS dev team to implement rolling updates
>
> https://issues.apache.org/jira/browse/HDFS-5535
>
> design reviews, code and tests welcome
>
> -steve
>
>
> On 21 November 2013 06:27, Azuryy Yu <az...@gmail.com> wrote:
>
> > No. I don't do any upgrade before this.
> >
> > I just want to rolling upgrade HDFS to hadoop-2.2.0, any further ideas?
> > Thanks.
> >
> >
> > On Thu, Nov 21, 2013 at 1:28 PM, Vinayakumar B <vinayakumar.b@huawei.com
> > >wrote:
> >
> > > Looks like you have already have upgraded cluster.. And you are trying
> to
> > > upgrade one more time.
> > >
> > >
> > > -----Original Message-----
> > > From: Azuryy Yu [mailto:azuryyyu@gmail.com]
> > > Sent: 21 November 2013 09:49
> > > To: hdfs-dev@hadoop.apache.org; user@hadoop.apache.org
> > > Subject: HDFS upgrade problem of fsImage
> > >
> > > Hi Dear,
> > >
> > > I have a small test cluster with hadoop-2.0x, and HA configuraded, but
> I
> > > want to upgrade to hadoop-2.2.
> > >
> > > I dont't want to stop cluster during upgrade, so my steps are:
> > >
> > > 1) on standby NN: hadoop-dameon.sh stop namenode
> > > 2) remove HA configuration in the conf
> > > 3) hadoop-daemon.sh start namenode -upgrade -clusterID test-cluster
> > >
> > > but Exception in the NN log, so how to upgrade and don't stop the whole
> > > cluster.
> > > Thanks.
> > >
> > >
> > > org.apache.hadoop.hdfs.server.common.InconsistentFSStateException:
> > > Directory /hdfs/name is in an inconsistent state: previous fs state
> > should
> > > not exist during upgrade. Finalize or rollback first.
> > > at
> > >
> >
> org.apache.hadoop.hdfs.server.namenode.FSImage.doUpgrade(FSImage.java:323)
> > > at
> > >
> > >
> >
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:248)
> > > at
> > >
> > >
> >
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:858)
> > > at
> > >
> > >
> >
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:620)
> > > at
> > >
> > >
> >
> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:445)
> > > at
> > >
> > >
> >
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:494)
> > > at
> > >
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:692)
> > > at
> > >
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:677)
> > > at
> > >
> > >
> >
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1279)
> > > at
> > >
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1345)
> > >
> >
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
Re: HDFS upgrade problem of fsImage
Posted by Steve Loughran <st...@hortonworks.com>.
Given you are asking questions on hdfs-dev rather than hadoop-user, the
normal home of user-level questions, i'd like to extend to you the
opportunity to help the HDFS dev team to implement rolling updates
https://issues.apache.org/jira/browse/HDFS-5535
design reviews, code and tests welcome
-steve
On 21 November 2013 06:27, Azuryy Yu <az...@gmail.com> wrote:
> No. I don't do any upgrade before this.
>
> I just want to rolling upgrade HDFS to hadoop-2.2.0, any further ideas?
> Thanks.
>
>
> On Thu, Nov 21, 2013 at 1:28 PM, Vinayakumar B <vinayakumar.b@huawei.com
> >wrote:
>
> > Looks like you have already have upgraded cluster.. And you are trying to
> > upgrade one more time.
> >
> >
> > -----Original Message-----
> > From: Azuryy Yu [mailto:azuryyyu@gmail.com]
> > Sent: 21 November 2013 09:49
> > To: hdfs-dev@hadoop.apache.org; user@hadoop.apache.org
> > Subject: HDFS upgrade problem of fsImage
> >
> > Hi Dear,
> >
> > I have a small test cluster with hadoop-2.0x, and HA configuraded, but I
> > want to upgrade to hadoop-2.2.
> >
> > I dont't want to stop cluster during upgrade, so my steps are:
> >
> > 1) on standby NN: hadoop-dameon.sh stop namenode
> > 2) remove HA configuration in the conf
> > 3) hadoop-daemon.sh start namenode -upgrade -clusterID test-cluster
> >
> > but Exception in the NN log, so how to upgrade and don't stop the whole
> > cluster.
> > Thanks.
> >
> >
> > org.apache.hadoop.hdfs.server.common.InconsistentFSStateException:
> > Directory /hdfs/name is in an inconsistent state: previous fs state
> should
> > not exist during upgrade. Finalize or rollback first.
> > at
> >
> org.apache.hadoop.hdfs.server.namenode.FSImage.doUpgrade(FSImage.java:323)
> > at
> >
> >
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:248)
> > at
> >
> >
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:858)
> > at
> >
> >
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:620)
> > at
> >
> >
> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:445)
> > at
> >
> >
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:494)
> > at
> > org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:692)
> > at
> > org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:677)
> > at
> >
> >
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1279)
> > at
> > org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1345)
> >
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: HDFS upgrade problem of fsImage
Posted by Azuryy Yu <az...@gmail.com>.
No. I don't do any upgrade before this.
I just want to rolling upgrade HDFS to hadoop-2.2.0, any further ideas?
Thanks.
On Thu, Nov 21, 2013 at 1:28 PM, Vinayakumar B <vi...@huawei.com>wrote:
> Looks like you have already have upgraded cluster.. And you are trying to
> upgrade one more time.
>
>
> -----Original Message-----
> From: Azuryy Yu [mailto:azuryyyu@gmail.com]
> Sent: 21 November 2013 09:49
> To: hdfs-dev@hadoop.apache.org; user@hadoop.apache.org
> Subject: HDFS upgrade problem of fsImage
>
> Hi Dear,
>
> I have a small test cluster with hadoop-2.0x, and HA configuraded, but I
> want to upgrade to hadoop-2.2.
>
> I dont't want to stop cluster during upgrade, so my steps are:
>
> 1) on standby NN: hadoop-dameon.sh stop namenode
> 2) remove HA configuration in the conf
> 3) hadoop-daemon.sh start namenode -upgrade -clusterID test-cluster
>
> but Exception in the NN log, so how to upgrade and don't stop the whole
> cluster.
> Thanks.
>
>
> org.apache.hadoop.hdfs.server.common.InconsistentFSStateException:
> Directory /hdfs/name is in an inconsistent state: previous fs state should
> not exist during upgrade. Finalize or rollback first.
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.doUpgrade(FSImage.java:323)
> at
>
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:248)
> at
>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:858)
> at
>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:620)
> at
>
> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:445)
> at
>
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:494)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:692)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:677)
> at
>
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1279)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1345)
>
RE: HDFS upgrade problem of fsImage
Posted by Vinayakumar B <vi...@huawei.com>.
Looks like you have already have upgraded cluster.. And you are trying to upgrade one more time.
-----Original Message-----
From: Azuryy Yu [mailto:azuryyyu@gmail.com]
Sent: 21 November 2013 09:49
To: hdfs-dev@hadoop.apache.org; user@hadoop.apache.org
Subject: HDFS upgrade problem of fsImage
Hi Dear,
I have a small test cluster with hadoop-2.0x, and HA configuraded, but I want to upgrade to hadoop-2.2.
I dont't want to stop cluster during upgrade, so my steps are:
1) on standby NN: hadoop-dameon.sh stop namenode
2) remove HA configuration in the conf
3) hadoop-daemon.sh start namenode -upgrade -clusterID test-cluster
but Exception in the NN log, so how to upgrade and don't stop the whole cluster.
Thanks.
org.apache.hadoop.hdfs.server.common.InconsistentFSStateException:
Directory /hdfs/name is in an inconsistent state: previous fs state should not exist during upgrade. Finalize or rollback first.
at
org.apache.hadoop.hdfs.server.namenode.FSImage.doUpgrade(FSImage.java:323)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:248)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:858)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:620)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:445)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:494)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:692)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:677)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1279)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1345)
Re: HDFS upgrade problem of fsImage
Posted by Azuryy Yu <az...@gmail.com>.
Thanks Joshi,
Maybe I pasted wrong log messages.
please looked at here for the real story.
https://issues.apache.org/jira/browse/HDFS-5550
On Fri, Nov 22, 2013 at 6:25 PM, Joshi, Rekha <Re...@intuit.com>wrote:
> Yes realized that and I see your point :-) However seems like some fs
> inconsistency present, did you attempt rollback/finalizeUpgrade and check?
>
> For that error, FSImage.java/code finds a previous fs state -
>
> // Upgrade is allowed only if there are
>
> // no previous fs states in any of the directories
>
> for (Iterator<StorageDirectory> it = storage.dirIterator();
> it.hasNext();) {
>
> StorageDirectory sd = it.next();
>
> if (sd.getPreviousDir().exists())
>
> throw new InconsistentFSStateException(sd.getRoot(),
>
> "previous fs state should not exist during upgrade. "
>
> + "Finalize or rollback first.");
>
> }
>
>
> Thanks
>
> Rekha
>
>
> From: Azuryy Yu <az...@gmail.com>
> Reply-To: "user@hadoop.apache.org" <us...@hadoop.apache.org>
> Date: Thursday 21 November 2013 5:19 PM
> To: "user@hadoop.apache.org" <us...@hadoop.apache.org>
> Cc: "hdfs-dev@hadoop.apache.org" <hd...@hadoop.apache.org>
> Subject: Re: HDFS upgrade problem of fsImage
>
> I insist hot upgrade on the test cluster because I want hot upgrade on
> the prod cluster.
> On 2013-11-21 7:23 PM, "Joshi, Rekha" <Re...@intuit.com> wrote:
>
>> Hi Azurry,
>>
>> This error occurs when FSImage finds previous fs state, and as log states you would need to either finalizeUpgrade or rollback to proceed.Below -
>>
>> bin/hadoop dfsadmin –finalizeUpgrade
>> hadoop dfsadmin –rollback
>>
>> On side note for a small test cluster on which one might suspect you are the only user, why wouldn't you insist on hot upgrade? :-)
>>
>> Thanks
>> Rekha
>>
>> Some helpful guidelines for upgrade here -
>>
>> http://wiki.apache.org/hadoop/Hadoop_Upgrade
>>
>> https://twiki.grid.iu.edu/bin/view/Storage/HadoopUpgrade
>>
>> http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/Federation.html#Upgrading_from_older_release_to_0.23_and_configuring_federation
>>
>>
>> From: Azuryy Yu <az...@gmail.com>
>> Reply-To: "user@hadoop.apache.org" <us...@hadoop.apache.org>
>> Date: Thursday 21 November 2013 9:48 AM
>> To: "hdfs-dev@hadoop.apache.org" <hd...@hadoop.apache.org>, "
>> user@hadoop.apache.org" <us...@hadoop.apache.org>
>> Subject: HDFS upgrade problem of fsImage
>>
>> Hi Dear,
>>
>> I have a small test cluster with hadoop-2.0x, and HA configuraded, but
>> I want to upgrade to hadoop-2.2.
>>
>> I dont't want to stop cluster during upgrade, so my steps are:
>>
>> 1) on standby NN: hadoop-dameon.sh stop namenode
>> 2) remove HA configuration in the conf
>> 3) hadoop-daemon.sh start namenode -upgrade -clusterID test-cluster
>>
>> but Exception in the NN log, so how to upgrade and don't stop the whole
>> cluster.
>> Thanks.
>>
>>
>> org.apache.hadoop.hdfs.server.common.InconsistentFSStateException:
>> Directory /hdfs/name is in an inconsistent state: previous fs state should
>> not exist during upgrade. Finalize or rollback first.
>> at
>> org.apache.hadoop.hdfs.server.namenode.FSImage.doUpgrade(FSImage.java:323)
>> at
>> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:248)
>> at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:858)
>> at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:620)
>> at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:445)
>> at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:494)
>> at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:692)
>> at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:677)
>> at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1279)
>> at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1345)
>>
>
Re: HDFS upgrade problem of fsImage
Posted by Azuryy Yu <az...@gmail.com>.
Thanks Joshi,
Maybe I pasted wrong log messages.
please looked at here for the real story.
https://issues.apache.org/jira/browse/HDFS-5550
On Fri, Nov 22, 2013 at 6:25 PM, Joshi, Rekha <Re...@intuit.com>wrote:
> Yes realized that and I see your point :-) However seems like some fs
> inconsistency present, did you attempt rollback/finalizeUpgrade and check?
>
> For that error, FSImage.java/code finds a previous fs state -
>
> // Upgrade is allowed only if there are
>
> // no previous fs states in any of the directories
>
> for (Iterator<StorageDirectory> it = storage.dirIterator();
> it.hasNext();) {
>
> StorageDirectory sd = it.next();
>
> if (sd.getPreviousDir().exists())
>
> throw new InconsistentFSStateException(sd.getRoot(),
>
> "previous fs state should not exist during upgrade. "
>
> + "Finalize or rollback first.");
>
> }
>
>
> Thanks
>
> Rekha
>
>
> From: Azuryy Yu <az...@gmail.com>
> Reply-To: "user@hadoop.apache.org" <us...@hadoop.apache.org>
> Date: Thursday 21 November 2013 5:19 PM
> To: "user@hadoop.apache.org" <us...@hadoop.apache.org>
> Cc: "hdfs-dev@hadoop.apache.org" <hd...@hadoop.apache.org>
> Subject: Re: HDFS upgrade problem of fsImage
>
> I insist hot upgrade on the test cluster because I want hot upgrade on
> the prod cluster.
> On 2013-11-21 7:23 PM, "Joshi, Rekha" <Re...@intuit.com> wrote:
>
>> Hi Azurry,
>>
>> This error occurs when FSImage finds previous fs state, and as log states you would need to either finalizeUpgrade or rollback to proceed.Below -
>>
>> bin/hadoop dfsadmin –finalizeUpgrade
>> hadoop dfsadmin –rollback
>>
>> On side note for a small test cluster on which one might suspect you are the only user, why wouldn't you insist on hot upgrade? :-)
>>
>> Thanks
>> Rekha
>>
>> Some helpful guidelines for upgrade here -
>>
>> http://wiki.apache.org/hadoop/Hadoop_Upgrade
>>
>> https://twiki.grid.iu.edu/bin/view/Storage/HadoopUpgrade
>>
>> http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/Federation.html#Upgrading_from_older_release_to_0.23_and_configuring_federation
>>
>>
>> From: Azuryy Yu <az...@gmail.com>
>> Reply-To: "user@hadoop.apache.org" <us...@hadoop.apache.org>
>> Date: Thursday 21 November 2013 9:48 AM
>> To: "hdfs-dev@hadoop.apache.org" <hd...@hadoop.apache.org>, "
>> user@hadoop.apache.org" <us...@hadoop.apache.org>
>> Subject: HDFS upgrade problem of fsImage
>>
>> Hi Dear,
>>
>> I have a small test cluster with hadoop-2.0x, and HA configuraded, but
>> I want to upgrade to hadoop-2.2.
>>
>> I dont't want to stop cluster during upgrade, so my steps are:
>>
>> 1) on standby NN: hadoop-dameon.sh stop namenode
>> 2) remove HA configuration in the conf
>> 3) hadoop-daemon.sh start namenode -upgrade -clusterID test-cluster
>>
>> but Exception in the NN log, so how to upgrade and don't stop the whole
>> cluster.
>> Thanks.
>>
>>
>> org.apache.hadoop.hdfs.server.common.InconsistentFSStateException:
>> Directory /hdfs/name is in an inconsistent state: previous fs state should
>> not exist during upgrade. Finalize or rollback first.
>> at
>> org.apache.hadoop.hdfs.server.namenode.FSImage.doUpgrade(FSImage.java:323)
>> at
>> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:248)
>> at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:858)
>> at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:620)
>> at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:445)
>> at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:494)
>> at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:692)
>> at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:677)
>> at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1279)
>> at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1345)
>>
>
Re: HDFS upgrade problem of fsImage
Posted by Azuryy Yu <az...@gmail.com>.
Thanks Joshi,
Maybe I pasted wrong log messages.
please looked at here for the real story.
https://issues.apache.org/jira/browse/HDFS-5550
On Fri, Nov 22, 2013 at 6:25 PM, Joshi, Rekha <Re...@intuit.com>wrote:
> Yes realized that and I see your point :-) However seems like some fs
> inconsistency present, did you attempt rollback/finalizeUpgrade and check?
>
> For that error, FSImage.java/code finds a previous fs state -
>
> // Upgrade is allowed only if there are
>
> // no previous fs states in any of the directories
>
> for (Iterator<StorageDirectory> it = storage.dirIterator();
> it.hasNext();) {
>
> StorageDirectory sd = it.next();
>
> if (sd.getPreviousDir().exists())
>
> throw new InconsistentFSStateException(sd.getRoot(),
>
> "previous fs state should not exist during upgrade. "
>
> + "Finalize or rollback first.");
>
> }
>
>
> Thanks
>
> Rekha
>
>
> From: Azuryy Yu <az...@gmail.com>
> Reply-To: "user@hadoop.apache.org" <us...@hadoop.apache.org>
> Date: Thursday 21 November 2013 5:19 PM
> To: "user@hadoop.apache.org" <us...@hadoop.apache.org>
> Cc: "hdfs-dev@hadoop.apache.org" <hd...@hadoop.apache.org>
> Subject: Re: HDFS upgrade problem of fsImage
>
> I insist hot upgrade on the test cluster because I want hot upgrade on
> the prod cluster.
> On 2013-11-21 7:23 PM, "Joshi, Rekha" <Re...@intuit.com> wrote:
>
>> Hi Azurry,
>>
>> This error occurs when FSImage finds previous fs state, and as log states you would need to either finalizeUpgrade or rollback to proceed.Below -
>>
>> bin/hadoop dfsadmin –finalizeUpgrade
>> hadoop dfsadmin –rollback
>>
>> On side note for a small test cluster on which one might suspect you are the only user, why wouldn't you insist on hot upgrade? :-)
>>
>> Thanks
>> Rekha
>>
>> Some helpful guidelines for upgrade here -
>>
>> http://wiki.apache.org/hadoop/Hadoop_Upgrade
>>
>> https://twiki.grid.iu.edu/bin/view/Storage/HadoopUpgrade
>>
>> http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/Federation.html#Upgrading_from_older_release_to_0.23_and_configuring_federation
>>
>>
>> From: Azuryy Yu <az...@gmail.com>
>> Reply-To: "user@hadoop.apache.org" <us...@hadoop.apache.org>
>> Date: Thursday 21 November 2013 9:48 AM
>> To: "hdfs-dev@hadoop.apache.org" <hd...@hadoop.apache.org>, "
>> user@hadoop.apache.org" <us...@hadoop.apache.org>
>> Subject: HDFS upgrade problem of fsImage
>>
>> Hi Dear,
>>
>> I have a small test cluster with hadoop-2.0x, and HA configuraded, but
>> I want to upgrade to hadoop-2.2.
>>
>> I dont't want to stop cluster during upgrade, so my steps are:
>>
>> 1) on standby NN: hadoop-dameon.sh stop namenode
>> 2) remove HA configuration in the conf
>> 3) hadoop-daemon.sh start namenode -upgrade -clusterID test-cluster
>>
>> but Exception in the NN log, so how to upgrade and don't stop the whole
>> cluster.
>> Thanks.
>>
>>
>> org.apache.hadoop.hdfs.server.common.InconsistentFSStateException:
>> Directory /hdfs/name is in an inconsistent state: previous fs state should
>> not exist during upgrade. Finalize or rollback first.
>> at
>> org.apache.hadoop.hdfs.server.namenode.FSImage.doUpgrade(FSImage.java:323)
>> at
>> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:248)
>> at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:858)
>> at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:620)
>> at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:445)
>> at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:494)
>> at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:692)
>> at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:677)
>> at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1279)
>> at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1345)
>>
>
Re: HDFS upgrade problem of fsImage
Posted by Azuryy Yu <az...@gmail.com>.
Thanks Joshi,
Maybe I pasted wrong log messages.
please looked at here for the real story.
https://issues.apache.org/jira/browse/HDFS-5550
On Fri, Nov 22, 2013 at 6:25 PM, Joshi, Rekha <Re...@intuit.com>wrote:
> Yes realized that and I see your point :-) However seems like some fs
> inconsistency present, did you attempt rollback/finalizeUpgrade and check?
>
> For that error, FSImage.java/code finds a previous fs state -
>
> // Upgrade is allowed only if there are
>
> // no previous fs states in any of the directories
>
> for (Iterator<StorageDirectory> it = storage.dirIterator();
> it.hasNext();) {
>
> StorageDirectory sd = it.next();
>
> if (sd.getPreviousDir().exists())
>
> throw new InconsistentFSStateException(sd.getRoot(),
>
> "previous fs state should not exist during upgrade. "
>
> + "Finalize or rollback first.");
>
> }
>
>
> Thanks
>
> Rekha
>
>
> From: Azuryy Yu <az...@gmail.com>
> Reply-To: "user@hadoop.apache.org" <us...@hadoop.apache.org>
> Date: Thursday 21 November 2013 5:19 PM
> To: "user@hadoop.apache.org" <us...@hadoop.apache.org>
> Cc: "hdfs-dev@hadoop.apache.org" <hd...@hadoop.apache.org>
> Subject: Re: HDFS upgrade problem of fsImage
>
> I insist hot upgrade on the test cluster because I want hot upgrade on
> the prod cluster.
> On 2013-11-21 7:23 PM, "Joshi, Rekha" <Re...@intuit.com> wrote:
>
>> Hi Azurry,
>>
>> This error occurs when FSImage finds previous fs state, and as log states you would need to either finalizeUpgrade or rollback to proceed.Below -
>>
>> bin/hadoop dfsadmin –finalizeUpgrade
>> hadoop dfsadmin –rollback
>>
>> On side note for a small test cluster on which one might suspect you are the only user, why wouldn't you insist on hot upgrade? :-)
>>
>> Thanks
>> Rekha
>>
>> Some helpful guidelines for upgrade here -
>>
>> http://wiki.apache.org/hadoop/Hadoop_Upgrade
>>
>> https://twiki.grid.iu.edu/bin/view/Storage/HadoopUpgrade
>>
>> http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/Federation.html#Upgrading_from_older_release_to_0.23_and_configuring_federation
>>
>>
>> From: Azuryy Yu <az...@gmail.com>
>> Reply-To: "user@hadoop.apache.org" <us...@hadoop.apache.org>
>> Date: Thursday 21 November 2013 9:48 AM
>> To: "hdfs-dev@hadoop.apache.org" <hd...@hadoop.apache.org>, "
>> user@hadoop.apache.org" <us...@hadoop.apache.org>
>> Subject: HDFS upgrade problem of fsImage
>>
>> Hi Dear,
>>
>> I have a small test cluster with hadoop-2.0x, and HA configuraded, but
>> I want to upgrade to hadoop-2.2.
>>
>> I dont't want to stop cluster during upgrade, so my steps are:
>>
>> 1) on standby NN: hadoop-dameon.sh stop namenode
>> 2) remove HA configuration in the conf
>> 3) hadoop-daemon.sh start namenode -upgrade -clusterID test-cluster
>>
>> but Exception in the NN log, so how to upgrade and don't stop the whole
>> cluster.
>> Thanks.
>>
>>
>> org.apache.hadoop.hdfs.server.common.InconsistentFSStateException:
>> Directory /hdfs/name is in an inconsistent state: previous fs state should
>> not exist during upgrade. Finalize or rollback first.
>> at
>> org.apache.hadoop.hdfs.server.namenode.FSImage.doUpgrade(FSImage.java:323)
>> at
>> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:248)
>> at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:858)
>> at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:620)
>> at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:445)
>> at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:494)
>> at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:692)
>> at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:677)
>> at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1279)
>> at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1345)
>>
>
Re: HDFS upgrade problem of fsImage
Posted by "Joshi, Rekha" <Re...@intuit.com>.
Yes realized that and I see your point :-) However seems like some fs inconsistency present, did you attempt rollback/finalizeUpgrade and check?
For that error, FSImage.java/code finds a previous fs state -
// Upgrade is allowed only if there are
// no previous fs states in any of the directories
for (Iterator<StorageDirectory> it = storage.dirIterator(); it.hasNext();) {
StorageDirectory sd = it.next();
if (sd.getPreviousDir().exists())
throw new InconsistentFSStateException(sd.getRoot(),
"previous fs state should not exist during upgrade. "
+ "Finalize or rollback first.");
}
Thanks
Rekha
From: Azuryy Yu <az...@gmail.com>>
Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Date: Thursday 21 November 2013 5:19 PM
To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Cc: "hdfs-dev@hadoop.apache.org<ma...@hadoop.apache.org>" <hd...@hadoop.apache.org>>
Subject: Re: HDFS upgrade problem of fsImage
I insist hot upgrade on the test cluster because I want hot upgrade on the prod cluster.
On 2013-11-21 7:23 PM, "Joshi, Rekha" <Re...@intuit.com>> wrote:
Hi Azurry,
This error occurs when FSImage finds previous fs state, and as log states you would need to either finalizeUpgrade or rollback to proceed.Below -
bin/hadoop dfsadmin –finalizeUpgrade
hadoop dfsadmin –rollback
On side note for a small test cluster on which one might suspect you are the only user, why wouldn't you insist on hot upgrade? :-)
Thanks
Rekha
Some helpful guidelines for upgrade here -
http://wiki.apache.org/hadoop/Hadoop_Upgrade
https://twiki.grid.iu.edu/bin/view/Storage/HadoopUpgrade
http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/Federation.html#Upgrading_from_older_release_to_0.23_and_configuring_federation
From: Azuryy Yu <az...@gmail.com>>
Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Date: Thursday 21 November 2013 9:48 AM
To: "hdfs-dev@hadoop.apache.org<ma...@hadoop.apache.org>" <hd...@hadoop.apache.org>>, "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Subject: HDFS upgrade problem of fsImage
Hi Dear,
I have a small test cluster with hadoop-2.0x, and HA configuraded, but I want to upgrade to hadoop-2.2.
I dont't want to stop cluster during upgrade, so my steps are:
1) on standby NN: hadoop-dameon.sh stop namenode
2) remove HA configuration in the conf
3) hadoop-daemon.sh start namenode -upgrade -clusterID test-cluster
but Exception in the NN log, so how to upgrade and don't stop the whole cluster.
Thanks.
org.apache.hadoop.hdfs.server.common.InconsistentFSStateException: Directory /hdfs/name is in an inconsistent state: previous fs state should not exist during upgrade. Finalize or rollback first.
at org.apache.hadoop.hdfs.server.namenode.FSImage.doUpgrade(FSImage.java:323)
at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:248)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:858)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:620)
at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:445)
at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:494)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:692)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:677)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1279)
at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1345)
Re: HDFS upgrade problem of fsImage
Posted by "Joshi, Rekha" <Re...@intuit.com>.
Yes realized that and I see your point :-) However seems like some fs inconsistency present, did you attempt rollback/finalizeUpgrade and check?
For that error, FSImage.java/code finds a previous fs state -
// Upgrade is allowed only if there are
// no previous fs states in any of the directories
for (Iterator<StorageDirectory> it = storage.dirIterator(); it.hasNext();) {
StorageDirectory sd = it.next();
if (sd.getPreviousDir().exists())
throw new InconsistentFSStateException(sd.getRoot(),
"previous fs state should not exist during upgrade. "
+ "Finalize or rollback first.");
}
Thanks
Rekha
From: Azuryy Yu <az...@gmail.com>>
Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Date: Thursday 21 November 2013 5:19 PM
To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Cc: "hdfs-dev@hadoop.apache.org<ma...@hadoop.apache.org>" <hd...@hadoop.apache.org>>
Subject: Re: HDFS upgrade problem of fsImage
I insist hot upgrade on the test cluster because I want hot upgrade on the prod cluster.
On 2013-11-21 7:23 PM, "Joshi, Rekha" <Re...@intuit.com>> wrote:
Hi Azurry,
This error occurs when FSImage finds previous fs state, and as log states you would need to either finalizeUpgrade or rollback to proceed.Below -
bin/hadoop dfsadmin –finalizeUpgrade
hadoop dfsadmin –rollback
On side note for a small test cluster on which one might suspect you are the only user, why wouldn't you insist on hot upgrade? :-)
Thanks
Rekha
Some helpful guidelines for upgrade here -
http://wiki.apache.org/hadoop/Hadoop_Upgrade
https://twiki.grid.iu.edu/bin/view/Storage/HadoopUpgrade
http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/Federation.html#Upgrading_from_older_release_to_0.23_and_configuring_federation
From: Azuryy Yu <az...@gmail.com>>
Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Date: Thursday 21 November 2013 9:48 AM
To: "hdfs-dev@hadoop.apache.org<ma...@hadoop.apache.org>" <hd...@hadoop.apache.org>>, "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Subject: HDFS upgrade problem of fsImage
Hi Dear,
I have a small test cluster with hadoop-2.0x, and HA configuraded, but I want to upgrade to hadoop-2.2.
I dont't want to stop cluster during upgrade, so my steps are:
1) on standby NN: hadoop-dameon.sh stop namenode
2) remove HA configuration in the conf
3) hadoop-daemon.sh start namenode -upgrade -clusterID test-cluster
but Exception in the NN log, so how to upgrade and don't stop the whole cluster.
Thanks.
org.apache.hadoop.hdfs.server.common.InconsistentFSStateException: Directory /hdfs/name is in an inconsistent state: previous fs state should not exist during upgrade. Finalize or rollback first.
at org.apache.hadoop.hdfs.server.namenode.FSImage.doUpgrade(FSImage.java:323)
at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:248)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:858)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:620)
at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:445)
at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:494)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:692)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:677)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1279)
at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1345)
Re: HDFS upgrade problem of fsImage
Posted by "Joshi, Rekha" <Re...@intuit.com>.
Yes realized that and I see your point :-) However seems like some fs inconsistency present, did you attempt rollback/finalizeUpgrade and check?
For that error, FSImage.java/code finds a previous fs state -
// Upgrade is allowed only if there are
// no previous fs states in any of the directories
for (Iterator<StorageDirectory> it = storage.dirIterator(); it.hasNext();) {
StorageDirectory sd = it.next();
if (sd.getPreviousDir().exists())
throw new InconsistentFSStateException(sd.getRoot(),
"previous fs state should not exist during upgrade. "
+ "Finalize or rollback first.");
}
Thanks
Rekha
From: Azuryy Yu <az...@gmail.com>>
Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Date: Thursday 21 November 2013 5:19 PM
To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Cc: "hdfs-dev@hadoop.apache.org<ma...@hadoop.apache.org>" <hd...@hadoop.apache.org>>
Subject: Re: HDFS upgrade problem of fsImage
I insist hot upgrade on the test cluster because I want hot upgrade on the prod cluster.
On 2013-11-21 7:23 PM, "Joshi, Rekha" <Re...@intuit.com>> wrote:
Hi Azurry,
This error occurs when FSImage finds previous fs state, and as log states you would need to either finalizeUpgrade or rollback to proceed.Below -
bin/hadoop dfsadmin –finalizeUpgrade
hadoop dfsadmin –rollback
On side note for a small test cluster on which one might suspect you are the only user, why wouldn't you insist on hot upgrade? :-)
Thanks
Rekha
Some helpful guidelines for upgrade here -
http://wiki.apache.org/hadoop/Hadoop_Upgrade
https://twiki.grid.iu.edu/bin/view/Storage/HadoopUpgrade
http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/Federation.html#Upgrading_from_older_release_to_0.23_and_configuring_federation
From: Azuryy Yu <az...@gmail.com>>
Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Date: Thursday 21 November 2013 9:48 AM
To: "hdfs-dev@hadoop.apache.org<ma...@hadoop.apache.org>" <hd...@hadoop.apache.org>>, "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Subject: HDFS upgrade problem of fsImage
Hi Dear,
I have a small test cluster with hadoop-2.0x, and HA configuraded, but I want to upgrade to hadoop-2.2.
I dont't want to stop cluster during upgrade, so my steps are:
1) on standby NN: hadoop-dameon.sh stop namenode
2) remove HA configuration in the conf
3) hadoop-daemon.sh start namenode -upgrade -clusterID test-cluster
but Exception in the NN log, so how to upgrade and don't stop the whole cluster.
Thanks.
org.apache.hadoop.hdfs.server.common.InconsistentFSStateException: Directory /hdfs/name is in an inconsistent state: previous fs state should not exist during upgrade. Finalize or rollback first.
at org.apache.hadoop.hdfs.server.namenode.FSImage.doUpgrade(FSImage.java:323)
at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:248)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:858)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:620)
at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:445)
at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:494)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:692)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:677)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1279)
at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1345)
Re: HDFS upgrade problem of fsImage
Posted by "Joshi, Rekha" <Re...@intuit.com>.
Yes realized that and I see your point :-) However seems like some fs inconsistency present, did you attempt rollback/finalizeUpgrade and check?
For that error, FSImage.java/code finds a previous fs state -
// Upgrade is allowed only if there are
// no previous fs states in any of the directories
for (Iterator<StorageDirectory> it = storage.dirIterator(); it.hasNext();) {
StorageDirectory sd = it.next();
if (sd.getPreviousDir().exists())
throw new InconsistentFSStateException(sd.getRoot(),
"previous fs state should not exist during upgrade. "
+ "Finalize or rollback first.");
}
Thanks
Rekha
From: Azuryy Yu <az...@gmail.com>>
Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Date: Thursday 21 November 2013 5:19 PM
To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Cc: "hdfs-dev@hadoop.apache.org<ma...@hadoop.apache.org>" <hd...@hadoop.apache.org>>
Subject: Re: HDFS upgrade problem of fsImage
I insist hot upgrade on the test cluster because I want hot upgrade on the prod cluster.
On 2013-11-21 7:23 PM, "Joshi, Rekha" <Re...@intuit.com>> wrote:
Hi Azurry,
This error occurs when FSImage finds previous fs state, and as log states you would need to either finalizeUpgrade or rollback to proceed.Below -
bin/hadoop dfsadmin –finalizeUpgrade
hadoop dfsadmin –rollback
On side note for a small test cluster on which one might suspect you are the only user, why wouldn't you insist on hot upgrade? :-)
Thanks
Rekha
Some helpful guidelines for upgrade here -
http://wiki.apache.org/hadoop/Hadoop_Upgrade
https://twiki.grid.iu.edu/bin/view/Storage/HadoopUpgrade
http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/Federation.html#Upgrading_from_older_release_to_0.23_and_configuring_federation
From: Azuryy Yu <az...@gmail.com>>
Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Date: Thursday 21 November 2013 9:48 AM
To: "hdfs-dev@hadoop.apache.org<ma...@hadoop.apache.org>" <hd...@hadoop.apache.org>>, "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Subject: HDFS upgrade problem of fsImage
Hi Dear,
I have a small test cluster with hadoop-2.0x, and HA configuraded, but I want to upgrade to hadoop-2.2.
I dont't want to stop cluster during upgrade, so my steps are:
1) on standby NN: hadoop-dameon.sh stop namenode
2) remove HA configuration in the conf
3) hadoop-daemon.sh start namenode -upgrade -clusterID test-cluster
but Exception in the NN log, so how to upgrade and don't stop the whole cluster.
Thanks.
org.apache.hadoop.hdfs.server.common.InconsistentFSStateException: Directory /hdfs/name is in an inconsistent state: previous fs state should not exist during upgrade. Finalize or rollback first.
at org.apache.hadoop.hdfs.server.namenode.FSImage.doUpgrade(FSImage.java:323)
at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:248)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:858)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:620)
at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:445)
at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:494)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:692)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:677)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1279)
at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1345)
Re: HDFS upgrade problem of fsImage
Posted by Azuryy Yu <az...@gmail.com>.
I insist hot upgrade on the test cluster because I want hot upgrade on the
prod cluster.
On 2013-11-21 7:23 PM, "Joshi, Rekha" <Re...@intuit.com> wrote:
> Hi Azurry,
>
> This error occurs when FSImage finds previous fs state, and as log states you would need to either finalizeUpgrade or rollback to proceed.Below -
>
> bin/hadoop dfsadmin –finalizeUpgrade
> hadoop dfsadmin –rollback
>
> On side note for a small test cluster on which one might suspect you are the only user, why wouldn't you insist on hot upgrade? :-)
>
> Thanks
> Rekha
>
> Some helpful guidelines for upgrade here -
>
> http://wiki.apache.org/hadoop/Hadoop_Upgrade
>
> https://twiki.grid.iu.edu/bin/view/Storage/HadoopUpgrade
>
> http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/Federation.html#Upgrading_from_older_release_to_0.23_and_configuring_federation
>
>
> From: Azuryy Yu <az...@gmail.com>
> Reply-To: "user@hadoop.apache.org" <us...@hadoop.apache.org>
> Date: Thursday 21 November 2013 9:48 AM
> To: "hdfs-dev@hadoop.apache.org" <hd...@hadoop.apache.org>, "
> user@hadoop.apache.org" <us...@hadoop.apache.org>
> Subject: HDFS upgrade problem of fsImage
>
> Hi Dear,
>
> I have a small test cluster with hadoop-2.0x, and HA configuraded, but I
> want to upgrade to hadoop-2.2.
>
> I dont't want to stop cluster during upgrade, so my steps are:
>
> 1) on standby NN: hadoop-dameon.sh stop namenode
> 2) remove HA configuration in the conf
> 3) hadoop-daemon.sh start namenode -upgrade -clusterID test-cluster
>
> but Exception in the NN log, so how to upgrade and don't stop the whole
> cluster.
> Thanks.
>
>
> org.apache.hadoop.hdfs.server.common.InconsistentFSStateException:
> Directory /hdfs/name is in an inconsistent state: previous fs state should
> not exist during upgrade. Finalize or rollback first.
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.doUpgrade(FSImage.java:323)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:248)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:858)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:620)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:445)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:494)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:692)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:677)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1279)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1345)
>
Re: HDFS upgrade problem of fsImage
Posted by Azuryy Yu <az...@gmail.com>.
Thanks Joshi,
I don't have upgrade before. the test cluser is the new cluster with
hadoop-2.0.3. so I don't need to 'bin/hadoop dfsadmin –finalizeUpgrade'
On Thu, Nov 21, 2013 at 7:22 PM, Joshi, Rekha <Re...@intuit.com>wrote:
> Hi Azurry,
>
> This error occurs when FSImage finds previous fs state, and as log states you would need to either finalizeUpgrade or rollback to proceed.Below -
>
> bin/hadoop dfsadmin –finalizeUpgrade
> hadoop dfsadmin –rollback
>
> On side note for a small test cluster on which one might suspect you are the only user, why wouldn't you insist on hot upgrade? :-)
>
> Thanks
> Rekha
>
> Some helpful guidelines for upgrade here -
>
> http://wiki.apache.org/hadoop/Hadoop_Upgrade
>
> https://twiki.grid.iu.edu/bin/view/Storage/HadoopUpgrade
>
> http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/Federation.html#Upgrading_from_older_release_to_0.23_and_configuring_federation
>
>
> From: Azuryy Yu <az...@gmail.com>
> Reply-To: "user@hadoop.apache.org" <us...@hadoop.apache.org>
> Date: Thursday 21 November 2013 9:48 AM
> To: "hdfs-dev@hadoop.apache.org" <hd...@hadoop.apache.org>, "
> user@hadoop.apache.org" <us...@hadoop.apache.org>
>
> Subject: HDFS upgrade problem of fsImage
>
> Hi Dear,
>
> I have a small test cluster with hadoop-2.0x, and HA configuraded, but I
> want to upgrade to hadoop-2.2.
>
> I dont't want to stop cluster during upgrade, so my steps are:
>
> 1) on standby NN: hadoop-dameon.sh stop namenode
> 2) remove HA configuration in the conf
> 3) hadoop-daemon.sh start namenode -upgrade -clusterID test-cluster
>
> but Exception in the NN log, so how to upgrade and don't stop the whole
> cluster.
> Thanks.
>
>
> org.apache.hadoop.hdfs.server.common.InconsistentFSStateException:
> Directory /hdfs/name is in an inconsistent state: previous fs state should
> not exist during upgrade. Finalize or rollback first.
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.doUpgrade(FSImage.java:323)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:248)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:858)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:620)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:445)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:494)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:692)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:677)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1279)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1345)
>
Re: HDFS upgrade problem of fsImage
Posted by Azuryy Yu <az...@gmail.com>.
Thanks Joshi,
I don't have upgrade before. the test cluser is the new cluster with
hadoop-2.0.3. so I don't need to 'bin/hadoop dfsadmin –finalizeUpgrade'
On Thu, Nov 21, 2013 at 7:22 PM, Joshi, Rekha <Re...@intuit.com>wrote:
> Hi Azurry,
>
> This error occurs when FSImage finds previous fs state, and as log states you would need to either finalizeUpgrade or rollback to proceed.Below -
>
> bin/hadoop dfsadmin –finalizeUpgrade
> hadoop dfsadmin –rollback
>
> On side note for a small test cluster on which one might suspect you are the only user, why wouldn't you insist on hot upgrade? :-)
>
> Thanks
> Rekha
>
> Some helpful guidelines for upgrade here -
>
> http://wiki.apache.org/hadoop/Hadoop_Upgrade
>
> https://twiki.grid.iu.edu/bin/view/Storage/HadoopUpgrade
>
> http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/Federation.html#Upgrading_from_older_release_to_0.23_and_configuring_federation
>
>
> From: Azuryy Yu <az...@gmail.com>
> Reply-To: "user@hadoop.apache.org" <us...@hadoop.apache.org>
> Date: Thursday 21 November 2013 9:48 AM
> To: "hdfs-dev@hadoop.apache.org" <hd...@hadoop.apache.org>, "
> user@hadoop.apache.org" <us...@hadoop.apache.org>
>
> Subject: HDFS upgrade problem of fsImage
>
> Hi Dear,
>
> I have a small test cluster with hadoop-2.0x, and HA configuraded, but I
> want to upgrade to hadoop-2.2.
>
> I dont't want to stop cluster during upgrade, so my steps are:
>
> 1) on standby NN: hadoop-dameon.sh stop namenode
> 2) remove HA configuration in the conf
> 3) hadoop-daemon.sh start namenode -upgrade -clusterID test-cluster
>
> but Exception in the NN log, so how to upgrade and don't stop the whole
> cluster.
> Thanks.
>
>
> org.apache.hadoop.hdfs.server.common.InconsistentFSStateException:
> Directory /hdfs/name is in an inconsistent state: previous fs state should
> not exist during upgrade. Finalize or rollback first.
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.doUpgrade(FSImage.java:323)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:248)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:858)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:620)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:445)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:494)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:692)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:677)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1279)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1345)
>
Re: HDFS upgrade problem of fsImage
Posted by Azuryy Yu <az...@gmail.com>.
I insist hot upgrade on the test cluster because I want hot upgrade on the
prod cluster.
On 2013-11-21 7:23 PM, "Joshi, Rekha" <Re...@intuit.com> wrote:
> Hi Azurry,
>
> This error occurs when FSImage finds previous fs state, and as log states you would need to either finalizeUpgrade or rollback to proceed.Below -
>
> bin/hadoop dfsadmin –finalizeUpgrade
> hadoop dfsadmin –rollback
>
> On side note for a small test cluster on which one might suspect you are the only user, why wouldn't you insist on hot upgrade? :-)
>
> Thanks
> Rekha
>
> Some helpful guidelines for upgrade here -
>
> http://wiki.apache.org/hadoop/Hadoop_Upgrade
>
> https://twiki.grid.iu.edu/bin/view/Storage/HadoopUpgrade
>
> http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/Federation.html#Upgrading_from_older_release_to_0.23_and_configuring_federation
>
>
> From: Azuryy Yu <az...@gmail.com>
> Reply-To: "user@hadoop.apache.org" <us...@hadoop.apache.org>
> Date: Thursday 21 November 2013 9:48 AM
> To: "hdfs-dev@hadoop.apache.org" <hd...@hadoop.apache.org>, "
> user@hadoop.apache.org" <us...@hadoop.apache.org>
> Subject: HDFS upgrade problem of fsImage
>
> Hi Dear,
>
> I have a small test cluster with hadoop-2.0x, and HA configuraded, but I
> want to upgrade to hadoop-2.2.
>
> I dont't want to stop cluster during upgrade, so my steps are:
>
> 1) on standby NN: hadoop-dameon.sh stop namenode
> 2) remove HA configuration in the conf
> 3) hadoop-daemon.sh start namenode -upgrade -clusterID test-cluster
>
> but Exception in the NN log, so how to upgrade and don't stop the whole
> cluster.
> Thanks.
>
>
> org.apache.hadoop.hdfs.server.common.InconsistentFSStateException:
> Directory /hdfs/name is in an inconsistent state: previous fs state should
> not exist during upgrade. Finalize or rollback first.
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.doUpgrade(FSImage.java:323)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:248)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:858)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:620)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:445)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:494)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:692)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:677)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1279)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1345)
>
Re: HDFS upgrade problem of fsImage
Posted by Azuryy Yu <az...@gmail.com>.
I insist hot upgrade on the test cluster because I want hot upgrade on the
prod cluster.
On 2013-11-21 7:23 PM, "Joshi, Rekha" <Re...@intuit.com> wrote:
> Hi Azurry,
>
> This error occurs when FSImage finds previous fs state, and as log states you would need to either finalizeUpgrade or rollback to proceed.Below -
>
> bin/hadoop dfsadmin –finalizeUpgrade
> hadoop dfsadmin –rollback
>
> On side note for a small test cluster on which one might suspect you are the only user, why wouldn't you insist on hot upgrade? :-)
>
> Thanks
> Rekha
>
> Some helpful guidelines for upgrade here -
>
> http://wiki.apache.org/hadoop/Hadoop_Upgrade
>
> https://twiki.grid.iu.edu/bin/view/Storage/HadoopUpgrade
>
> http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/Federation.html#Upgrading_from_older_release_to_0.23_and_configuring_federation
>
>
> From: Azuryy Yu <az...@gmail.com>
> Reply-To: "user@hadoop.apache.org" <us...@hadoop.apache.org>
> Date: Thursday 21 November 2013 9:48 AM
> To: "hdfs-dev@hadoop.apache.org" <hd...@hadoop.apache.org>, "
> user@hadoop.apache.org" <us...@hadoop.apache.org>
> Subject: HDFS upgrade problem of fsImage
>
> Hi Dear,
>
> I have a small test cluster with hadoop-2.0x, and HA configuraded, but I
> want to upgrade to hadoop-2.2.
>
> I dont't want to stop cluster during upgrade, so my steps are:
>
> 1) on standby NN: hadoop-dameon.sh stop namenode
> 2) remove HA configuration in the conf
> 3) hadoop-daemon.sh start namenode -upgrade -clusterID test-cluster
>
> but Exception in the NN log, so how to upgrade and don't stop the whole
> cluster.
> Thanks.
>
>
> org.apache.hadoop.hdfs.server.common.InconsistentFSStateException:
> Directory /hdfs/name is in an inconsistent state: previous fs state should
> not exist during upgrade. Finalize or rollback first.
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.doUpgrade(FSImage.java:323)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:248)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:858)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:620)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:445)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:494)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:692)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:677)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1279)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1345)
>
Re: HDFS upgrade problem of fsImage
Posted by Azuryy Yu <az...@gmail.com>.
I insist hot upgrade on the test cluster because I want hot upgrade on the
prod cluster.
On 2013-11-21 7:23 PM, "Joshi, Rekha" <Re...@intuit.com> wrote:
> Hi Azurry,
>
> This error occurs when FSImage finds previous fs state, and as log states you would need to either finalizeUpgrade or rollback to proceed.Below -
>
> bin/hadoop dfsadmin –finalizeUpgrade
> hadoop dfsadmin –rollback
>
> On side note for a small test cluster on which one might suspect you are the only user, why wouldn't you insist on hot upgrade? :-)
>
> Thanks
> Rekha
>
> Some helpful guidelines for upgrade here -
>
> http://wiki.apache.org/hadoop/Hadoop_Upgrade
>
> https://twiki.grid.iu.edu/bin/view/Storage/HadoopUpgrade
>
> http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/Federation.html#Upgrading_from_older_release_to_0.23_and_configuring_federation
>
>
> From: Azuryy Yu <az...@gmail.com>
> Reply-To: "user@hadoop.apache.org" <us...@hadoop.apache.org>
> Date: Thursday 21 November 2013 9:48 AM
> To: "hdfs-dev@hadoop.apache.org" <hd...@hadoop.apache.org>, "
> user@hadoop.apache.org" <us...@hadoop.apache.org>
> Subject: HDFS upgrade problem of fsImage
>
> Hi Dear,
>
> I have a small test cluster with hadoop-2.0x, and HA configuraded, but I
> want to upgrade to hadoop-2.2.
>
> I dont't want to stop cluster during upgrade, so my steps are:
>
> 1) on standby NN: hadoop-dameon.sh stop namenode
> 2) remove HA configuration in the conf
> 3) hadoop-daemon.sh start namenode -upgrade -clusterID test-cluster
>
> but Exception in the NN log, so how to upgrade and don't stop the whole
> cluster.
> Thanks.
>
>
> org.apache.hadoop.hdfs.server.common.InconsistentFSStateException:
> Directory /hdfs/name is in an inconsistent state: previous fs state should
> not exist during upgrade. Finalize or rollback first.
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.doUpgrade(FSImage.java:323)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:248)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:858)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:620)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:445)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:494)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:692)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:677)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1279)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1345)
>
Re: HDFS upgrade problem of fsImage
Posted by Azuryy Yu <az...@gmail.com>.
Thanks Joshi,
I don't have upgrade before. the test cluser is the new cluster with
hadoop-2.0.3. so I don't need to 'bin/hadoop dfsadmin –finalizeUpgrade'
On Thu, Nov 21, 2013 at 7:22 PM, Joshi, Rekha <Re...@intuit.com>wrote:
> Hi Azurry,
>
> This error occurs when FSImage finds previous fs state, and as log states you would need to either finalizeUpgrade or rollback to proceed.Below -
>
> bin/hadoop dfsadmin –finalizeUpgrade
> hadoop dfsadmin –rollback
>
> On side note for a small test cluster on which one might suspect you are the only user, why wouldn't you insist on hot upgrade? :-)
>
> Thanks
> Rekha
>
> Some helpful guidelines for upgrade here -
>
> http://wiki.apache.org/hadoop/Hadoop_Upgrade
>
> https://twiki.grid.iu.edu/bin/view/Storage/HadoopUpgrade
>
> http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/Federation.html#Upgrading_from_older_release_to_0.23_and_configuring_federation
>
>
> From: Azuryy Yu <az...@gmail.com>
> Reply-To: "user@hadoop.apache.org" <us...@hadoop.apache.org>
> Date: Thursday 21 November 2013 9:48 AM
> To: "hdfs-dev@hadoop.apache.org" <hd...@hadoop.apache.org>, "
> user@hadoop.apache.org" <us...@hadoop.apache.org>
>
> Subject: HDFS upgrade problem of fsImage
>
> Hi Dear,
>
> I have a small test cluster with hadoop-2.0x, and HA configuraded, but I
> want to upgrade to hadoop-2.2.
>
> I dont't want to stop cluster during upgrade, so my steps are:
>
> 1) on standby NN: hadoop-dameon.sh stop namenode
> 2) remove HA configuration in the conf
> 3) hadoop-daemon.sh start namenode -upgrade -clusterID test-cluster
>
> but Exception in the NN log, so how to upgrade and don't stop the whole
> cluster.
> Thanks.
>
>
> org.apache.hadoop.hdfs.server.common.InconsistentFSStateException:
> Directory /hdfs/name is in an inconsistent state: previous fs state should
> not exist during upgrade. Finalize or rollback first.
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.doUpgrade(FSImage.java:323)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:248)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:858)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:620)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:445)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:494)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:692)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:677)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1279)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1345)
>
Re: HDFS upgrade problem of fsImage
Posted by Azuryy Yu <az...@gmail.com>.
I insist hot upgrade on the test cluster because I want hot upgrade on the
prod cluster.
On 2013-11-21 7:23 PM, "Joshi, Rekha" <Re...@intuit.com> wrote:
> Hi Azurry,
>
> This error occurs when FSImage finds previous fs state, and as log states you would need to either finalizeUpgrade or rollback to proceed.Below -
>
> bin/hadoop dfsadmin –finalizeUpgrade
> hadoop dfsadmin –rollback
>
> On side note for a small test cluster on which one might suspect you are the only user, why wouldn't you insist on hot upgrade? :-)
>
> Thanks
> Rekha
>
> Some helpful guidelines for upgrade here -
>
> http://wiki.apache.org/hadoop/Hadoop_Upgrade
>
> https://twiki.grid.iu.edu/bin/view/Storage/HadoopUpgrade
>
> http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/Federation.html#Upgrading_from_older_release_to_0.23_and_configuring_federation
>
>
> From: Azuryy Yu <az...@gmail.com>
> Reply-To: "user@hadoop.apache.org" <us...@hadoop.apache.org>
> Date: Thursday 21 November 2013 9:48 AM
> To: "hdfs-dev@hadoop.apache.org" <hd...@hadoop.apache.org>, "
> user@hadoop.apache.org" <us...@hadoop.apache.org>
> Subject: HDFS upgrade problem of fsImage
>
> Hi Dear,
>
> I have a small test cluster with hadoop-2.0x, and HA configuraded, but I
> want to upgrade to hadoop-2.2.
>
> I dont't want to stop cluster during upgrade, so my steps are:
>
> 1) on standby NN: hadoop-dameon.sh stop namenode
> 2) remove HA configuration in the conf
> 3) hadoop-daemon.sh start namenode -upgrade -clusterID test-cluster
>
> but Exception in the NN log, so how to upgrade and don't stop the whole
> cluster.
> Thanks.
>
>
> org.apache.hadoop.hdfs.server.common.InconsistentFSStateException:
> Directory /hdfs/name is in an inconsistent state: previous fs state should
> not exist during upgrade. Finalize or rollback first.
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.doUpgrade(FSImage.java:323)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:248)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:858)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:620)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:445)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:494)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:692)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:677)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1279)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1345)
>
Re: HDFS upgrade problem of fsImage
Posted by Azuryy Yu <az...@gmail.com>.
Thanks Joshi,
I don't have upgrade before. the test cluser is the new cluster with
hadoop-2.0.3. so I don't need to 'bin/hadoop dfsadmin –finalizeUpgrade'
On Thu, Nov 21, 2013 at 7:22 PM, Joshi, Rekha <Re...@intuit.com>wrote:
> Hi Azurry,
>
> This error occurs when FSImage finds previous fs state, and as log states you would need to either finalizeUpgrade or rollback to proceed.Below -
>
> bin/hadoop dfsadmin –finalizeUpgrade
> hadoop dfsadmin –rollback
>
> On side note for a small test cluster on which one might suspect you are the only user, why wouldn't you insist on hot upgrade? :-)
>
> Thanks
> Rekha
>
> Some helpful guidelines for upgrade here -
>
> http://wiki.apache.org/hadoop/Hadoop_Upgrade
>
> https://twiki.grid.iu.edu/bin/view/Storage/HadoopUpgrade
>
> http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/Federation.html#Upgrading_from_older_release_to_0.23_and_configuring_federation
>
>
> From: Azuryy Yu <az...@gmail.com>
> Reply-To: "user@hadoop.apache.org" <us...@hadoop.apache.org>
> Date: Thursday 21 November 2013 9:48 AM
> To: "hdfs-dev@hadoop.apache.org" <hd...@hadoop.apache.org>, "
> user@hadoop.apache.org" <us...@hadoop.apache.org>
>
> Subject: HDFS upgrade problem of fsImage
>
> Hi Dear,
>
> I have a small test cluster with hadoop-2.0x, and HA configuraded, but I
> want to upgrade to hadoop-2.2.
>
> I dont't want to stop cluster during upgrade, so my steps are:
>
> 1) on standby NN: hadoop-dameon.sh stop namenode
> 2) remove HA configuration in the conf
> 3) hadoop-daemon.sh start namenode -upgrade -clusterID test-cluster
>
> but Exception in the NN log, so how to upgrade and don't stop the whole
> cluster.
> Thanks.
>
>
> org.apache.hadoop.hdfs.server.common.InconsistentFSStateException:
> Directory /hdfs/name is in an inconsistent state: previous fs state should
> not exist during upgrade. Finalize or rollback first.
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.doUpgrade(FSImage.java:323)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:248)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:858)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:620)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:445)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:494)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:692)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:677)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1279)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1345)
>
Re: HDFS upgrade problem of fsImage
Posted by "Joshi, Rekha" <Re...@intuit.com>.
Hi Azurry,
This error occurs when FSImage finds previous fs state, and as log states you would need to either finalizeUpgrade or rollback to proceed.Below -
bin/hadoop dfsadmin –finalizeUpgrade
hadoop dfsadmin –rollback
On side note for a small test cluster on which one might suspect you are the only user, why wouldn't you insist on hot upgrade? :-)
Thanks
Rekha
Some helpful guidelines for upgrade here -
http://wiki.apache.org/hadoop/Hadoop_Upgrade
https://twiki.grid.iu.edu/bin/view/Storage/HadoopUpgrade
http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/Federation.html#Upgrading_from_older_release_to_0.23_and_configuring_federation
From: Azuryy Yu <az...@gmail.com>>
Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Date: Thursday 21 November 2013 9:48 AM
To: "hdfs-dev@hadoop.apache.org<ma...@hadoop.apache.org>" <hd...@hadoop.apache.org>>, "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Subject: HDFS upgrade problem of fsImage
Hi Dear,
I have a small test cluster with hadoop-2.0x, and HA configuraded, but I want to upgrade to hadoop-2.2.
I dont't want to stop cluster during upgrade, so my steps are:
1) on standby NN: hadoop-dameon.sh stop namenode
2) remove HA configuration in the conf
3) hadoop-daemon.sh start namenode -upgrade -clusterID test-cluster
but Exception in the NN log, so how to upgrade and don't stop the whole cluster.
Thanks.
org.apache.hadoop.hdfs.server.common.InconsistentFSStateException: Directory /hdfs/name is in an inconsistent state: previous fs state should not exist during upgrade. Finalize or rollback first.
at org.apache.hadoop.hdfs.server.namenode.FSImage.doUpgrade(FSImage.java:323)
at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:248)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:858)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:620)
at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:445)
at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:494)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:692)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:677)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1279)
at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1345)
Re: HDFS upgrade problem of fsImage
Posted by "Joshi, Rekha" <Re...@intuit.com>.
Hi Azurry,
This error occurs when FSImage finds previous fs state, and as log states you would need to either finalizeUpgrade or rollback to proceed.Below -
bin/hadoop dfsadmin –finalizeUpgrade
hadoop dfsadmin –rollback
On side note for a small test cluster on which one might suspect you are the only user, why wouldn't you insist on hot upgrade? :-)
Thanks
Rekha
Some helpful guidelines for upgrade here -
http://wiki.apache.org/hadoop/Hadoop_Upgrade
https://twiki.grid.iu.edu/bin/view/Storage/HadoopUpgrade
http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/Federation.html#Upgrading_from_older_release_to_0.23_and_configuring_federation
From: Azuryy Yu <az...@gmail.com>>
Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Date: Thursday 21 November 2013 9:48 AM
To: "hdfs-dev@hadoop.apache.org<ma...@hadoop.apache.org>" <hd...@hadoop.apache.org>>, "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Subject: HDFS upgrade problem of fsImage
Hi Dear,
I have a small test cluster with hadoop-2.0x, and HA configuraded, but I want to upgrade to hadoop-2.2.
I dont't want to stop cluster during upgrade, so my steps are:
1) on standby NN: hadoop-dameon.sh stop namenode
2) remove HA configuration in the conf
3) hadoop-daemon.sh start namenode -upgrade -clusterID test-cluster
but Exception in the NN log, so how to upgrade and don't stop the whole cluster.
Thanks.
org.apache.hadoop.hdfs.server.common.InconsistentFSStateException: Directory /hdfs/name is in an inconsistent state: previous fs state should not exist during upgrade. Finalize or rollback first.
at org.apache.hadoop.hdfs.server.namenode.FSImage.doUpgrade(FSImage.java:323)
at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:248)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:858)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:620)
at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:445)
at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:494)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:692)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:677)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1279)
at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1345)
Re: HDFS upgrade problem of fsImage
Posted by "Joshi, Rekha" <Re...@intuit.com>.
Hi Azurry,
This error occurs when FSImage finds previous fs state, and as log states you would need to either finalizeUpgrade or rollback to proceed.Below -
bin/hadoop dfsadmin –finalizeUpgrade
hadoop dfsadmin –rollback
On side note for a small test cluster on which one might suspect you are the only user, why wouldn't you insist on hot upgrade? :-)
Thanks
Rekha
Some helpful guidelines for upgrade here -
http://wiki.apache.org/hadoop/Hadoop_Upgrade
https://twiki.grid.iu.edu/bin/view/Storage/HadoopUpgrade
http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/Federation.html#Upgrading_from_older_release_to_0.23_and_configuring_federation
From: Azuryy Yu <az...@gmail.com>>
Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Date: Thursday 21 November 2013 9:48 AM
To: "hdfs-dev@hadoop.apache.org<ma...@hadoop.apache.org>" <hd...@hadoop.apache.org>>, "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Subject: HDFS upgrade problem of fsImage
Hi Dear,
I have a small test cluster with hadoop-2.0x, and HA configuraded, but I want to upgrade to hadoop-2.2.
I dont't want to stop cluster during upgrade, so my steps are:
1) on standby NN: hadoop-dameon.sh stop namenode
2) remove HA configuration in the conf
3) hadoop-daemon.sh start namenode -upgrade -clusterID test-cluster
but Exception in the NN log, so how to upgrade and don't stop the whole cluster.
Thanks.
org.apache.hadoop.hdfs.server.common.InconsistentFSStateException: Directory /hdfs/name is in an inconsistent state: previous fs state should not exist during upgrade. Finalize or rollback first.
at org.apache.hadoop.hdfs.server.namenode.FSImage.doUpgrade(FSImage.java:323)
at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:248)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:858)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:620)
at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:445)
at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:494)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:692)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:677)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1279)
at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1345)
Re: HDFS upgrade problem of fsImage
Posted by "Joshi, Rekha" <Re...@intuit.com>.
Hi Azurry,
This error occurs when FSImage finds previous fs state, and as log states you would need to either finalizeUpgrade or rollback to proceed.Below -
bin/hadoop dfsadmin –finalizeUpgrade
hadoop dfsadmin –rollback
On side note for a small test cluster on which one might suspect you are the only user, why wouldn't you insist on hot upgrade? :-)
Thanks
Rekha
Some helpful guidelines for upgrade here -
http://wiki.apache.org/hadoop/Hadoop_Upgrade
https://twiki.grid.iu.edu/bin/view/Storage/HadoopUpgrade
http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/Federation.html#Upgrading_from_older_release_to_0.23_and_configuring_federation
From: Azuryy Yu <az...@gmail.com>>
Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Date: Thursday 21 November 2013 9:48 AM
To: "hdfs-dev@hadoop.apache.org<ma...@hadoop.apache.org>" <hd...@hadoop.apache.org>>, "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Subject: HDFS upgrade problem of fsImage
Hi Dear,
I have a small test cluster with hadoop-2.0x, and HA configuraded, but I want to upgrade to hadoop-2.2.
I dont't want to stop cluster during upgrade, so my steps are:
1) on standby NN: hadoop-dameon.sh stop namenode
2) remove HA configuration in the conf
3) hadoop-daemon.sh start namenode -upgrade -clusterID test-cluster
but Exception in the NN log, so how to upgrade and don't stop the whole cluster.
Thanks.
org.apache.hadoop.hdfs.server.common.InconsistentFSStateException: Directory /hdfs/name is in an inconsistent state: previous fs state should not exist during upgrade. Finalize or rollback first.
at org.apache.hadoop.hdfs.server.namenode.FSImage.doUpgrade(FSImage.java:323)
at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:248)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:858)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:620)
at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:445)
at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:494)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:692)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:677)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1279)
at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1345)