You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by Dinesh Kumar Prabakaran <di...@gmail.com> on 2016/10/03 05:13:25 UTC
Issue in Rollback (after rolling upgrade) from hadoop 2.7.2 to 2.5.2
I tried to do rolling upgrade from hadoop 2.5.2 to hadoop 2.7.2.
As per reference
<http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HdfsRollingUpgrade.html#Rollback>
one can rollback to previous release provided the finalize step is not
done.
I upgraded the setup, did not finalize the upgrade and tried to rollback
HDFS to 2.5.2 version. Facing following exception after starting *Name node
1* as Active with the “*-rollingUpgrade rollback*” option.
*org.apache.hadoop.hdfs.server.common.IncorrectVersionException: Unexpected
version of storage directory ..\Metadata\data\dfs\namenode. Reported: -63.
Expecting = -57.*
Faced similar exception when I started *Data node* based on Downgrade
<http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HdfsRollingUpgrade.html#Downgrade>
approach
also.
*org.apache.hadoop.hdfs.server.common.IncorrectVersionException: Unexpected
version of storage directory ..\Metadata\data\dfs\datanode. Reported: -56.
Expecting = -55.*
There is already a task regarding this but its in Open status.
https://issues.apache.org/jira/browse/HDFS-9096
Is this known issue? Please let me know are there any other steps to
perform rollback (from rolling upgrade) or is it not supported to rollback
to Hadoop 2.5.2.
Thanks,
Dinesh Kumar P
Re: Issue in Rollback (after rolling upgrade) from hadoop 2.7.2 to
2.5.2
Posted by Kihwal Lee <ki...@yahoo-inc.com.INVALID>.
It looks like the NN is having trouble after reading in the VERSION file, which contains the new layout version. We can continue the discussion in the jira.
Kihwal
From: Ravi Prakash <ra...@gmail.com>
To: Dinesh Kumar Prabakaran <di...@gmail.com>
Cc: user <us...@hadoop.apache.org>; hdfs-dev <hd...@hadoop.apache.org>
Sent: Thursday, October 13, 2016 3:29 PM
Subject: Re: Issue in Rollback (after rolling upgrade) from hadoop 2.7.2 to 2.5.2
Hi Dinesh!
This is obviously a very hazardous situation you are in (if your data is
important), so I'd suggest moving carefully. Make as many backups of as
many things you can.
The usual mechanism that Hadoop uses when upgrading is to rename
directories of the old format and keep them around until the admin
finalizes the upgrade. Here is the relevant method :
https://github.com/apache/hadoop/blob/branch-2.7.2/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java#L388
. You will probably have to dig into code and see what operations were
performed, where the failure occurred, and figure out how best to fix it.
In fact for your particular upgrade there were quite substantial changes
not just in the Namenode formats, but also the layouts on the datanodes
(which you may have to work on once you recover your namenode) .
https://issues.apache.org/jira/browse/HDFS-6482 . I'm guessing operations
will take a long time and may not work. Why do you need to rollback? We are
on 2.7.2 and its working fine for us.
HTH
Ravi
On Wed, Oct 12, 2016 at 11:01 PM, Dinesh Kumar Prabakaran <
dineshpvino@gmail.com> wrote:
> Hi Guys,
>
> Did rolling upgrade from hadoop 2.5.2 to hadoop 2.7.2 and *did not
> finalize* the upgrade. Now I wished to rollback to 2.5.2 version based on
> reference
> <http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HdfsRollingUpgrade.html#Rollback>
> .
>
> Starting Name node 1 as active with *-rollingUpgrade rollback*, it gets
> shuts down with the following exception,
>
> *org.apache.hadoop.hdfs.server.common.IncorrectVersionException:
> Unexpected version of storage directory ..\Metadata\data\dfs\namenode.
> Reported: -63. Expecting = -57.*
>
> There is already a task regarding this but the status is *Open*.
> https://issues.apache.org/jira/browse/HDFS-9096
>
> Please let me know are there any *work around *to rollback HDFS from
> rolling upgrade without any issues.
>
> Thanks,
> Dinesh Kumar P
>
>
>
>
Re: Issue in Rollback (after rolling upgrade) from hadoop 2.7.2 to
2.5.2
Posted by Kihwal Lee <ki...@yahoo-inc.com.INVALID>.
It looks like the NN is having trouble after reading in the VERSION file, which contains the new layout version. We can continue the discussion in the jira.
Kihwal
From: Ravi Prakash <ra...@gmail.com>
To: Dinesh Kumar Prabakaran <di...@gmail.com>
Cc: user <us...@hadoop.apache.org>; hdfs-dev <hd...@hadoop.apache.org>
Sent: Thursday, October 13, 2016 3:29 PM
Subject: Re: Issue in Rollback (after rolling upgrade) from hadoop 2.7.2 to 2.5.2
Hi Dinesh!
This is obviously a very hazardous situation you are in (if your data is
important), so I'd suggest moving carefully. Make as many backups of as
many things you can.
The usual mechanism that Hadoop uses when upgrading is to rename
directories of the old format and keep them around until the admin
finalizes the upgrade. Here is the relevant method :
https://github.com/apache/hadoop/blob/branch-2.7.2/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java#L388
. You will probably have to dig into code and see what operations were
performed, where the failure occurred, and figure out how best to fix it.
In fact for your particular upgrade there were quite substantial changes
not just in the Namenode formats, but also the layouts on the datanodes
(which you may have to work on once you recover your namenode) .
https://issues.apache.org/jira/browse/HDFS-6482 . I'm guessing operations
will take a long time and may not work. Why do you need to rollback? We are
on 2.7.2 and its working fine for us.
HTH
Ravi
On Wed, Oct 12, 2016 at 11:01 PM, Dinesh Kumar Prabakaran <
dineshpvino@gmail.com> wrote:
> Hi Guys,
>
> Did rolling upgrade from hadoop 2.5.2 to hadoop 2.7.2 and *did not
> finalize* the upgrade. Now I wished to rollback to 2.5.2 version based on
> reference
> <http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HdfsRollingUpgrade.html#Rollback>
> .
>
> Starting Name node 1 as active with *-rollingUpgrade rollback*, it gets
> shuts down with the following exception,
>
> *org.apache.hadoop.hdfs.server.common.IncorrectVersionException:
> Unexpected version of storage directory ..\Metadata\data\dfs\namenode.
> Reported: -63. Expecting = -57.*
>
> There is already a task regarding this but the status is *Open*.
> https://issues.apache.org/jira/browse/HDFS-9096
>
> Please let me know are there any *work around *to rollback HDFS from
> rolling upgrade without any issues.
>
> Thanks,
> Dinesh Kumar P
>
>
>
>
Re: Issue in Rollback (after rolling upgrade) from hadoop 2.7.2 to 2.5.2
Posted by Ravi Prakash <ra...@gmail.com>.
Hi Dinesh!
This is obviously a very hazardous situation you are in (if your data is
important), so I'd suggest moving carefully. Make as many backups of as
many things you can.
The usual mechanism that Hadoop uses when upgrading is to rename
directories of the old format and keep them around until the admin
finalizes the upgrade. Here is the relevant method :
https://github.com/apache/hadoop/blob/branch-2.7.2/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java#L388
. You will probably have to dig into code and see what operations were
performed, where the failure occurred, and figure out how best to fix it.
In fact for your particular upgrade there were quite substantial changes
not just in the Namenode formats, but also the layouts on the datanodes
(which you may have to work on once you recover your namenode) .
https://issues.apache.org/jira/browse/HDFS-6482 . I'm guessing operations
will take a long time and may not work. Why do you need to rollback? We are
on 2.7.2 and its working fine for us.
HTH
Ravi
On Wed, Oct 12, 2016 at 11:01 PM, Dinesh Kumar Prabakaran <
dineshpvino@gmail.com> wrote:
> Hi Guys,
>
> Did rolling upgrade from hadoop 2.5.2 to hadoop 2.7.2 and *did not
> finalize* the upgrade. Now I wished to rollback to 2.5.2 version based on
> reference
> <http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HdfsRollingUpgrade.html#Rollback>
> .
>
> Starting Name node 1 as active with *-rollingUpgrade rollback*, it gets
> shuts down with the following exception,
>
> *org.apache.hadoop.hdfs.server.common.IncorrectVersionException:
> Unexpected version of storage directory ..\Metadata\data\dfs\namenode.
> Reported: -63. Expecting = -57.*
>
> There is already a task regarding this but the status is *Open*.
> https://issues.apache.org/jira/browse/HDFS-9096
>
> Please let me know are there any *work around *to rollback HDFS from
> rolling upgrade without any issues.
>
> Thanks,
> Dinesh Kumar P
>
>
>
>
Re: Issue in Rollback (after rolling upgrade) from hadoop 2.7.2 to 2.5.2
Posted by Ravi Prakash <ra...@gmail.com>.
Hi Dinesh!
This is obviously a very hazardous situation you are in (if your data is
important), so I'd suggest moving carefully. Make as many backups of as
many things you can.
The usual mechanism that Hadoop uses when upgrading is to rename
directories of the old format and keep them around until the admin
finalizes the upgrade. Here is the relevant method :
https://github.com/apache/hadoop/blob/branch-2.7.2/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java#L388
. You will probably have to dig into code and see what operations were
performed, where the failure occurred, and figure out how best to fix it.
In fact for your particular upgrade there were quite substantial changes
not just in the Namenode formats, but also the layouts on the datanodes
(which you may have to work on once you recover your namenode) .
https://issues.apache.org/jira/browse/HDFS-6482 . I'm guessing operations
will take a long time and may not work. Why do you need to rollback? We are
on 2.7.2 and its working fine for us.
HTH
Ravi
On Wed, Oct 12, 2016 at 11:01 PM, Dinesh Kumar Prabakaran <
dineshpvino@gmail.com> wrote:
> Hi Guys,
>
> Did rolling upgrade from hadoop 2.5.2 to hadoop 2.7.2 and *did not
> finalize* the upgrade. Now I wished to rollback to 2.5.2 version based on
> reference
> <http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HdfsRollingUpgrade.html#Rollback>
> .
>
> Starting Name node 1 as active with *-rollingUpgrade rollback*, it gets
> shuts down with the following exception,
>
> *org.apache.hadoop.hdfs.server.common.IncorrectVersionException:
> Unexpected version of storage directory ..\Metadata\data\dfs\namenode.
> Reported: -63. Expecting = -57.*
>
> There is already a task regarding this but the status is *Open*.
> https://issues.apache.org/jira/browse/HDFS-9096
>
> Please let me know are there any *work around *to rollback HDFS from
> rolling upgrade without any issues.
>
> Thanks,
> Dinesh Kumar P
>
>
>
>
Fwd: Issue in Rollback (after rolling upgrade) from hadoop 2.7.2 to 2.5.2
Posted by Dinesh Kumar Prabakaran <di...@gmail.com>.
Hi Guys,
Did rolling upgrade from hadoop 2.5.2 to hadoop 2.7.2 and *did not finalize*
the upgrade. Now I wished to rollback to 2.5.2 version based on reference
<http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HdfsRollingUpgrade.html#Rollback>
.
Starting Name node 1 as active with *-rollingUpgrade rollback*, it gets
shuts down with the following exception,
*org.apache.hadoop.hdfs.server.common.IncorrectVersionException: Unexpected
version of storage directory ..\Metadata\data\dfs\namenode. Reported: -63.
Expecting = -57.*
There is already a task regarding this but the status is *Open*.
https://issues.apache.org/jira/browse/HDFS-9096
Please let me know are there any *work around *to rollback HDFS from
rolling upgrade without any issues.
Thanks,
Dinesh Kumar P