You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by Bing Jiang <ji...@gmail.com> on 2013/07/26 08:28:21 UTC
【data migrate from hdfs0.20.* to hdfs-2.0.5(HA)】
hi,all
Have you tried to find out a way to make data transformation between two
hdfs cluster, which are the different version.
In our environment, one data in 0.20.*, and we want to migrate into
2.0.5(HA).
I try to use distcp, but it occurs to rpc version dismatch.
org.apache.hadoop.ipc.RemoteException: Server IPC version 8 cannot
> communicate with client version 4
>
Could anyone give me an efficient way to make it?
Thanks.
--
Bing Jiang
weibo: http://weibo.com/jiangbinglover
BLOG: www.binospace.com
BLOG: http://blog.sina.com.cn/jiangbinglover
Focus on distributed computing, HDFS/HBase
RE: 【data migrate from hdfs0.20.* to hdfs-2.0.5(HA)】
Posted by Vinayakumar B <vi...@huawei.com>.
Upgrade needs to be done in two phases.
1. Upgrade the data to 2.0.5 (Non-HA)
a. Just install the 2.0.5 Hadoop NonHA cluster by pointing to old Name dir and data dirs and start the cluster with ./start-dfs.sh -upgrade
b. Verify the upgraded data.
c. Once the upgrade is fine, then finalize the upgrade by calling ./hdfs dfsadmin –finalizeUpgrade
d. After finalizing stop the cluster
2. Migrating cluster to HA
a. Now install one more NameNode and zookeeper for HA and update the configurations in all other machines for HA
b. Call ./hdfs namenode –initializeSharedEdits to initialize the shared storage
c. Start the previous NameNode first along with ZKFC and let it become active.
d. Call ./hdfs namenode –bootstrapStandby from the new Namenode. This will copy the data from previous namenode.
e. After this start namenode and zkfc in new machine.
f. Start all datanodes.
Hope this solves your issue.
Regards,
Vinayakumar B
From: Binglin Chang [mailto:decstery@gmail.com]
Sent: 26 July 2013 12:53
To: user@hadoop.apache.org
Subject: Re: 【data migrate from hdfs0.20.* to hdfs-2.0.5(HA)】
have you looked at distcp over hftp?
http://hadoop.apache.org/docs/r1.0.4/distcp.html#cpver
On Fri, Jul 26, 2013 at 2:28 PM, Bing Jiang <ji...@gmail.com>> wrote:
hi,all
Have you tried to find out a way to make data transformation between two hdfs cluster, which are the different version.
In our environment, one data in 0.20.*, and we want to migrate into 2.0.5(HA).
I try to use distcp, but it occurs to rpc version dismatch.
org.apache.hadoop.ipc.RemoteException: Server IPC version 8 cannot communicate with client version 4
Could anyone give me an efficient way to make it?
Thanks.
--
Bing Jiang
weibo: http://weibo.com/jiangbinglover
BLOG: www.binospace.com<http://www.binospace.com>
BLOG: http://blog.sina.com.cn/jiangbinglover
Focus on distributed computing, HDFS/HBase
RE: 【data migrate from hdfs0.20.* to hdfs-2.0.5(HA)】
Posted by Vinayakumar B <vi...@huawei.com>.
Upgrade needs to be done in two phases.
1. Upgrade the data to 2.0.5 (Non-HA)
a. Just install the 2.0.5 Hadoop NonHA cluster by pointing to old Name dir and data dirs and start the cluster with ./start-dfs.sh -upgrade
b. Verify the upgraded data.
c. Once the upgrade is fine, then finalize the upgrade by calling ./hdfs dfsadmin –finalizeUpgrade
d. After finalizing stop the cluster
2. Migrating cluster to HA
a. Now install one more NameNode and zookeeper for HA and update the configurations in all other machines for HA
b. Call ./hdfs namenode –initializeSharedEdits to initialize the shared storage
c. Start the previous NameNode first along with ZKFC and let it become active.
d. Call ./hdfs namenode –bootstrapStandby from the new Namenode. This will copy the data from previous namenode.
e. After this start namenode and zkfc in new machine.
f. Start all datanodes.
Hope this solves your issue.
Regards,
Vinayakumar B
From: Binglin Chang [mailto:decstery@gmail.com]
Sent: 26 July 2013 12:53
To: user@hadoop.apache.org
Subject: Re: 【data migrate from hdfs0.20.* to hdfs-2.0.5(HA)】
have you looked at distcp over hftp?
http://hadoop.apache.org/docs/r1.0.4/distcp.html#cpver
On Fri, Jul 26, 2013 at 2:28 PM, Bing Jiang <ji...@gmail.com>> wrote:
hi,all
Have you tried to find out a way to make data transformation between two hdfs cluster, which are the different version.
In our environment, one data in 0.20.*, and we want to migrate into 2.0.5(HA).
I try to use distcp, but it occurs to rpc version dismatch.
org.apache.hadoop.ipc.RemoteException: Server IPC version 8 cannot communicate with client version 4
Could anyone give me an efficient way to make it?
Thanks.
--
Bing Jiang
weibo: http://weibo.com/jiangbinglover
BLOG: www.binospace.com<http://www.binospace.com>
BLOG: http://blog.sina.com.cn/jiangbinglover
Focus on distributed computing, HDFS/HBase
RE: 【data migrate from hdfs0.20.* to hdfs-2.0.5(HA)】
Posted by Vinayakumar B <vi...@huawei.com>.
Upgrade needs to be done in two phases.
1. Upgrade the data to 2.0.5 (Non-HA)
a. Just install the 2.0.5 Hadoop NonHA cluster by pointing to old Name dir and data dirs and start the cluster with ./start-dfs.sh -upgrade
b. Verify the upgraded data.
c. Once the upgrade is fine, then finalize the upgrade by calling ./hdfs dfsadmin –finalizeUpgrade
d. After finalizing stop the cluster
2. Migrating cluster to HA
a. Now install one more NameNode and zookeeper for HA and update the configurations in all other machines for HA
b. Call ./hdfs namenode –initializeSharedEdits to initialize the shared storage
c. Start the previous NameNode first along with ZKFC and let it become active.
d. Call ./hdfs namenode –bootstrapStandby from the new Namenode. This will copy the data from previous namenode.
e. After this start namenode and zkfc in new machine.
f. Start all datanodes.
Hope this solves your issue.
Regards,
Vinayakumar B
From: Binglin Chang [mailto:decstery@gmail.com]
Sent: 26 July 2013 12:53
To: user@hadoop.apache.org
Subject: Re: 【data migrate from hdfs0.20.* to hdfs-2.0.5(HA)】
have you looked at distcp over hftp?
http://hadoop.apache.org/docs/r1.0.4/distcp.html#cpver
On Fri, Jul 26, 2013 at 2:28 PM, Bing Jiang <ji...@gmail.com>> wrote:
hi,all
Have you tried to find out a way to make data transformation between two hdfs cluster, which are the different version.
In our environment, one data in 0.20.*, and we want to migrate into 2.0.5(HA).
I try to use distcp, but it occurs to rpc version dismatch.
org.apache.hadoop.ipc.RemoteException: Server IPC version 8 cannot communicate with client version 4
Could anyone give me an efficient way to make it?
Thanks.
--
Bing Jiang
weibo: http://weibo.com/jiangbinglover
BLOG: www.binospace.com<http://www.binospace.com>
BLOG: http://blog.sina.com.cn/jiangbinglover
Focus on distributed computing, HDFS/HBase
RE: 【data migrate from hdfs0.20.* to hdfs-2.0.5(HA)】
Posted by Vinayakumar B <vi...@huawei.com>.
Upgrade needs to be done in two phases.
1. Upgrade the data to 2.0.5 (Non-HA)
a. Just install the 2.0.5 Hadoop NonHA cluster by pointing to old Name dir and data dirs and start the cluster with ./start-dfs.sh -upgrade
b. Verify the upgraded data.
c. Once the upgrade is fine, then finalize the upgrade by calling ./hdfs dfsadmin –finalizeUpgrade
d. After finalizing stop the cluster
2. Migrating cluster to HA
a. Now install one more NameNode and zookeeper for HA and update the configurations in all other machines for HA
b. Call ./hdfs namenode –initializeSharedEdits to initialize the shared storage
c. Start the previous NameNode first along with ZKFC and let it become active.
d. Call ./hdfs namenode –bootstrapStandby from the new Namenode. This will copy the data from previous namenode.
e. After this start namenode and zkfc in new machine.
f. Start all datanodes.
Hope this solves your issue.
Regards,
Vinayakumar B
From: Binglin Chang [mailto:decstery@gmail.com]
Sent: 26 July 2013 12:53
To: user@hadoop.apache.org
Subject: Re: 【data migrate from hdfs0.20.* to hdfs-2.0.5(HA)】
have you looked at distcp over hftp?
http://hadoop.apache.org/docs/r1.0.4/distcp.html#cpver
On Fri, Jul 26, 2013 at 2:28 PM, Bing Jiang <ji...@gmail.com>> wrote:
hi,all
Have you tried to find out a way to make data transformation between two hdfs cluster, which are the different version.
In our environment, one data in 0.20.*, and we want to migrate into 2.0.5(HA).
I try to use distcp, but it occurs to rpc version dismatch.
org.apache.hadoop.ipc.RemoteException: Server IPC version 8 cannot communicate with client version 4
Could anyone give me an efficient way to make it?
Thanks.
--
Bing Jiang
weibo: http://weibo.com/jiangbinglover
BLOG: www.binospace.com<http://www.binospace.com>
BLOG: http://blog.sina.com.cn/jiangbinglover
Focus on distributed computing, HDFS/HBase
Re: 【data migrate from hdfs0.20.* to hdfs-2.0.5(HA)】
Posted by Binglin Chang <de...@gmail.com>.
have you looked at distcp over hftp?
http://hadoop.apache.org/docs/r1.0.4/distcp.html#cpver
On Fri, Jul 26, 2013 at 2:28 PM, Bing Jiang <ji...@gmail.com>wrote:
>
> hi,all
>
> Have you tried to find out a way to make data transformation between two
> hdfs cluster, which are the different version.
>
> In our environment, one data in 0.20.*, and we want to migrate into
> 2.0.5(HA).
>
> I try to use distcp, but it occurs to rpc version dismatch.
>
> org.apache.hadoop.ipc.RemoteException: Server IPC version 8 cannot
>> communicate with client version 4
>>
>
> Could anyone give me an efficient way to make it?
>
> Thanks.
>
> --
> Bing Jiang
> weibo: http://weibo.com/jiangbinglover
> BLOG: www.binospace.com
> BLOG: http://blog.sina.com.cn/jiangbinglover
> Focus on distributed computing, HDFS/HBase
>
Re: 【data migrate from hdfs0.20.* to hdfs-2.0.5(HA)】
Posted by Binglin Chang <de...@gmail.com>.
have you looked at distcp over hftp?
http://hadoop.apache.org/docs/r1.0.4/distcp.html#cpver
On Fri, Jul 26, 2013 at 2:28 PM, Bing Jiang <ji...@gmail.com>wrote:
>
> hi,all
>
> Have you tried to find out a way to make data transformation between two
> hdfs cluster, which are the different version.
>
> In our environment, one data in 0.20.*, and we want to migrate into
> 2.0.5(HA).
>
> I try to use distcp, but it occurs to rpc version dismatch.
>
> org.apache.hadoop.ipc.RemoteException: Server IPC version 8 cannot
>> communicate with client version 4
>>
>
> Could anyone give me an efficient way to make it?
>
> Thanks.
>
> --
> Bing Jiang
> weibo: http://weibo.com/jiangbinglover
> BLOG: www.binospace.com
> BLOG: http://blog.sina.com.cn/jiangbinglover
> Focus on distributed computing, HDFS/HBase
>
Re: 【data migrate from hdfs0.20.* to hdfs-2.0.5(HA)】
Posted by Binglin Chang <de...@gmail.com>.
have you looked at distcp over hftp?
http://hadoop.apache.org/docs/r1.0.4/distcp.html#cpver
On Fri, Jul 26, 2013 at 2:28 PM, Bing Jiang <ji...@gmail.com>wrote:
>
> hi,all
>
> Have you tried to find out a way to make data transformation between two
> hdfs cluster, which are the different version.
>
> In our environment, one data in 0.20.*, and we want to migrate into
> 2.0.5(HA).
>
> I try to use distcp, but it occurs to rpc version dismatch.
>
> org.apache.hadoop.ipc.RemoteException: Server IPC version 8 cannot
>> communicate with client version 4
>>
>
> Could anyone give me an efficient way to make it?
>
> Thanks.
>
> --
> Bing Jiang
> weibo: http://weibo.com/jiangbinglover
> BLOG: www.binospace.com
> BLOG: http://blog.sina.com.cn/jiangbinglover
> Focus on distributed computing, HDFS/HBase
>
Re: 【data migrate from hdfs0.20.* to hdfs-2.0.5(HA)】
Posted by Binglin Chang <de...@gmail.com>.
have you looked at distcp over hftp?
http://hadoop.apache.org/docs/r1.0.4/distcp.html#cpver
On Fri, Jul 26, 2013 at 2:28 PM, Bing Jiang <ji...@gmail.com>wrote:
>
> hi,all
>
> Have you tried to find out a way to make data transformation between two
> hdfs cluster, which are the different version.
>
> In our environment, one data in 0.20.*, and we want to migrate into
> 2.0.5(HA).
>
> I try to use distcp, but it occurs to rpc version dismatch.
>
> org.apache.hadoop.ipc.RemoteException: Server IPC version 8 cannot
>> communicate with client version 4
>>
>
> Could anyone give me an efficient way to make it?
>
> Thanks.
>
> --
> Bing Jiang
> weibo: http://weibo.com/jiangbinglover
> BLOG: www.binospace.com
> BLOG: http://blog.sina.com.cn/jiangbinglover
> Focus on distributed computing, HDFS/HBase
>