You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Yongjun Zhang (JIRA)" <ji...@apache.org> on 2017/02/23 03:14:44 UTC

[jira] [Comment Edited] (HDFS-9868) Add ability for DistCp to run between 2 clusters

    [ https://issues.apache.org/jira/browse/HDFS-9868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15879780#comment-15879780 ] 

Yongjun Zhang edited comment on HDFS-9868 at 2/23/17 3:14 AM:
--------------------------------------------------------------

Hi [~iceberg565] and [~xiaochen],

Thanks for the work here. I have not got chance to look at the patch details yet, but one high level comment, instead of using a command line switch to indicate source or target conf, I think we could have a conf map that maps from a qualified cluster name to conf path. There are two reasons for that:
# distcp can have multiple sources, and one target. Each source path could be even a different cluster. 
# distcp could be run at one of the source clusters or target cluster

Specifically, we can do {{distcp -confMap <pathToConfMapFile> ...}}, inside the confMapFile, we can have one entry per line:
{code}
hdfs://x.y.z:8020 <path1>
hdfs//p.q.r:8020 <path2>
webhdfs://a.b.c:50070 <path3>
......
{code}

For cluster not specified in this map, use the default.

This way, we don't need to worry about whether the conf is for the source, or for the target. Does this sound good to you?

[~qwertymaniac]: what do you think Harsh?

Thanks.






was (Author: yzhangal):
Hi [~iceberg565] and [~xiaochen],

Thanks for the work here. I have not got chance to look at the patch details yet, but one high level comment, instead of using a command line switch to indicate source or target conf, I think we could have a conf map that maps from a qualified cluster name to conf path. There are two reasons for that:
# distcp can have multiple sources, and one target. Each source path could be even a different cluster. 
# distcp could be run at one of the source clusters or target cluster

Specifically, we can do {{distcp -confMap <pathToConfMapFile> ...}}, inside the confMapFile, we can have one entry per line:
{code}
hdfs://x.y.z:8020 <path1>
hdfs//p.q.r:8020 <path2>
webhdfs://a.b.c:50070 <path3>
......
{code}

For cluster not specified in this map, use the default.

This way, we don't need to worry about whether the conf is for the source, or for the target. Does this sound good to you?

Thanks.





> Add ability for DistCp to run between 2 clusters
> ------------------------------------------------
>
>                 Key: HDFS-9868
>                 URL: https://issues.apache.org/jira/browse/HDFS-9868
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: distcp
>    Affects Versions: 2.7.1
>            Reporter: NING DING
>            Assignee: NING DING
>         Attachments: HDFS-9868.05.patch, HDFS-9868.06.patch, HDFS-9868.07.patch, HDFS-9868.1.patch, HDFS-9868.2.patch, HDFS-9868.3.patch, HDFS-9868.4.patch
>
>
> Normally the HDFS cluster is HA enabled. It could take a long time when coping huge data by distp. If the source cluster changes active namenode, the distp will run failed. This patch supports the DistCp can read source cluster files in HA access mode. A source cluster configuration file needs to be specified (via the -sourceClusterConf option).
>   The following is an example of the contents of a source cluster configuration
>   file:
> {code:xml}
>     <configuration>
>       <property>
> 		<name>fs.defaultFS</name>
> 		<value>hdfs://mycluster</value>
> 	  </property>
> 	  <property>
> 		<name>dfs.nameservices</name>
> 		<value>mycluster</value>
> 	  </property>
> 	  <property>
> 		<name>dfs.ha.namenodes.mycluster</name>
> 		<value>nn1,nn2</value>
> 	  </property>
> 	  <property>
> 		<name>dfs.namenode.rpc-address.mycluster.nn1</name>
> 		<value>host1:9000</value>
> 	  </property>
> 	  <property>
> 		<name>dfs.namenode.rpc-address.mycluster.nn2</name>
> 		<value>host2:9000</value>
> 	  </property>
> 	  <property>
> 		<name>dfs.namenode.http-address.mycluster.nn1</name>
> 		<value>host1:50070</value>
> 	  </property>
> 	  <property>
> 		<name>dfs.namenode.http-address.mycluster.nn2</name>
> 		<value>host2:50070</value>
> 	  </property>
> 	  <property>
> 		<name>dfs.client.failover.proxy.provider.mycluster</name>
> 		<value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
> 	  </property>
> 	</configuration>
> {code}
>   The invocation of DistCp is as below:
> {code}
>     bash$ hadoop distcp -sourceClusterConf sourceCluster.xml /foo/bar hdfs://nn2:8020/bar/foo
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org