You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-dev@hadoop.apache.org by Loudongfeng <lo...@huawei.com> on 2013/02/25 14:08:58 UTC

Distcp doesn't work when destination cluster not configured for ResourceManager (Hadoop 2.0)

For example,when I type this command:
bin/hadoop  distcp  hdfs://hacluster:8020/mr-history/done/2013/02/18/000000/  hdfs://hacluster-slave:8020/mr-history/done/2013/02/18/000000/

hdfs://hacluster-slave is the destination HDFS cluster in HA mode. when it's not configured for resourcemanager, distcp will fail with the following error:
13/02/25 12:00:27 ERROR tools.DistCp: Exception encountered
java.io.IOException: Failed to run job : Unable to map logical nameservice URI 'hdfs://hacluster-slave' to a NameNode. Local configuration does not have a failover proxy provider configured.
        at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:301)
        at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:384)
        at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1215)
        at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1212)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
        at org.apache.hadoop.mapreduce.Job.submit(Job.java:1212)
        at org.apache.hadoop.tools.DistCp.execute(DistCp.java:148)
        at org.apache.hadoop.tools.DistCp.run(DistCp.java:113)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
        at org.apache.hadoop.tools.DistCp.main(DistCp.java:368)


And RM's log has this:
2013-02-25 12:00:26,857 ERROR org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1235) PriviledgedActionException as:mapred/hadoop@HADOOP.COM (auth:KERBEROS) cause:java.io.IOException: Unable to map logical nameservice URI 'hdfs://hacluster-slave' to a NameNode. Local configuration does not have a failover proxy provider configured.
2013-02-25 12:00:26,857 INFO  org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.submitApplication(RMAppManager.java:290) RMAppManager submit application exception
java.io.IOException: Unable to map logical nameservice URI 'hdfs://hacluster-slave' to a NameNode. Local configuration does not have a failover proxy provider configured.
        at org.apache.hadoop.hdfs.DFSClient$Renewer.getNNProxy(DFSClient.java:875)
        at org.apache.hadoop.hdfs.DFSClient$Renewer.renew(DFSClient.java:840)
        at org.apache.hadoop.security.token.Token.renew(Token.java:372)
        at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$1.run(DelegationTokenRenewer.java:368)
        at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$1.run(DelegationTokenRenewer.java:1)
        ...  at org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.submitApplication(RMAppManager.java:280)
        ...

After I put remote cluster hacluster-slave's configraion into RM's hdfs-site.xml file and restart RM, then distcp works.

My question is ,shall RM support reading configuration from client side when doing delegation renew? Then no need to configure remote cluster configurations and restart RM.


Thank you!