You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-dev@hadoop.apache.org by Loudongfeng <lo...@huawei.com> on 2013/02/25 14:08:58 UTC
Distcp doesn't work when destination cluster not configured for
ResourceManager (Hadoop 2.0)
For example,when I type this command:
bin/hadoop distcp hdfs://hacluster:8020/mr-history/done/2013/02/18/000000/ hdfs://hacluster-slave:8020/mr-history/done/2013/02/18/000000/
hdfs://hacluster-slave is the destination HDFS cluster in HA mode. when it's not configured for resourcemanager, distcp will fail with the following error:
13/02/25 12:00:27 ERROR tools.DistCp: Exception encountered
java.io.IOException: Failed to run job : Unable to map logical nameservice URI 'hdfs://hacluster-slave' to a NameNode. Local configuration does not have a failover proxy provider configured.
at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:301)
at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:384)
at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1215)
at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1212)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1212)
at org.apache.hadoop.tools.DistCp.execute(DistCp.java:148)
at org.apache.hadoop.tools.DistCp.run(DistCp.java:113)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.tools.DistCp.main(DistCp.java:368)
And RM's log has this:
2013-02-25 12:00:26,857 ERROR org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1235) PriviledgedActionException as:mapred/hadoop@HADOOP.COM (auth:KERBEROS) cause:java.io.IOException: Unable to map logical nameservice URI 'hdfs://hacluster-slave' to a NameNode. Local configuration does not have a failover proxy provider configured.
2013-02-25 12:00:26,857 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.submitApplication(RMAppManager.java:290) RMAppManager submit application exception
java.io.IOException: Unable to map logical nameservice URI 'hdfs://hacluster-slave' to a NameNode. Local configuration does not have a failover proxy provider configured.
at org.apache.hadoop.hdfs.DFSClient$Renewer.getNNProxy(DFSClient.java:875)
at org.apache.hadoop.hdfs.DFSClient$Renewer.renew(DFSClient.java:840)
at org.apache.hadoop.security.token.Token.renew(Token.java:372)
at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$1.run(DelegationTokenRenewer.java:368)
at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$1.run(DelegationTokenRenewer.java:1)
... at org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.submitApplication(RMAppManager.java:280)
...
After I put remote cluster hacluster-slave's configraion into RM's hdfs-site.xml file and restart RM, then distcp works.
My question is ,shall RM support reading configuration from client side when doing delegation renew? Then no need to configure remote cluster configurations and restart RM.
Thank you!