You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Benoy Antony (JIRA)" <ji...@apache.org> on 2013/10/17 19:51:42 UTC
[jira] [Commented] (MAPREDUCE-5589) MapReduce Job setup error
leaves no useful info to users (when LinuxTaskController is used)
[ https://issues.apache.org/jira/browse/MAPREDUCE-5589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13798169#comment-13798169 ]
Benoy Antony commented on MAPREDUCE-5589:
-----------------------------------------
Proper error output by JobLocalizer is displayed when the patch is applied.
Error initializing attempt_201310140952_0001_m_000002_0:
java.io.IOException: Job initialization failed (255) with output: Reading task controller config from /etc/hadoop/taskcontroller.cfg
main : command provided 0
main : user is bantony
Good mapred-local-dirs are /hadoop12/scratch,/hadoop04/scratch,/hadoop09/scratch,/hadoop05/scratch,/hadoop01/scratch,/hadoop11/scratch,/hadoop10/scratch,/hadoop08/scratch,/hadoop06/scratch,/hadoop07/scratch,/hadoop02/scratch
java.io.IOException: Call to localhost/127.0.0.1:60487 failed on local exception: java.io.IOException: javax.security.sasl.SaslException: DIGEST-MD5: No common protection layer between client and server
at org.apache.hadoop.ipc.Client.wrapException(Client.java:1155)
at org.apache.hadoop.ipc.Client.call(Client.java:1123)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)
at $Proxy5.getProtocolVersion(Unknown Source)
at org.apache.hadoop.ipc.RPC.checkVersion(RPC.java:422)
at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:414)
at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:392)
at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:374)
at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:444)
at org.apache.hadoop.mapred.JobLocalizer$2.run(JobLocalizer.java:529)
at org.apache.hadoop.mapred.JobLocalizer$2.run(JobLocalizer.java:527)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
at org.apache.hadoop.mapred.JobLocalizer.main(JobLocalizer.java:526)
Caused by: java.io.IOException: javax.security.sasl.SaslException: DIGEST-MD5: No common protection layer between client and server
at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:568)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
at org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:513)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:616)
at org.apache.hadoop.ipc.Client$Connection.access$2100(Client.java:203)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1254)
at org.apache.hadoop.ipc.Client.call(Client.java:1098)
... 13 more
Caused by: javax.security.sasl.SaslException: DIGEST-MD5: No common protection layer between client and server
at com.sun.security.sasl.digest.DigestMD5Client.checkQopSupport(DigestMD5Client.java:396)
at com.sun.security.sasl.digest.DigestMD5Client.evaluateChallenge(DigestMD5Client.java:208)
at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:168)
at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:410)
at org.apache.hadoop.ipc.Client$Connection.access$1300(Client.java:203)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:609)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:606)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:605)
... 16 more
at org.apache.hadoop.mapred.LinuxTaskController.initializeJob(LinuxTaskController.java:193)
at org.apache.hadoop.mapred.TaskTracker$4.run(TaskTracker.java:1340)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1315)
at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1230)
at org.apache.hadoop.mapred.TaskTracker$5.run(TaskTracker.java:2641)
at java.lang.Thread.run(Thread.java:662)
Caused by: org.apache.hadoop.util.Shell$ExitCodeException:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:255)
at org.apache.hadoop.util.Shell.run(Shell.java:182)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:375)
at org.apache.hadoop.mapred.LinuxTaskController.initializeJob(LinuxTaskController.java:186)
... 8 more
This is not a problem in Yarn.
> MapReduce Job setup error leaves no useful info to users (when LinuxTaskController is used)
> --------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-5589
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5589
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv1
> Affects Versions: 1.2.1
> Environment: I found that a significant portion of the main() in JobLocalizer will not have its output displayed in the tasklogs.
> Before fix :
> Error initializing attempt_201310122204_0002_m_000002_0:
> java.io.IOException: Job initialization failed (1) with output: Reading task controller config from /etc/hadoop/taskcontroller.cfg
> main : command provided 0
> main : user is bantony
> Good mapred-local-dirs are /hadoop12/scratch,/hadoop04/scratch,/hadoop09/scratch,/hadoop03/scratch,/hadoop05/scratch,/hadoop01/scratch,/hadoop11/scratch,/hadoop10/scratch,/hadoop08/scratch,/hadoop06/scratch,/hadoop07/scratch,/hadoop02/scratch
> at org.apache.hadoop.mapred.LinuxTaskController.initializeJob(LinuxTaskController.java:193)
> at org.apache.hadoop.mapred.TaskTracker$4.run(TaskTracker.java:1340)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
> at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1315)
> at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1230)
> at org.apache.hadoop.mapred.TaskTracker$5.run(TaskTracker.java:2641)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: org.apache.hadoop.util.Shell$ExitCodeException:
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:255)
> at org.apache.hadoop.util.Shell.run(Shell.java:182)
> at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:375)
> at org.apache.hadoop.mapred.LinuxTaskController.initializeJob(LinuxTaskController.java:186)
> ... 8 more
> Reporter: Benoy Antony
> Assignee: Benoy Antony
> Fix For: 1.3.0
>
> Attachments: MAPREDUCE-5368.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.1#6144)