You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by rohit sarewar <ro...@gmail.com> on 2014/07/23 10:39:11 UTC
Redirecting to job history server fails in MapReduceV2
Hi
I am facing an error during Java Map reduce code execution in using Hadoop
2.2
The FinalApplicationStatus is successful. I can find the output in desired
location but It throws error while redirecting to job history server
(please refer to red line in the logs below).
*Explanation in detail:*
Input File to the Java MR job:
100 files ~ each with 300 MB size , Total size = 34 GB
*Case 1: (1 Job ) Error*
Run one MR job for all 100 files as input to the job ( Run time - 1 hour)
It throws error as given in job logs - marked in red color below.
*Case 2: (100 job)Successful*
Run one MR job for each 100 files as input to the job ( Run time - 40 sec
per job)
I can see all 100 output and all job ran successfully.Also for all these
jobs it successfully redirected to job history server and displayed all
counter in terminal. (No error was seen in this case)
After Case 2, I cannot blame to my cluster configuration.
*How do I fix this ? Please help.*
I could also find few unresolved JIRA tickets and discussion for similar
case(links below):
1) Similar case
http://mail-archives.apache.org/mod_mbox/oozie-user/201402.mbox/%3CCAAu13zFQWQuVo-ShJYVT-o+D=SC6-4Zhwn8UKPsB31wS62=0Jw@mail.gmail.com%3E
2) Exact Same Case
http://mail-archives.apache.org/mod_mbox/hadoop-hdfs-user/201311.mbox/%3CCANXCz3RC5aCgeAMro=wS6TbwjsQVx9kemJd28kpmjMTp5hdX=Q@mail.gmail.com%3E
3) Similar Case
http://stackoverflow.com/questions/17930644/connection-error-in-apache-pig
*Job Logs:*
14/07/22 06:37:11 INFO mapreduce.Job: map 100% reduce 98%
14/07/22 06:37:44 INFO mapreduce.Job: map 100% reduce 99%
14/07/22 06:38:30 INFO mapreduce.Job: map 100% reduce 100%
14/07/22 06:39:02 INFO mapred.ClientServiceDelegate: Application state is
completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history
server
14/07/22 06:39:02 INFO mapred.ClientServiceDelegate: Application state is
completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history
server
14/07/22 06:39:02 INFO mapred.ClientServiceDelegate: Application state is
completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history
server
14/07/22 06:39:02 ERROR security.UserGroupInformation:
PriviledgedActionException as: rohitsarewar (auth:SIMPLE)
cause:java.io.IOException:
org.apache.hadoop.ipc.RemoteException(java.lang.NullPointerException):
java.lang.NullPointerException
at
org.apache.hadoop.mapreduce.v2.hs.HistoryClientService$HSClientProtocolHandler.getTaskAttemptCompletionEvents(HistoryClientService.java:269)
at
org.apache.hadoop.mapreduce.v2.api.impl.pb.service.MRClientProtocolPBServiceImpl.getTaskAttemptCompletionEvents(MRClientProtocolPBServiceImpl.java:173)
at
org.apache.hadoop.yarn.proto.MRClientProtocol$MRClientProtocolService$2.callBlockingMethod(MRClientProtocol.java:283)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2053)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2047)
Exception in thread "main" java.io.IOException:
org.apache.hadoop.ipc.RemoteException(java.lang.NullPointerException):
java.lang.NullPointerException
at
org.apache.hadoop.mapreduce.v2.hs.HistoryClientService$HSClientProtocolHandler.getTaskAttemptCompletionEvents(HistoryClientService.java:269)
at
org.apache.hadoop.mapreduce.v2.api.impl.pb.service.MRClientProtocolPBServiceImpl.getTaskAttemptCompletionEvents(MRClientProtocolPBServiceImpl.java:173)
at
org.apache.hadoop.yarn.proto.MRClientProtocol$MRClientProtocolService$2.callBlockingMethod(MRClientProtocol.java:283)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2053)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2047)
at
org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:330)
at
org.apache.hadoop.mapred.ClientServiceDelegate.getTaskCompletionEvents(ClientServiceDelegate.java:382)
at
org.apache.hadoop.mapred.YARNRunner.getTaskCompletionEvents(YARNRunner.java:529)
at org.apache.hadoop.mapreduce.Job$5.run(Job.java:668)
at org.apache.hadoop.mapreduce.Job$5.run(Job.java:665)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
at
org.apache.hadoop.mapreduce.Job.getTaskCompletionEvents(Job.java:665)
at org.apache.hadoop.mapreduce.Job.monitorAndPrintJob(Job.java:1349)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1289)
at com.bigdata.mapreduce.esc.escDriver.main(escDriver.java:23)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
Caused by:
org.apache.hadoop.ipc.RemoteException(java.lang.NullPointerException): j
at
org.apache.hadoop.mapreduce.v2.hs.HistoryClientService$HSClientProtocolH
at
org.apache.hadoop.mapreduce.v2.api.impl.pb.service.MRClientProtocolPBSer
at
org.apache.hadoop.yarn.proto.MRClientProtocol$MRClientProtocolService$2.
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(P
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2053)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformatio
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2047)
at org.apache.hadoop.ipc.Client.call(Client.java:1347)
at org.apache.hadoop.ipc.Client.call(Client.java:1300)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine
at com.sun.proxy.$Proxy12.getTaskAttemptCompletionEvents(Unknown
Source)
at
org.apache.hadoop.mapreduce.v2.api.impl.pb.client.MRClientProtocolPBClie
at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDeleg
... 16 more
Regards
Rohit Sarewar