You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@oozie.apache.org by "Mohammad Kamrul Islam (Created) (JIRA)" <ji...@apache.org> on 2011/11/04 02:03:32 UTC
[jira] [Created] (OOZIE-597) Issues in Oozie-Hadoop.next/0.23
integration
Issues in Oozie-Hadoop.next/0.23 integration
--------------------------------------------
Key: OOZIE-597
URL: https://issues.apache.org/jira/browse/OOZIE-597
Project: Oozie
Issue Type: Task
Reporter: Mohammad Kamrul Islam
Oozie team at Y! is working to make upcoming hadoop release working.
Since Hadoop 0.23 is a big overhaul, the integration task is very large as well.
This JIRA is to :
* Track the updates of the integration status.
* Exchange error details and solicit help from others.
* It primarily verifies if the ___existing___ features work properly.
* It ___doesn't___ verify the __new__ features added into 0.23. There is a separate JIRA for new feature support (Ref #Oozie-477).
Current status:
* Our dev and QE already made some progress and more work is in progress.
* Mona (from dev) and Marcy and Michelle (from QE) are actively working and expected to update this JIRA with status/issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-597) Issues in Oozie-Hadoop.next/0.23
integration
Posted by "Mona Chitnis (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13144216#comment-13144216 ]
Mona Chitnis commented on OOZIE-597:
------------------------------------
1. Oozie is receiving a callback query from Hadoop indicating the submitted job has reached its end state
(SUCCEEDED/FAILED) following which, Oozie will check with that job_id with the ResourceManager for status.
At this point in time, the RM logs start showing this after every 10 seconds
Call: protocol=org.apache.hadoop.yarn.proto.ClientRMProtocol.ClientRMProtocolService.BlockingInterface,
method=getApplicationReport
Need more clarity on how the RM handles this incoming request about job status. Does the 10 second re-occurrence
of the above statement point to an RM-specific polling interval? It seems that the Oozie server thread which is
checking this keeps spinning on this call to the RM.
2. MR job submitted via Oozie gets the external ID which is something of the form "job_0123456_001". Note the
"job_" prefix. Now looking through the RM web console for the submitted jobs, we see IDs of the form
"application_0123456_001" corresponding to that job. Need to address this discrepancy for consequences.
> Issues in Oozie-Hadoop.next/0.23 integration
> --------------------------------------------
>
> Key: OOZIE-597
> URL: https://issues.apache.org/jira/browse/OOZIE-597
> Project: Oozie
> Issue Type: Task
> Reporter: Mohammad Kamrul Islam
>
> Oozie team at Y! is working to make upcoming hadoop release working.
> Since Hadoop 0.23 is a big overhaul, the integration task is very large as well.
>
> This JIRA is to :
> * Track the updates of the integration status.
> * Exchange error details and solicit help from others.
> * It primarily verifies if the ___existing___ features work properly.
> * It ___doesn't___ verify the __new__ features added into 0.23. There is a separate JIRA for new feature support (Ref #Oozie-477).
> Current status:
> * Our dev and QE already made some progress and more work is in progress.
> * Mona (from dev) and Marcy and Michelle (from QE) are actively working and expected to update this JIRA with status/issues.
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-597) Issues in Oozie-Hadoop.next/0.23
integration
Posted by "Mona Chitnis (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13147428#comment-13147428 ]
Mona Chitnis commented on OOZIE-597:
------------------------------------
We encountered this error a while ago:
Oozie job is submitted to Hadoop, gets the Job Id, but fails to update status. After a while, when Oozie server
repeatedly starts polling for action status, this exception is seen which prevents any update.
Exception in thread "pool-2-thread-143" java.lang.NoSuchMethodError:
org.apache.hadoop.mapred.Counters.getGroup(Ljava/lang/String;)Lorg/apache/hadoop/mapred/Counters$Group;
at org.apache.oozie.action.hadoop.LauncherMapper.hasIdSwap(LauncherMapper.java:274)
at org.apache.oozie.action.hadoop.JavaActionExecutor.check(JavaActionExecutor.java:842)
at org.apache.oozie.command.wf.ActionCheckXCommand.execute(ActionCheckXCommand.java:162)
at org.apache.oozie.command.wf.ActionCheckXCommand.execute(ActionCheckXCommand.java:51)
at org.apache.oozie.command.XCommand.call(XCommand.java:260)
at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:166)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Now, after further work, this issue is resolved after having compiled the latest Hadoop code available from the Apache svn repo (branch-0.23) and using those JARs during Oozie compilation. Oozie compiles successfully with those JARs.
The aforementioned Counters.getGroup NoSuchMethodError (which was preventing Oozie from updating job status and spinning on the thread) has been resolved by linking Oozie library with the latest available Hadoop JARs.
However, Oozie job fails owing to an NPE (refer to MAPREDUCE-3380).
2011-11-09 16:20:48,873 DEBUG MapReduceActionExecutor:545 Creating Job Client for action
2011-11-09 16:20:49,022 DEBUG KerberosHadoopAccessorService:545
2011-11-09 16:20:49,026 WARN ActionStartXCommand:542 Error starting action. ErrorType [ERROR], ErrorCode [NullPointerException], Message [NullPointerException: null]
org.apache.oozie.action.ActionExecutorException: NullPointerException: null
at org.apache.oozie.action.ActionExecutor.convertException(ActionExecutor.java:370)
at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:608)
at org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:767)
at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:193)
at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:55)
at org.apache.oozie.command.XCommand.call(XCommand.java:260)
at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:300)
at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:247)
at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:166)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.NullPointerException
at org.apache.hadoop.mapreduce.Cluster.getDelegationToken(Cluster.java:399)
at org.apache.hadoop.mapred.JobClient.getDelegationToken(JobClient.java:1074)
at
org.apache.oozie.service.KerberosHadoopAccessorService.createJobClient(KerberosHadoopAccessorService.java:139)
at org.apache.oozie.action.hadoop.JavaActionExecutor.createJobClient(JavaActionExecutor.java:810)
at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:551)
... 10 more
Trying the fix proposed in MAPREDUCE-3380 helped towards successful Oozie job completion and status update
> Issues in Oozie-Hadoop.next/0.23 integration
> --------------------------------------------
>
> Key: OOZIE-597
> URL: https://issues.apache.org/jira/browse/OOZIE-597
> Project: Oozie
> Issue Type: Task
> Reporter: Mohammad Kamrul Islam
>
> Oozie team at Y! is working to make upcoming hadoop release working.
> Since Hadoop 0.23 is a big overhaul, the integration task is very large as well.
>
> This JIRA is to :
> * Track the updates of the integration status.
> * Exchange error details and solicit help from others.
> * It primarily verifies if the ___existing___ features work properly.
> * It ___doesn't___ verify the __new__ features added into 0.23. There is a separate JIRA for new feature support (Ref #Oozie-477).
> Current status:
> * Our dev and QE already made some progress and more work is in progress.
> * Mona (from dev) and Marcy and Michelle (from QE) are actively working and expected to update this JIRA with status/issues.
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira