You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@oozie.apache.org by "Srikanth Sundarrajan (JIRA)" <ji...@apache.org> on 2013/09/16 18:32:54 UTC
[jira] [Commented] (OOZIE-1534) Launcher job might run do hadoop
attempt relaunch - possibly causing correctness issues
[ https://issues.apache.org/jira/browse/OOZIE-1534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13768465#comment-13768465 ]
Srikanth Sundarrajan commented on OOZIE-1534:
---------------------------------------------
Notes from my discussion with [~tucu00] offline:
This could be done if the log scavenger logic with use to harvest MR jobs started by pig/hive/sqoop is done realtime (as opposed after pig/hive/sqoop finishes) and the captured job IDs are written/fsync to a file in HDFS in the action subdir. Then the action main class would look for this file container job ids at start time and if it exists, it would kill all those jobs before proceeding. This would make the launcher job idempotent.
> Launcher job might run do hadoop attempt relaunch - possibly causing correctness issues
> ---------------------------------------------------------------------------------------
>
> Key: OOZIE-1534
> URL: https://issues.apache.org/jira/browse/OOZIE-1534
> Project: Oozie
> Issue Type: Improvement
> Reporter: Srikanth Sundarrajan
>
> The <prepare> section of the action allow to clean up the output dir. This is not sufficient as MR jobs started by Pig/Hive may be still running.We should look to kill child MR jobs if any before launching new ones.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira