You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@oozie.apache.org by "Kiran Nagasubramanian (Created) (JIRA)" <ji...@apache.org> on 2011/11/24 00:56:39 UTC
[jira] [Created] (OOZIE-616) Moving action prepare FS execution to
LauncherMapper
Moving action prepare FS execution to LauncherMapper
----------------------------------------------------
Key: OOZIE-616
URL: https://issues.apache.org/jira/browse/OOZIE-616
Project: Oozie
Issue Type: Improvement
Reporter: Kiran Nagasubramanian
1) Motivations:
----------------
a) Currently Oozie Server executes the Prepare logic before posting the Launcher job to Hadoop. If the launcher fails and the execution is retried by Hadoop automatically, the prepare logic is not re-executed. So, for Java/MR/Pig actions, if the launcher fails and it is retried by Hadoop automatically, the prepare logic is not executed again. Once the prepare logic execution is moved to the Launcher, the prepare logic is also re-executed during every retry.
b) Heavy duty operations like copy that are currently not supported can be supported once the prepare logic has been moved to the Launcher.
2) Design choices for the proposed model:
------------------------------------------------
a) Separate classes for different types of actions
---------------------------------------------------------
The different types of actions like FS actions, HCat related actions, etc. can be grouped together in separate classes like FSActions, HCatActions, etc.
b) Passing the prepare logic to the Launcher
---------------------------------------------------
Launcher needs the Prepare XML block to execute the actions. Oozie server can write the XML block to a file on DFS and then the Launcher could read from there.
c) Execution of actions through a "Driver"
-----------------------------------------------
The Launcher can pass the XML block to the Driver which parses the content and calls corresponding methods that are grouped in different classes.
3) Pros and Cons of the design change:
--------------------------------------------
a) User facing impacts:
-------------------------
i) At present: In case of prepare block failure, there is no impact on the retry of map reduce job since the prepare block would have got executed earlier itself. The impact is seen only when Launcher fails.
As per the proposed design: When the prepare block fails it will impact the retry of the map reduce job.
ii) At present: In the case of prepare block failure, the launcher mapper is not launched at all.
As per the proposed design: The cost of executing the prepare block would go up since it requires the launch of a map only task. This is highly pronounced in case of failures
Thanks a lot for all the input from Oozie team members.
Please post your suggestions. Thanks.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-616) Moving action prepare FS execution
to LauncherMapper
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13171158#comment-13171158 ]
jiraposter@reviews.apache.org commented on OOZIE-616:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3143/
-----------------------------------------------------------
(Updated 2011-12-16 19:34:53.428355)
Review request for oozie, Mohammad Islam and Angelo K. Huang.
Changes
-------
Please ignore the revision 4.
Summary
-------
a) Separate classes for different types of actions
------------------------------------------------
The different types of actions like FS actions, HCat related actions, etc. can be grouped together in separate classes like FSActions, HCatActions, etc.
b) Passing the prepare logic to the Launcher
-----------------------------------------
Launcher needs the Prepare XML block to execute the actions. Oozie server can write the XML block to a file on DFS and then the Launcher could read from there.
c) Execution of actions through a "Driver"
---------------------------------------
The Launcher can pass the XML block to the Driver which parses the content and calls corresponding methods that are grouped in different classes.
This addresses bug OOZIE-616.
https://issues.apache.org/jira/browse/OOZIE-616
Diffs (updated)
-----
trunk/core/src/main/java/org/apache/oozie/action/hadoop/FileSystemActions.java PRE-CREATION
trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1213901
trunk/core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java 1213901
trunk/core/src/main/java/org/apache/oozie/action/hadoop/PrepareActionsDriver.java PRE-CREATION
trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestFileSystemActions.java PRE-CREATION
trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 1213901
trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestLauncher.java 1213901
trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestMapReduceActionError.java 1213901
trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestMapReduceActionExecutor.java 1213901
trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestPigActionExecutor.java 1213901
trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestPrepareActionsDriver.java PRE-CREATION
trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestShellActionExecutor.java 1213901
Diff: https://reviews.apache.org/r/3143/diff
Testing
-------
Yes
Thanks,
Kiran
> Moving action prepare FS execution to LauncherMapper
> ----------------------------------------------------
>
> Key: OOZIE-616
> URL: https://issues.apache.org/jira/browse/OOZIE-616
> Project: Oozie
> Issue Type: Improvement
> Reporter: Kiran Nagasubramanian
>
> 1) Motivations:
> ----------------
> a) Currently Oozie Server executes the Prepare logic before posting the Launcher job to Hadoop. If the launcher fails and the execution is retried by Hadoop automatically, the prepare logic is not re-executed. So, for Java/MR/Pig actions, if the launcher fails and it is retried by Hadoop automatically, the prepare logic is not executed again. Once the prepare logic execution is moved to the Launcher, the prepare logic is also re-executed during every retry.
> b) Heavy duty operations like copy that are currently not supported can be supported once the prepare logic has been moved to the Launcher.
> 2) Design choices for the proposed model:
> ------------------------------------------------
> a) Separate classes for different types of actions
> ---------------------------------------------------------
> The different types of actions like FS actions, HCat related actions, etc. can be grouped together in separate classes like FSActions, HCatActions, etc.
> b) Passing the prepare logic to the Launcher
> ---------------------------------------------------
> Launcher needs the Prepare XML block to execute the actions. Oozie server can write the XML block to a file on DFS and then the Launcher could read from there.
> c) Execution of actions through a "Driver"
> -----------------------------------------------
> The Launcher can pass the XML block to the Driver which parses the content and calls corresponding methods that are grouped in different classes.
> 3) Pros and Cons of the design change:
> --------------------------------------------
> a) User facing impacts:
> -------------------------
> i) At present: In case of prepare block failure, there is no impact on the retry of map reduce job since the prepare block would have got executed earlier itself. The impact is seen only when Launcher fails.
>
> As per the proposed design: When the prepare block fails it will impact the retry of the map reduce job.
>
> ii) At present: In the case of prepare block failure, the launcher mapper is not launched at all.
>
> As per the proposed design: The cost of executing the prepare block would go up since it requires the launch of a map only task. This is highly pronounced in case of failures
>
> Thanks a lot for all the input from Oozie team members.
> Please post your suggestions. Thanks.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (OOZIE-616) Moving action prepare FS execution to
LauncherMapper
Posted by "Alejandro Abdelnur (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alejandro Abdelnur updated OOZIE-616:
-------------------------------------
Fix Version/s: 3.2.0
> Moving action prepare FS execution to LauncherMapper
> ----------------------------------------------------
>
> Key: OOZIE-616
> URL: https://issues.apache.org/jira/browse/OOZIE-616
> Project: Oozie
> Issue Type: Improvement
> Reporter: Kiran Nagasubramanian
> Fix For: 3.2.0
>
>
> 1) Motivations:
> ----------------
> a) Currently Oozie Server executes the Prepare logic before posting the Launcher job to Hadoop. If the launcher fails and the execution is retried by Hadoop automatically, the prepare logic is not re-executed. So, for Java/MR/Pig actions, if the launcher fails and it is retried by Hadoop automatically, the prepare logic is not executed again. Once the prepare logic execution is moved to the Launcher, the prepare logic is also re-executed during every retry.
> b) Heavy duty operations like copy that are currently not supported can be supported once the prepare logic has been moved to the Launcher.
> 2) Design choices for the proposed model:
> ------------------------------------------------
> a) Separate classes for different types of actions
> ---------------------------------------------------------
> The different types of actions like FS actions, HCat related actions, etc. can be grouped together in separate classes like FSActions, HCatActions, etc.
> b) Passing the prepare logic to the Launcher
> ---------------------------------------------------
> Launcher needs the Prepare XML block to execute the actions. Oozie server can write the XML block to a file on DFS and then the Launcher could read from there.
> c) Execution of actions through a "Driver"
> -----------------------------------------------
> The Launcher can pass the XML block to the Driver which parses the content and calls corresponding methods that are grouped in different classes.
> 3) Pros and Cons of the design change:
> --------------------------------------------
> a) User facing impacts:
> -------------------------
> i) At present: In case of prepare block failure, there is no impact on the retry of map reduce job since the prepare block would have got executed earlier itself. The impact is seen only when Launcher fails.
>
> As per the proposed design: When the prepare block fails it will impact the retry of the map reduce job.
>
> ii) At present: In the case of prepare block failure, the launcher mapper is not launched at all.
>
> As per the proposed design: The cost of executing the prepare block would go up since it requires the launch of a map only task. This is highly pronounced in case of failures
>
> Thanks a lot for all the input from Oozie team members.
> Please post your suggestions. Thanks.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-616) Moving action prepare FS execution
to LauncherMapper
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13205203#comment-13205203 ]
jiraposter@reviews.apache.org commented on OOZIE-616:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3827/
-----------------------------------------------------------
(Updated 2012-02-10 03:48:04.469602)
Review request for oozie, Santhosh Srinivasan and Alejandro Abdelnur.
Changes
-------
Integrating Santhosh's feedback
Summary
-------
This is Kiran's patch. I am uploading the diff as he is not here. Original diff at https://reviews.apache.org/r/3143/
Thanks Alejandro for testing the patch out. The testcases are fixed now.
This addresses bug oozie-616.
https://issues.apache.org/jira/browse/oozie-616
Diffs (updated)
-----
trunk/core/src/main/java/org/apache/oozie/action/hadoop/FileSystemActions.java PRE-CREATION
trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1242619
trunk/core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java 1242619
trunk/core/src/main/java/org/apache/oozie/action/hadoop/PrepareActionsDriver.java PRE-CREATION
trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestFileSystemActions.java PRE-CREATION
trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 1242619
trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestLauncher.java 1242619
trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestMapReduceActionError.java 1242619
trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestMapReduceActionExecutor.java 1242619
trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestPigActionExecutor.java 1242619
trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestPrepareActionsDriver.java PRE-CREATION
trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestShellActionExecutor.java 1242619
Diff: https://reviews.apache.org/r/3827/diff
Testing
-------
Ran the entire test suite
Thanks,
Virag
> Moving action prepare FS execution to LauncherMapper
> ----------------------------------------------------
>
> Key: OOZIE-616
> URL: https://issues.apache.org/jira/browse/OOZIE-616
> Project: Oozie
> Issue Type: Improvement
> Reporter: Kiran Nagasubramanian
>
> 1) Motivations:
> ----------------
> a) Currently Oozie Server executes the Prepare logic before posting the Launcher job to Hadoop. If the launcher fails and the execution is retried by Hadoop automatically, the prepare logic is not re-executed. So, for Java/MR/Pig actions, if the launcher fails and it is retried by Hadoop automatically, the prepare logic is not executed again. Once the prepare logic execution is moved to the Launcher, the prepare logic is also re-executed during every retry.
> b) Heavy duty operations like copy that are currently not supported can be supported once the prepare logic has been moved to the Launcher.
> 2) Design choices for the proposed model:
> ------------------------------------------------
> a) Separate classes for different types of actions
> ---------------------------------------------------------
> The different types of actions like FS actions, HCat related actions, etc. can be grouped together in separate classes like FSActions, HCatActions, etc.
> b) Passing the prepare logic to the Launcher
> ---------------------------------------------------
> Launcher needs the Prepare XML block to execute the actions. Oozie server can write the XML block to a file on DFS and then the Launcher could read from there.
> c) Execution of actions through a "Driver"
> -----------------------------------------------
> The Launcher can pass the XML block to the Driver which parses the content and calls corresponding methods that are grouped in different classes.
> 3) Pros and Cons of the design change:
> --------------------------------------------
> a) User facing impacts:
> -------------------------
> i) At present: In case of prepare block failure, there is no impact on the retry of map reduce job since the prepare block would have got executed earlier itself. The impact is seen only when Launcher fails.
>
> As per the proposed design: When the prepare block fails it will impact the retry of the map reduce job.
>
> ii) At present: In the case of prepare block failure, the launcher mapper is not launched at all.
>
> As per the proposed design: The cost of executing the prepare block would go up since it requires the launch of a map only task. This is highly pronounced in case of failures
>
> Thanks a lot for all the input from Oozie team members.
> Please post your suggestions. Thanks.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-616) Moving action prepare FS execution
to LauncherMapper
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13205154#comment-13205154 ]
jiraposter@reviews.apache.org commented on OOZIE-616:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3827/#review5000
-----------------------------------------------------------
Some more comments to make the error messages cleaner and usage of the LauncherException tighter.
trunk/core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java
<https://reviews.apache.org/r/3827/#comment11003>
Can we remove the default constructor and force developers to use one of the other two constructors?
trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestFileSystemActions.java
<https://reviews.apache.org/r/3827/#comment10997>
Can you add a message here - fail("Expected to catch an exception but did not encounter any") ?
trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestFileSystemActions.java
<https://reviews.apache.org/r/3827/#comment10998>
Can you add a message here - fail("Expected a LauncherException but received an Exception") ?
trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestFileSystemActions.java
<https://reviews.apache.org/r/3827/#comment10999>
Can you add a message here - fail("Expected to catch an exception but did not encounter any") ?
trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestFileSystemActions.java
<https://reviews.apache.org/r/3827/#comment11000>
Can you add a message here - fail("Expected a LauncherException but received an Exception") ?
trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestPrepareActionsDriver.java
<https://reviews.apache.org/r/3827/#comment11002>
Can you add a message here - fail("Expected to catch an exception but did not encounter any") ?
trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestPrepareActionsDriver.java
<https://reviews.apache.org/r/3827/#comment11001>
Can you add a message here - fail("Expected a LauncherException but received an Exception") ?
- Santhosh
On 2012-02-10 00:41:55, Virag Kothari wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/3827/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2012-02-10 00:41:55)
bq.
bq.
bq. Review request for oozie, Santhosh Srinivasan and Alejandro Abdelnur.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. This is Kiran's patch. I am uploading the diff as he is not here. Original diff at https://reviews.apache.org/r/3143/
bq. Thanks Alejandro for testing the patch out. The testcases are fixed now.
bq.
bq.
bq. This addresses bug oozie-616.
bq. https://issues.apache.org/jira/browse/oozie-616
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1242619
bq. trunk/core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java 1242619
bq. trunk/core/src/main/java/org/apache/oozie/action/hadoop/FileSystemActions.java PRE-CREATION
bq. trunk/core/src/main/java/org/apache/oozie/action/hadoop/PrepareActionsDriver.java PRE-CREATION
bq. trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestFileSystemActions.java PRE-CREATION
bq. trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 1242619
bq. trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestLauncher.java 1242619
bq. trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestMapReduceActionError.java 1242619
bq. trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestMapReduceActionExecutor.java 1242619
bq. trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestPigActionExecutor.java 1242619
bq. trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestPrepareActionsDriver.java PRE-CREATION
bq. trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestShellActionExecutor.java 1242619
bq.
bq. Diff: https://reviews.apache.org/r/3827/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. Ran the entire test suite
bq.
bq.
bq. Thanks,
bq.
bq. Virag
bq.
bq.
> Moving action prepare FS execution to LauncherMapper
> ----------------------------------------------------
>
> Key: OOZIE-616
> URL: https://issues.apache.org/jira/browse/OOZIE-616
> Project: Oozie
> Issue Type: Improvement
> Reporter: Kiran Nagasubramanian
>
> 1) Motivations:
> ----------------
> a) Currently Oozie Server executes the Prepare logic before posting the Launcher job to Hadoop. If the launcher fails and the execution is retried by Hadoop automatically, the prepare logic is not re-executed. So, for Java/MR/Pig actions, if the launcher fails and it is retried by Hadoop automatically, the prepare logic is not executed again. Once the prepare logic execution is moved to the Launcher, the prepare logic is also re-executed during every retry.
> b) Heavy duty operations like copy that are currently not supported can be supported once the prepare logic has been moved to the Launcher.
> 2) Design choices for the proposed model:
> ------------------------------------------------
> a) Separate classes for different types of actions
> ---------------------------------------------------------
> The different types of actions like FS actions, HCat related actions, etc. can be grouped together in separate classes like FSActions, HCatActions, etc.
> b) Passing the prepare logic to the Launcher
> ---------------------------------------------------
> Launcher needs the Prepare XML block to execute the actions. Oozie server can write the XML block to a file on DFS and then the Launcher could read from there.
> c) Execution of actions through a "Driver"
> -----------------------------------------------
> The Launcher can pass the XML block to the Driver which parses the content and calls corresponding methods that are grouped in different classes.
> 3) Pros and Cons of the design change:
> --------------------------------------------
> a) User facing impacts:
> -------------------------
> i) At present: In case of prepare block failure, there is no impact on the retry of map reduce job since the prepare block would have got executed earlier itself. The impact is seen only when Launcher fails.
>
> As per the proposed design: When the prepare block fails it will impact the retry of the map reduce job.
>
> ii) At present: In the case of prepare block failure, the launcher mapper is not launched at all.
>
> As per the proposed design: The cost of executing the prepare block would go up since it requires the launch of a map only task. This is highly pronounced in case of failures
>
> Thanks a lot for all the input from Oozie team members.
> Please post your suggestions. Thanks.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (OOZIE-616) Moving action prepare FS execution to
LauncherMapper
Posted by "Virag Kothari (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Virag Kothari updated OOZIE-616:
--------------------------------
Assignee: Kiran Nagasubramanian
> Moving action prepare FS execution to LauncherMapper
> ----------------------------------------------------
>
> Key: OOZIE-616
> URL: https://issues.apache.org/jira/browse/OOZIE-616
> Project: Oozie
> Issue Type: Improvement
> Reporter: Kiran Nagasubramanian
> Assignee: Kiran Nagasubramanian
> Fix For: 3.2.0
>
>
> 1) Motivations:
> ----------------
> a) Currently Oozie Server executes the Prepare logic before posting the Launcher job to Hadoop. If the launcher fails and the execution is retried by Hadoop automatically, the prepare logic is not re-executed. So, for Java/MR/Pig actions, if the launcher fails and it is retried by Hadoop automatically, the prepare logic is not executed again. Once the prepare logic execution is moved to the Launcher, the prepare logic is also re-executed during every retry.
> b) Heavy duty operations like copy that are currently not supported can be supported once the prepare logic has been moved to the Launcher.
> 2) Design choices for the proposed model:
> ------------------------------------------------
> a) Separate classes for different types of actions
> ---------------------------------------------------------
> The different types of actions like FS actions, HCat related actions, etc. can be grouped together in separate classes like FSActions, HCatActions, etc.
> b) Passing the prepare logic to the Launcher
> ---------------------------------------------------
> Launcher needs the Prepare XML block to execute the actions. Oozie server can write the XML block to a file on DFS and then the Launcher could read from there.
> c) Execution of actions through a "Driver"
> -----------------------------------------------
> The Launcher can pass the XML block to the Driver which parses the content and calls corresponding methods that are grouped in different classes.
> 3) Pros and Cons of the design change:
> --------------------------------------------
> a) User facing impacts:
> -------------------------
> i) At present: In case of prepare block failure, there is no impact on the retry of map reduce job since the prepare block would have got executed earlier itself. The impact is seen only when Launcher fails.
>
> As per the proposed design: When the prepare block fails it will impact the retry of the map reduce job.
>
> ii) At present: In the case of prepare block failure, the launcher mapper is not launched at all.
>
> As per the proposed design: The cost of executing the prepare block would go up since it requires the launch of a map only task. This is highly pronounced in case of failures
>
> Thanks a lot for all the input from Oozie team members.
> Please post your suggestions. Thanks.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-616) Moving action prepare FS execution
to LauncherMapper
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13171340#comment-13171340 ]
jiraposter@reviews.apache.org commented on OOZIE-616:
-----------------------------------------------------
bq. On 2011-12-16 18:59:53, Kiran Nagasubramanian wrote:
bq. > trunk/core/src/main/java/org/apache/oozie/action/hadoop/PrepareActionsDriver.java, line 32
bq. > <https://reviews.apache.org/r/3143/diff/3/?file=64229#file64229line32>
bq. >
bq. > If we use JDOM or XMLUtils we need to have the corresponding JAR files in the distributed cache. This becomes heavy. This is pronounced even more in case of XMLUtils since the class by itself needs many JARs to be included. Hence, decided to parse the XML using the functionalities that came as a part of the standard java distribution. In addition to this, this class acts an intermediate driver, as the name suggests, and calls the corresponding type of actions(File system/HCat etc.) which are/will be grouped together in separate classes.
Oozie uses JDOM all over the place, is a lightweight JAR, no additional dependencies. And with OOZIE-628 you can have the JDOM JAR in Oozie sharelib and the launcher will always have that JAR in the classpath
- Alejandro
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3143/#review3949
-----------------------------------------------------------
On 2011-12-16 23:55:14, Kiran Nagasubramanian wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/3143/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2011-12-16 23:55:14)
bq.
bq.
bq. Review request for oozie, Mohammad Islam and Angelo K. Huang.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. a) Separate classes for different types of actions
bq. ------------------------------------------------
bq. The different types of actions like FS actions, HCat related actions, etc. can be grouped together in separate classes like FSActions, HCatActions, etc.
bq.
bq. b) Passing the prepare logic to the Launcher
bq. -----------------------------------------
bq. Launcher needs the Prepare XML block to execute the actions. Oozie server can write the XML block to a file on DFS and then the Launcher could read from there.
bq.
bq. c) Execution of actions through a "Driver"
bq. ---------------------------------------
bq. The Launcher can pass the XML block to the Driver which parses the content and calls corresponding methods that are grouped in different classes.
bq.
bq.
bq. This addresses bug OOZIE-616.
bq. https://issues.apache.org/jira/browse/OOZIE-616
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. trunk/core/src/main/java/org/apache/oozie/action/hadoop/FileSystemActions.java PRE-CREATION
bq. trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1213901
bq. trunk/core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java 1213901
bq. trunk/core/src/main/java/org/apache/oozie/action/hadoop/PrepareActionsDriver.java PRE-CREATION
bq. trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestFileSystemActions.java PRE-CREATION
bq. trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 1213901
bq. trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestLauncher.java 1213901
bq. trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestMapReduceActionError.java 1213901
bq. trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestMapReduceActionExecutor.java 1213901
bq. trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestPigActionExecutor.java 1213901
bq. trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestPrepareActionsDriver.java PRE-CREATION
bq. trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestShellActionExecutor.java 1213901
bq.
bq. Diff: https://reviews.apache.org/r/3143/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. Yes
bq.
bq.
bq. Thanks,
bq.
bq. Kiran
bq.
bq.
> Moving action prepare FS execution to LauncherMapper
> ----------------------------------------------------
>
> Key: OOZIE-616
> URL: https://issues.apache.org/jira/browse/OOZIE-616
> Project: Oozie
> Issue Type: Improvement
> Reporter: Kiran Nagasubramanian
>
> 1) Motivations:
> ----------------
> a) Currently Oozie Server executes the Prepare logic before posting the Launcher job to Hadoop. If the launcher fails and the execution is retried by Hadoop automatically, the prepare logic is not re-executed. So, for Java/MR/Pig actions, if the launcher fails and it is retried by Hadoop automatically, the prepare logic is not executed again. Once the prepare logic execution is moved to the Launcher, the prepare logic is also re-executed during every retry.
> b) Heavy duty operations like copy that are currently not supported can be supported once the prepare logic has been moved to the Launcher.
> 2) Design choices for the proposed model:
> ------------------------------------------------
> a) Separate classes for different types of actions
> ---------------------------------------------------------
> The different types of actions like FS actions, HCat related actions, etc. can be grouped together in separate classes like FSActions, HCatActions, etc.
> b) Passing the prepare logic to the Launcher
> ---------------------------------------------------
> Launcher needs the Prepare XML block to execute the actions. Oozie server can write the XML block to a file on DFS and then the Launcher could read from there.
> c) Execution of actions through a "Driver"
> -----------------------------------------------
> The Launcher can pass the XML block to the Driver which parses the content and calls corresponding methods that are grouped in different classes.
> 3) Pros and Cons of the design change:
> --------------------------------------------
> a) User facing impacts:
> -------------------------
> i) At present: In case of prepare block failure, there is no impact on the retry of map reduce job since the prepare block would have got executed earlier itself. The impact is seen only when Launcher fails.
>
> As per the proposed design: When the prepare block fails it will impact the retry of the map reduce job.
>
> ii) At present: In the case of prepare block failure, the launcher mapper is not launched at all.
>
> As per the proposed design: The cost of executing the prepare block would go up since it requires the launch of a map only task. This is highly pronounced in case of failures
>
> Thanks a lot for all the input from Oozie team members.
> Please post your suggestions. Thanks.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-616) Moving action prepare FS execution
to LauncherMapper
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13171137#comment-13171137 ]
jiraposter@reviews.apache.org commented on OOZIE-616:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3143/
-----------------------------------------------------------
(Updated 2011-12-16 18:59:50.565968)
Review request for oozie, Mohammad Islam and Angelo K. Huang.
Changes
-------
Have incorporated the review comments
Summary
-------
a) Separate classes for different types of actions
------------------------------------------------
The different types of actions like FS actions, HCat related actions, etc. can be grouped together in separate classes like FSActions, HCatActions, etc.
b) Passing the prepare logic to the Launcher
-----------------------------------------
Launcher needs the Prepare XML block to execute the actions. Oozie server can write the XML block to a file on DFS and then the Launcher could read from there.
c) Execution of actions through a "Driver"
---------------------------------------
The Launcher can pass the XML block to the Driver which parses the content and calls corresponding methods that are grouped in different classes.
This addresses bug OOZIE-616.
https://issues.apache.org/jira/browse/OOZIE-616
Diffs (updated)
-----
trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 1213901
trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestFileSystemActions.java PRE-CREATION
trunk/core/src/main/java/org/apache/oozie/action/hadoop/PrepareActionsDriver.java PRE-CREATION
trunk/core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java 1213901
trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1213901
trunk/core/src/main/java/org/apache/oozie/action/hadoop/FileSystemActions.java PRE-CREATION
trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestLauncher.java 1213901
trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestMapReduceActionError.java 1213901
trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestMapReduceActionExecutor.java 1213901
trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestPigActionExecutor.java 1213901
trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestPrepareActionsDriver.java PRE-CREATION
trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestShellActionExecutor.java 1213901
Diff: https://reviews.apache.org/r/3143/diff
Testing
-------
Yes
Thanks,
Kiran
> Moving action prepare FS execution to LauncherMapper
> ----------------------------------------------------
>
> Key: OOZIE-616
> URL: https://issues.apache.org/jira/browse/OOZIE-616
> Project: Oozie
> Issue Type: Improvement
> Reporter: Kiran Nagasubramanian
>
> 1) Motivations:
> ----------------
> a) Currently Oozie Server executes the Prepare logic before posting the Launcher job to Hadoop. If the launcher fails and the execution is retried by Hadoop automatically, the prepare logic is not re-executed. So, for Java/MR/Pig actions, if the launcher fails and it is retried by Hadoop automatically, the prepare logic is not executed again. Once the prepare logic execution is moved to the Launcher, the prepare logic is also re-executed during every retry.
> b) Heavy duty operations like copy that are currently not supported can be supported once the prepare logic has been moved to the Launcher.
> 2) Design choices for the proposed model:
> ------------------------------------------------
> a) Separate classes for different types of actions
> ---------------------------------------------------------
> The different types of actions like FS actions, HCat related actions, etc. can be grouped together in separate classes like FSActions, HCatActions, etc.
> b) Passing the prepare logic to the Launcher
> ---------------------------------------------------
> Launcher needs the Prepare XML block to execute the actions. Oozie server can write the XML block to a file on DFS and then the Launcher could read from there.
> c) Execution of actions through a "Driver"
> -----------------------------------------------
> The Launcher can pass the XML block to the Driver which parses the content and calls corresponding methods that are grouped in different classes.
> 3) Pros and Cons of the design change:
> --------------------------------------------
> a) User facing impacts:
> -------------------------
> i) At present: In case of prepare block failure, there is no impact on the retry of map reduce job since the prepare block would have got executed earlier itself. The impact is seen only when Launcher fails.
>
> As per the proposed design: When the prepare block fails it will impact the retry of the map reduce job.
>
> ii) At present: In the case of prepare block failure, the launcher mapper is not launched at all.
>
> As per the proposed design: The cost of executing the prepare block would go up since it requires the launch of a map only task. This is highly pronounced in case of failures
>
> Thanks a lot for all the input from Oozie team members.
> Please post your suggestions. Thanks.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-616) Moving action prepare FS execution
to LauncherMapper
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13205926#comment-13205926 ]
jiraposter@reviews.apache.org commented on OOZIE-616:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3827/
-----------------------------------------------------------
(Updated 2012-02-11 00:33:59.395574)
Review request for oozie, Santhosh Srinivasan and Alejandro Abdelnur.
Changes
-------
Addressing the last comment. Santhosh, you are right. No need to wrap that one.
Summary
-------
This is Kiran's patch. I am uploading the diff as he is not here. Original diff at https://reviews.apache.org/r/3143/
Thanks Alejandro for testing the patch out. The testcases are fixed now.
This addresses bug oozie-616.
https://issues.apache.org/jira/browse/oozie-616
Diffs (updated)
-----
trunk/core/src/main/java/org/apache/oozie/action/hadoop/FileSystemActions.java PRE-CREATION
trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1242619
trunk/core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java 1242619
trunk/core/src/main/java/org/apache/oozie/action/hadoop/PrepareActionsDriver.java PRE-CREATION
trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestFileSystemActions.java PRE-CREATION
trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 1242619
trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestLauncher.java 1242619
trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestMapReduceActionError.java 1242619
trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestMapReduceActionExecutor.java 1242619
trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestPigActionExecutor.java 1242619
trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestPrepareActionsDriver.java PRE-CREATION
trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestShellActionExecutor.java 1242619
Diff: https://reviews.apache.org/r/3827/diff
Testing
-------
Ran the entire test suite
Thanks,
Virag
> Moving action prepare FS execution to LauncherMapper
> ----------------------------------------------------
>
> Key: OOZIE-616
> URL: https://issues.apache.org/jira/browse/OOZIE-616
> Project: Oozie
> Issue Type: Improvement
> Reporter: Kiran Nagasubramanian
>
> 1) Motivations:
> ----------------
> a) Currently Oozie Server executes the Prepare logic before posting the Launcher job to Hadoop. If the launcher fails and the execution is retried by Hadoop automatically, the prepare logic is not re-executed. So, for Java/MR/Pig actions, if the launcher fails and it is retried by Hadoop automatically, the prepare logic is not executed again. Once the prepare logic execution is moved to the Launcher, the prepare logic is also re-executed during every retry.
> b) Heavy duty operations like copy that are currently not supported can be supported once the prepare logic has been moved to the Launcher.
> 2) Design choices for the proposed model:
> ------------------------------------------------
> a) Separate classes for different types of actions
> ---------------------------------------------------------
> The different types of actions like FS actions, HCat related actions, etc. can be grouped together in separate classes like FSActions, HCatActions, etc.
> b) Passing the prepare logic to the Launcher
> ---------------------------------------------------
> Launcher needs the Prepare XML block to execute the actions. Oozie server can write the XML block to a file on DFS and then the Launcher could read from there.
> c) Execution of actions through a "Driver"
> -----------------------------------------------
> The Launcher can pass the XML block to the Driver which parses the content and calls corresponding methods that are grouped in different classes.
> 3) Pros and Cons of the design change:
> --------------------------------------------
> a) User facing impacts:
> -------------------------
> i) At present: In case of prepare block failure, there is no impact on the retry of map reduce job since the prepare block would have got executed earlier itself. The impact is seen only when Launcher fails.
>
> As per the proposed design: When the prepare block fails it will impact the retry of the map reduce job.
>
> ii) At present: In the case of prepare block failure, the launcher mapper is not launched at all.
>
> As per the proposed design: The cost of executing the prepare block would go up since it requires the launch of a map only task. This is highly pronounced in case of failures
>
> Thanks a lot for all the input from Oozie team members.
> Please post your suggestions. Thanks.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-616) Moving action prepare FS execution
to LauncherMapper
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13167644#comment-13167644 ]
jiraposter@reviews.apache.org commented on OOZIE-616:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3143/
-----------------------------------------------------------
(Updated 2011-12-12 18:23:46.548715)
Review request for oozie, Mohammad Islam and Angelo K. Huang.
Summary
-------
a) Separate classes for different types of actions
------------------------------------------------
The different types of actions like FS actions, HCat related actions, etc. can be grouped together in separate classes like FSActions, HCatActions, etc.
b) Passing the prepare logic to the Launcher
-----------------------------------------
Launcher needs the Prepare XML block to execute the actions. Oozie server can write the XML block to a file on DFS and then the Launcher could read from there.
c) Execution of actions through a "Driver"
---------------------------------------
The Launcher can pass the XML block to the Driver which parses the content and calls corresponding methods that are grouped in different classes.
This addresses bug OOZIE-616.
https://issues.apache.org/jira/browse/OOZIE-616
Diffs (updated)
-----
trunk/core/src/main/java/org/apache/oozie/action/hadoop/FileSystemActions.java PRE-CREATION
trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1211045
trunk/core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java 1211045
trunk/core/src/main/java/org/apache/oozie/action/hadoop/PrepareActionsDriver.java PRE-CREATION
trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestFileSystemActions.java PRE-CREATION
trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 1211045
trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestLauncher.java 1211045
trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestMapReduceActionError.java 1211045
trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestMapReduceActionExecutor.java 1211045
trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestPigActionExecutor.java 1211045
trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestPrepareActionsDriver.java PRE-CREATION
Diff: https://reviews.apache.org/r/3143/diff
Testing
-------
Yes
Thanks,
Kiran
> Moving action prepare FS execution to LauncherMapper
> ----------------------------------------------------
>
> Key: OOZIE-616
> URL: https://issues.apache.org/jira/browse/OOZIE-616
> Project: Oozie
> Issue Type: Improvement
> Reporter: Kiran Nagasubramanian
>
> 1) Motivations:
> ----------------
> a) Currently Oozie Server executes the Prepare logic before posting the Launcher job to Hadoop. If the launcher fails and the execution is retried by Hadoop automatically, the prepare logic is not re-executed. So, for Java/MR/Pig actions, if the launcher fails and it is retried by Hadoop automatically, the prepare logic is not executed again. Once the prepare logic execution is moved to the Launcher, the prepare logic is also re-executed during every retry.
> b) Heavy duty operations like copy that are currently not supported can be supported once the prepare logic has been moved to the Launcher.
> 2) Design choices for the proposed model:
> ------------------------------------------------
> a) Separate classes for different types of actions
> ---------------------------------------------------------
> The different types of actions like FS actions, HCat related actions, etc. can be grouped together in separate classes like FSActions, HCatActions, etc.
> b) Passing the prepare logic to the Launcher
> ---------------------------------------------------
> Launcher needs the Prepare XML block to execute the actions. Oozie server can write the XML block to a file on DFS and then the Launcher could read from there.
> c) Execution of actions through a "Driver"
> -----------------------------------------------
> The Launcher can pass the XML block to the Driver which parses the content and calls corresponding methods that are grouped in different classes.
> 3) Pros and Cons of the design change:
> --------------------------------------------
> a) User facing impacts:
> -------------------------
> i) At present: In case of prepare block failure, there is no impact on the retry of map reduce job since the prepare block would have got executed earlier itself. The impact is seen only when Launcher fails.
>
> As per the proposed design: When the prepare block fails it will impact the retry of the map reduce job.
>
> ii) At present: In the case of prepare block failure, the launcher mapper is not launched at all.
>
> As per the proposed design: The cost of executing the prepare block would go up since it requires the launch of a map only task. This is highly pronounced in case of failures
>
> Thanks a lot for all the input from Oozie team members.
> Please post your suggestions. Thanks.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-616) Moving action prepare FS execution
to LauncherMapper
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13204860#comment-13204860 ]
jiraposter@reviews.apache.org commented on OOZIE-616:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3143/#review4986
-----------------------------------------------------------
The error handling needs improvement as exceptions are not chained.
trunk/core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java
<https://reviews.apache.org/r/3143/#comment10959>
Exceptions should be chained to prevent loss of details. Any reason to not chain the exception here?
trunk/core/src/main/java/org/apache/oozie/action/hadoop/FileSystemActions.java
<https://reviews.apache.org/r/3143/#comment10975>
Can you add a description of the class?
trunk/core/src/main/java/org/apache/oozie/action/hadoop/FileSystemActions.java
<https://reviews.apache.org/r/3143/#comment10968>
Can we chain the exception?
trunk/core/src/main/java/org/apache/oozie/action/hadoop/FileSystemActions.java
<https://reviews.apache.org/r/3143/#comment10969>
Can we chain the exception?
trunk/core/src/main/java/org/apache/oozie/action/hadoop/FileSystemActions.java
<https://reviews.apache.org/r/3143/#comment10970>
Lack of details will hurt debugging - please include the appropriate error message from the previous line.
trunk/core/src/main/java/org/apache/oozie/action/hadoop/FileSystemActions.java
<https://reviews.apache.org/r/3143/#comment10971>
Lack of details will hurt debugging - please include the appropriate error message from the previous line.
trunk/core/src/main/java/org/apache/oozie/action/hadoop/FileSystemActions.java
<https://reviews.apache.org/r/3143/#comment10972>
Lack of details will hurt debugging - please include the appropriate error message from the previous line.
trunk/core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java
<https://reviews.apache.org/r/3143/#comment10973>
Can you chain the exception here?
trunk/core/src/main/java/org/apache/oozie/action/hadoop/PrepareActionsDriver.java
<https://reviews.apache.org/r/3143/#comment10976>
Can you add a description of the class?
trunk/core/src/main/java/org/apache/oozie/action/hadoop/PrepareActionsDriver.java
<https://reviews.apache.org/r/3143/#comment10978>
The error message may not be representative of the cause since Exception is being caught. If the error message is to be retained then try catching the appropriate exception thrown by getDocumentFromXML, i.e., ParserConfigurationException, SAXException, IOException
trunk/core/src/main/java/org/apache/oozie/action/hadoop/PrepareActionsDriver.java
<https://reviews.apache.org/r/3143/#comment10977>
Can you chain the exception?
trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestFileSystemActions.java
<https://reviews.apache.org/r/3143/#comment10983>
Similar to the tests for PrepareActionsDriver, delete the directory prior to the creation.
trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestFileSystemActions.java
<https://reviews.apache.org/r/3143/#comment10979>
A better test would be to catch LauncherException and check for a reasonable error message and fail for Exception - something like:
} catch (LauncherException le) {
assertEquals(<check for message>);
return;
} catch (Exception ex) {
<fail here>
}
<fail here too> if no exception was thrown
trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestFileSystemActions.java
<https://reviews.apache.org/r/3143/#comment10980>
A better test would be to catch LauncherException and check for a reasonable error message and fail for Exception - something like:
} catch (LauncherException le) {
assertEquals(<check for message>);
return;
} catch (Exception ex) {
<fail here>
}
<fail here too> if no exception was thrown
trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestFileSystemActions.java
<https://reviews.apache.org/r/3143/#comment10981>
This code looks like a copy/paste of the method from PrepareActionsDriver - can this be refactored to avoid the copy?
trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestPrepareActionsDriver.java
<https://reviews.apache.org/r/3143/#comment10982>
A better test would be to catch LauncherException and check for a reasonable error message and fail for Exception - something like:
} catch (LauncherException le) {
assertEquals(<check for message>);
} catch (Exception ex) {
<fail here>
}
- Santhosh
On 2012-02-08 21:06:03, Kiran Nagasubramanian wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/3143/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2012-02-08 21:06:03)
bq.
bq.
bq. Review request for oozie, Mohammad Islam and Angelo K. Huang.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. a) Separate classes for different types of actions
bq. ------------------------------------------------
bq. The different types of actions like FS actions, HCat related actions, etc. can be grouped together in separate classes like FSActions, HCatActions, etc.
bq.
bq. b) Passing the prepare logic to the Launcher
bq. -----------------------------------------
bq. Launcher needs the Prepare XML block to execute the actions. Oozie server can write the XML block to a file on DFS and then the Launcher could read from there.
bq.
bq. c) Execution of actions through a "Driver"
bq. ---------------------------------------
bq. The Launcher can pass the XML block to the Driver which parses the content and calls corresponding methods that are grouped in different classes.
bq.
bq.
bq. This addresses bug OOZIE-616.
bq. https://issues.apache.org/jira/browse/OOZIE-616
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. trunk/core/src/main/java/org/apache/oozie/action/hadoop/FileSystemActions.java PRE-CREATION
bq. trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1213901
bq. trunk/core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java 1213901
bq. trunk/core/src/main/java/org/apache/oozie/action/hadoop/PrepareActionsDriver.java PRE-CREATION
bq. trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestFileSystemActions.java PRE-CREATION
bq. trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 1213901
bq. trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestLauncher.java 1213901
bq. trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestMapReduceActionError.java 1213901
bq. trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestMapReduceActionExecutor.java 1213901
bq. trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestPigActionExecutor.java 1213901
bq. trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestPrepareActionsDriver.java PRE-CREATION
bq. trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestShellActionExecutor.java 1213901
bq.
bq. Diff: https://reviews.apache.org/r/3143/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. Yes
bq.
bq.
bq. Thanks,
bq.
bq. Kiran
bq.
bq.
> Moving action prepare FS execution to LauncherMapper
> ----------------------------------------------------
>
> Key: OOZIE-616
> URL: https://issues.apache.org/jira/browse/OOZIE-616
> Project: Oozie
> Issue Type: Improvement
> Reporter: Kiran Nagasubramanian
>
> 1) Motivations:
> ----------------
> a) Currently Oozie Server executes the Prepare logic before posting the Launcher job to Hadoop. If the launcher fails and the execution is retried by Hadoop automatically, the prepare logic is not re-executed. So, for Java/MR/Pig actions, if the launcher fails and it is retried by Hadoop automatically, the prepare logic is not executed again. Once the prepare logic execution is moved to the Launcher, the prepare logic is also re-executed during every retry.
> b) Heavy duty operations like copy that are currently not supported can be supported once the prepare logic has been moved to the Launcher.
> 2) Design choices for the proposed model:
> ------------------------------------------------
> a) Separate classes for different types of actions
> ---------------------------------------------------------
> The different types of actions like FS actions, HCat related actions, etc. can be grouped together in separate classes like FSActions, HCatActions, etc.
> b) Passing the prepare logic to the Launcher
> ---------------------------------------------------
> Launcher needs the Prepare XML block to execute the actions. Oozie server can write the XML block to a file on DFS and then the Launcher could read from there.
> c) Execution of actions through a "Driver"
> -----------------------------------------------
> The Launcher can pass the XML block to the Driver which parses the content and calls corresponding methods that are grouped in different classes.
> 3) Pros and Cons of the design change:
> --------------------------------------------
> a) User facing impacts:
> -------------------------
> i) At present: In case of prepare block failure, there is no impact on the retry of map reduce job since the prepare block would have got executed earlier itself. The impact is seen only when Launcher fails.
>
> As per the proposed design: When the prepare block fails it will impact the retry of the map reduce job.
>
> ii) At present: In the case of prepare block failure, the launcher mapper is not launched at all.
>
> As per the proposed design: The cost of executing the prepare block would go up since it requires the launch of a map only task. This is highly pronounced in case of failures
>
> Thanks a lot for all the input from Oozie team members.
> Please post your suggestions. Thanks.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (OOZIE-616) Moving action prepare FS execution to
LauncherMapper
Posted by "Alejandro Abdelnur (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alejandro Abdelnur resolved OOZIE-616.
--------------------------------------
Resolution: Fixed
> Moving action prepare FS execution to LauncherMapper
> ----------------------------------------------------
>
> Key: OOZIE-616
> URL: https://issues.apache.org/jira/browse/OOZIE-616
> Project: Oozie
> Issue Type: Improvement
> Reporter: Kiran Nagasubramanian
>
> 1) Motivations:
> ----------------
> a) Currently Oozie Server executes the Prepare logic before posting the Launcher job to Hadoop. If the launcher fails and the execution is retried by Hadoop automatically, the prepare logic is not re-executed. So, for Java/MR/Pig actions, if the launcher fails and it is retried by Hadoop automatically, the prepare logic is not executed again. Once the prepare logic execution is moved to the Launcher, the prepare logic is also re-executed during every retry.
> b) Heavy duty operations like copy that are currently not supported can be supported once the prepare logic has been moved to the Launcher.
> 2) Design choices for the proposed model:
> ------------------------------------------------
> a) Separate classes for different types of actions
> ---------------------------------------------------------
> The different types of actions like FS actions, HCat related actions, etc. can be grouped together in separate classes like FSActions, HCatActions, etc.
> b) Passing the prepare logic to the Launcher
> ---------------------------------------------------
> Launcher needs the Prepare XML block to execute the actions. Oozie server can write the XML block to a file on DFS and then the Launcher could read from there.
> c) Execution of actions through a "Driver"
> -----------------------------------------------
> The Launcher can pass the XML block to the Driver which parses the content and calls corresponding methods that are grouped in different classes.
> 3) Pros and Cons of the design change:
> --------------------------------------------
> a) User facing impacts:
> -------------------------
> i) At present: In case of prepare block failure, there is no impact on the retry of map reduce job since the prepare block would have got executed earlier itself. The impact is seen only when Launcher fails.
>
> As per the proposed design: When the prepare block fails it will impact the retry of the map reduce job.
>
> ii) At present: In the case of prepare block failure, the launcher mapper is not launched at all.
>
> As per the proposed design: The cost of executing the prepare block would go up since it requires the launch of a map only task. This is highly pronounced in case of failures
>
> Thanks a lot for all the input from Oozie team members.
> Please post your suggestions. Thanks.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-616) Moving action prepare FS execution
to LauncherMapper
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13167715#comment-13167715 ]
jiraposter@reviews.apache.org commented on OOZIE-616:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3143/#review3848
-----------------------------------------------------------
trunk/core/src/main/java/org/apache/oozie/action/hadoop/FileSystemActions.java
<https://reviews.apache.org/r/3143/#comment8667>
this should be done in a static {} so it done once and no risk or concurrency issues in the HashSet. Or the variables should be instance variables. and the isSupportedFileSsytemSetConstructor is not needed.
trunk/core/src/main/java/org/apache/oozie/action/hadoop/FileSystemActions.java
<https://reviews.apache.org/r/3143/#comment8668>
why equalIgnoreCase(), equal() should be used, per schema def.
trunk/core/src/main/java/org/apache/oozie/action/hadoop/FileSystemActions.java
<https://reviews.apache.org/r/3143/#comment8669>
ditto as prev comment
trunk/core/src/main/java/org/apache/oozie/action/hadoop/FileSystemActions.java
<https://reviews.apache.org/r/3143/#comment8670>
why this?, this is already done in the constructor
trunk/core/src/main/java/org/apache/oozie/action/hadoop/FileSystemActions.java
<https://reviews.apache.org/r/3143/#comment8671>
remove
trunk/core/src/main/java/org/apache/oozie/action/hadoop/FileSystemActions.java
<https://reviews.apache.org/r/3143/#comment8672>
this eats the LauncherException thrown within the try/catch, it should not. use a catch for it before and rethrow it from there
trunk/core/src/main/java/org/apache/oozie/action/hadoop/FileSystemActions.java
<https://reviews.apache.org/r/3143/#comment8673>
the cause exception should be passed as param
trunk/core/src/main/java/org/apache/oozie/action/hadoop/FileSystemActions.java
<https://reviews.apache.org/r/3143/#comment8674>
same comments as in delete
trunk/core/src/main/java/org/apache/oozie/action/hadoop/FileSystemActions.java
<https://reviews.apache.org/r/3143/#comment8675>
what is this? seems overly complex, we should have a static fixed set of supported FS and we are good, no?
trunk/core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java
<https://reviews.apache.org/r/3143/#comment8676>
as this is to be used by the FS action as well, the more correct name & value would be FILE_SYSTEM_OPS_XML
trunk/core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java
<https://reviews.apache.org/r/3143/#comment8677>
why writing to a separate file instead setting it as property in the action.xml ? (we have to start thinking about less file in HDFS)
trunk/core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java
<https://reviews.apache.org/r/3143/#comment8678>
remove, this is on oozie server
trunk/core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java
<https://reviews.apache.org/r/3143/#comment8679>
executeFileSystemOperations is more appropriate as method name
trunk/core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java
<https://reviews.apache.org/r/3143/#comment8680>
again, if using a property to encode all the fs operations, this is much simpler
trunk/core/src/main/java/org/apache/oozie/action/hadoop/PrepareActionsDriver.java
<https://reviews.apache.org/r/3143/#comment8681>
Why not using JDOM and/or Oozie XML utils? then this class is not needed
- Alejandro
On 2011-12-12 18:23:46, Kiran Nagasubramanian wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/3143/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2011-12-12 18:23:46)
bq.
bq.
bq. Review request for oozie, Mohammad Islam and Angelo K. Huang.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. a) Separate classes for different types of actions
bq. ------------------------------------------------
bq. The different types of actions like FS actions, HCat related actions, etc. can be grouped together in separate classes like FSActions, HCatActions, etc.
bq.
bq. b) Passing the prepare logic to the Launcher
bq. -----------------------------------------
bq. Launcher needs the Prepare XML block to execute the actions. Oozie server can write the XML block to a file on DFS and then the Launcher could read from there.
bq.
bq. c) Execution of actions through a "Driver"
bq. ---------------------------------------
bq. The Launcher can pass the XML block to the Driver which parses the content and calls corresponding methods that are grouped in different classes.
bq.
bq.
bq. This addresses bug OOZIE-616.
bq. https://issues.apache.org/jira/browse/OOZIE-616
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. trunk/core/src/main/java/org/apache/oozie/action/hadoop/FileSystemActions.java PRE-CREATION
bq. trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1211045
bq. trunk/core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java 1211045
bq. trunk/core/src/main/java/org/apache/oozie/action/hadoop/PrepareActionsDriver.java PRE-CREATION
bq. trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestFileSystemActions.java PRE-CREATION
bq. trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 1211045
bq. trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestLauncher.java 1211045
bq. trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestMapReduceActionError.java 1211045
bq. trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestMapReduceActionExecutor.java 1211045
bq. trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestPigActionExecutor.java 1211045
bq. trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestPrepareActionsDriver.java PRE-CREATION
bq.
bq. Diff: https://reviews.apache.org/r/3143/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. Yes
bq.
bq.
bq. Thanks,
bq.
bq. Kiran
bq.
bq.
> Moving action prepare FS execution to LauncherMapper
> ----------------------------------------------------
>
> Key: OOZIE-616
> URL: https://issues.apache.org/jira/browse/OOZIE-616
> Project: Oozie
> Issue Type: Improvement
> Reporter: Kiran Nagasubramanian
>
> 1) Motivations:
> ----------------
> a) Currently Oozie Server executes the Prepare logic before posting the Launcher job to Hadoop. If the launcher fails and the execution is retried by Hadoop automatically, the prepare logic is not re-executed. So, for Java/MR/Pig actions, if the launcher fails and it is retried by Hadoop automatically, the prepare logic is not executed again. Once the prepare logic execution is moved to the Launcher, the prepare logic is also re-executed during every retry.
> b) Heavy duty operations like copy that are currently not supported can be supported once the prepare logic has been moved to the Launcher.
> 2) Design choices for the proposed model:
> ------------------------------------------------
> a) Separate classes for different types of actions
> ---------------------------------------------------------
> The different types of actions like FS actions, HCat related actions, etc. can be grouped together in separate classes like FSActions, HCatActions, etc.
> b) Passing the prepare logic to the Launcher
> ---------------------------------------------------
> Launcher needs the Prepare XML block to execute the actions. Oozie server can write the XML block to a file on DFS and then the Launcher could read from there.
> c) Execution of actions through a "Driver"
> -----------------------------------------------
> The Launcher can pass the XML block to the Driver which parses the content and calls corresponding methods that are grouped in different classes.
> 3) Pros and Cons of the design change:
> --------------------------------------------
> a) User facing impacts:
> -------------------------
> i) At present: In case of prepare block failure, there is no impact on the retry of map reduce job since the prepare block would have got executed earlier itself. The impact is seen only when Launcher fails.
>
> As per the proposed design: When the prepare block fails it will impact the retry of the map reduce job.
>
> ii) At present: In the case of prepare block failure, the launcher mapper is not launched at all.
>
> As per the proposed design: The cost of executing the prepare block would go up since it requires the launch of a map only task. This is highly pronounced in case of failures
>
> Thanks a lot for all the input from Oozie team members.
> Please post your suggestions. Thanks.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-616) Moving action prepare FS execution
to LauncherMapper
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13205916#comment-13205916 ]
jiraposter@reviews.apache.org commented on OOZIE-616:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3827/#review5018
-----------------------------------------------------------
One comment - if you can address/answer it then its good to go.
trunk/core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java
<https://reviews.apache.org/r/3827/#comment11019>
configureFailureEx can be thrown directly. Are you wrapping it to retain the context from configure(jobConf) ?
- Santhosh
On 2012-02-10 03:48:04, Virag Kothari wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/3827/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2012-02-10 03:48:04)
bq.
bq.
bq. Review request for oozie, Santhosh Srinivasan and Alejandro Abdelnur.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. This is Kiran's patch. I am uploading the diff as he is not here. Original diff at https://reviews.apache.org/r/3143/
bq. Thanks Alejandro for testing the patch out. The testcases are fixed now.
bq.
bq.
bq. This addresses bug oozie-616.
bq. https://issues.apache.org/jira/browse/oozie-616
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. trunk/core/src/main/java/org/apache/oozie/action/hadoop/FileSystemActions.java PRE-CREATION
bq. trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1242619
bq. trunk/core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java 1242619
bq. trunk/core/src/main/java/org/apache/oozie/action/hadoop/PrepareActionsDriver.java PRE-CREATION
bq. trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestFileSystemActions.java PRE-CREATION
bq. trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 1242619
bq. trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestLauncher.java 1242619
bq. trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestMapReduceActionError.java 1242619
bq. trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestMapReduceActionExecutor.java 1242619
bq. trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestPigActionExecutor.java 1242619
bq. trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestPrepareActionsDriver.java PRE-CREATION
bq. trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestShellActionExecutor.java 1242619
bq.
bq. Diff: https://reviews.apache.org/r/3827/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. Ran the entire test suite
bq.
bq.
bq. Thanks,
bq.
bq. Virag
bq.
bq.
> Moving action prepare FS execution to LauncherMapper
> ----------------------------------------------------
>
> Key: OOZIE-616
> URL: https://issues.apache.org/jira/browse/OOZIE-616
> Project: Oozie
> Issue Type: Improvement
> Reporter: Kiran Nagasubramanian
>
> 1) Motivations:
> ----------------
> a) Currently Oozie Server executes the Prepare logic before posting the Launcher job to Hadoop. If the launcher fails and the execution is retried by Hadoop automatically, the prepare logic is not re-executed. So, for Java/MR/Pig actions, if the launcher fails and it is retried by Hadoop automatically, the prepare logic is not executed again. Once the prepare logic execution is moved to the Launcher, the prepare logic is also re-executed during every retry.
> b) Heavy duty operations like copy that are currently not supported can be supported once the prepare logic has been moved to the Launcher.
> 2) Design choices for the proposed model:
> ------------------------------------------------
> a) Separate classes for different types of actions
> ---------------------------------------------------------
> The different types of actions like FS actions, HCat related actions, etc. can be grouped together in separate classes like FSActions, HCatActions, etc.
> b) Passing the prepare logic to the Launcher
> ---------------------------------------------------
> Launcher needs the Prepare XML block to execute the actions. Oozie server can write the XML block to a file on DFS and then the Launcher could read from there.
> c) Execution of actions through a "Driver"
> -----------------------------------------------
> The Launcher can pass the XML block to the Driver which parses the content and calls corresponding methods that are grouped in different classes.
> 3) Pros and Cons of the design change:
> --------------------------------------------
> a) User facing impacts:
> -------------------------
> i) At present: In case of prepare block failure, there is no impact on the retry of map reduce job since the prepare block would have got executed earlier itself. The impact is seen only when Launcher fails.
>
> As per the proposed design: When the prepare block fails it will impact the retry of the map reduce job.
>
> ii) At present: In the case of prepare block failure, the launcher mapper is not launched at all.
>
> As per the proposed design: The cost of executing the prepare block would go up since it requires the launch of a map only task. This is highly pronounced in case of failures
>
> Thanks a lot for all the input from Oozie team members.
> Please post your suggestions. Thanks.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-616) Moving action prepare FS execution
to LauncherMapper
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13204020#comment-13204020 ]
jiraposter@reviews.apache.org commented on OOZIE-616:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3143/#review4939
-----------------------------------------------------------
All related testcases are failing after applying the patch to trunk:
Running org.apache.oozie.action.hadoop.TestJavaActionExecutor
Tests run: 14, Failures: 6, Errors: 0, Skipped: 0, Time elapsed: 211.483 sec <<< FAILURE!
Running org.apache.oozie.action.hadoop.TestLauncher
Tests run: 9, Failures: 5, Errors: 0, Skipped: 0, Time elapsed: 169.621 sec <<< FAILURE!
Running org.apache.oozie.action.hadoop.TestMapReduceActionError
Tests run: 3, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 23.445 sec <<< FAILURE!
Running org.apache.oozie.action.hadoop.TestMapReduceActionExecutor
Tests run: 8, Failures: 5, Errors: 0, Skipped: 0, Time elapsed: 117.263 sec <<< FAILURE!
Running org.apache.oozie.action.hadoop.TestPigActionExecutor
Tests run: 9, Failures: 5, Errors: 0, Skipped: 0, Time elapsed: 185.404 sec <<< FAILURE!
Running org.apache.oozie.action.hadoop.TestPrepareActionsDriver
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.791 sec
Running org.apache.oozie.action.hadoop.TestShellActionExecutor
Tests run: 5, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 75.039 sec <<< FAILURE!
Results :
Failed tests:
testSimpestSleSubmitOK(org.apache.oozie.action.hadoop.TestJavaActionExecutor)
testOutputSubmitOK(org.apache.oozie.action.hadoop.TestJavaActionExecutor)
testAdditionalJarSubmitOK(org.apache.oozie.action.hadoop.TestJavaActionExecutor)
testExit0SubmitOK(org.apache.oozie.action.hadoop.TestJavaActionExecutor)
testExit1SubmitError(org.apache.oozie.action.hadoop.TestJavaActionExecutor)
testRecovery(org.apache.oozie.action.hadoop.TestJavaActionExecutor)
testEmpty(org.apache.oozie.action.hadoop.TestLauncher)
testExit0(org.apache.oozie.action.hadoop.TestLauncher)
testOutput(org.apache.oozie.action.hadoop.TestLauncher)
testNewId(org.apache.oozie.action.hadoop.TestLauncher)
testSecurityManager(org.apache.oozie.action.hadoop.TestLauncher)
testMapReduce(org.apache.oozie.action.hadoop.TestMapReduceActionError)
testMapReduce(org.apache.oozie.action.hadoop.TestMapReduceActionExecutor)
testMapReduceWithCredentials(org.apache.oozie.action.hadoop.TestMapReduceActionExecutor)
testStreaming(org.apache.oozie.action.hadoop.TestMapReduceActionExecutor)
testSetExecutionStats_when_user_has_specified_stats_write_TRUE(org.apache.oozie.action.hadoop.TestMapReduceActionExecutor)
testSetExecutionStats_when_user_has_specified_stats_write_FALSE(org.apache.oozie.action.hadoop.TestMapReduceActionExecutor)
testExecutionStats(org.apache.oozie.action.hadoop.TestPigActionExecutor)
testExternalChildIds(org.apache.oozie.action.hadoop.TestPigActionExecutor)
testExecutionStatsWithRetrieveStatsFalse(org.apache.oozie.action.hadoop.TestPigActionExecutor)
testPig(org.apache.oozie.action.hadoop.TestPigActionExecutor)
testUdfPig(org.apache.oozie.action.hadoop.TestPigActionExecutor)
testShellScript(org.apache.oozie.action.hadoop.TestShellActionExecutor)
testPerlScript(org.apache.oozie.action.hadoop.TestShellActionExecutor)
Tests run: 50, Failures: 24, Errors: 0, Skipped: 0
- Alejandro
On 2012-02-08 21:06:03, Kiran Nagasubramanian wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/3143/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2012-02-08 21:06:03)
bq.
bq.
bq. Review request for oozie, Mohammad Islam and Angelo K. Huang.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. a) Separate classes for different types of actions
bq. ------------------------------------------------
bq. The different types of actions like FS actions, HCat related actions, etc. can be grouped together in separate classes like FSActions, HCatActions, etc.
bq.
bq. b) Passing the prepare logic to the Launcher
bq. -----------------------------------------
bq. Launcher needs the Prepare XML block to execute the actions. Oozie server can write the XML block to a file on DFS and then the Launcher could read from there.
bq.
bq. c) Execution of actions through a "Driver"
bq. ---------------------------------------
bq. The Launcher can pass the XML block to the Driver which parses the content and calls corresponding methods that are grouped in different classes.
bq.
bq.
bq. This addresses bug OOZIE-616.
bq. https://issues.apache.org/jira/browse/OOZIE-616
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. trunk/core/src/main/java/org/apache/oozie/action/hadoop/FileSystemActions.java PRE-CREATION
bq. trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1213901
bq. trunk/core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java 1213901
bq. trunk/core/src/main/java/org/apache/oozie/action/hadoop/PrepareActionsDriver.java PRE-CREATION
bq. trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestFileSystemActions.java PRE-CREATION
bq. trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 1213901
bq. trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestLauncher.java 1213901
bq. trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestMapReduceActionError.java 1213901
bq. trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestMapReduceActionExecutor.java 1213901
bq. trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestPigActionExecutor.java 1213901
bq. trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestPrepareActionsDriver.java PRE-CREATION
bq. trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestShellActionExecutor.java 1213901
bq.
bq. Diff: https://reviews.apache.org/r/3143/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. Yes
bq.
bq.
bq. Thanks,
bq.
bq. Kiran
bq.
bq.
> Moving action prepare FS execution to LauncherMapper
> ----------------------------------------------------
>
> Key: OOZIE-616
> URL: https://issues.apache.org/jira/browse/OOZIE-616
> Project: Oozie
> Issue Type: Improvement
> Reporter: Kiran Nagasubramanian
>
> 1) Motivations:
> ----------------
> a) Currently Oozie Server executes the Prepare logic before posting the Launcher job to Hadoop. If the launcher fails and the execution is retried by Hadoop automatically, the prepare logic is not re-executed. So, for Java/MR/Pig actions, if the launcher fails and it is retried by Hadoop automatically, the prepare logic is not executed again. Once the prepare logic execution is moved to the Launcher, the prepare logic is also re-executed during every retry.
> b) Heavy duty operations like copy that are currently not supported can be supported once the prepare logic has been moved to the Launcher.
> 2) Design choices for the proposed model:
> ------------------------------------------------
> a) Separate classes for different types of actions
> ---------------------------------------------------------
> The different types of actions like FS actions, HCat related actions, etc. can be grouped together in separate classes like FSActions, HCatActions, etc.
> b) Passing the prepare logic to the Launcher
> ---------------------------------------------------
> Launcher needs the Prepare XML block to execute the actions. Oozie server can write the XML block to a file on DFS and then the Launcher could read from there.
> c) Execution of actions through a "Driver"
> -----------------------------------------------
> The Launcher can pass the XML block to the Driver which parses the content and calls corresponding methods that are grouped in different classes.
> 3) Pros and Cons of the design change:
> --------------------------------------------
> a) User facing impacts:
> -------------------------
> i) At present: In case of prepare block failure, there is no impact on the retry of map reduce job since the prepare block would have got executed earlier itself. The impact is seen only when Launcher fails.
>
> As per the proposed design: When the prepare block fails it will impact the retry of the map reduce job.
>
> ii) At present: In the case of prepare block failure, the launcher mapper is not launched at all.
>
> As per the proposed design: The cost of executing the prepare block would go up since it requires the launch of a map only task. This is highly pronounced in case of failures
>
> Thanks a lot for all the input from Oozie team members.
> Please post your suggestions. Thanks.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-616) Moving action prepare FS execution
to LauncherMapper
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13171321#comment-13171321 ]
jiraposter@reviews.apache.org commented on OOZIE-616:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3143/
-----------------------------------------------------------
(Updated 2011-12-16 23:55:14.096242)
Review request for oozie, Mohammad Islam and Angelo K. Huang.
Changes
-------
Have incorporated the suggested change.
Summary
-------
a) Separate classes for different types of actions
------------------------------------------------
The different types of actions like FS actions, HCat related actions, etc. can be grouped together in separate classes like FSActions, HCatActions, etc.
b) Passing the prepare logic to the Launcher
-----------------------------------------
Launcher needs the Prepare XML block to execute the actions. Oozie server can write the XML block to a file on DFS and then the Launcher could read from there.
c) Execution of actions through a "Driver"
---------------------------------------
The Launcher can pass the XML block to the Driver which parses the content and calls corresponding methods that are grouped in different classes.
This addresses bug OOZIE-616.
https://issues.apache.org/jira/browse/OOZIE-616
Diffs (updated)
-----
trunk/core/src/main/java/org/apache/oozie/action/hadoop/FileSystemActions.java PRE-CREATION
trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1213901
trunk/core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java 1213901
trunk/core/src/main/java/org/apache/oozie/action/hadoop/PrepareActionsDriver.java PRE-CREATION
trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestFileSystemActions.java PRE-CREATION
trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 1213901
trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestLauncher.java 1213901
trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestMapReduceActionError.java 1213901
trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestMapReduceActionExecutor.java 1213901
trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestPigActionExecutor.java 1213901
trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestPrepareActionsDriver.java PRE-CREATION
trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestShellActionExecutor.java 1213901
Diff: https://reviews.apache.org/r/3143/diff
Testing
-------
Yes
Thanks,
Kiran
> Moving action prepare FS execution to LauncherMapper
> ----------------------------------------------------
>
> Key: OOZIE-616
> URL: https://issues.apache.org/jira/browse/OOZIE-616
> Project: Oozie
> Issue Type: Improvement
> Reporter: Kiran Nagasubramanian
>
> 1) Motivations:
> ----------------
> a) Currently Oozie Server executes the Prepare logic before posting the Launcher job to Hadoop. If the launcher fails and the execution is retried by Hadoop automatically, the prepare logic is not re-executed. So, for Java/MR/Pig actions, if the launcher fails and it is retried by Hadoop automatically, the prepare logic is not executed again. Once the prepare logic execution is moved to the Launcher, the prepare logic is also re-executed during every retry.
> b) Heavy duty operations like copy that are currently not supported can be supported once the prepare logic has been moved to the Launcher.
> 2) Design choices for the proposed model:
> ------------------------------------------------
> a) Separate classes for different types of actions
> ---------------------------------------------------------
> The different types of actions like FS actions, HCat related actions, etc. can be grouped together in separate classes like FSActions, HCatActions, etc.
> b) Passing the prepare logic to the Launcher
> ---------------------------------------------------
> Launcher needs the Prepare XML block to execute the actions. Oozie server can write the XML block to a file on DFS and then the Launcher could read from there.
> c) Execution of actions through a "Driver"
> -----------------------------------------------
> The Launcher can pass the XML block to the Driver which parses the content and calls corresponding methods that are grouped in different classes.
> 3) Pros and Cons of the design change:
> --------------------------------------------
> a) User facing impacts:
> -------------------------
> i) At present: In case of prepare block failure, there is no impact on the retry of map reduce job since the prepare block would have got executed earlier itself. The impact is seen only when Launcher fails.
>
> As per the proposed design: When the prepare block fails it will impact the retry of the map reduce job.
>
> ii) At present: In the case of prepare block failure, the launcher mapper is not launched at all.
>
> As per the proposed design: The cost of executing the prepare block would go up since it requires the launch of a map only task. This is highly pronounced in case of failures
>
> Thanks a lot for all the input from Oozie team members.
> Please post your suggestions. Thanks.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-616) Moving action prepare FS execution
to LauncherMapper
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13206592#comment-13206592 ]
jiraposter@reviews.apache.org commented on OOZIE-616:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3827/#review5037
-----------------------------------------------------------
Ship it!
+1
- Santhosh
On 2012-02-11 00:33:59, Virag Kothari wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/3827/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2012-02-11 00:33:59)
bq.
bq.
bq. Review request for oozie, Santhosh Srinivasan and Alejandro Abdelnur.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. This is Kiran's patch. I am uploading the diff as he is not here. Original diff at https://reviews.apache.org/r/3143/
bq. Thanks Alejandro for testing the patch out. The testcases are fixed now.
bq.
bq.
bq. This addresses bug oozie-616.
bq. https://issues.apache.org/jira/browse/oozie-616
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. trunk/core/src/main/java/org/apache/oozie/action/hadoop/FileSystemActions.java PRE-CREATION
bq. trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1242619
bq. trunk/core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java 1242619
bq. trunk/core/src/main/java/org/apache/oozie/action/hadoop/PrepareActionsDriver.java PRE-CREATION
bq. trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestFileSystemActions.java PRE-CREATION
bq. trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 1242619
bq. trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestLauncher.java 1242619
bq. trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestMapReduceActionError.java 1242619
bq. trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestMapReduceActionExecutor.java 1242619
bq. trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestPigActionExecutor.java 1242619
bq. trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestPrepareActionsDriver.java PRE-CREATION
bq. trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestShellActionExecutor.java 1242619
bq.
bq. Diff: https://reviews.apache.org/r/3827/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. Ran the entire test suite
bq.
bq.
bq. Thanks,
bq.
bq. Virag
bq.
bq.
> Moving action prepare FS execution to LauncherMapper
> ----------------------------------------------------
>
> Key: OOZIE-616
> URL: https://issues.apache.org/jira/browse/OOZIE-616
> Project: Oozie
> Issue Type: Improvement
> Reporter: Kiran Nagasubramanian
>
> 1) Motivations:
> ----------------
> a) Currently Oozie Server executes the Prepare logic before posting the Launcher job to Hadoop. If the launcher fails and the execution is retried by Hadoop automatically, the prepare logic is not re-executed. So, for Java/MR/Pig actions, if the launcher fails and it is retried by Hadoop automatically, the prepare logic is not executed again. Once the prepare logic execution is moved to the Launcher, the prepare logic is also re-executed during every retry.
> b) Heavy duty operations like copy that are currently not supported can be supported once the prepare logic has been moved to the Launcher.
> 2) Design choices for the proposed model:
> ------------------------------------------------
> a) Separate classes for different types of actions
> ---------------------------------------------------------
> The different types of actions like FS actions, HCat related actions, etc. can be grouped together in separate classes like FSActions, HCatActions, etc.
> b) Passing the prepare logic to the Launcher
> ---------------------------------------------------
> Launcher needs the Prepare XML block to execute the actions. Oozie server can write the XML block to a file on DFS and then the Launcher could read from there.
> c) Execution of actions through a "Driver"
> -----------------------------------------------
> The Launcher can pass the XML block to the Driver which parses the content and calls corresponding methods that are grouped in different classes.
> 3) Pros and Cons of the design change:
> --------------------------------------------
> a) User facing impacts:
> -------------------------
> i) At present: In case of prepare block failure, there is no impact on the retry of map reduce job since the prepare block would have got executed earlier itself. The impact is seen only when Launcher fails.
>
> As per the proposed design: When the prepare block fails it will impact the retry of the map reduce job.
>
> ii) At present: In the case of prepare block failure, the launcher mapper is not launched at all.
>
> As per the proposed design: The cost of executing the prepare block would go up since it requires the launch of a map only task. This is highly pronounced in case of failures
>
> Thanks a lot for all the input from Oozie team members.
> Please post your suggestions. Thanks.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-616) Moving action prepare FS execution
to LauncherMapper
Posted by "Mohammad Kamrul Islam (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13160594#comment-13160594 ]
Mohammad Kamrul Islam commented on OOZIE-616:
---------------------------------------------
I agree that FS action should also move to LM.
In the first step, we could move the prepare and then in another stage we could move the FS action.
I would suggest Kiran to create a new JIRA for FS action.
> Moving action prepare FS execution to LauncherMapper
> ----------------------------------------------------
>
> Key: OOZIE-616
> URL: https://issues.apache.org/jira/browse/OOZIE-616
> Project: Oozie
> Issue Type: Improvement
> Reporter: Kiran Nagasubramanian
>
> 1) Motivations:
> ----------------
> a) Currently Oozie Server executes the Prepare logic before posting the Launcher job to Hadoop. If the launcher fails and the execution is retried by Hadoop automatically, the prepare logic is not re-executed. So, for Java/MR/Pig actions, if the launcher fails and it is retried by Hadoop automatically, the prepare logic is not executed again. Once the prepare logic execution is moved to the Launcher, the prepare logic is also re-executed during every retry.
> b) Heavy duty operations like copy that are currently not supported can be supported once the prepare logic has been moved to the Launcher.
> 2) Design choices for the proposed model:
> ------------------------------------------------
> a) Separate classes for different types of actions
> ---------------------------------------------------------
> The different types of actions like FS actions, HCat related actions, etc. can be grouped together in separate classes like FSActions, HCatActions, etc.
> b) Passing the prepare logic to the Launcher
> ---------------------------------------------------
> Launcher needs the Prepare XML block to execute the actions. Oozie server can write the XML block to a file on DFS and then the Launcher could read from there.
> c) Execution of actions through a "Driver"
> -----------------------------------------------
> The Launcher can pass the XML block to the Driver which parses the content and calls corresponding methods that are grouped in different classes.
> 3) Pros and Cons of the design change:
> --------------------------------------------
> a) User facing impacts:
> -------------------------
> i) At present: In case of prepare block failure, there is no impact on the retry of map reduce job since the prepare block would have got executed earlier itself. The impact is seen only when Launcher fails.
>
> As per the proposed design: When the prepare block fails it will impact the retry of the map reduce job.
>
> ii) At present: In the case of prepare block failure, the launcher mapper is not launched at all.
>
> As per the proposed design: The cost of executing the prepare block would go up since it requires the launch of a map only task. This is highly pronounced in case of failures
>
> Thanks a lot for all the input from Oozie team members.
> Please post your suggestions. Thanks.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-616) Moving action prepare FS execution
to LauncherMapper
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13171136#comment-13171136 ]
jiraposter@reviews.apache.org commented on OOZIE-616:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3143/#review3949
-----------------------------------------------------------
trunk/core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java
<https://reviews.apache.org/r/3143/#comment8918>
A generic name independent of File system actions has been given since there are plans of supporting HCat actions also, as a part of prepare block, going forward.
trunk/core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java
<https://reviews.apache.org/r/3143/#comment8919>
Great suggestion! Have changed this..
trunk/core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java
<https://reviews.apache.org/r/3143/#comment8920>
Same as earlier. Since we are planning to support HCat actions also in th prepare block, the name needs to be generic.
trunk/core/src/main/java/org/apache/oozie/action/hadoop/PrepareActionsDriver.java
<https://reviews.apache.org/r/3143/#comment8921>
If we use JDOM or XMLUtils we need to have the corresponding JAR files in the distributed cache. This becomes heavy. This is pronounced even more in case of XMLUtils since the class by itself needs many JARs to be included. Hence, decided to parse the XML using the functionalities that came as a part of the standard java distribution. In addition to this, this class acts an intermediate driver, as the name suggests, and calls the corresponding type of actions(File system/HCat etc.) which are/will be grouped together in separate classes.
- Kiran
On 2011-12-16 18:59:50, Kiran Nagasubramanian wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/3143/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2011-12-16 18:59:50)
bq.
bq.
bq. Review request for oozie, Mohammad Islam and Angelo K. Huang.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. a) Separate classes for different types of actions
bq. ------------------------------------------------
bq. The different types of actions like FS actions, HCat related actions, etc. can be grouped together in separate classes like FSActions, HCatActions, etc.
bq.
bq. b) Passing the prepare logic to the Launcher
bq. -----------------------------------------
bq. Launcher needs the Prepare XML block to execute the actions. Oozie server can write the XML block to a file on DFS and then the Launcher could read from there.
bq.
bq. c) Execution of actions through a "Driver"
bq. ---------------------------------------
bq. The Launcher can pass the XML block to the Driver which parses the content and calls corresponding methods that are grouped in different classes.
bq.
bq.
bq. This addresses bug OOZIE-616.
bq. https://issues.apache.org/jira/browse/OOZIE-616
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 1213901
bq. trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestFileSystemActions.java PRE-CREATION
bq. trunk/core/src/main/java/org/apache/oozie/action/hadoop/PrepareActionsDriver.java PRE-CREATION
bq. trunk/core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java 1213901
bq. trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1213901
bq. trunk/core/src/main/java/org/apache/oozie/action/hadoop/FileSystemActions.java PRE-CREATION
bq. trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestLauncher.java 1213901
bq. trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestMapReduceActionError.java 1213901
bq. trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestMapReduceActionExecutor.java 1213901
bq. trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestPigActionExecutor.java 1213901
bq. trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestPrepareActionsDriver.java PRE-CREATION
bq. trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestShellActionExecutor.java 1213901
bq.
bq. Diff: https://reviews.apache.org/r/3143/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. Yes
bq.
bq.
bq. Thanks,
bq.
bq. Kiran
bq.
bq.
> Moving action prepare FS execution to LauncherMapper
> ----------------------------------------------------
>
> Key: OOZIE-616
> URL: https://issues.apache.org/jira/browse/OOZIE-616
> Project: Oozie
> Issue Type: Improvement
> Reporter: Kiran Nagasubramanian
>
> 1) Motivations:
> ----------------
> a) Currently Oozie Server executes the Prepare logic before posting the Launcher job to Hadoop. If the launcher fails and the execution is retried by Hadoop automatically, the prepare logic is not re-executed. So, for Java/MR/Pig actions, if the launcher fails and it is retried by Hadoop automatically, the prepare logic is not executed again. Once the prepare logic execution is moved to the Launcher, the prepare logic is also re-executed during every retry.
> b) Heavy duty operations like copy that are currently not supported can be supported once the prepare logic has been moved to the Launcher.
> 2) Design choices for the proposed model:
> ------------------------------------------------
> a) Separate classes for different types of actions
> ---------------------------------------------------------
> The different types of actions like FS actions, HCat related actions, etc. can be grouped together in separate classes like FSActions, HCatActions, etc.
> b) Passing the prepare logic to the Launcher
> ---------------------------------------------------
> Launcher needs the Prepare XML block to execute the actions. Oozie server can write the XML block to a file on DFS and then the Launcher could read from there.
> c) Execution of actions through a "Driver"
> -----------------------------------------------
> The Launcher can pass the XML block to the Driver which parses the content and calls corresponding methods that are grouped in different classes.
> 3) Pros and Cons of the design change:
> --------------------------------------------
> a) User facing impacts:
> -------------------------
> i) At present: In case of prepare block failure, there is no impact on the retry of map reduce job since the prepare block would have got executed earlier itself. The impact is seen only when Launcher fails.
>
> As per the proposed design: When the prepare block fails it will impact the retry of the map reduce job.
>
> ii) At present: In the case of prepare block failure, the launcher mapper is not launched at all.
>
> As per the proposed design: The cost of executing the prepare block would go up since it requires the launch of a map only task. This is highly pronounced in case of failures
>
> Thanks a lot for all the input from Oozie team members.
> Please post your suggestions. Thanks.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-616) Moving action prepare FS execution
to LauncherMapper
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13205110#comment-13205110 ]
jiraposter@reviews.apache.org commented on OOZIE-616:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3827/
-----------------------------------------------------------
(Updated 2012-02-10 00:41:55.595215)
Review request for oozie, Santhosh Srinivasan and Alejandro Abdelnur.
Changes
-------
Addressing Santhosh's comments
Summary
-------
This is Kiran's patch. I am uploading the diff as he is not here. Original diff at https://reviews.apache.org/r/3143/
Thanks Alejandro for testing the patch out. The testcases are fixed now.
This addresses bug oozie-616.
https://issues.apache.org/jira/browse/oozie-616
Diffs (updated)
-----
trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1242619
trunk/core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java 1242619
trunk/core/src/main/java/org/apache/oozie/action/hadoop/FileSystemActions.java PRE-CREATION
trunk/core/src/main/java/org/apache/oozie/action/hadoop/PrepareActionsDriver.java PRE-CREATION
trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestFileSystemActions.java PRE-CREATION
trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 1242619
trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestLauncher.java 1242619
trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestMapReduceActionError.java 1242619
trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestMapReduceActionExecutor.java 1242619
trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestPigActionExecutor.java 1242619
trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestPrepareActionsDriver.java PRE-CREATION
trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestShellActionExecutor.java 1242619
Diff: https://reviews.apache.org/r/3827/diff
Testing
-------
Ran the entire test suite
Thanks,
Virag
> Moving action prepare FS execution to LauncherMapper
> ----------------------------------------------------
>
> Key: OOZIE-616
> URL: https://issues.apache.org/jira/browse/OOZIE-616
> Project: Oozie
> Issue Type: Improvement
> Reporter: Kiran Nagasubramanian
>
> 1) Motivations:
> ----------------
> a) Currently Oozie Server executes the Prepare logic before posting the Launcher job to Hadoop. If the launcher fails and the execution is retried by Hadoop automatically, the prepare logic is not re-executed. So, for Java/MR/Pig actions, if the launcher fails and it is retried by Hadoop automatically, the prepare logic is not executed again. Once the prepare logic execution is moved to the Launcher, the prepare logic is also re-executed during every retry.
> b) Heavy duty operations like copy that are currently not supported can be supported once the prepare logic has been moved to the Launcher.
> 2) Design choices for the proposed model:
> ------------------------------------------------
> a) Separate classes for different types of actions
> ---------------------------------------------------------
> The different types of actions like FS actions, HCat related actions, etc. can be grouped together in separate classes like FSActions, HCatActions, etc.
> b) Passing the prepare logic to the Launcher
> ---------------------------------------------------
> Launcher needs the Prepare XML block to execute the actions. Oozie server can write the XML block to a file on DFS and then the Launcher could read from there.
> c) Execution of actions through a "Driver"
> -----------------------------------------------
> The Launcher can pass the XML block to the Driver which parses the content and calls corresponding methods that are grouped in different classes.
> 3) Pros and Cons of the design change:
> --------------------------------------------
> a) User facing impacts:
> -------------------------
> i) At present: In case of prepare block failure, there is no impact on the retry of map reduce job since the prepare block would have got executed earlier itself. The impact is seen only when Launcher fails.
>
> As per the proposed design: When the prepare block fails it will impact the retry of the map reduce job.
>
> ii) At present: In the case of prepare block failure, the launcher mapper is not launched at all.
>
> As per the proposed design: The cost of executing the prepare block would go up since it requires the launch of a map only task. This is highly pronounced in case of failures
>
> Thanks a lot for all the input from Oozie team members.
> Please post your suggestions. Thanks.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-616) Moving action prepare FS execution
to LauncherMapper
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13203983#comment-13203983 ]
jiraposter@reviews.apache.org commented on OOZIE-616:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3143/
-----------------------------------------------------------
(Updated 2012-02-08 21:06:03.849960)
Review request for oozie, Mohammad Islam and Angelo K. Huang.
Summary (updated)
-------
a) Separate classes for different types of actions
------------------------------------------------
The different types of actions like FS actions, HCat related actions, etc. can be grouped together in separate classes like FSActions, HCatActions, etc.
b) Passing the prepare logic to the Launcher
-----------------------------------------
Launcher needs the Prepare XML block to execute the actions. Oozie server can write the XML block to a file on DFS and then the Launcher could read from there.
c) Execution of actions through a "Driver"
---------------------------------------
The Launcher can pass the XML block to the Driver which parses the content and calls corresponding methods that are grouped in different classes.
This addresses bug OOZIE-616.
https://issues.apache.org/jira/browse/OOZIE-616
Diffs
-----
trunk/core/src/main/java/org/apache/oozie/action/hadoop/FileSystemActions.java PRE-CREATION
trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1213901
trunk/core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java 1213901
trunk/core/src/main/java/org/apache/oozie/action/hadoop/PrepareActionsDriver.java PRE-CREATION
trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestFileSystemActions.java PRE-CREATION
trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 1213901
trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestLauncher.java 1213901
trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestMapReduceActionError.java 1213901
trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestMapReduceActionExecutor.java 1213901
trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestPigActionExecutor.java 1213901
trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestPrepareActionsDriver.java PRE-CREATION
trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestShellActionExecutor.java 1213901
Diff: https://reviews.apache.org/r/3143/diff
Testing
-------
Yes
Thanks,
Kiran
> Moving action prepare FS execution to LauncherMapper
> ----------------------------------------------------
>
> Key: OOZIE-616
> URL: https://issues.apache.org/jira/browse/OOZIE-616
> Project: Oozie
> Issue Type: Improvement
> Reporter: Kiran Nagasubramanian
>
> 1) Motivations:
> ----------------
> a) Currently Oozie Server executes the Prepare logic before posting the Launcher job to Hadoop. If the launcher fails and the execution is retried by Hadoop automatically, the prepare logic is not re-executed. So, for Java/MR/Pig actions, if the launcher fails and it is retried by Hadoop automatically, the prepare logic is not executed again. Once the prepare logic execution is moved to the Launcher, the prepare logic is also re-executed during every retry.
> b) Heavy duty operations like copy that are currently not supported can be supported once the prepare logic has been moved to the Launcher.
> 2) Design choices for the proposed model:
> ------------------------------------------------
> a) Separate classes for different types of actions
> ---------------------------------------------------------
> The different types of actions like FS actions, HCat related actions, etc. can be grouped together in separate classes like FSActions, HCatActions, etc.
> b) Passing the prepare logic to the Launcher
> ---------------------------------------------------
> Launcher needs the Prepare XML block to execute the actions. Oozie server can write the XML block to a file on DFS and then the Launcher could read from there.
> c) Execution of actions through a "Driver"
> -----------------------------------------------
> The Launcher can pass the XML block to the Driver which parses the content and calls corresponding methods that are grouped in different classes.
> 3) Pros and Cons of the design change:
> --------------------------------------------
> a) User facing impacts:
> -------------------------
> i) At present: In case of prepare block failure, there is no impact on the retry of map reduce job since the prepare block would have got executed earlier itself. The impact is seen only when Launcher fails.
>
> As per the proposed design: When the prepare block fails it will impact the retry of the map reduce job.
>
> ii) At present: In the case of prepare block failure, the launcher mapper is not launched at all.
>
> As per the proposed design: The cost of executing the prepare block would go up since it requires the launch of a map only task. This is highly pronounced in case of failures
>
> Thanks a lot for all the input from Oozie team members.
> Please post your suggestions. Thanks.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-616) Moving action prepare FS execution
to LauncherMapper
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13171214#comment-13171214 ]
jiraposter@reviews.apache.org commented on OOZIE-616:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3143/#review3952
-----------------------------------------------------------
trunk/core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java
<https://reviews.apache.org/r/3143/#comment8923>
we should print/propagate the exception, else the root cause is lost and we won't know why is failing
- Alejandro
On 2011-12-16 19:34:53, Kiran Nagasubramanian wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/3143/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2011-12-16 19:34:53)
bq.
bq.
bq. Review request for oozie, Mohammad Islam and Angelo K. Huang.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. a) Separate classes for different types of actions
bq. ------------------------------------------------
bq. The different types of actions like FS actions, HCat related actions, etc. can be grouped together in separate classes like FSActions, HCatActions, etc.
bq.
bq. b) Passing the prepare logic to the Launcher
bq. -----------------------------------------
bq. Launcher needs the Prepare XML block to execute the actions. Oozie server can write the XML block to a file on DFS and then the Launcher could read from there.
bq.
bq. c) Execution of actions through a "Driver"
bq. ---------------------------------------
bq. The Launcher can pass the XML block to the Driver which parses the content and calls corresponding methods that are grouped in different classes.
bq.
bq.
bq. This addresses bug OOZIE-616.
bq. https://issues.apache.org/jira/browse/OOZIE-616
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. trunk/core/src/main/java/org/apache/oozie/action/hadoop/FileSystemActions.java PRE-CREATION
bq. trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1213901
bq. trunk/core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java 1213901
bq. trunk/core/src/main/java/org/apache/oozie/action/hadoop/PrepareActionsDriver.java PRE-CREATION
bq. trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestFileSystemActions.java PRE-CREATION
bq. trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 1213901
bq. trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestLauncher.java 1213901
bq. trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestMapReduceActionError.java 1213901
bq. trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestMapReduceActionExecutor.java 1213901
bq. trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestPigActionExecutor.java 1213901
bq. trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestPrepareActionsDriver.java PRE-CREATION
bq. trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestShellActionExecutor.java 1213901
bq.
bq. Diff: https://reviews.apache.org/r/3143/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. Yes
bq.
bq.
bq. Thanks,
bq.
bq. Kiran
bq.
bq.
> Moving action prepare FS execution to LauncherMapper
> ----------------------------------------------------
>
> Key: OOZIE-616
> URL: https://issues.apache.org/jira/browse/OOZIE-616
> Project: Oozie
> Issue Type: Improvement
> Reporter: Kiran Nagasubramanian
>
> 1) Motivations:
> ----------------
> a) Currently Oozie Server executes the Prepare logic before posting the Launcher job to Hadoop. If the launcher fails and the execution is retried by Hadoop automatically, the prepare logic is not re-executed. So, for Java/MR/Pig actions, if the launcher fails and it is retried by Hadoop automatically, the prepare logic is not executed again. Once the prepare logic execution is moved to the Launcher, the prepare logic is also re-executed during every retry.
> b) Heavy duty operations like copy that are currently not supported can be supported once the prepare logic has been moved to the Launcher.
> 2) Design choices for the proposed model:
> ------------------------------------------------
> a) Separate classes for different types of actions
> ---------------------------------------------------------
> The different types of actions like FS actions, HCat related actions, etc. can be grouped together in separate classes like FSActions, HCatActions, etc.
> b) Passing the prepare logic to the Launcher
> ---------------------------------------------------
> Launcher needs the Prepare XML block to execute the actions. Oozie server can write the XML block to a file on DFS and then the Launcher could read from there.
> c) Execution of actions through a "Driver"
> -----------------------------------------------
> The Launcher can pass the XML block to the Driver which parses the content and calls corresponding methods that are grouped in different classes.
> 3) Pros and Cons of the design change:
> --------------------------------------------
> a) User facing impacts:
> -------------------------
> i) At present: In case of prepare block failure, there is no impact on the retry of map reduce job since the prepare block would have got executed earlier itself. The impact is seen only when Launcher fails.
>
> As per the proposed design: When the prepare block fails it will impact the retry of the map reduce job.
>
> ii) At present: In the case of prepare block failure, the launcher mapper is not launched at all.
>
> As per the proposed design: The cost of executing the prepare block would go up since it requires the launch of a map only task. This is highly pronounced in case of failures
>
> Thanks a lot for all the input from Oozie team members.
> Please post your suggestions. Thanks.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-616) Moving action prepare FS execution
to LauncherMapper
Posted by "Alejandro Abdelnur (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13160648#comment-13160648 ]
Alejandro Abdelnur commented on OOZIE-616:
------------------------------------------
doing in two steps is fine. Just keep in mind that the code doing prepare and fs action is exactly the same.
> Moving action prepare FS execution to LauncherMapper
> ----------------------------------------------------
>
> Key: OOZIE-616
> URL: https://issues.apache.org/jira/browse/OOZIE-616
> Project: Oozie
> Issue Type: Improvement
> Reporter: Kiran Nagasubramanian
>
> 1) Motivations:
> ----------------
> a) Currently Oozie Server executes the Prepare logic before posting the Launcher job to Hadoop. If the launcher fails and the execution is retried by Hadoop automatically, the prepare logic is not re-executed. So, for Java/MR/Pig actions, if the launcher fails and it is retried by Hadoop automatically, the prepare logic is not executed again. Once the prepare logic execution is moved to the Launcher, the prepare logic is also re-executed during every retry.
> b) Heavy duty operations like copy that are currently not supported can be supported once the prepare logic has been moved to the Launcher.
> 2) Design choices for the proposed model:
> ------------------------------------------------
> a) Separate classes for different types of actions
> ---------------------------------------------------------
> The different types of actions like FS actions, HCat related actions, etc. can be grouped together in separate classes like FSActions, HCatActions, etc.
> b) Passing the prepare logic to the Launcher
> ---------------------------------------------------
> Launcher needs the Prepare XML block to execute the actions. Oozie server can write the XML block to a file on DFS and then the Launcher could read from there.
> c) Execution of actions through a "Driver"
> -----------------------------------------------
> The Launcher can pass the XML block to the Driver which parses the content and calls corresponding methods that are grouped in different classes.
> 3) Pros and Cons of the design change:
> --------------------------------------------
> a) User facing impacts:
> -------------------------
> i) At present: In case of prepare block failure, there is no impact on the retry of map reduce job since the prepare block would have got executed earlier itself. The impact is seen only when Launcher fails.
>
> As per the proposed design: When the prepare block fails it will impact the retry of the map reduce job.
>
> ii) At present: In the case of prepare block failure, the launcher mapper is not launched at all.
>
> As per the proposed design: The cost of executing the prepare block would go up since it requires the launch of a map only task. This is highly pronounced in case of failures
>
> Thanks a lot for all the input from Oozie team members.
> Please post your suggestions. Thanks.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-616) Moving action prepare FS execution
to LauncherMapper
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13204720#comment-13204720 ]
jiraposter@reviews.apache.org commented on OOZIE-616:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3827/
-----------------------------------------------------------
Review request for oozie and Alejandro Abdelnur.
Summary
-------
This is Kiran's patch. I am uploading the diff as he is not here. Original diff at https://reviews.apache.org/r/3143/
Thanks Alejandro for testing the patch out. The testcases are fixed now.
This addresses bug oozie-616.
https://issues.apache.org/jira/browse/oozie-616
Diffs
-----
trunk/core/src/main/java/org/apache/oozie/action/hadoop/FileSystemActions.java PRE-CREATION
trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1242416
trunk/core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java 1242416
trunk/core/src/main/java/org/apache/oozie/action/hadoop/PrepareActionsDriver.java PRE-CREATION
trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestFileSystemActions.java PRE-CREATION
trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 1242416
trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestLauncher.java 1242416
trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestMapReduceActionError.java 1242416
trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestMapReduceActionExecutor.java 1242416
trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestPigActionExecutor.java 1242416
trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestPrepareActionsDriver.java PRE-CREATION
trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestShellActionExecutor.java 1242416
Diff: https://reviews.apache.org/r/3827/diff
Testing
-------
Ran the entire test suite
Thanks,
Virag
> Moving action prepare FS execution to LauncherMapper
> ----------------------------------------------------
>
> Key: OOZIE-616
> URL: https://issues.apache.org/jira/browse/OOZIE-616
> Project: Oozie
> Issue Type: Improvement
> Reporter: Kiran Nagasubramanian
>
> 1) Motivations:
> ----------------
> a) Currently Oozie Server executes the Prepare logic before posting the Launcher job to Hadoop. If the launcher fails and the execution is retried by Hadoop automatically, the prepare logic is not re-executed. So, for Java/MR/Pig actions, if the launcher fails and it is retried by Hadoop automatically, the prepare logic is not executed again. Once the prepare logic execution is moved to the Launcher, the prepare logic is also re-executed during every retry.
> b) Heavy duty operations like copy that are currently not supported can be supported once the prepare logic has been moved to the Launcher.
> 2) Design choices for the proposed model:
> ------------------------------------------------
> a) Separate classes for different types of actions
> ---------------------------------------------------------
> The different types of actions like FS actions, HCat related actions, etc. can be grouped together in separate classes like FSActions, HCatActions, etc.
> b) Passing the prepare logic to the Launcher
> ---------------------------------------------------
> Launcher needs the Prepare XML block to execute the actions. Oozie server can write the XML block to a file on DFS and then the Launcher could read from there.
> c) Execution of actions through a "Driver"
> -----------------------------------------------
> The Launcher can pass the XML block to the Driver which parses the content and calls corresponding methods that are grouped in different classes.
> 3) Pros and Cons of the design change:
> --------------------------------------------
> a) User facing impacts:
> -------------------------
> i) At present: In case of prepare block failure, there is no impact on the retry of map reduce job since the prepare block would have got executed earlier itself. The impact is seen only when Launcher fails.
>
> As per the proposed design: When the prepare block fails it will impact the retry of the map reduce job.
>
> ii) At present: In the case of prepare block failure, the launcher mapper is not launched at all.
>
> As per the proposed design: The cost of executing the prepare block would go up since it requires the launch of a map only task. This is highly pronounced in case of failures
>
> Thanks a lot for all the input from Oozie team members.
> Please post your suggestions. Thanks.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-616) Moving action prepare FS execution
to LauncherMapper
Posted by "Alejandro Abdelnur (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13156436#comment-13156436 ]
Alejandro Abdelnur commented on OOZIE-616:
------------------------------------------
Agree, this is a needed change that will also help simplify failures handling in Oozie side.
My take is:
The fs action should run in a launcher.
The prepare sections of all actions should run in their launcher (not a new one but the same).
Effectively we'd be moving the FS code to the LauncherMapper and we'd just have to pass the FS operations to it.
In the case of FS actions, main class would just do a NOP
I would not augment the FS operations to support copy as that can be a heavy operation. Instead I'd suggest users to use the distcp action for that.
> Moving action prepare FS execution to LauncherMapper
> ----------------------------------------------------
>
> Key: OOZIE-616
> URL: https://issues.apache.org/jira/browse/OOZIE-616
> Project: Oozie
> Issue Type: Improvement
> Reporter: Kiran Nagasubramanian
>
> 1) Motivations:
> ----------------
> a) Currently Oozie Server executes the Prepare logic before posting the Launcher job to Hadoop. If the launcher fails and the execution is retried by Hadoop automatically, the prepare logic is not re-executed. So, for Java/MR/Pig actions, if the launcher fails and it is retried by Hadoop automatically, the prepare logic is not executed again. Once the prepare logic execution is moved to the Launcher, the prepare logic is also re-executed during every retry.
> b) Heavy duty operations like copy that are currently not supported can be supported once the prepare logic has been moved to the Launcher.
> 2) Design choices for the proposed model:
> ------------------------------------------------
> a) Separate classes for different types of actions
> ---------------------------------------------------------
> The different types of actions like FS actions, HCat related actions, etc. can be grouped together in separate classes like FSActions, HCatActions, etc.
> b) Passing the prepare logic to the Launcher
> ---------------------------------------------------
> Launcher needs the Prepare XML block to execute the actions. Oozie server can write the XML block to a file on DFS and then the Launcher could read from there.
> c) Execution of actions through a "Driver"
> -----------------------------------------------
> The Launcher can pass the XML block to the Driver which parses the content and calls corresponding methods that are grouped in different classes.
> 3) Pros and Cons of the design change:
> --------------------------------------------
> a) User facing impacts:
> -------------------------
> i) At present: In case of prepare block failure, there is no impact on the retry of map reduce job since the prepare block would have got executed earlier itself. The impact is seen only when Launcher fails.
>
> As per the proposed design: When the prepare block fails it will impact the retry of the map reduce job.
>
> ii) At present: In the case of prepare block failure, the launcher mapper is not launched at all.
>
> As per the proposed design: The cost of executing the prepare block would go up since it requires the launch of a map only task. This is highly pronounced in case of failures
>
> Thanks a lot for all the input from Oozie team members.
> Please post your suggestions. Thanks.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-616) Moving action prepare FS execution
to LauncherMapper
Posted by "Virag Kothari (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13274762#comment-13274762 ]
Virag Kothari commented on OOZIE-616:
-------------------------------------
For the patch committed in trunk with this JIRA ID (also available under the same JIRA ID in Apache ReviewBoard system), I grant license to ASF for inclusion in ASF works (as per the Apache License ยง5).
> Moving action prepare FS execution to LauncherMapper
> ----------------------------------------------------
>
> Key: OOZIE-616
> URL: https://issues.apache.org/jira/browse/OOZIE-616
> Project: Oozie
> Issue Type: Improvement
> Reporter: Kiran Nagasubramanian
> Assignee: Kiran Nagasubramanian
> Fix For: 3.2.0
>
>
> 1) Motivations:
> ----------------
> a) Currently Oozie Server executes the Prepare logic before posting the Launcher job to Hadoop. If the launcher fails and the execution is retried by Hadoop automatically, the prepare logic is not re-executed. So, for Java/MR/Pig actions, if the launcher fails and it is retried by Hadoop automatically, the prepare logic is not executed again. Once the prepare logic execution is moved to the Launcher, the prepare logic is also re-executed during every retry.
> b) Heavy duty operations like copy that are currently not supported can be supported once the prepare logic has been moved to the Launcher.
> 2) Design choices for the proposed model:
> ------------------------------------------------
> a) Separate classes for different types of actions
> ---------------------------------------------------------
> The different types of actions like FS actions, HCat related actions, etc. can be grouped together in separate classes like FSActions, HCatActions, etc.
> b) Passing the prepare logic to the Launcher
> ---------------------------------------------------
> Launcher needs the Prepare XML block to execute the actions. Oozie server can write the XML block to a file on DFS and then the Launcher could read from there.
> c) Execution of actions through a "Driver"
> -----------------------------------------------
> The Launcher can pass the XML block to the Driver which parses the content and calls corresponding methods that are grouped in different classes.
> 3) Pros and Cons of the design change:
> --------------------------------------------
> a) User facing impacts:
> -------------------------
> i) At present: In case of prepare block failure, there is no impact on the retry of map reduce job since the prepare block would have got executed earlier itself. The impact is seen only when Launcher fails.
>
> As per the proposed design: When the prepare block fails it will impact the retry of the map reduce job.
>
> ii) At present: In the case of prepare block failure, the launcher mapper is not launched at all.
>
> As per the proposed design: The cost of executing the prepare block would go up since it requires the launch of a map only task. This is highly pronounced in case of failures
>
> Thanks a lot for all the input from Oozie team members.
> Please post your suggestions. Thanks.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-616) Moving action prepare FS execution
to LauncherMapper
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13166793#comment-13166793 ]
jiraposter@reviews.apache.org commented on OOZIE-616:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3143/
-----------------------------------------------------------
Review request for oozie, Mohammad Islam and Angelo K. Huang.
Summary
-------
a) Separate classes for different types of actions
------------------------------------------------
The different types of actions like FS actions, HCat related actions, etc. can be grouped together in separate classes like FSActions, HCatActions, etc.
b) Passing the prepare logic to the Launcher
-----------------------------------------
Launcher needs the Prepare XML block to execute the actions. Oozie server can write the XML block to a file on DFS and then the Launcher could read from there.
c) Execution of actions through a "Driver"
---------------------------------------
The Launcher can pass the XML block to the Driver which parses the content and calls corresponding methods that are grouped in different classes.
This addresses bug OOZIE-616.
https://issues.apache.org/jira/browse/OOZIE-616
Diffs
-----
trunk/core/src/main/java/org/apache/oozie/action/hadoop/FileSystemActions.java PRE-CREATION
trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1211045
trunk/core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java 1211045
trunk/core/src/main/java/org/apache/oozie/action/hadoop/PrepareActionsDriver.java PRE-CREATION
trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestFileSystemActions.java PRE-CREATION
trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 1211045
trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestLauncher.java 1211045
trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestMapReduceActionError.java 1211045
trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestMapReduceActionExecutor.java 1211045
trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestPigActionExecutor.java 1211045
trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestPrepareActionsDriver.java PRE-CREATION
Diff: https://reviews.apache.org/r/3143/diff
Testing
-------
Yes
Thanks,
Kiran
> Moving action prepare FS execution to LauncherMapper
> ----------------------------------------------------
>
> Key: OOZIE-616
> URL: https://issues.apache.org/jira/browse/OOZIE-616
> Project: Oozie
> Issue Type: Improvement
> Reporter: Kiran Nagasubramanian
>
> 1) Motivations:
> ----------------
> a) Currently Oozie Server executes the Prepare logic before posting the Launcher job to Hadoop. If the launcher fails and the execution is retried by Hadoop automatically, the prepare logic is not re-executed. So, for Java/MR/Pig actions, if the launcher fails and it is retried by Hadoop automatically, the prepare logic is not executed again. Once the prepare logic execution is moved to the Launcher, the prepare logic is also re-executed during every retry.
> b) Heavy duty operations like copy that are currently not supported can be supported once the prepare logic has been moved to the Launcher.
> 2) Design choices for the proposed model:
> ------------------------------------------------
> a) Separate classes for different types of actions
> ---------------------------------------------------------
> The different types of actions like FS actions, HCat related actions, etc. can be grouped together in separate classes like FSActions, HCatActions, etc.
> b) Passing the prepare logic to the Launcher
> ---------------------------------------------------
> Launcher needs the Prepare XML block to execute the actions. Oozie server can write the XML block to a file on DFS and then the Launcher could read from there.
> c) Execution of actions through a "Driver"
> -----------------------------------------------
> The Launcher can pass the XML block to the Driver which parses the content and calls corresponding methods that are grouped in different classes.
> 3) Pros and Cons of the design change:
> --------------------------------------------
> a) User facing impacts:
> -------------------------
> i) At present: In case of prepare block failure, there is no impact on the retry of map reduce job since the prepare block would have got executed earlier itself. The impact is seen only when Launcher fails.
>
> As per the proposed design: When the prepare block fails it will impact the retry of the map reduce job.
>
> ii) At present: In the case of prepare block failure, the launcher mapper is not launched at all.
>
> As per the proposed design: The cost of executing the prepare block would go up since it requires the launch of a map only task. This is highly pronounced in case of failures
>
> Thanks a lot for all the input from Oozie team members.
> Please post your suggestions. Thanks.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira