You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@oozie.apache.org by Robert Kanter <rk...@cloudera.com> on 2014/03/01 00:48:15 UTC

Review Request 18653: OOZIE-1722: When an ApplicationMaster restarts, it restarts the launcher job

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18653/
-----------------------------------------------------------

Review request for oozie.


Bugs: OOZIE-1722
    https://issues.apache.org/jira/browse/OOZIE-1722


Repository: oozie-git


Description
-------

See https://issues.apache.org/jira/browse/OOZIE-1722


Diffs
-----

  core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 68d77a8 
  core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 07de214 
  hadooplibs/hadoop-utils-0.23/pom.xml PRE-CREATION 
  hadooplibs/hadoop-utils-0.23/src/main/java/org/apache/oozie/action/hadoop/LauncherMainHadoopUtils.java PRE-CREATION 
  hadooplibs/hadoop-utils-1/pom.xml PRE-CREATION 
  hadooplibs/hadoop-utils-1/src/main/java/org/apache/oozie/action/hadoop/LauncherMainHadoopUtils.java PRE-CREATION 
  hadooplibs/hadoop-utils-2/pom.xml PRE-CREATION 
  hadooplibs/hadoop-utils-2/src/main/java/org/apache/oozie/action/hadoop/LauncherMainHadoopUtils.java PRE-CREATION 
  hadooplibs/hadoop-utils-3/pom.xml PRE-CREATION 
  hadooplibs/hadoop-utils-3/src/main/java/org/apache/oozie/action/hadoop/LauncherMainHadoopUtils.java PRE-CREATION 
  hadooplibs/pom.xml 35875c8 
  pom.xml 1223ec2 
  sharelib/hive/pom.xml e355d97 
  sharelib/hive/src/main/java/org/apache/oozie/action/hadoop/HiveMain.java 8fa5ec8 
  sharelib/oozie/pom.xml c70582d 
  sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java b5fa4a1 
  sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/MapReduceMain.java 6230e64 
  sharelib/pig/pom.xml a2bbeaf 
  sharelib/pig/src/main/java/org/apache/oozie/action/hadoop/PigMain.java 9c1e659 
  sharelib/sqoop/pom.xml a962565 
  sharelib/sqoop/src/main/java/org/apache/oozie/action/hadoop/SqoopMain.java e45c8b2 

Diff: https://reviews.apache.org/r/18653/diff/


Testing
-------


Thanks,

Robert Kanter


Re: Review Request 18653: OOZIE-1722: When an ApplicationMaster restarts, it restarts the launcher job

Posted by Rohini Palaniswamy <ro...@gmail.com>.

> On March 3, 2014, 7:30 a.m., Rohini Palaniswamy wrote:
> > We need to add the killChildJobs in Java, Shell and Distcp as well. Many people launch hbase jobs from java action and instead of each user adding killChildJobs doing it in JavaActionExecutor should be good. Currently the configuration is not passed properly to distcp action. For killing child jobs in distcp to work, we need to fix that or pass the configuration as arg (i.e -Dmapreduce.job.tags=xxx) so that child jobs are launched with that tag. I think we can pass it as arg for now.
> 
> Robert Kanter wrote:
>     The reason I didn't add the killChildJobs to the Distcp action is that we don't have a "DistcpMain" to put it in.  I didn't want to put the killChildJobs in the calling class LauncherMapper because (a) we handle this differently for the MR action and (b) someone writing a custom action (or us in a future action) may also want to handle this differently.  I guess I could write a wrapper "DistcpMain" to call killChildJobs...
>     
>     For the Java and Shell actions, they don't necessarily launch any child jobs.  And in general, we don't implement some of the "Oozie magic" stuff for the Java and Shell actions, leaving it up to the user instead (for example, propagating the action's <configuration> or the delegation token).  The Java action also has the same problem as the Distcp action, in that there's no "JavaMain".  What if I added documentation on how to call killChildJobs so the user can do this in their Main?
> 
> Rohini Palaniswamy wrote:
>     Many users launch child jobs in their java actions (especially with hbase). Would prefer as much less work for user as possible. If you are writing a DistcpMain, you can just write a JavaMain that calls killChildJobs and make DistcpMain just extend that. That way it will just work without users having to bother about it and writing different code for different versions of hadoop.
> 
> Robert Kanter wrote:
>     I can add a wrapper for Java and Distcp that does the killing.  Though for the Java action, it will still be up to the user to propagate the actionConf prepared by Oozie, or at least just set the tag property.
> 
> Rohini Palaniswamy wrote:
>     Sounds good and thanks. Most of them usually propagate the actionConf and would benefit from this.
> 
> Robert Kanter wrote:
>     Getting this to work with DistCp is trickier than I thought.  I tried passing the tags property as a -D argument, but that didn't work (in fact, it caused an error).  Looking at DistCp's code, we could try to invoke it programmatically from a "DistCpMain" instead of JavaMain; this would allow us to pass it a Configuration.  However, DistCp v1 (Hadoop 1) and v2 (Hadoop 0.23, Hadoop 2+) don't have a common constructor (v2's constructor takes an additional argument), so the "DistCpMain" would have to use Reflection and check which constructor is available.  MAPREDUCE-5081 did backport v2 to Hadoop 1, but its only in Hadoop 1.2.0 and we're using 1.1.1
>     
>     So we have a few options:
>     (1) Don't support the tags with DistCp
>     (2) Use a "DistCpMain"
>        (2a) Use reflection to call the appropriate constructor and run it programmatically
>        (2b) Update to Hadoop 1.2.x and use DistCp v2 for each version of Hadoop
>     
>     I think we should do (1) for now so we can get the tags stuff in and create a follow-up JIRA to do (2b), which will require a lot of additional testing.
>     
>     Thoughts?

Option 3:
 <arg>-Dmapred.job.queue.name=${queueName}</arg> is how queue information is currently passed to Distcp from the distcp action. LauncherMapperHelper.setupMainArguments and LauncherMapper.getMainArguments handle the main method arguments. We need to introduce it somewhere there for Distcp. But it is hacky and don't like it much. Just pointing out as an option.

 Your option 2 is cleaner and like it better. We definitely need to fix DistcpMain to call the constructor instead of the main() method to pass the JobConf. Users have been passing even the queue information as above because the distcp job does not get the action configuration. We had this reported as an issue, but have always gone with the <arg> workaround. So option 2 will fix more issues. We can definitely do (2a). (2b) of course can be a different jira.


- Rohini


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18653/#review35947
-----------------------------------------------------------


On March 4, 2014, 3:11 a.m., Robert Kanter wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/18653/
> -----------------------------------------------------------
> 
> (Updated March 4, 2014, 3:11 a.m.)
> 
> 
> Review request for oozie.
> 
> 
> Bugs: OOZIE-1722
>     https://issues.apache.org/jira/browse/OOZIE-1722
> 
> 
> Repository: oozie-git
> 
> 
> Description
> -------
> 
> See https://issues.apache.org/jira/browse/OOZIE-1722
> 
> 
> Diffs
> -----
> 
>   core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java fcf69e7 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 68d77a8 
>   core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 07de214 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 7841076 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestLauncher.java b55b9d7 
>   core/src/test/java/org/apache/oozie/service/TestShareLibService.java e7d7a26 
>   core/src/test/java/org/apache/oozie/workflow/lite/TestLiteWorkflowAppParser.java 41e6205 
>   hadooplibs/hadoop-utils-0.23/pom.xml PRE-CREATION 
>   hadooplibs/hadoop-utils-0.23/src/main/java/org/apache/oozie/action/hadoop/LauncherMainHadoopUtils.java PRE-CREATION 
>   hadooplibs/hadoop-utils-1/pom.xml PRE-CREATION 
>   hadooplibs/hadoop-utils-1/src/main/java/org/apache/oozie/action/hadoop/LauncherMainHadoopUtils.java PRE-CREATION 
>   hadooplibs/hadoop-utils-2/pom.xml PRE-CREATION 
>   hadooplibs/hadoop-utils-2/src/main/java/org/apache/oozie/action/hadoop/LauncherMainHadoopUtils.java PRE-CREATION 
>   hadooplibs/hadoop-utils-3/pom.xml PRE-CREATION 
>   hadooplibs/hadoop-utils-3/src/main/java/org/apache/oozie/action/hadoop/LauncherMainHadoopUtils.java PRE-CREATION 
>   hadooplibs/pom.xml 35875c8 
>   pom.xml 1223ec2 
>   release-log.txt f6810bc 
>   sharelib/distcp/src/test/java/org/apache/oozie/action/hadoop/TestDistCpActionExecutor.java 25b1a23 
>   sharelib/hive/pom.xml e355d97 
>   sharelib/hive/src/main/java/org/apache/oozie/action/hadoop/HiveMain.java 8fa5ec8 
>   sharelib/oozie/pom.xml c70582d 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/JavaMain.java PRE-CREATION 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java b5fa4a1 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/MapReduceMain.java 6230e64 
>   sharelib/pig/pom.xml a2bbeaf 
>   sharelib/pig/src/main/java/org/apache/oozie/action/hadoop/PigMain.java 9c1e659 
>   sharelib/sqoop/pom.xml a962565 
>   sharelib/sqoop/src/main/java/org/apache/oozie/action/hadoop/SqoopMain.java e45c8b2 
>   sharelib/streaming/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/18653/diff/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Robert Kanter
> 
>


Re: Review Request 18653: OOZIE-1722: When an ApplicationMaster restarts, it restarts the launcher job

Posted by Rohini Palaniswamy <ro...@gmail.com>.

> On March 3, 2014, 7:30 a.m., Rohini Palaniswamy wrote:
> > We need to add the killChildJobs in Java, Shell and Distcp as well. Many people launch hbase jobs from java action and instead of each user adding killChildJobs doing it in JavaActionExecutor should be good. Currently the configuration is not passed properly to distcp action. For killing child jobs in distcp to work, we need to fix that or pass the configuration as arg (i.e -Dmapreduce.job.tags=xxx) so that child jobs are launched with that tag. I think we can pass it as arg for now.
> 
> Robert Kanter wrote:
>     The reason I didn't add the killChildJobs to the Distcp action is that we don't have a "DistcpMain" to put it in.  I didn't want to put the killChildJobs in the calling class LauncherMapper because (a) we handle this differently for the MR action and (b) someone writing a custom action (or us in a future action) may also want to handle this differently.  I guess I could write a wrapper "DistcpMain" to call killChildJobs...
>     
>     For the Java and Shell actions, they don't necessarily launch any child jobs.  And in general, we don't implement some of the "Oozie magic" stuff for the Java and Shell actions, leaving it up to the user instead (for example, propagating the action's <configuration> or the delegation token).  The Java action also has the same problem as the Distcp action, in that there's no "JavaMain".  What if I added documentation on how to call killChildJobs so the user can do this in their Main?
> 
> Rohini Palaniswamy wrote:
>     Many users launch child jobs in their java actions (especially with hbase). Would prefer as much less work for user as possible. If you are writing a DistcpMain, you can just write a JavaMain that calls killChildJobs and make DistcpMain just extend that. That way it will just work without users having to bother about it and writing different code for different versions of hadoop.
> 
> Robert Kanter wrote:
>     I can add a wrapper for Java and Distcp that does the killing.  Though for the Java action, it will still be up to the user to propagate the actionConf prepared by Oozie, or at least just set the tag property.

Sounds good and thanks. Most of them usually propagate the actionConf and would benefit from this. 


- Rohini


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18653/#review35947
-----------------------------------------------------------


On March 1, 2014, 2:01 a.m., Robert Kanter wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/18653/
> -----------------------------------------------------------
> 
> (Updated March 1, 2014, 2:01 a.m.)
> 
> 
> Review request for oozie.
> 
> 
> Bugs: OOZIE-1722
>     https://issues.apache.org/jira/browse/OOZIE-1722
> 
> 
> Repository: oozie-git
> 
> 
> Description
> -------
> 
> See https://issues.apache.org/jira/browse/OOZIE-1722
> 
> 
> Diffs
> -----
> 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 68d77a8 
>   core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 07de214 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestLauncher.java b55b9d7 
>   hadooplibs/hadoop-utils-0.23/pom.xml PRE-CREATION 
>   hadooplibs/hadoop-utils-0.23/src/main/java/org/apache/oozie/action/hadoop/LauncherMainHadoopUtils.java PRE-CREATION 
>   hadooplibs/hadoop-utils-1/pom.xml PRE-CREATION 
>   hadooplibs/hadoop-utils-1/src/main/java/org/apache/oozie/action/hadoop/LauncherMainHadoopUtils.java PRE-CREATION 
>   hadooplibs/hadoop-utils-2/pom.xml PRE-CREATION 
>   hadooplibs/hadoop-utils-2/src/main/java/org/apache/oozie/action/hadoop/LauncherMainHadoopUtils.java PRE-CREATION 
>   hadooplibs/hadoop-utils-3/pom.xml PRE-CREATION 
>   hadooplibs/hadoop-utils-3/src/main/java/org/apache/oozie/action/hadoop/LauncherMainHadoopUtils.java PRE-CREATION 
>   hadooplibs/pom.xml 35875c8 
>   pom.xml 1223ec2 
>   sharelib/hive/pom.xml e355d97 
>   sharelib/hive/src/main/java/org/apache/oozie/action/hadoop/HiveMain.java 8fa5ec8 
>   sharelib/oozie/pom.xml c70582d 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java b5fa4a1 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/MapReduceMain.java 6230e64 
>   sharelib/pig/pom.xml a2bbeaf 
>   sharelib/pig/src/main/java/org/apache/oozie/action/hadoop/PigMain.java 9c1e659 
>   sharelib/sqoop/pom.xml a962565 
>   sharelib/sqoop/src/main/java/org/apache/oozie/action/hadoop/SqoopMain.java e45c8b2 
> 
> Diff: https://reviews.apache.org/r/18653/diff/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Robert Kanter
> 
>


Re: Review Request 18653: OOZIE-1722: When an ApplicationMaster restarts, it restarts the launcher job

Posted by Robert Kanter <rk...@cloudera.com>.

> On March 3, 2014, 7:30 a.m., Rohini Palaniswamy wrote:
> > We need to add the killChildJobs in Java, Shell and Distcp as well. Many people launch hbase jobs from java action and instead of each user adding killChildJobs doing it in JavaActionExecutor should be good. Currently the configuration is not passed properly to distcp action. For killing child jobs in distcp to work, we need to fix that or pass the configuration as arg (i.e -Dmapreduce.job.tags=xxx) so that child jobs are launched with that tag. I think we can pass it as arg for now.

The reason I didn't add the killChildJobs to the Distcp action is that we don't have a "DistcpMain" to put it in.  I didn't want to put the killChildJobs in the calling class LauncherMapper because (a) we handle this differently for the MR action and (b) someone writing a custom action (or us in a future action) may also want to handle this differently.  I guess I could write a wrapper "DistcpMain" to call killChildJobs...

For the Java and Shell actions, they don't necessarily launch any child jobs.  And in general, we don't implement some of the "Oozie magic" stuff for the Java and Shell actions, leaving it up to the user instead (for example, propagating the action's <configuration> or the delegation token).  The Java action also has the same problem as the Distcp action, in that there's no "JavaMain".  What if I added documentation on how to call killChildJobs so the user can do this in their Main?


- Robert


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18653/#review35947
-----------------------------------------------------------


On March 1, 2014, 2:01 a.m., Robert Kanter wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/18653/
> -----------------------------------------------------------
> 
> (Updated March 1, 2014, 2:01 a.m.)
> 
> 
> Review request for oozie.
> 
> 
> Bugs: OOZIE-1722
>     https://issues.apache.org/jira/browse/OOZIE-1722
> 
> 
> Repository: oozie-git
> 
> 
> Description
> -------
> 
> See https://issues.apache.org/jira/browse/OOZIE-1722
> 
> 
> Diffs
> -----
> 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 68d77a8 
>   core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 07de214 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestLauncher.java b55b9d7 
>   hadooplibs/hadoop-utils-0.23/pom.xml PRE-CREATION 
>   hadooplibs/hadoop-utils-0.23/src/main/java/org/apache/oozie/action/hadoop/LauncherMainHadoopUtils.java PRE-CREATION 
>   hadooplibs/hadoop-utils-1/pom.xml PRE-CREATION 
>   hadooplibs/hadoop-utils-1/src/main/java/org/apache/oozie/action/hadoop/LauncherMainHadoopUtils.java PRE-CREATION 
>   hadooplibs/hadoop-utils-2/pom.xml PRE-CREATION 
>   hadooplibs/hadoop-utils-2/src/main/java/org/apache/oozie/action/hadoop/LauncherMainHadoopUtils.java PRE-CREATION 
>   hadooplibs/hadoop-utils-3/pom.xml PRE-CREATION 
>   hadooplibs/hadoop-utils-3/src/main/java/org/apache/oozie/action/hadoop/LauncherMainHadoopUtils.java PRE-CREATION 
>   hadooplibs/pom.xml 35875c8 
>   pom.xml 1223ec2 
>   sharelib/hive/pom.xml e355d97 
>   sharelib/hive/src/main/java/org/apache/oozie/action/hadoop/HiveMain.java 8fa5ec8 
>   sharelib/oozie/pom.xml c70582d 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java b5fa4a1 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/MapReduceMain.java 6230e64 
>   sharelib/pig/pom.xml a2bbeaf 
>   sharelib/pig/src/main/java/org/apache/oozie/action/hadoop/PigMain.java 9c1e659 
>   sharelib/sqoop/pom.xml a962565 
>   sharelib/sqoop/src/main/java/org/apache/oozie/action/hadoop/SqoopMain.java e45c8b2 
> 
> Diff: https://reviews.apache.org/r/18653/diff/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Robert Kanter
> 
>


Re: Review Request 18653: OOZIE-1722: When an ApplicationMaster restarts, it restarts the launcher job

Posted by Rohini Palaniswamy <ro...@gmail.com>.

> On March 3, 2014, 7:30 a.m., Rohini Palaniswamy wrote:
> > We need to add the killChildJobs in Java, Shell and Distcp as well. Many people launch hbase jobs from java action and instead of each user adding killChildJobs doing it in JavaActionExecutor should be good. Currently the configuration is not passed properly to distcp action. For killing child jobs in distcp to work, we need to fix that or pass the configuration as arg (i.e -Dmapreduce.job.tags=xxx) so that child jobs are launched with that tag. I think we can pass it as arg for now.
> 
> Robert Kanter wrote:
>     The reason I didn't add the killChildJobs to the Distcp action is that we don't have a "DistcpMain" to put it in.  I didn't want to put the killChildJobs in the calling class LauncherMapper because (a) we handle this differently for the MR action and (b) someone writing a custom action (or us in a future action) may also want to handle this differently.  I guess I could write a wrapper "DistcpMain" to call killChildJobs...
>     
>     For the Java and Shell actions, they don't necessarily launch any child jobs.  And in general, we don't implement some of the "Oozie magic" stuff for the Java and Shell actions, leaving it up to the user instead (for example, propagating the action's <configuration> or the delegation token).  The Java action also has the same problem as the Distcp action, in that there's no "JavaMain".  What if I added documentation on how to call killChildJobs so the user can do this in their Main?
> 
> Rohini Palaniswamy wrote:
>     Many users launch child jobs in their java actions (especially with hbase). Would prefer as much less work for user as possible. If you are writing a DistcpMain, you can just write a JavaMain that calls killChildJobs and make DistcpMain just extend that. That way it will just work without users having to bother about it and writing different code for different versions of hadoop.
> 
> Robert Kanter wrote:
>     I can add a wrapper for Java and Distcp that does the killing.  Though for the Java action, it will still be up to the user to propagate the actionConf prepared by Oozie, or at least just set the tag property.
> 
> Rohini Palaniswamy wrote:
>     Sounds good and thanks. Most of them usually propagate the actionConf and would benefit from this.
> 
> Robert Kanter wrote:
>     Getting this to work with DistCp is trickier than I thought.  I tried passing the tags property as a -D argument, but that didn't work (in fact, it caused an error).  Looking at DistCp's code, we could try to invoke it programmatically from a "DistCpMain" instead of JavaMain; this would allow us to pass it a Configuration.  However, DistCp v1 (Hadoop 1) and v2 (Hadoop 0.23, Hadoop 2+) don't have a common constructor (v2's constructor takes an additional argument), so the "DistCpMain" would have to use Reflection and check which constructor is available.  MAPREDUCE-5081 did backport v2 to Hadoop 1, but its only in Hadoop 1.2.0 and we're using 1.1.1
>     
>     So we have a few options:
>     (1) Don't support the tags with DistCp
>     (2) Use a "DistCpMain"
>        (2a) Use reflection to call the appropriate constructor and run it programmatically
>        (2b) Update to Hadoop 1.2.x and use DistCp v2 for each version of Hadoop
>     
>     I think we should do (1) for now so we can get the tags stuff in and create a follow-up JIRA to do (2b), which will require a lot of additional testing.
>     
>     Thoughts?
> 
> Rohini Palaniswamy wrote:
>     Option 3:
>      <arg>-Dmapred.job.queue.name=${queueName}</arg> is how queue information is currently passed to Distcp from the distcp action. LauncherMapperHelper.setupMainArguments and LauncherMapper.getMainArguments handle the main method arguments. We need to introduce it somewhere there for Distcp. But it is hacky and don't like it much. Just pointing out as an option.
>     
>      Your option 2 is cleaner and like it better. We definitely need to fix DistcpMain to call the constructor instead of the main() method to pass the JobConf. Users have been passing even the queue information as above because the distcp job does not get the action configuration. We had this reported as an issue, but have always gone with the <arg> workaround. So option 2 will fix more issues. We can definitely do (2a). (2b) of course can be a different jira.
> 
> Rohini Palaniswamy wrote:
>     Actually with your current patch you can easily do Option 3 (passing -Dmapreduce.job.tags=xxx as main args) with DistcpMain as you would be invoking the Distcp main class from there. Was that what you tried and did not work?
> 
> Robert Kanter wrote:
>     I did try option 3, using LauncherMapperHelper.setupMainArguments.  It gave an error about not understanding the -D argument.  I think that only works from the hadoop CLI, which parses out the -D args.  
>     
>     I can try 2a, but I think this will be very brittle.  The main() in each DistCp is different (e.g. v2 has a shutdown hook it configures).  So while I can try to duplicate each main() in the reflection, it can easily break if DistCp changes or if someone is using a different version of DistCp.  I think we should wait and do 1 and do 2b later.

How does <arg>-Dmapred.job.queue.name=${queueName}</arg> get passed to distcp then?


- Rohini


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18653/#review35947
-----------------------------------------------------------


On March 4, 2014, 9:48 p.m., Robert Kanter wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/18653/
> -----------------------------------------------------------
> 
> (Updated March 4, 2014, 9:48 p.m.)
> 
> 
> Review request for oozie.
> 
> 
> Bugs: OOZIE-1722
>     https://issues.apache.org/jira/browse/OOZIE-1722
> 
> 
> Repository: oozie-git
> 
> 
> Description
> -------
> 
> See https://issues.apache.org/jira/browse/OOZIE-1722
> 
> 
> Diffs
> -----
> 
>   core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java fcf69e7 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 68d77a8 
>   core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 07de214 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 7841076 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestLauncher.java b55b9d7 
>   core/src/test/java/org/apache/oozie/service/TestShareLibService.java e7d7a26 
>   hadooplibs/hadoop-utils-0.23/pom.xml PRE-CREATION 
>   hadooplibs/hadoop-utils-0.23/src/main/java/org/apache/oozie/action/hadoop/LauncherMainHadoopUtils.java PRE-CREATION 
>   hadooplibs/hadoop-utils-1/pom.xml PRE-CREATION 
>   hadooplibs/hadoop-utils-1/src/main/java/org/apache/oozie/action/hadoop/LauncherMainHadoopUtils.java PRE-CREATION 
>   hadooplibs/hadoop-utils-2/pom.xml PRE-CREATION 
>   hadooplibs/hadoop-utils-2/src/main/java/org/apache/oozie/action/hadoop/LauncherMainHadoopUtils.java PRE-CREATION 
>   hadooplibs/hadoop-utils-3/pom.xml PRE-CREATION 
>   hadooplibs/hadoop-utils-3/src/main/java/org/apache/oozie/action/hadoop/LauncherMainHadoopUtils.java PRE-CREATION 
>   hadooplibs/pom.xml 35875c8 
>   pom.xml 1223ec2 
>   sharelib/distcp/src/test/java/org/apache/oozie/action/hadoop/TestDistCpActionExecutor.java 25b1a23 
>   sharelib/hive/pom.xml e355d97 
>   sharelib/hive/src/main/java/org/apache/oozie/action/hadoop/HiveMain.java 8fa5ec8 
>   sharelib/oozie/pom.xml c70582d 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/JavaMain.java PRE-CREATION 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java b5fa4a1 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/MapReduceMain.java 6230e64 
>   sharelib/pig/pom.xml a2bbeaf 
>   sharelib/pig/src/main/java/org/apache/oozie/action/hadoop/PigMain.java 9c1e659 
>   sharelib/sqoop/pom.xml a962565 
>   sharelib/sqoop/src/main/java/org/apache/oozie/action/hadoop/SqoopMain.java e45c8b2 
>   sharelib/streaming/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/18653/diff/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Robert Kanter
> 
>


Re: Review Request 18653: OOZIE-1722: When an ApplicationMaster restarts, it restarts the launcher job

Posted by Robert Kanter <rk...@cloudera.com>.

> On March 3, 2014, 7:30 a.m., Rohini Palaniswamy wrote:
> > We need to add the killChildJobs in Java, Shell and Distcp as well. Many people launch hbase jobs from java action and instead of each user adding killChildJobs doing it in JavaActionExecutor should be good. Currently the configuration is not passed properly to distcp action. For killing child jobs in distcp to work, we need to fix that or pass the configuration as arg (i.e -Dmapreduce.job.tags=xxx) so that child jobs are launched with that tag. I think we can pass it as arg for now.
> 
> Robert Kanter wrote:
>     The reason I didn't add the killChildJobs to the Distcp action is that we don't have a "DistcpMain" to put it in.  I didn't want to put the killChildJobs in the calling class LauncherMapper because (a) we handle this differently for the MR action and (b) someone writing a custom action (or us in a future action) may also want to handle this differently.  I guess I could write a wrapper "DistcpMain" to call killChildJobs...
>     
>     For the Java and Shell actions, they don't necessarily launch any child jobs.  And in general, we don't implement some of the "Oozie magic" stuff for the Java and Shell actions, leaving it up to the user instead (for example, propagating the action's <configuration> or the delegation token).  The Java action also has the same problem as the Distcp action, in that there's no "JavaMain".  What if I added documentation on how to call killChildJobs so the user can do this in their Main?
> 
> Rohini Palaniswamy wrote:
>     Many users launch child jobs in their java actions (especially with hbase). Would prefer as much less work for user as possible. If you are writing a DistcpMain, you can just write a JavaMain that calls killChildJobs and make DistcpMain just extend that. That way it will just work without users having to bother about it and writing different code for different versions of hadoop.
> 
> Robert Kanter wrote:
>     I can add a wrapper for Java and Distcp that does the killing.  Though for the Java action, it will still be up to the user to propagate the actionConf prepared by Oozie, or at least just set the tag property.
> 
> Rohini Palaniswamy wrote:
>     Sounds good and thanks. Most of them usually propagate the actionConf and would benefit from this.
> 
> Robert Kanter wrote:
>     Getting this to work with DistCp is trickier than I thought.  I tried passing the tags property as a -D argument, but that didn't work (in fact, it caused an error).  Looking at DistCp's code, we could try to invoke it programmatically from a "DistCpMain" instead of JavaMain; this would allow us to pass it a Configuration.  However, DistCp v1 (Hadoop 1) and v2 (Hadoop 0.23, Hadoop 2+) don't have a common constructor (v2's constructor takes an additional argument), so the "DistCpMain" would have to use Reflection and check which constructor is available.  MAPREDUCE-5081 did backport v2 to Hadoop 1, but its only in Hadoop 1.2.0 and we're using 1.1.1
>     
>     So we have a few options:
>     (1) Don't support the tags with DistCp
>     (2) Use a "DistCpMain"
>        (2a) Use reflection to call the appropriate constructor and run it programmatically
>        (2b) Update to Hadoop 1.2.x and use DistCp v2 for each version of Hadoop
>     
>     I think we should do (1) for now so we can get the tags stuff in and create a follow-up JIRA to do (2b), which will require a lot of additional testing.
>     
>     Thoughts?
> 
> Rohini Palaniswamy wrote:
>     Option 3:
>      <arg>-Dmapred.job.queue.name=${queueName}</arg> is how queue information is currently passed to Distcp from the distcp action. LauncherMapperHelper.setupMainArguments and LauncherMapper.getMainArguments handle the main method arguments. We need to introduce it somewhere there for Distcp. But it is hacky and don't like it much. Just pointing out as an option.
>     
>      Your option 2 is cleaner and like it better. We definitely need to fix DistcpMain to call the constructor instead of the main() method to pass the JobConf. Users have been passing even the queue information as above because the distcp job does not get the action configuration. We had this reported as an issue, but have always gone with the <arg> workaround. So option 2 will fix more issues. We can definitely do (2a). (2b) of course can be a different jira.
> 
> Rohini Palaniswamy wrote:
>     Actually with your current patch you can easily do Option 3 (passing -Dmapreduce.job.tags=xxx as main args) with DistcpMain as you would be invoking the Distcp main class from there. Was that what you tried and did not work?

I did try option 3, using LauncherMapperHelper.setupMainArguments.  It gave an error about not understanding the -D argument.  I think that only works from the hadoop CLI, which parses out the -D args.  

I can try 2a, but I think this will be very brittle.  The main() in each DistCp is different (e.g. v2 has a shutdown hook it configures).  So while I can try to duplicate each main() in the reflection, it can easily break if DistCp changes or if someone is using a different version of DistCp.  I think we should wait and do 1 and do 2b later.


- Robert


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18653/#review35947
-----------------------------------------------------------


On March 4, 2014, 3:11 a.m., Robert Kanter wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/18653/
> -----------------------------------------------------------
> 
> (Updated March 4, 2014, 3:11 a.m.)
> 
> 
> Review request for oozie.
> 
> 
> Bugs: OOZIE-1722
>     https://issues.apache.org/jira/browse/OOZIE-1722
> 
> 
> Repository: oozie-git
> 
> 
> Description
> -------
> 
> See https://issues.apache.org/jira/browse/OOZIE-1722
> 
> 
> Diffs
> -----
> 
>   core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java fcf69e7 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 68d77a8 
>   core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 07de214 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 7841076 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestLauncher.java b55b9d7 
>   core/src/test/java/org/apache/oozie/service/TestShareLibService.java e7d7a26 
>   core/src/test/java/org/apache/oozie/workflow/lite/TestLiteWorkflowAppParser.java 41e6205 
>   hadooplibs/hadoop-utils-0.23/pom.xml PRE-CREATION 
>   hadooplibs/hadoop-utils-0.23/src/main/java/org/apache/oozie/action/hadoop/LauncherMainHadoopUtils.java PRE-CREATION 
>   hadooplibs/hadoop-utils-1/pom.xml PRE-CREATION 
>   hadooplibs/hadoop-utils-1/src/main/java/org/apache/oozie/action/hadoop/LauncherMainHadoopUtils.java PRE-CREATION 
>   hadooplibs/hadoop-utils-2/pom.xml PRE-CREATION 
>   hadooplibs/hadoop-utils-2/src/main/java/org/apache/oozie/action/hadoop/LauncherMainHadoopUtils.java PRE-CREATION 
>   hadooplibs/hadoop-utils-3/pom.xml PRE-CREATION 
>   hadooplibs/hadoop-utils-3/src/main/java/org/apache/oozie/action/hadoop/LauncherMainHadoopUtils.java PRE-CREATION 
>   hadooplibs/pom.xml 35875c8 
>   pom.xml 1223ec2 
>   release-log.txt f6810bc 
>   sharelib/distcp/src/test/java/org/apache/oozie/action/hadoop/TestDistCpActionExecutor.java 25b1a23 
>   sharelib/hive/pom.xml e355d97 
>   sharelib/hive/src/main/java/org/apache/oozie/action/hadoop/HiveMain.java 8fa5ec8 
>   sharelib/oozie/pom.xml c70582d 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/JavaMain.java PRE-CREATION 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java b5fa4a1 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/MapReduceMain.java 6230e64 
>   sharelib/pig/pom.xml a2bbeaf 
>   sharelib/pig/src/main/java/org/apache/oozie/action/hadoop/PigMain.java 9c1e659 
>   sharelib/sqoop/pom.xml a962565 
>   sharelib/sqoop/src/main/java/org/apache/oozie/action/hadoop/SqoopMain.java e45c8b2 
>   sharelib/streaming/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/18653/diff/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Robert Kanter
> 
>


Re: Review Request 18653: OOZIE-1722: When an ApplicationMaster restarts, it restarts the launcher job

Posted by Robert Kanter <rk...@cloudera.com>.

> On March 3, 2014, 7:30 a.m., Rohini Palaniswamy wrote:
> > We need to add the killChildJobs in Java, Shell and Distcp as well. Many people launch hbase jobs from java action and instead of each user adding killChildJobs doing it in JavaActionExecutor should be good. Currently the configuration is not passed properly to distcp action. For killing child jobs in distcp to work, we need to fix that or pass the configuration as arg (i.e -Dmapreduce.job.tags=xxx) so that child jobs are launched with that tag. I think we can pass it as arg for now.
> 
> Robert Kanter wrote:
>     The reason I didn't add the killChildJobs to the Distcp action is that we don't have a "DistcpMain" to put it in.  I didn't want to put the killChildJobs in the calling class LauncherMapper because (a) we handle this differently for the MR action and (b) someone writing a custom action (or us in a future action) may also want to handle this differently.  I guess I could write a wrapper "DistcpMain" to call killChildJobs...
>     
>     For the Java and Shell actions, they don't necessarily launch any child jobs.  And in general, we don't implement some of the "Oozie magic" stuff for the Java and Shell actions, leaving it up to the user instead (for example, propagating the action's <configuration> or the delegation token).  The Java action also has the same problem as the Distcp action, in that there's no "JavaMain".  What if I added documentation on how to call killChildJobs so the user can do this in their Main?
> 
> Rohini Palaniswamy wrote:
>     Many users launch child jobs in their java actions (especially with hbase). Would prefer as much less work for user as possible. If you are writing a DistcpMain, you can just write a JavaMain that calls killChildJobs and make DistcpMain just extend that. That way it will just work without users having to bother about it and writing different code for different versions of hadoop.
> 
> Robert Kanter wrote:
>     I can add a wrapper for Java and Distcp that does the killing.  Though for the Java action, it will still be up to the user to propagate the actionConf prepared by Oozie, or at least just set the tag property.
> 
> Rohini Palaniswamy wrote:
>     Sounds good and thanks. Most of them usually propagate the actionConf and would benefit from this.

Getting this to work with DistCp is trickier than I thought.  I tried passing the tags property as a -D argument, but that didn't work (in fact, it caused an error).  Looking at DistCp's code, we could try to invoke it programmatically from a "DistCpMain" instead of JavaMain; this would allow us to pass it a Configuration.  However, DistCp v1 (Hadoop 1) and v2 (Hadoop 0.23, Hadoop 2+) don't have a common constructor (v2's constructor takes an additional argument), so the "DistCpMain" would have to use Reflection and check which constructor is available.  MAPREDUCE-5081 did backport v2 to Hadoop 1, but its only in Hadoop 1.2.0 and we're using 1.1.1

So we have a few options:
(1) Don't support the tags with DistCp
(2) Use a "DistCpMain"
   (2a) Use reflection to call the appropriate constructor and run it programmatically
   (2b) Update to Hadoop 1.2.x and use DistCp v2 for each version of Hadoop

I think we should do (1) for now so we can get the tags stuff in and create a follow-up JIRA to do (2b), which will require a lot of additional testing.

Thoughts?


- Robert


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18653/#review35947
-----------------------------------------------------------


On March 4, 2014, 3:11 a.m., Robert Kanter wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/18653/
> -----------------------------------------------------------
> 
> (Updated March 4, 2014, 3:11 a.m.)
> 
> 
> Review request for oozie.
> 
> 
> Bugs: OOZIE-1722
>     https://issues.apache.org/jira/browse/OOZIE-1722
> 
> 
> Repository: oozie-git
> 
> 
> Description
> -------
> 
> See https://issues.apache.org/jira/browse/OOZIE-1722
> 
> 
> Diffs
> -----
> 
>   core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java fcf69e7 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 68d77a8 
>   core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 07de214 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 7841076 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestLauncher.java b55b9d7 
>   core/src/test/java/org/apache/oozie/service/TestShareLibService.java e7d7a26 
>   core/src/test/java/org/apache/oozie/workflow/lite/TestLiteWorkflowAppParser.java 41e6205 
>   hadooplibs/hadoop-utils-0.23/pom.xml PRE-CREATION 
>   hadooplibs/hadoop-utils-0.23/src/main/java/org/apache/oozie/action/hadoop/LauncherMainHadoopUtils.java PRE-CREATION 
>   hadooplibs/hadoop-utils-1/pom.xml PRE-CREATION 
>   hadooplibs/hadoop-utils-1/src/main/java/org/apache/oozie/action/hadoop/LauncherMainHadoopUtils.java PRE-CREATION 
>   hadooplibs/hadoop-utils-2/pom.xml PRE-CREATION 
>   hadooplibs/hadoop-utils-2/src/main/java/org/apache/oozie/action/hadoop/LauncherMainHadoopUtils.java PRE-CREATION 
>   hadooplibs/hadoop-utils-3/pom.xml PRE-CREATION 
>   hadooplibs/hadoop-utils-3/src/main/java/org/apache/oozie/action/hadoop/LauncherMainHadoopUtils.java PRE-CREATION 
>   hadooplibs/pom.xml 35875c8 
>   pom.xml 1223ec2 
>   release-log.txt f6810bc 
>   sharelib/distcp/src/test/java/org/apache/oozie/action/hadoop/TestDistCpActionExecutor.java 25b1a23 
>   sharelib/hive/pom.xml e355d97 
>   sharelib/hive/src/main/java/org/apache/oozie/action/hadoop/HiveMain.java 8fa5ec8 
>   sharelib/oozie/pom.xml c70582d 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/JavaMain.java PRE-CREATION 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java b5fa4a1 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/MapReduceMain.java 6230e64 
>   sharelib/pig/pom.xml a2bbeaf 
>   sharelib/pig/src/main/java/org/apache/oozie/action/hadoop/PigMain.java 9c1e659 
>   sharelib/sqoop/pom.xml a962565 
>   sharelib/sqoop/src/main/java/org/apache/oozie/action/hadoop/SqoopMain.java e45c8b2 
>   sharelib/streaming/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/18653/diff/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Robert Kanter
> 
>


Re: Review Request 18653: OOZIE-1722: When an ApplicationMaster restarts, it restarts the launcher job

Posted by Rohini Palaniswamy <ro...@gmail.com>.

> On March 3, 2014, 7:30 a.m., Rohini Palaniswamy wrote:
> > We need to add the killChildJobs in Java, Shell and Distcp as well. Many people launch hbase jobs from java action and instead of each user adding killChildJobs doing it in JavaActionExecutor should be good. Currently the configuration is not passed properly to distcp action. For killing child jobs in distcp to work, we need to fix that or pass the configuration as arg (i.e -Dmapreduce.job.tags=xxx) so that child jobs are launched with that tag. I think we can pass it as arg for now.
> 
> Robert Kanter wrote:
>     The reason I didn't add the killChildJobs to the Distcp action is that we don't have a "DistcpMain" to put it in.  I didn't want to put the killChildJobs in the calling class LauncherMapper because (a) we handle this differently for the MR action and (b) someone writing a custom action (or us in a future action) may also want to handle this differently.  I guess I could write a wrapper "DistcpMain" to call killChildJobs...
>     
>     For the Java and Shell actions, they don't necessarily launch any child jobs.  And in general, we don't implement some of the "Oozie magic" stuff for the Java and Shell actions, leaving it up to the user instead (for example, propagating the action's <configuration> or the delegation token).  The Java action also has the same problem as the Distcp action, in that there's no "JavaMain".  What if I added documentation on how to call killChildJobs so the user can do this in their Main?

Many users launch child jobs in their java actions (especially with hbase). Would prefer as much less work for user as possible. If you are writing a DistcpMain, you can just write a JavaMain that calls killChildJobs and make DistcpMain just extend that. That way it will just work without users having to bother about it and writing different code for different versions of hadoop. 


- Rohini


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18653/#review35947
-----------------------------------------------------------


On March 1, 2014, 2:01 a.m., Robert Kanter wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/18653/
> -----------------------------------------------------------
> 
> (Updated March 1, 2014, 2:01 a.m.)
> 
> 
> Review request for oozie.
> 
> 
> Bugs: OOZIE-1722
>     https://issues.apache.org/jira/browse/OOZIE-1722
> 
> 
> Repository: oozie-git
> 
> 
> Description
> -------
> 
> See https://issues.apache.org/jira/browse/OOZIE-1722
> 
> 
> Diffs
> -----
> 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 68d77a8 
>   core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 07de214 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestLauncher.java b55b9d7 
>   hadooplibs/hadoop-utils-0.23/pom.xml PRE-CREATION 
>   hadooplibs/hadoop-utils-0.23/src/main/java/org/apache/oozie/action/hadoop/LauncherMainHadoopUtils.java PRE-CREATION 
>   hadooplibs/hadoop-utils-1/pom.xml PRE-CREATION 
>   hadooplibs/hadoop-utils-1/src/main/java/org/apache/oozie/action/hadoop/LauncherMainHadoopUtils.java PRE-CREATION 
>   hadooplibs/hadoop-utils-2/pom.xml PRE-CREATION 
>   hadooplibs/hadoop-utils-2/src/main/java/org/apache/oozie/action/hadoop/LauncherMainHadoopUtils.java PRE-CREATION 
>   hadooplibs/hadoop-utils-3/pom.xml PRE-CREATION 
>   hadooplibs/hadoop-utils-3/src/main/java/org/apache/oozie/action/hadoop/LauncherMainHadoopUtils.java PRE-CREATION 
>   hadooplibs/pom.xml 35875c8 
>   pom.xml 1223ec2 
>   sharelib/hive/pom.xml e355d97 
>   sharelib/hive/src/main/java/org/apache/oozie/action/hadoop/HiveMain.java 8fa5ec8 
>   sharelib/oozie/pom.xml c70582d 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java b5fa4a1 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/MapReduceMain.java 6230e64 
>   sharelib/pig/pom.xml a2bbeaf 
>   sharelib/pig/src/main/java/org/apache/oozie/action/hadoop/PigMain.java 9c1e659 
>   sharelib/sqoop/pom.xml a962565 
>   sharelib/sqoop/src/main/java/org/apache/oozie/action/hadoop/SqoopMain.java e45c8b2 
> 
> Diff: https://reviews.apache.org/r/18653/diff/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Robert Kanter
> 
>


Re: Review Request 18653: OOZIE-1722: When an ApplicationMaster restarts, it restarts the launcher job

Posted by Rohini Palaniswamy <ro...@gmail.com>.

> On March 3, 2014, 7:30 a.m., Rohini Palaniswamy wrote:
> > We need to add the killChildJobs in Java, Shell and Distcp as well. Many people launch hbase jobs from java action and instead of each user adding killChildJobs doing it in JavaActionExecutor should be good. Currently the configuration is not passed properly to distcp action. For killing child jobs in distcp to work, we need to fix that or pass the configuration as arg (i.e -Dmapreduce.job.tags=xxx) so that child jobs are launched with that tag. I think we can pass it as arg for now.
> 
> Robert Kanter wrote:
>     The reason I didn't add the killChildJobs to the Distcp action is that we don't have a "DistcpMain" to put it in.  I didn't want to put the killChildJobs in the calling class LauncherMapper because (a) we handle this differently for the MR action and (b) someone writing a custom action (or us in a future action) may also want to handle this differently.  I guess I could write a wrapper "DistcpMain" to call killChildJobs...
>     
>     For the Java and Shell actions, they don't necessarily launch any child jobs.  And in general, we don't implement some of the "Oozie magic" stuff for the Java and Shell actions, leaving it up to the user instead (for example, propagating the action's <configuration> or the delegation token).  The Java action also has the same problem as the Distcp action, in that there's no "JavaMain".  What if I added documentation on how to call killChildJobs so the user can do this in their Main?
> 
> Rohini Palaniswamy wrote:
>     Many users launch child jobs in their java actions (especially with hbase). Would prefer as much less work for user as possible. If you are writing a DistcpMain, you can just write a JavaMain that calls killChildJobs and make DistcpMain just extend that. That way it will just work without users having to bother about it and writing different code for different versions of hadoop.
> 
> Robert Kanter wrote:
>     I can add a wrapper for Java and Distcp that does the killing.  Though for the Java action, it will still be up to the user to propagate the actionConf prepared by Oozie, or at least just set the tag property.
> 
> Rohini Palaniswamy wrote:
>     Sounds good and thanks. Most of them usually propagate the actionConf and would benefit from this.
> 
> Robert Kanter wrote:
>     Getting this to work with DistCp is trickier than I thought.  I tried passing the tags property as a -D argument, but that didn't work (in fact, it caused an error).  Looking at DistCp's code, we could try to invoke it programmatically from a "DistCpMain" instead of JavaMain; this would allow us to pass it a Configuration.  However, DistCp v1 (Hadoop 1) and v2 (Hadoop 0.23, Hadoop 2+) don't have a common constructor (v2's constructor takes an additional argument), so the "DistCpMain" would have to use Reflection and check which constructor is available.  MAPREDUCE-5081 did backport v2 to Hadoop 1, but its only in Hadoop 1.2.0 and we're using 1.1.1
>     
>     So we have a few options:
>     (1) Don't support the tags with DistCp
>     (2) Use a "DistCpMain"
>        (2a) Use reflection to call the appropriate constructor and run it programmatically
>        (2b) Update to Hadoop 1.2.x and use DistCp v2 for each version of Hadoop
>     
>     I think we should do (1) for now so we can get the tags stuff in and create a follow-up JIRA to do (2b), which will require a lot of additional testing.
>     
>     Thoughts?
> 
> Rohini Palaniswamy wrote:
>     Option 3:
>      <arg>-Dmapred.job.queue.name=${queueName}</arg> is how queue information is currently passed to Distcp from the distcp action. LauncherMapperHelper.setupMainArguments and LauncherMapper.getMainArguments handle the main method arguments. We need to introduce it somewhere there for Distcp. But it is hacky and don't like it much. Just pointing out as an option.
>     
>      Your option 2 is cleaner and like it better. We definitely need to fix DistcpMain to call the constructor instead of the main() method to pass the JobConf. Users have been passing even the queue information as above because the distcp job does not get the action configuration. We had this reported as an issue, but have always gone with the <arg> workaround. So option 2 will fix more issues. We can definitely do (2a). (2b) of course can be a different jira.

Actually with your current patch you can easily do Option 3 (passing -Dmapreduce.job.tags=xxx as main args) with DistcpMain as you would be invoking the Distcp main class from there. Was that what you tried and did not work?


- Rohini


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18653/#review35947
-----------------------------------------------------------


On March 4, 2014, 3:11 a.m., Robert Kanter wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/18653/
> -----------------------------------------------------------
> 
> (Updated March 4, 2014, 3:11 a.m.)
> 
> 
> Review request for oozie.
> 
> 
> Bugs: OOZIE-1722
>     https://issues.apache.org/jira/browse/OOZIE-1722
> 
> 
> Repository: oozie-git
> 
> 
> Description
> -------
> 
> See https://issues.apache.org/jira/browse/OOZIE-1722
> 
> 
> Diffs
> -----
> 
>   core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java fcf69e7 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 68d77a8 
>   core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 07de214 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 7841076 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestLauncher.java b55b9d7 
>   core/src/test/java/org/apache/oozie/service/TestShareLibService.java e7d7a26 
>   core/src/test/java/org/apache/oozie/workflow/lite/TestLiteWorkflowAppParser.java 41e6205 
>   hadooplibs/hadoop-utils-0.23/pom.xml PRE-CREATION 
>   hadooplibs/hadoop-utils-0.23/src/main/java/org/apache/oozie/action/hadoop/LauncherMainHadoopUtils.java PRE-CREATION 
>   hadooplibs/hadoop-utils-1/pom.xml PRE-CREATION 
>   hadooplibs/hadoop-utils-1/src/main/java/org/apache/oozie/action/hadoop/LauncherMainHadoopUtils.java PRE-CREATION 
>   hadooplibs/hadoop-utils-2/pom.xml PRE-CREATION 
>   hadooplibs/hadoop-utils-2/src/main/java/org/apache/oozie/action/hadoop/LauncherMainHadoopUtils.java PRE-CREATION 
>   hadooplibs/hadoop-utils-3/pom.xml PRE-CREATION 
>   hadooplibs/hadoop-utils-3/src/main/java/org/apache/oozie/action/hadoop/LauncherMainHadoopUtils.java PRE-CREATION 
>   hadooplibs/pom.xml 35875c8 
>   pom.xml 1223ec2 
>   release-log.txt f6810bc 
>   sharelib/distcp/src/test/java/org/apache/oozie/action/hadoop/TestDistCpActionExecutor.java 25b1a23 
>   sharelib/hive/pom.xml e355d97 
>   sharelib/hive/src/main/java/org/apache/oozie/action/hadoop/HiveMain.java 8fa5ec8 
>   sharelib/oozie/pom.xml c70582d 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/JavaMain.java PRE-CREATION 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java b5fa4a1 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/MapReduceMain.java 6230e64 
>   sharelib/pig/pom.xml a2bbeaf 
>   sharelib/pig/src/main/java/org/apache/oozie/action/hadoop/PigMain.java 9c1e659 
>   sharelib/sqoop/pom.xml a962565 
>   sharelib/sqoop/src/main/java/org/apache/oozie/action/hadoop/SqoopMain.java e45c8b2 
>   sharelib/streaming/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/18653/diff/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Robert Kanter
> 
>


Re: Review Request 18653: OOZIE-1722: When an ApplicationMaster restarts, it restarts the launcher job

Posted by Robert Kanter <rk...@cloudera.com>.

> On March 3, 2014, 7:30 a.m., Rohini Palaniswamy wrote:
> > We need to add the killChildJobs in Java, Shell and Distcp as well. Many people launch hbase jobs from java action and instead of each user adding killChildJobs doing it in JavaActionExecutor should be good. Currently the configuration is not passed properly to distcp action. For killing child jobs in distcp to work, we need to fix that or pass the configuration as arg (i.e -Dmapreduce.job.tags=xxx) so that child jobs are launched with that tag. I think we can pass it as arg for now.
> 
> Robert Kanter wrote:
>     The reason I didn't add the killChildJobs to the Distcp action is that we don't have a "DistcpMain" to put it in.  I didn't want to put the killChildJobs in the calling class LauncherMapper because (a) we handle this differently for the MR action and (b) someone writing a custom action (or us in a future action) may also want to handle this differently.  I guess I could write a wrapper "DistcpMain" to call killChildJobs...
>     
>     For the Java and Shell actions, they don't necessarily launch any child jobs.  And in general, we don't implement some of the "Oozie magic" stuff for the Java and Shell actions, leaving it up to the user instead (for example, propagating the action's <configuration> or the delegation token).  The Java action also has the same problem as the Distcp action, in that there's no "JavaMain".  What if I added documentation on how to call killChildJobs so the user can do this in their Main?
> 
> Rohini Palaniswamy wrote:
>     Many users launch child jobs in their java actions (especially with hbase). Would prefer as much less work for user as possible. If you are writing a DistcpMain, you can just write a JavaMain that calls killChildJobs and make DistcpMain just extend that. That way it will just work without users having to bother about it and writing different code for different versions of hadoop.

I can add a wrapper for Java and Distcp that does the killing.  Though for the Java action, it will still be up to the user to propagate the actionConf prepared by Oozie, or at least just set the tag property.


- Robert


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18653/#review35947
-----------------------------------------------------------


On March 1, 2014, 2:01 a.m., Robert Kanter wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/18653/
> -----------------------------------------------------------
> 
> (Updated March 1, 2014, 2:01 a.m.)
> 
> 
> Review request for oozie.
> 
> 
> Bugs: OOZIE-1722
>     https://issues.apache.org/jira/browse/OOZIE-1722
> 
> 
> Repository: oozie-git
> 
> 
> Description
> -------
> 
> See https://issues.apache.org/jira/browse/OOZIE-1722
> 
> 
> Diffs
> -----
> 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 68d77a8 
>   core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 07de214 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestLauncher.java b55b9d7 
>   hadooplibs/hadoop-utils-0.23/pom.xml PRE-CREATION 
>   hadooplibs/hadoop-utils-0.23/src/main/java/org/apache/oozie/action/hadoop/LauncherMainHadoopUtils.java PRE-CREATION 
>   hadooplibs/hadoop-utils-1/pom.xml PRE-CREATION 
>   hadooplibs/hadoop-utils-1/src/main/java/org/apache/oozie/action/hadoop/LauncherMainHadoopUtils.java PRE-CREATION 
>   hadooplibs/hadoop-utils-2/pom.xml PRE-CREATION 
>   hadooplibs/hadoop-utils-2/src/main/java/org/apache/oozie/action/hadoop/LauncherMainHadoopUtils.java PRE-CREATION 
>   hadooplibs/hadoop-utils-3/pom.xml PRE-CREATION 
>   hadooplibs/hadoop-utils-3/src/main/java/org/apache/oozie/action/hadoop/LauncherMainHadoopUtils.java PRE-CREATION 
>   hadooplibs/pom.xml 35875c8 
>   pom.xml 1223ec2 
>   sharelib/hive/pom.xml e355d97 
>   sharelib/hive/src/main/java/org/apache/oozie/action/hadoop/HiveMain.java 8fa5ec8 
>   sharelib/oozie/pom.xml c70582d 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java b5fa4a1 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/MapReduceMain.java 6230e64 
>   sharelib/pig/pom.xml a2bbeaf 
>   sharelib/pig/src/main/java/org/apache/oozie/action/hadoop/PigMain.java 9c1e659 
>   sharelib/sqoop/pom.xml a962565 
>   sharelib/sqoop/src/main/java/org/apache/oozie/action/hadoop/SqoopMain.java e45c8b2 
> 
> Diff: https://reviews.apache.org/r/18653/diff/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Robert Kanter
> 
>


Re: Review Request 18653: OOZIE-1722: When an ApplicationMaster restarts, it restarts the launcher job

Posted by Rohini Palaniswamy <ro...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18653/#review35947
-----------------------------------------------------------


We need to add the killChildJobs in Java, Shell and Distcp as well. Many people launch hbase jobs from java action and instead of each user adding killChildJobs doing it in JavaActionExecutor should be good. Currently the configuration is not passed properly to distcp action. For killing child jobs in distcp to work, we need to fix that or pass the configuration as arg (i.e -Dmapreduce.job.tags=xxx) so that child jobs are launched with that tag. I think we can pass it as arg for now.  

- Rohini Palaniswamy


On March 1, 2014, 2:01 a.m., Robert Kanter wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/18653/
> -----------------------------------------------------------
> 
> (Updated March 1, 2014, 2:01 a.m.)
> 
> 
> Review request for oozie.
> 
> 
> Bugs: OOZIE-1722
>     https://issues.apache.org/jira/browse/OOZIE-1722
> 
> 
> Repository: oozie-git
> 
> 
> Description
> -------
> 
> See https://issues.apache.org/jira/browse/OOZIE-1722
> 
> 
> Diffs
> -----
> 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 68d77a8 
>   core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 07de214 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestLauncher.java b55b9d7 
>   hadooplibs/hadoop-utils-0.23/pom.xml PRE-CREATION 
>   hadooplibs/hadoop-utils-0.23/src/main/java/org/apache/oozie/action/hadoop/LauncherMainHadoopUtils.java PRE-CREATION 
>   hadooplibs/hadoop-utils-1/pom.xml PRE-CREATION 
>   hadooplibs/hadoop-utils-1/src/main/java/org/apache/oozie/action/hadoop/LauncherMainHadoopUtils.java PRE-CREATION 
>   hadooplibs/hadoop-utils-2/pom.xml PRE-CREATION 
>   hadooplibs/hadoop-utils-2/src/main/java/org/apache/oozie/action/hadoop/LauncherMainHadoopUtils.java PRE-CREATION 
>   hadooplibs/hadoop-utils-3/pom.xml PRE-CREATION 
>   hadooplibs/hadoop-utils-3/src/main/java/org/apache/oozie/action/hadoop/LauncherMainHadoopUtils.java PRE-CREATION 
>   hadooplibs/pom.xml 35875c8 
>   pom.xml 1223ec2 
>   sharelib/hive/pom.xml e355d97 
>   sharelib/hive/src/main/java/org/apache/oozie/action/hadoop/HiveMain.java 8fa5ec8 
>   sharelib/oozie/pom.xml c70582d 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java b5fa4a1 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/MapReduceMain.java 6230e64 
>   sharelib/pig/pom.xml a2bbeaf 
>   sharelib/pig/src/main/java/org/apache/oozie/action/hadoop/PigMain.java 9c1e659 
>   sharelib/sqoop/pom.xml a962565 
>   sharelib/sqoop/src/main/java/org/apache/oozie/action/hadoop/SqoopMain.java e45c8b2 
> 
> Diff: https://reviews.apache.org/r/18653/diff/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Robert Kanter
> 
>


Re: Review Request 18653: OOZIE-1722: When an ApplicationMaster restarts, it restarts the launcher job

Posted by Robert Kanter <rk...@cloudera.com>.

> On March 4, 2014, 10:05 p.m., Rohini Palaniswamy wrote:
> > Missed a question earlier. Why do we have to move TEstJavaActionExecutor to streaming module?

I had originally moved TestJavaActionExecutor to the Oozie sharelib (to be in the same module as JavaMain), but that won't compile because TestJavaActionExecutor requires stuff in Oozie core; so I moved it to the streaming sharelib, forgetting that I could have just moved it back to Oozie core :)

I'll upload a new patch.


- Robert


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18653/#review36177
-----------------------------------------------------------


On March 4, 2014, 9:48 p.m., Robert Kanter wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/18653/
> -----------------------------------------------------------
> 
> (Updated March 4, 2014, 9:48 p.m.)
> 
> 
> Review request for oozie.
> 
> 
> Bugs: OOZIE-1722
>     https://issues.apache.org/jira/browse/OOZIE-1722
> 
> 
> Repository: oozie-git
> 
> 
> Description
> -------
> 
> See https://issues.apache.org/jira/browse/OOZIE-1722
> 
> 
> Diffs
> -----
> 
>   core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java fcf69e7 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 68d77a8 
>   core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 07de214 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 7841076 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestLauncher.java b55b9d7 
>   core/src/test/java/org/apache/oozie/service/TestShareLibService.java e7d7a26 
>   hadooplibs/hadoop-utils-0.23/pom.xml PRE-CREATION 
>   hadooplibs/hadoop-utils-0.23/src/main/java/org/apache/oozie/action/hadoop/LauncherMainHadoopUtils.java PRE-CREATION 
>   hadooplibs/hadoop-utils-1/pom.xml PRE-CREATION 
>   hadooplibs/hadoop-utils-1/src/main/java/org/apache/oozie/action/hadoop/LauncherMainHadoopUtils.java PRE-CREATION 
>   hadooplibs/hadoop-utils-2/pom.xml PRE-CREATION 
>   hadooplibs/hadoop-utils-2/src/main/java/org/apache/oozie/action/hadoop/LauncherMainHadoopUtils.java PRE-CREATION 
>   hadooplibs/hadoop-utils-3/pom.xml PRE-CREATION 
>   hadooplibs/hadoop-utils-3/src/main/java/org/apache/oozie/action/hadoop/LauncherMainHadoopUtils.java PRE-CREATION 
>   hadooplibs/pom.xml 35875c8 
>   pom.xml 1223ec2 
>   sharelib/distcp/src/test/java/org/apache/oozie/action/hadoop/TestDistCpActionExecutor.java 25b1a23 
>   sharelib/hive/pom.xml e355d97 
>   sharelib/hive/src/main/java/org/apache/oozie/action/hadoop/HiveMain.java 8fa5ec8 
>   sharelib/oozie/pom.xml c70582d 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/JavaMain.java PRE-CREATION 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java b5fa4a1 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/MapReduceMain.java 6230e64 
>   sharelib/pig/pom.xml a2bbeaf 
>   sharelib/pig/src/main/java/org/apache/oozie/action/hadoop/PigMain.java 9c1e659 
>   sharelib/sqoop/pom.xml a962565 
>   sharelib/sqoop/src/main/java/org/apache/oozie/action/hadoop/SqoopMain.java e45c8b2 
>   sharelib/streaming/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/18653/diff/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Robert Kanter
> 
>


Re: Review Request 18653: OOZIE-1722: When an ApplicationMaster restarts, it restarts the launcher job

Posted by Rohini Palaniswamy <ro...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18653/#review36177
-----------------------------------------------------------


Missed a question earlier. Why do we have to move TEstJavaActionExecutor to streaming module?

- Rohini Palaniswamy


On March 4, 2014, 9:48 p.m., Robert Kanter wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/18653/
> -----------------------------------------------------------
> 
> (Updated March 4, 2014, 9:48 p.m.)
> 
> 
> Review request for oozie.
> 
> 
> Bugs: OOZIE-1722
>     https://issues.apache.org/jira/browse/OOZIE-1722
> 
> 
> Repository: oozie-git
> 
> 
> Description
> -------
> 
> See https://issues.apache.org/jira/browse/OOZIE-1722
> 
> 
> Diffs
> -----
> 
>   core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java fcf69e7 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 68d77a8 
>   core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 07de214 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 7841076 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestLauncher.java b55b9d7 
>   core/src/test/java/org/apache/oozie/service/TestShareLibService.java e7d7a26 
>   hadooplibs/hadoop-utils-0.23/pom.xml PRE-CREATION 
>   hadooplibs/hadoop-utils-0.23/src/main/java/org/apache/oozie/action/hadoop/LauncherMainHadoopUtils.java PRE-CREATION 
>   hadooplibs/hadoop-utils-1/pom.xml PRE-CREATION 
>   hadooplibs/hadoop-utils-1/src/main/java/org/apache/oozie/action/hadoop/LauncherMainHadoopUtils.java PRE-CREATION 
>   hadooplibs/hadoop-utils-2/pom.xml PRE-CREATION 
>   hadooplibs/hadoop-utils-2/src/main/java/org/apache/oozie/action/hadoop/LauncherMainHadoopUtils.java PRE-CREATION 
>   hadooplibs/hadoop-utils-3/pom.xml PRE-CREATION 
>   hadooplibs/hadoop-utils-3/src/main/java/org/apache/oozie/action/hadoop/LauncherMainHadoopUtils.java PRE-CREATION 
>   hadooplibs/pom.xml 35875c8 
>   pom.xml 1223ec2 
>   sharelib/distcp/src/test/java/org/apache/oozie/action/hadoop/TestDistCpActionExecutor.java 25b1a23 
>   sharelib/hive/pom.xml e355d97 
>   sharelib/hive/src/main/java/org/apache/oozie/action/hadoop/HiveMain.java 8fa5ec8 
>   sharelib/oozie/pom.xml c70582d 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/JavaMain.java PRE-CREATION 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java b5fa4a1 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/MapReduceMain.java 6230e64 
>   sharelib/pig/pom.xml a2bbeaf 
>   sharelib/pig/src/main/java/org/apache/oozie/action/hadoop/PigMain.java 9c1e659 
>   sharelib/sqoop/pom.xml a962565 
>   sharelib/sqoop/src/main/java/org/apache/oozie/action/hadoop/SqoopMain.java e45c8b2 
>   sharelib/streaming/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/18653/diff/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Robert Kanter
> 
>


Re: Review Request 18653: OOZIE-1722: When an ApplicationMaster restarts, it restarts the launcher job

Posted by Rohini Palaniswamy <ro...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18653/#review36183
-----------------------------------------------------------

Ship it!


Ship It!

- Rohini Palaniswamy


On March 4, 2014, 10:27 p.m., Robert Kanter wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/18653/
> -----------------------------------------------------------
> 
> (Updated March 4, 2014, 10:27 p.m.)
> 
> 
> Review request for oozie.
> 
> 
> Bugs: OOZIE-1722
>     https://issues.apache.org/jira/browse/OOZIE-1722
> 
> 
> Repository: oozie-git
> 
> 
> Description
> -------
> 
> See https://issues.apache.org/jira/browse/OOZIE-1722
> 
> 
> Diffs
> -----
> 
>   core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java fcf69e7 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 68d77a8 
>   core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 07de214 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 7841076 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestLauncher.java b55b9d7 
>   core/src/test/java/org/apache/oozie/service/TestShareLibService.java e7d7a26 
>   hadooplibs/hadoop-utils-0.23/pom.xml PRE-CREATION 
>   hadooplibs/hadoop-utils-0.23/src/main/java/org/apache/oozie/action/hadoop/LauncherMainHadoopUtils.java PRE-CREATION 
>   hadooplibs/hadoop-utils-1/pom.xml PRE-CREATION 
>   hadooplibs/hadoop-utils-1/src/main/java/org/apache/oozie/action/hadoop/LauncherMainHadoopUtils.java PRE-CREATION 
>   hadooplibs/hadoop-utils-2/pom.xml PRE-CREATION 
>   hadooplibs/hadoop-utils-2/src/main/java/org/apache/oozie/action/hadoop/LauncherMainHadoopUtils.java PRE-CREATION 
>   hadooplibs/hadoop-utils-3/pom.xml PRE-CREATION 
>   hadooplibs/hadoop-utils-3/src/main/java/org/apache/oozie/action/hadoop/LauncherMainHadoopUtils.java PRE-CREATION 
>   hadooplibs/pom.xml 35875c8 
>   pom.xml 1223ec2 
>   sharelib/distcp/src/test/java/org/apache/oozie/action/hadoop/TestDistCpActionExecutor.java 25b1a23 
>   sharelib/hive/pom.xml e355d97 
>   sharelib/hive/src/main/java/org/apache/oozie/action/hadoop/HiveMain.java 8fa5ec8 
>   sharelib/oozie/pom.xml c70582d 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/JavaMain.java PRE-CREATION 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java b5fa4a1 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/MapReduceMain.java 6230e64 
>   sharelib/pig/pom.xml a2bbeaf 
>   sharelib/pig/src/main/java/org/apache/oozie/action/hadoop/PigMain.java 9c1e659 
>   sharelib/sqoop/pom.xml a962565 
>   sharelib/sqoop/src/main/java/org/apache/oozie/action/hadoop/SqoopMain.java e45c8b2 
> 
> Diff: https://reviews.apache.org/r/18653/diff/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Robert Kanter
> 
>


Re: Review Request 18653: OOZIE-1722: When an ApplicationMaster restarts, it restarts the launcher job

Posted by Robert Kanter <rk...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18653/
-----------------------------------------------------------

(Updated March 4, 2014, 10:27 p.m.)


Review request for oozie.


Changes
-------

Moved TestJavaActionExecutor back to Oozie core.


Bugs: OOZIE-1722
    https://issues.apache.org/jira/browse/OOZIE-1722


Repository: oozie-git


Description
-------

See https://issues.apache.org/jira/browse/OOZIE-1722


Diffs (updated)
-----

  core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java fcf69e7 
  core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 68d77a8 
  core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 07de214 
  core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 7841076 
  core/src/test/java/org/apache/oozie/action/hadoop/TestLauncher.java b55b9d7 
  core/src/test/java/org/apache/oozie/service/TestShareLibService.java e7d7a26 
  hadooplibs/hadoop-utils-0.23/pom.xml PRE-CREATION 
  hadooplibs/hadoop-utils-0.23/src/main/java/org/apache/oozie/action/hadoop/LauncherMainHadoopUtils.java PRE-CREATION 
  hadooplibs/hadoop-utils-1/pom.xml PRE-CREATION 
  hadooplibs/hadoop-utils-1/src/main/java/org/apache/oozie/action/hadoop/LauncherMainHadoopUtils.java PRE-CREATION 
  hadooplibs/hadoop-utils-2/pom.xml PRE-CREATION 
  hadooplibs/hadoop-utils-2/src/main/java/org/apache/oozie/action/hadoop/LauncherMainHadoopUtils.java PRE-CREATION 
  hadooplibs/hadoop-utils-3/pom.xml PRE-CREATION 
  hadooplibs/hadoop-utils-3/src/main/java/org/apache/oozie/action/hadoop/LauncherMainHadoopUtils.java PRE-CREATION 
  hadooplibs/pom.xml 35875c8 
  pom.xml 1223ec2 
  sharelib/distcp/src/test/java/org/apache/oozie/action/hadoop/TestDistCpActionExecutor.java 25b1a23 
  sharelib/hive/pom.xml e355d97 
  sharelib/hive/src/main/java/org/apache/oozie/action/hadoop/HiveMain.java 8fa5ec8 
  sharelib/oozie/pom.xml c70582d 
  sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/JavaMain.java PRE-CREATION 
  sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java b5fa4a1 
  sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/MapReduceMain.java 6230e64 
  sharelib/pig/pom.xml a2bbeaf 
  sharelib/pig/src/main/java/org/apache/oozie/action/hadoop/PigMain.java 9c1e659 
  sharelib/sqoop/pom.xml a962565 
  sharelib/sqoop/src/main/java/org/apache/oozie/action/hadoop/SqoopMain.java e45c8b2 

Diff: https://reviews.apache.org/r/18653/diff/


Testing
-------


Thanks,

Robert Kanter


Re: Review Request 18653: OOZIE-1722: When an ApplicationMaster restarts, it restarts the launcher job

Posted by Robert Kanter <rk...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18653/
-----------------------------------------------------------

(Updated March 4, 2014, 9:48 p.m.)


Review request for oozie.


Bugs: OOZIE-1722
    https://issues.apache.org/jira/browse/OOZIE-1722


Repository: oozie-git


Description
-------

See https://issues.apache.org/jira/browse/OOZIE-1722


Diffs (updated)
-----

  core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java fcf69e7 
  core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 68d77a8 
  core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 07de214 
  core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 7841076 
  core/src/test/java/org/apache/oozie/action/hadoop/TestLauncher.java b55b9d7 
  core/src/test/java/org/apache/oozie/service/TestShareLibService.java e7d7a26 
  hadooplibs/hadoop-utils-0.23/pom.xml PRE-CREATION 
  hadooplibs/hadoop-utils-0.23/src/main/java/org/apache/oozie/action/hadoop/LauncherMainHadoopUtils.java PRE-CREATION 
  hadooplibs/hadoop-utils-1/pom.xml PRE-CREATION 
  hadooplibs/hadoop-utils-1/src/main/java/org/apache/oozie/action/hadoop/LauncherMainHadoopUtils.java PRE-CREATION 
  hadooplibs/hadoop-utils-2/pom.xml PRE-CREATION 
  hadooplibs/hadoop-utils-2/src/main/java/org/apache/oozie/action/hadoop/LauncherMainHadoopUtils.java PRE-CREATION 
  hadooplibs/hadoop-utils-3/pom.xml PRE-CREATION 
  hadooplibs/hadoop-utils-3/src/main/java/org/apache/oozie/action/hadoop/LauncherMainHadoopUtils.java PRE-CREATION 
  hadooplibs/pom.xml 35875c8 
  pom.xml 1223ec2 
  sharelib/distcp/src/test/java/org/apache/oozie/action/hadoop/TestDistCpActionExecutor.java 25b1a23 
  sharelib/hive/pom.xml e355d97 
  sharelib/hive/src/main/java/org/apache/oozie/action/hadoop/HiveMain.java 8fa5ec8 
  sharelib/oozie/pom.xml c70582d 
  sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/JavaMain.java PRE-CREATION 
  sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java b5fa4a1 
  sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/MapReduceMain.java 6230e64 
  sharelib/pig/pom.xml a2bbeaf 
  sharelib/pig/src/main/java/org/apache/oozie/action/hadoop/PigMain.java 9c1e659 
  sharelib/sqoop/pom.xml a962565 
  sharelib/sqoop/src/main/java/org/apache/oozie/action/hadoop/SqoopMain.java e45c8b2 
  sharelib/streaming/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java PRE-CREATION 

Diff: https://reviews.apache.org/r/18653/diff/


Testing
-------


Thanks,

Robert Kanter


Re: Review Request 18653: OOZIE-1722: When an ApplicationMaster restarts, it restarts the launcher job

Posted by Rohini Palaniswamy <ro...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18653/#review36163
-----------------------------------------------------------


Patch is good. Just one minor variable name change comment. Will do final +1 after a decision on distcp based on whether you are able to get it working with (2a) or (3) or we have to go with (1) with no support for tags for distcp action.


core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
<https://reviews.apache.org/r/18653/#comment67083>

    actionConf



core/src/test/java/org/apache/oozie/workflow/lite/TestLiteWorkflowAppParser.java
<https://reviews.apache.org/r/18653/#comment67087>

    Unintended revert of earlier checkin



release-log.txt
<https://reviews.apache.org/r/18653/#comment67082>

    Unintended revert of earlier checkin


- Rohini Palaniswamy


On March 4, 2014, 3:11 a.m., Robert Kanter wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/18653/
> -----------------------------------------------------------
> 
> (Updated March 4, 2014, 3:11 a.m.)
> 
> 
> Review request for oozie.
> 
> 
> Bugs: OOZIE-1722
>     https://issues.apache.org/jira/browse/OOZIE-1722
> 
> 
> Repository: oozie-git
> 
> 
> Description
> -------
> 
> See https://issues.apache.org/jira/browse/OOZIE-1722
> 
> 
> Diffs
> -----
> 
>   core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java fcf69e7 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 68d77a8 
>   core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 07de214 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 7841076 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestLauncher.java b55b9d7 
>   core/src/test/java/org/apache/oozie/service/TestShareLibService.java e7d7a26 
>   core/src/test/java/org/apache/oozie/workflow/lite/TestLiteWorkflowAppParser.java 41e6205 
>   hadooplibs/hadoop-utils-0.23/pom.xml PRE-CREATION 
>   hadooplibs/hadoop-utils-0.23/src/main/java/org/apache/oozie/action/hadoop/LauncherMainHadoopUtils.java PRE-CREATION 
>   hadooplibs/hadoop-utils-1/pom.xml PRE-CREATION 
>   hadooplibs/hadoop-utils-1/src/main/java/org/apache/oozie/action/hadoop/LauncherMainHadoopUtils.java PRE-CREATION 
>   hadooplibs/hadoop-utils-2/pom.xml PRE-CREATION 
>   hadooplibs/hadoop-utils-2/src/main/java/org/apache/oozie/action/hadoop/LauncherMainHadoopUtils.java PRE-CREATION 
>   hadooplibs/hadoop-utils-3/pom.xml PRE-CREATION 
>   hadooplibs/hadoop-utils-3/src/main/java/org/apache/oozie/action/hadoop/LauncherMainHadoopUtils.java PRE-CREATION 
>   hadooplibs/pom.xml 35875c8 
>   pom.xml 1223ec2 
>   release-log.txt f6810bc 
>   sharelib/distcp/src/test/java/org/apache/oozie/action/hadoop/TestDistCpActionExecutor.java 25b1a23 
>   sharelib/hive/pom.xml e355d97 
>   sharelib/hive/src/main/java/org/apache/oozie/action/hadoop/HiveMain.java 8fa5ec8 
>   sharelib/oozie/pom.xml c70582d 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/JavaMain.java PRE-CREATION 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java b5fa4a1 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/MapReduceMain.java 6230e64 
>   sharelib/pig/pom.xml a2bbeaf 
>   sharelib/pig/src/main/java/org/apache/oozie/action/hadoop/PigMain.java 9c1e659 
>   sharelib/sqoop/pom.xml a962565 
>   sharelib/sqoop/src/main/java/org/apache/oozie/action/hadoop/SqoopMain.java e45c8b2 
>   sharelib/streaming/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/18653/diff/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Robert Kanter
> 
>


Re: Review Request 18653: OOZIE-1722: When an ApplicationMaster restarts, it restarts the launcher job

Posted by Robert Kanter <rk...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18653/
-----------------------------------------------------------

(Updated March 4, 2014, 3:11 a.m.)


Review request for oozie.


Changes
-------

I added a JavaMain that the Java and DistCp actions use.  The patch got much bigger because I unfortunately had to move TestJavaActionExecutor into the streaming sharelib.

I couldn't figure out a way to pass the tags property to DistCp itself (I think -Dmapreduce.job.tags=xxx only works when used from the 'hadoop' CLI) so while it currently uses the JavaMain, it doesn't actually use the tags properly yet.  I'll try to figure something out, unless someone has any ideas?


Bugs: OOZIE-1722
    https://issues.apache.org/jira/browse/OOZIE-1722


Repository: oozie-git


Description
-------

See https://issues.apache.org/jira/browse/OOZIE-1722


Diffs (updated)
-----

  core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java fcf69e7 
  core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 68d77a8 
  core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 07de214 
  core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 7841076 
  core/src/test/java/org/apache/oozie/action/hadoop/TestLauncher.java b55b9d7 
  core/src/test/java/org/apache/oozie/service/TestShareLibService.java e7d7a26 
  core/src/test/java/org/apache/oozie/workflow/lite/TestLiteWorkflowAppParser.java 41e6205 
  hadooplibs/hadoop-utils-0.23/pom.xml PRE-CREATION 
  hadooplibs/hadoop-utils-0.23/src/main/java/org/apache/oozie/action/hadoop/LauncherMainHadoopUtils.java PRE-CREATION 
  hadooplibs/hadoop-utils-1/pom.xml PRE-CREATION 
  hadooplibs/hadoop-utils-1/src/main/java/org/apache/oozie/action/hadoop/LauncherMainHadoopUtils.java PRE-CREATION 
  hadooplibs/hadoop-utils-2/pom.xml PRE-CREATION 
  hadooplibs/hadoop-utils-2/src/main/java/org/apache/oozie/action/hadoop/LauncherMainHadoopUtils.java PRE-CREATION 
  hadooplibs/hadoop-utils-3/pom.xml PRE-CREATION 
  hadooplibs/hadoop-utils-3/src/main/java/org/apache/oozie/action/hadoop/LauncherMainHadoopUtils.java PRE-CREATION 
  hadooplibs/pom.xml 35875c8 
  pom.xml 1223ec2 
  release-log.txt f6810bc 
  sharelib/distcp/src/test/java/org/apache/oozie/action/hadoop/TestDistCpActionExecutor.java 25b1a23 
  sharelib/hive/pom.xml e355d97 
  sharelib/hive/src/main/java/org/apache/oozie/action/hadoop/HiveMain.java 8fa5ec8 
  sharelib/oozie/pom.xml c70582d 
  sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/JavaMain.java PRE-CREATION 
  sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java b5fa4a1 
  sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/MapReduceMain.java 6230e64 
  sharelib/pig/pom.xml a2bbeaf 
  sharelib/pig/src/main/java/org/apache/oozie/action/hadoop/PigMain.java 9c1e659 
  sharelib/sqoop/pom.xml a962565 
  sharelib/sqoop/src/main/java/org/apache/oozie/action/hadoop/SqoopMain.java e45c8b2 
  sharelib/streaming/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java PRE-CREATION 

Diff: https://reviews.apache.org/r/18653/diff/


Testing
-------


Thanks,

Robert Kanter


Re: Review Request 18653: OOZIE-1722: When an ApplicationMaster restarts, it restarts the launcher job

Posted by Robert Kanter <rk...@cloudera.com>.

> On March 3, 2014, 7:24 a.m., Rohini Palaniswamy wrote:
> > I like the idea that for a mapreduce action we just exit if there is already a job. But can you check if the child job id of the mapreduce job is propagated back to the externalChildIds column in the database in that case? If not lot of things like stats, etc will be broken.

That would involve the launcher somehow checking the Oozie database (or with the Oozie server to check the OOzie database).  And the restart is transparent to the Oozie server, so it can't do anything either.  Would it be enough if we checked that the idFile was written to HDFS?  (and if not, we can then go and write it, still without killing the child job).


> On March 3, 2014, 7:24 a.m., Rohini Palaniswamy wrote:
> > hadooplibs/pom.xml, line 38
> > <https://reviews.apache.org/r/18653/diff/2/?file=507592#file507592line38>
> >
> >     Do we need separate modules? Can't we make it part of hadoop-1/0.23/2 itself? I am ok either way.

I tried that first, but the problem is that we want to pull in the jar with LauncherMainHadoopUtils in it as a compile dependency in the Oozie sharelib, but only want to pull in the hadooplibs as provided.  


- Robert


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18653/#review35945
-----------------------------------------------------------


On March 1, 2014, 2:01 a.m., Robert Kanter wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/18653/
> -----------------------------------------------------------
> 
> (Updated March 1, 2014, 2:01 a.m.)
> 
> 
> Review request for oozie.
> 
> 
> Bugs: OOZIE-1722
>     https://issues.apache.org/jira/browse/OOZIE-1722
> 
> 
> Repository: oozie-git
> 
> 
> Description
> -------
> 
> See https://issues.apache.org/jira/browse/OOZIE-1722
> 
> 
> Diffs
> -----
> 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 68d77a8 
>   core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 07de214 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestLauncher.java b55b9d7 
>   hadooplibs/hadoop-utils-0.23/pom.xml PRE-CREATION 
>   hadooplibs/hadoop-utils-0.23/src/main/java/org/apache/oozie/action/hadoop/LauncherMainHadoopUtils.java PRE-CREATION 
>   hadooplibs/hadoop-utils-1/pom.xml PRE-CREATION 
>   hadooplibs/hadoop-utils-1/src/main/java/org/apache/oozie/action/hadoop/LauncherMainHadoopUtils.java PRE-CREATION 
>   hadooplibs/hadoop-utils-2/pom.xml PRE-CREATION 
>   hadooplibs/hadoop-utils-2/src/main/java/org/apache/oozie/action/hadoop/LauncherMainHadoopUtils.java PRE-CREATION 
>   hadooplibs/hadoop-utils-3/pom.xml PRE-CREATION 
>   hadooplibs/hadoop-utils-3/src/main/java/org/apache/oozie/action/hadoop/LauncherMainHadoopUtils.java PRE-CREATION 
>   hadooplibs/pom.xml 35875c8 
>   pom.xml 1223ec2 
>   sharelib/hive/pom.xml e355d97 
>   sharelib/hive/src/main/java/org/apache/oozie/action/hadoop/HiveMain.java 8fa5ec8 
>   sharelib/oozie/pom.xml c70582d 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java b5fa4a1 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/MapReduceMain.java 6230e64 
>   sharelib/pig/pom.xml a2bbeaf 
>   sharelib/pig/src/main/java/org/apache/oozie/action/hadoop/PigMain.java 9c1e659 
>   sharelib/sqoop/pom.xml a962565 
>   sharelib/sqoop/src/main/java/org/apache/oozie/action/hadoop/SqoopMain.java e45c8b2 
> 
> Diff: https://reviews.apache.org/r/18653/diff/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Robert Kanter
> 
>


Re: Review Request 18653: OOZIE-1722: When an ApplicationMaster restarts, it restarts the launcher job

Posted by Karthik Kambatla <ka...@cloudera.com>.

> On March 3, 2014, 7:24 a.m., Rohini Palaniswamy wrote:
> > core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java, line 174
> > <https://reviews.apache.org/r/18653/diff/2/?file=507582#file507582line174>
> >
> >     Shouldn't the hadoop team be naming it as yarn.application.tags as it is not mapreduce specific but yarn specific? Can you check with Karthik Kambatla about the naming of the config? Don't want it to be deprecated soon and renamed.

YARN doesn't know about this config. The config is just a convenience in MR, and is used to set the corresponding field in ApplicationSubmissionContext (ASC). So, if a different AM (may be Oozie AM - that would be cool :)) uses this config, it has no effect unless the AM implements applying this config to the ASC. I don't see a risk of the config being deprecated/renamed. 


- Karthik


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18653/#review35945
-----------------------------------------------------------


On March 1, 2014, 2:01 a.m., Robert Kanter wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/18653/
> -----------------------------------------------------------
> 
> (Updated March 1, 2014, 2:01 a.m.)
> 
> 
> Review request for oozie.
> 
> 
> Bugs: OOZIE-1722
>     https://issues.apache.org/jira/browse/OOZIE-1722
> 
> 
> Repository: oozie-git
> 
> 
> Description
> -------
> 
> See https://issues.apache.org/jira/browse/OOZIE-1722
> 
> 
> Diffs
> -----
> 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 68d77a8 
>   core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 07de214 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestLauncher.java b55b9d7 
>   hadooplibs/hadoop-utils-0.23/pom.xml PRE-CREATION 
>   hadooplibs/hadoop-utils-0.23/src/main/java/org/apache/oozie/action/hadoop/LauncherMainHadoopUtils.java PRE-CREATION 
>   hadooplibs/hadoop-utils-1/pom.xml PRE-CREATION 
>   hadooplibs/hadoop-utils-1/src/main/java/org/apache/oozie/action/hadoop/LauncherMainHadoopUtils.java PRE-CREATION 
>   hadooplibs/hadoop-utils-2/pom.xml PRE-CREATION 
>   hadooplibs/hadoop-utils-2/src/main/java/org/apache/oozie/action/hadoop/LauncherMainHadoopUtils.java PRE-CREATION 
>   hadooplibs/hadoop-utils-3/pom.xml PRE-CREATION 
>   hadooplibs/hadoop-utils-3/src/main/java/org/apache/oozie/action/hadoop/LauncherMainHadoopUtils.java PRE-CREATION 
>   hadooplibs/pom.xml 35875c8 
>   pom.xml 1223ec2 
>   sharelib/hive/pom.xml e355d97 
>   sharelib/hive/src/main/java/org/apache/oozie/action/hadoop/HiveMain.java 8fa5ec8 
>   sharelib/oozie/pom.xml c70582d 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java b5fa4a1 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/MapReduceMain.java 6230e64 
>   sharelib/pig/pom.xml a2bbeaf 
>   sharelib/pig/src/main/java/org/apache/oozie/action/hadoop/PigMain.java 9c1e659 
>   sharelib/sqoop/pom.xml a962565 
>   sharelib/sqoop/src/main/java/org/apache/oozie/action/hadoop/SqoopMain.java e45c8b2 
> 
> Diff: https://reviews.apache.org/r/18653/diff/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Robert Kanter
> 
>


Re: Review Request 18653: OOZIE-1722: When an ApplicationMaster restarts, it restarts the launcher job

Posted by Rohini Palaniswamy <ro...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18653/#review35945
-----------------------------------------------------------


I like the idea that for a mapreduce action we just exit if there is already a job. But can you check if the child job id of the mapreduce job is propagated back to the externalChildIds column in the database in that case? If not lot of things like stats, etc will be broken.  


core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java
<https://reviews.apache.org/r/18653/#comment66751>

    Shouldn't the hadoop team be naming it as yarn.application.tags as it is not mapreduce specific but yarn specific? Can you check with Karthik Kambatla about the naming of the config? Don't want it to be deprecated soon and renamed.



hadooplibs/hadoop-utils-1/src/main/java/org/apache/oozie/action/hadoop/LauncherMainHadoopUtils.java
<https://reviews.apache.org/r/18653/#comment66753>

    Since all the methods are static can you remove the abstract classifier and just have a private constructor?



hadooplibs/pom.xml
<https://reviews.apache.org/r/18653/#comment66752>

    Do we need separate modules? Can't we make it part of hadoop-1/0.23/2 itself? I am ok either way.


- Rohini Palaniswamy


On March 1, 2014, 2:01 a.m., Robert Kanter wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/18653/
> -----------------------------------------------------------
> 
> (Updated March 1, 2014, 2:01 a.m.)
> 
> 
> Review request for oozie.
> 
> 
> Bugs: OOZIE-1722
>     https://issues.apache.org/jira/browse/OOZIE-1722
> 
> 
> Repository: oozie-git
> 
> 
> Description
> -------
> 
> See https://issues.apache.org/jira/browse/OOZIE-1722
> 
> 
> Diffs
> -----
> 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 68d77a8 
>   core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 07de214 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestLauncher.java b55b9d7 
>   hadooplibs/hadoop-utils-0.23/pom.xml PRE-CREATION 
>   hadooplibs/hadoop-utils-0.23/src/main/java/org/apache/oozie/action/hadoop/LauncherMainHadoopUtils.java PRE-CREATION 
>   hadooplibs/hadoop-utils-1/pom.xml PRE-CREATION 
>   hadooplibs/hadoop-utils-1/src/main/java/org/apache/oozie/action/hadoop/LauncherMainHadoopUtils.java PRE-CREATION 
>   hadooplibs/hadoop-utils-2/pom.xml PRE-CREATION 
>   hadooplibs/hadoop-utils-2/src/main/java/org/apache/oozie/action/hadoop/LauncherMainHadoopUtils.java PRE-CREATION 
>   hadooplibs/hadoop-utils-3/pom.xml PRE-CREATION 
>   hadooplibs/hadoop-utils-3/src/main/java/org/apache/oozie/action/hadoop/LauncherMainHadoopUtils.java PRE-CREATION 
>   hadooplibs/pom.xml 35875c8 
>   pom.xml 1223ec2 
>   sharelib/hive/pom.xml e355d97 
>   sharelib/hive/src/main/java/org/apache/oozie/action/hadoop/HiveMain.java 8fa5ec8 
>   sharelib/oozie/pom.xml c70582d 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java b5fa4a1 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/MapReduceMain.java 6230e64 
>   sharelib/pig/pom.xml a2bbeaf 
>   sharelib/pig/src/main/java/org/apache/oozie/action/hadoop/PigMain.java 9c1e659 
>   sharelib/sqoop/pom.xml a962565 
>   sharelib/sqoop/src/main/java/org/apache/oozie/action/hadoop/SqoopMain.java e45c8b2 
> 
> Diff: https://reviews.apache.org/r/18653/diff/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Robert Kanter
> 
>


Re: Review Request 18653: OOZIE-1722: When an ApplicationMaster restarts, it restarts the launcher job

Posted by Robert Kanter <rk...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18653/
-----------------------------------------------------------

(Updated March 1, 2014, 2:01 a.m.)


Review request for oozie.


Changes
-------

Fixes test failures


Bugs: OOZIE-1722
    https://issues.apache.org/jira/browse/OOZIE-1722


Repository: oozie-git


Description
-------

See https://issues.apache.org/jira/browse/OOZIE-1722


Diffs (updated)
-----

  core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 68d77a8 
  core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 07de214 
  core/src/test/java/org/apache/oozie/action/hadoop/TestLauncher.java b55b9d7 
  hadooplibs/hadoop-utils-0.23/pom.xml PRE-CREATION 
  hadooplibs/hadoop-utils-0.23/src/main/java/org/apache/oozie/action/hadoop/LauncherMainHadoopUtils.java PRE-CREATION 
  hadooplibs/hadoop-utils-1/pom.xml PRE-CREATION 
  hadooplibs/hadoop-utils-1/src/main/java/org/apache/oozie/action/hadoop/LauncherMainHadoopUtils.java PRE-CREATION 
  hadooplibs/hadoop-utils-2/pom.xml PRE-CREATION 
  hadooplibs/hadoop-utils-2/src/main/java/org/apache/oozie/action/hadoop/LauncherMainHadoopUtils.java PRE-CREATION 
  hadooplibs/hadoop-utils-3/pom.xml PRE-CREATION 
  hadooplibs/hadoop-utils-3/src/main/java/org/apache/oozie/action/hadoop/LauncherMainHadoopUtils.java PRE-CREATION 
  hadooplibs/pom.xml 35875c8 
  pom.xml 1223ec2 
  sharelib/hive/pom.xml e355d97 
  sharelib/hive/src/main/java/org/apache/oozie/action/hadoop/HiveMain.java 8fa5ec8 
  sharelib/oozie/pom.xml c70582d 
  sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java b5fa4a1 
  sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/MapReduceMain.java 6230e64 
  sharelib/pig/pom.xml a2bbeaf 
  sharelib/pig/src/main/java/org/apache/oozie/action/hadoop/PigMain.java 9c1e659 
  sharelib/sqoop/pom.xml a962565 
  sharelib/sqoop/src/main/java/org/apache/oozie/action/hadoop/SqoopMain.java e45c8b2 

Diff: https://reviews.apache.org/r/18653/diff/


Testing
-------


Thanks,

Robert Kanter