You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@oozie.apache.org by Robert Kanter <rk...@cloudera.com> on 2016/07/08 00:17:14 UTC

Review Request 49778: OOZIE-2590: OYA: Create basic Oozie Launcher Application Master

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49778/
-----------------------------------------------------------

Review request for oozie.


Bugs: OOZIE-2590
    https://issues.apache.org/jira/browse/OOZIE-2590


Repository: oozie-git


Description
-------

See OOZIE-2590


Diffs
-----

  core/pom.xml 6584af8123925e00f7b29d6b33201f4fc7dca231 
  core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 96726daa800861c15e4a0d97514fbd45e31fa000 
  core/src/main/java/org/apache/oozie/action/hadoop/Hive2ActionExecutor.java b5b1bf90891fde1b13ca058e0b5f538b1b683128 
  core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java c74e9e61c01a4b6430bbd5491c8cbaf0c4e90198 
  core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 99e33445d68ac6a99a08e02ecc182f35103a53ce 
  core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 69e1044e531a67c2d522f642376f4ad030423e53 
  core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java 252f461267818c0da91c7ad59380b2f18a074f9f 
  core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 6813a37d0ffc66da41636fe01e9cc7a37d6f6e79 
  core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java 794e8258472df5fe18299eb3c62471cf9fecc186 
  core/src/main/java/org/apache/oozie/util/ClasspathUtils.java PRE-CREATION 
  core/src/main/resources/oozie-default.xml 6c2f7d82e933614ea6b434f9b035dca3751d1c7f 
  core/src/test/java/org/apache/oozie/QueryServlet.java PRE-CREATION 
  core/src/test/java/org/apache/oozie/action/hadoop/TestCallbackNotifier.java PRE-CREATION 
  core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 879bfebf58acc12ce787b151a67eaaa0716ff2bb 
  core/src/test/java/org/apache/oozie/action/hadoop/TestLauncherAM.java PRE-CREATION 
  core/src/test/java/org/apache/oozie/action/hadoop/TestPrepareActionsDriver.java df9e939cb236e8b65e7b3aa9ab683c33f5fa02b7 
  core/src/test/java/org/apache/oozie/action/hadoop/TestShellActionExecutor.java 6a962a1030a16e61a2af27a6316c4d71b17ea5c9 
  core/src/test/java/org/apache/oozie/command/wf/HangServlet.java 3344cf97762c77f5660d94341e72df4243f76eb2 
  core/src/test/java/org/apache/oozie/service/TestConfigurationService.java 2153bf122ddbdf81acd90c4922fa8d0f619d2d41 
  core/src/test/java/org/apache/oozie/service/TestHadoopAccessorService.java 96faa4878111baafb81fdf0f3b38becba0c6f15e 
  core/src/test/java/org/apache/oozie/test/XTestCase.java e3603692b2ae2ebb741adfa3a6ab5a37ea84eb96 
  core/src/test/java/org/apache/oozie/util/TestClasspathUtils.java PRE-CREATION 
  sharelib/distcp/pom.xml c8cc47cddc6d2922766e2d68ddddee96f5f3cfec 
  sharelib/hcatalog/pom.xml 2b0c5042a6f295101f8c5fae0b08a4239b5e97d3 
  sharelib/hive/pom.xml d10d7b85b70cbeb1906c83484739f34c09038134 
  sharelib/hive2/pom.xml ce967c5575eb639335332dc6de7ce3f3c66a55ca 
  sharelib/oozie/pom.xml dd95b4537c00b516b124341bdacf4fbd1fd351f9 
  sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/CallbackNotifier.java PRE-CREATION 
  sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java PRE-CREATION 
  sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java f2cba13e89e8ddc16b59a04947309147ea6b34b7 
  sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/PrepareActionsDriver.java 21ae456b768fb1cb5334a9e23e6c6082240f8dff 
  sharelib/pig/pom.xml 562c5309cde72c7430fc03f4d342b6d30b5b4575 
  sharelib/spark/pom.xml 46c6375314ac314fc6e702695303bbc8eda4f87b 
  sharelib/sqoop/pom.xml d875c93ae599b990cdcbefdf2aa306eae41a9f0c 
  sharelib/streaming/pom.xml fd795182f11f0f884177a56592e99aaea01d195d 

Diff: https://reviews.apache.org/r/49778/diff/


Testing
-------

- Tried a bunch of stuff against a Hadoop 2.4.0 cluster, including an "action" that would have different kinds of Exceptions, System.exit, etc.
- Some new and old unit tests.


Thanks,

Robert Kanter

Re: Review Request 49778: OOZIE-2590: OYA: Create basic Oozie Launcher Application Master

Posted by Jaydeep Vishwakarma <ja...@gmail.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49778/#review142044
-----------------------------------------------------------




core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java (line 1219)
<https://reviews.apache.org/r/49778/#comment207527>

    No it should not be closed, As it is not opening by this code.



sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java (line 373)
<https://reviews.apache.org/r/49778/#comment207528>

    can you put this in finally block


- Jaydeep Vishwakarma


On July 13, 2016, 5:42 a.m., Robert Kanter wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/49778/
> -----------------------------------------------------------
> 
> (Updated July 13, 2016, 5:42 a.m.)
> 
> 
> Review request for oozie.
> 
> 
> Bugs: OOZIE-2590
>     https://issues.apache.org/jira/browse/OOZIE-2590
> 
> 
> Repository: oozie-git
> 
> 
> Description
> -------
> 
> See OOZIE-2590
> 
> 
> Diffs
> -----
> 
>   core/pom.xml 6584af8123925e00f7b29d6b33201f4fc7dca231 
>   core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 96726daa800861c15e4a0d97514fbd45e31fa000 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2ActionExecutor.java b5b1bf90891fde1b13ca058e0b5f538b1b683128 
>   core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java c74e9e61c01a4b6430bbd5491c8cbaf0c4e90198 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 99e33445d68ac6a99a08e02ecc182f35103a53ce 
>   core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 69e1044e531a67c2d522f642376f4ad030423e53 
>   core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java 252f461267818c0da91c7ad59380b2f18a074f9f 
>   core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 6813a37d0ffc66da41636fe01e9cc7a37d6f6e79 
>   core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java 794e8258472df5fe18299eb3c62471cf9fecc186 
>   core/src/main/java/org/apache/oozie/util/ClasspathUtils.java PRE-CREATION 
>   core/src/main/resources/oozie-default.xml 6c2f7d82e933614ea6b434f9b035dca3751d1c7f 
>   core/src/test/java/org/apache/oozie/QueryServlet.java PRE-CREATION 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestCallbackNotifier.java PRE-CREATION 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 879bfebf58acc12ce787b151a67eaaa0716ff2bb 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestLauncherAM.java PRE-CREATION 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestPrepareActionsDriver.java df9e939cb236e8b65e7b3aa9ab683c33f5fa02b7 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestShellActionExecutor.java 6a962a1030a16e61a2af27a6316c4d71b17ea5c9 
>   core/src/test/java/org/apache/oozie/command/wf/HangServlet.java 3344cf97762c77f5660d94341e72df4243f76eb2 
>   core/src/test/java/org/apache/oozie/service/TestConfigurationService.java 2153bf122ddbdf81acd90c4922fa8d0f619d2d41 
>   core/src/test/java/org/apache/oozie/service/TestHadoopAccessorService.java 96faa4878111baafb81fdf0f3b38becba0c6f15e 
>   core/src/test/java/org/apache/oozie/test/XTestCase.java e3603692b2ae2ebb741adfa3a6ab5a37ea84eb96 
>   core/src/test/java/org/apache/oozie/util/TestClasspathUtils.java PRE-CREATION 
>   sharelib/distcp/pom.xml c8cc47cddc6d2922766e2d68ddddee96f5f3cfec 
>   sharelib/hcatalog/pom.xml 2b0c5042a6f295101f8c5fae0b08a4239b5e97d3 
>   sharelib/hive/pom.xml d10d7b85b70cbeb1906c83484739f34c09038134 
>   sharelib/hive2/pom.xml ce967c5575eb639335332dc6de7ce3f3c66a55ca 
>   sharelib/oozie/pom.xml dd95b4537c00b516b124341bdacf4fbd1fd351f9 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/CallbackNotifier.java PRE-CREATION 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java PRE-CREATION 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java f2cba13e89e8ddc16b59a04947309147ea6b34b7 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/PrepareActionsDriver.java 21ae456b768fb1cb5334a9e23e6c6082240f8dff 
>   sharelib/pig/pom.xml 562c5309cde72c7430fc03f4d342b6d30b5b4575 
>   sharelib/spark/pom.xml 46c6375314ac314fc6e702695303bbc8eda4f87b 
>   sharelib/sqoop/pom.xml d875c93ae599b990cdcbefdf2aa306eae41a9f0c 
>   sharelib/streaming/pom.xml fd795182f11f0f884177a56592e99aaea01d195d 
> 
> Diff: https://reviews.apache.org/r/49778/diff/
> 
> 
> Testing
> -------
> 
> - Tried a bunch of stuff against a Hadoop 2.4.0 cluster, including an "action" that would have different kinds of Exceptions, System.exit, etc.
> - Some new and old unit tests.
> 
> 
> Thanks,
> 
> Robert Kanter
> 
>

Re: Review Request 49778: OOZIE-2590: OYA: Create basic Oozie Launcher Application Master

Posted by Karthik Kambatla <ka...@cloudera.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49778/#review142146
-----------------------------------------------------------



I primarily looked at the integration-with-YARN parts of the code.

- Karthik Kambatla


On July 13, 2016, 5:42 a.m., Robert Kanter wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/49778/
> -----------------------------------------------------------
> 
> (Updated July 13, 2016, 5:42 a.m.)
> 
> 
> Review request for oozie.
> 
> 
> Bugs: OOZIE-2590
>     https://issues.apache.org/jira/browse/OOZIE-2590
> 
> 
> Repository: oozie-git
> 
> 
> Description
> -------
> 
> See OOZIE-2590
> 
> 
> Diffs
> -----
> 
>   core/pom.xml 6584af8123925e00f7b29d6b33201f4fc7dca231 
>   core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 96726daa800861c15e4a0d97514fbd45e31fa000 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2ActionExecutor.java b5b1bf90891fde1b13ca058e0b5f538b1b683128 
>   core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java c74e9e61c01a4b6430bbd5491c8cbaf0c4e90198 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 99e33445d68ac6a99a08e02ecc182f35103a53ce 
>   core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 69e1044e531a67c2d522f642376f4ad030423e53 
>   core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java 252f461267818c0da91c7ad59380b2f18a074f9f 
>   core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 6813a37d0ffc66da41636fe01e9cc7a37d6f6e79 
>   core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java 794e8258472df5fe18299eb3c62471cf9fecc186 
>   core/src/main/java/org/apache/oozie/util/ClasspathUtils.java PRE-CREATION 
>   core/src/main/resources/oozie-default.xml 6c2f7d82e933614ea6b434f9b035dca3751d1c7f 
>   core/src/test/java/org/apache/oozie/QueryServlet.java PRE-CREATION 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestCallbackNotifier.java PRE-CREATION 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 879bfebf58acc12ce787b151a67eaaa0716ff2bb 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestLauncherAM.java PRE-CREATION 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestPrepareActionsDriver.java df9e939cb236e8b65e7b3aa9ab683c33f5fa02b7 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestShellActionExecutor.java 6a962a1030a16e61a2af27a6316c4d71b17ea5c9 
>   core/src/test/java/org/apache/oozie/command/wf/HangServlet.java 3344cf97762c77f5660d94341e72df4243f76eb2 
>   core/src/test/java/org/apache/oozie/service/TestConfigurationService.java 2153bf122ddbdf81acd90c4922fa8d0f619d2d41 
>   core/src/test/java/org/apache/oozie/service/TestHadoopAccessorService.java 96faa4878111baafb81fdf0f3b38becba0c6f15e 
>   core/src/test/java/org/apache/oozie/test/XTestCase.java e3603692b2ae2ebb741adfa3a6ab5a37ea84eb96 
>   core/src/test/java/org/apache/oozie/util/TestClasspathUtils.java PRE-CREATION 
>   sharelib/distcp/pom.xml c8cc47cddc6d2922766e2d68ddddee96f5f3cfec 
>   sharelib/hcatalog/pom.xml 2b0c5042a6f295101f8c5fae0b08a4239b5e97d3 
>   sharelib/hive/pom.xml d10d7b85b70cbeb1906c83484739f34c09038134 
>   sharelib/hive2/pom.xml ce967c5575eb639335332dc6de7ce3f3c66a55ca 
>   sharelib/oozie/pom.xml dd95b4537c00b516b124341bdacf4fbd1fd351f9 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/CallbackNotifier.java PRE-CREATION 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java PRE-CREATION 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java f2cba13e89e8ddc16b59a04947309147ea6b34b7 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/PrepareActionsDriver.java 21ae456b768fb1cb5334a9e23e6c6082240f8dff 
>   sharelib/pig/pom.xml 562c5309cde72c7430fc03f4d342b6d30b5b4575 
>   sharelib/spark/pom.xml 46c6375314ac314fc6e702695303bbc8eda4f87b 
>   sharelib/sqoop/pom.xml d875c93ae599b990cdcbefdf2aa306eae41a9f0c 
>   sharelib/streaming/pom.xml fd795182f11f0f884177a56592e99aaea01d195d 
> 
> Diff: https://reviews.apache.org/r/49778/diff/
> 
> 
> Testing
> -------
> 
> - Tried a bunch of stuff against a Hadoop 2.4.0 cluster, including an "action" that would have different kinds of Exceptions, System.exit, etc.
> - Some new and old unit tests.
> 
> 
> Thanks,
> 
> Robert Kanter
> 
>

Re: Review Request 49778: OOZIE-2590: OYA: Create basic Oozie Launcher Application Master

Posted by Robert Kanter <rk...@cloudera.com>.


> On July 13, 2016, 1:43 p.m., Peter Bacsko wrote:
> > core/src/test/java/org/apache/oozie/command/wf/HangServlet.java, line 33
> > <https://reviews.apache.org/r/49778/diff/2/?file=1443106#file1443106line33>
> >
> >     Is this another test-specific servlet?

Yup.


- Robert


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49778/#review142057
-----------------------------------------------------------


On July 13, 2016, 5:42 a.m., Robert Kanter wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/49778/
> -----------------------------------------------------------
> 
> (Updated July 13, 2016, 5:42 a.m.)
> 
> 
> Review request for oozie.
> 
> 
> Bugs: OOZIE-2590
>     https://issues.apache.org/jira/browse/OOZIE-2590
> 
> 
> Repository: oozie-git
> 
> 
> Description
> -------
> 
> See OOZIE-2590
> 
> 
> Diffs
> -----
> 
>   core/pom.xml 6584af8123925e00f7b29d6b33201f4fc7dca231 
>   core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 96726daa800861c15e4a0d97514fbd45e31fa000 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2ActionExecutor.java b5b1bf90891fde1b13ca058e0b5f538b1b683128 
>   core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java c74e9e61c01a4b6430bbd5491c8cbaf0c4e90198 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 99e33445d68ac6a99a08e02ecc182f35103a53ce 
>   core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 69e1044e531a67c2d522f642376f4ad030423e53 
>   core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java 252f461267818c0da91c7ad59380b2f18a074f9f 
>   core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 6813a37d0ffc66da41636fe01e9cc7a37d6f6e79 
>   core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java 794e8258472df5fe18299eb3c62471cf9fecc186 
>   core/src/main/java/org/apache/oozie/util/ClasspathUtils.java PRE-CREATION 
>   core/src/main/resources/oozie-default.xml 6c2f7d82e933614ea6b434f9b035dca3751d1c7f 
>   core/src/test/java/org/apache/oozie/QueryServlet.java PRE-CREATION 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestCallbackNotifier.java PRE-CREATION 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 879bfebf58acc12ce787b151a67eaaa0716ff2bb 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestLauncherAM.java PRE-CREATION 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestPrepareActionsDriver.java df9e939cb236e8b65e7b3aa9ab683c33f5fa02b7 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestShellActionExecutor.java 6a962a1030a16e61a2af27a6316c4d71b17ea5c9 
>   core/src/test/java/org/apache/oozie/command/wf/HangServlet.java 3344cf97762c77f5660d94341e72df4243f76eb2 
>   core/src/test/java/org/apache/oozie/service/TestConfigurationService.java 2153bf122ddbdf81acd90c4922fa8d0f619d2d41 
>   core/src/test/java/org/apache/oozie/service/TestHadoopAccessorService.java 96faa4878111baafb81fdf0f3b38becba0c6f15e 
>   core/src/test/java/org/apache/oozie/test/XTestCase.java e3603692b2ae2ebb741adfa3a6ab5a37ea84eb96 
>   core/src/test/java/org/apache/oozie/util/TestClasspathUtils.java PRE-CREATION 
>   sharelib/distcp/pom.xml c8cc47cddc6d2922766e2d68ddddee96f5f3cfec 
>   sharelib/hcatalog/pom.xml 2b0c5042a6f295101f8c5fae0b08a4239b5e97d3 
>   sharelib/hive/pom.xml d10d7b85b70cbeb1906c83484739f34c09038134 
>   sharelib/hive2/pom.xml ce967c5575eb639335332dc6de7ce3f3c66a55ca 
>   sharelib/oozie/pom.xml dd95b4537c00b516b124341bdacf4fbd1fd351f9 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/CallbackNotifier.java PRE-CREATION 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java PRE-CREATION 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java f2cba13e89e8ddc16b59a04947309147ea6b34b7 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/PrepareActionsDriver.java 21ae456b768fb1cb5334a9e23e6c6082240f8dff 
>   sharelib/pig/pom.xml 562c5309cde72c7430fc03f4d342b6d30b5b4575 
>   sharelib/spark/pom.xml 46c6375314ac314fc6e702695303bbc8eda4f87b 
>   sharelib/sqoop/pom.xml d875c93ae599b990cdcbefdf2aa306eae41a9f0c 
>   sharelib/streaming/pom.xml fd795182f11f0f884177a56592e99aaea01d195d 
> 
> Diff: https://reviews.apache.org/r/49778/diff/
> 
> 
> Testing
> -------
> 
> - Tried a bunch of stuff against a Hadoop 2.4.0 cluster, including an "action" that would have different kinds of Exceptions, System.exit, etc.
> - Some new and old unit tests.
> 
> 
> Thanks,
> 
> Robert Kanter
> 
>

Re: Review Request 49778: OOZIE-2590: OYA: Create basic Oozie Launcher Application Master

Posted by Robert Kanter <rk...@cloudera.com>.


> On July 13, 2016, 1:43 p.m., Peter Bacsko wrote:
> > core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java, line 1323
> > <https://reviews.apache.org/r/49778/diff/2/?file=1443093#file1443093line1323>
> >
> >     Just nitpicking, but any reason to use the oldie Vector here?
> >     
> >     Also I'd wrap it in an unmodifiable list.

That's what I get for copying old MR code.  I'll change it to an ArrayList.


- Robert


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49778/#review142057
-----------------------------------------------------------


On July 13, 2016, 11:44 p.m., Robert Kanter wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/49778/
> -----------------------------------------------------------
> 
> (Updated July 13, 2016, 11:44 p.m.)
> 
> 
> Review request for oozie.
> 
> 
> Bugs: OOZIE-2590
>     https://issues.apache.org/jira/browse/OOZIE-2590
> 
> 
> Repository: oozie-git
> 
> 
> Description
> -------
> 
> See OOZIE-2590
> 
> 
> Diffs
> -----
> 
>   core/pom.xml 6584af8123925e00f7b29d6b33201f4fc7dca231 
>   core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 96726daa800861c15e4a0d97514fbd45e31fa000 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2ActionExecutor.java b5b1bf90891fde1b13ca058e0b5f538b1b683128 
>   core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java c74e9e61c01a4b6430bbd5491c8cbaf0c4e90198 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 99e33445d68ac6a99a08e02ecc182f35103a53ce 
>   core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 69e1044e531a67c2d522f642376f4ad030423e53 
>   core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java 252f461267818c0da91c7ad59380b2f18a074f9f 
>   core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 6813a37d0ffc66da41636fe01e9cc7a37d6f6e79 
>   core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java 794e8258472df5fe18299eb3c62471cf9fecc186 
>   core/src/main/java/org/apache/oozie/util/ClasspathUtils.java PRE-CREATION 
>   core/src/main/resources/oozie-default.xml 6c2f7d82e933614ea6b434f9b035dca3751d1c7f 
>   core/src/test/java/org/apache/oozie/QueryServlet.java PRE-CREATION 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 879bfebf58acc12ce787b151a67eaaa0716ff2bb 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestLauncherAM.java PRE-CREATION 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestLauncherAMCallbackNotifier.java PRE-CREATION 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestPrepareActionsDriver.java df9e939cb236e8b65e7b3aa9ab683c33f5fa02b7 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestShellActionExecutor.java 6a962a1030a16e61a2af27a6316c4d71b17ea5c9 
>   core/src/test/java/org/apache/oozie/command/wf/HangServlet.java 3344cf97762c77f5660d94341e72df4243f76eb2 
>   core/src/test/java/org/apache/oozie/service/TestConfigurationService.java 2153bf122ddbdf81acd90c4922fa8d0f619d2d41 
>   core/src/test/java/org/apache/oozie/service/TestHadoopAccessorService.java 96faa4878111baafb81fdf0f3b38becba0c6f15e 
>   core/src/test/java/org/apache/oozie/test/XTestCase.java e3603692b2ae2ebb741adfa3a6ab5a37ea84eb96 
>   core/src/test/java/org/apache/oozie/util/TestClasspathUtils.java PRE-CREATION 
>   sharelib/distcp/pom.xml c8cc47cddc6d2922766e2d68ddddee96f5f3cfec 
>   sharelib/hcatalog/pom.xml 2b0c5042a6f295101f8c5fae0b08a4239b5e97d3 
>   sharelib/hive/pom.xml d10d7b85b70cbeb1906c83484739f34c09038134 
>   sharelib/hive2/pom.xml ce967c5575eb639335332dc6de7ce3f3c66a55ca 
>   sharelib/oozie/pom.xml dd95b4537c00b516b124341bdacf4fbd1fd351f9 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java PRE-CREATION 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAMCallbackNotifier.java PRE-CREATION 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java f2cba13e89e8ddc16b59a04947309147ea6b34b7 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/PrepareActionsDriver.java 21ae456b768fb1cb5334a9e23e6c6082240f8dff 
>   sharelib/pig/pom.xml 562c5309cde72c7430fc03f4d342b6d30b5b4575 
>   sharelib/spark/pom.xml 46c6375314ac314fc6e702695303bbc8eda4f87b 
>   sharelib/sqoop/pom.xml d875c93ae599b990cdcbefdf2aa306eae41a9f0c 
>   sharelib/streaming/pom.xml fd795182f11f0f884177a56592e99aaea01d195d 
> 
> Diff: https://reviews.apache.org/r/49778/diff/
> 
> 
> Testing
> -------
> 
> - Tried a bunch of stuff against a Hadoop 2.4.0 cluster, including an "action" that would have different kinds of Exceptions, System.exit, etc.
> - Some new and old unit tests.
> 
> 
> Thanks,
> 
> Robert Kanter
> 
>

Re: Review Request 49778: OOZIE-2590: OYA: Create basic Oozie Launcher Application Master

Posted by Peter Bacsko <pb...@cloudera.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49778/#review142057
-----------------------------------------------------------




core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java (line 1177)
<https://reviews.apache.org/r/49778/#comment207553>

    Just nitpicking, but any reason to use the oldie Vector here?
    
    Also I'd wrap it in an unmodifiable list.



core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java (line 1471)
<https://reviews.apache.org/r/49778/#comment207554>

    Shouldn't we use "==" instead of equals() to check app state? YarnApplicationState is an enum. I think it enhances readability.



core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java (line 1527)
<https://reviews.apache.org/r/49778/#comment207555>

    I know this is not a scope of this task, but these hard-coded error codes are not that nice. Could be a part of another JIRA to clean these up to an enum, just like ErrorCode. I've seen this in other actions, too.



core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java (line 1551)
<https://reviews.apache.org/r/49778/#comment207550>

    Hard-coded string



core/src/test/java/org/apache/oozie/command/wf/HangServlet.java (line 33)
<https://reviews.apache.org/r/49778/#comment207559>

    Is this another test-specific servlet?



sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java (line 180)
<https://reviews.apache.org/r/49778/#comment207551>

    Minor: Class<?> to suppress raw type warning


- Peter Bacsko


On j�l. 13, 2016, 5:42 de, Robert Kanter wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/49778/
> -----------------------------------------------------------
> 
> (Updated j�l. 13, 2016, 5:42 de)
> 
> 
> Review request for oozie.
> 
> 
> Bugs: OOZIE-2590
>     https://issues.apache.org/jira/browse/OOZIE-2590
> 
> 
> Repository: oozie-git
> 
> 
> Description
> -------
> 
> See OOZIE-2590
> 
> 
> Diffs
> -----
> 
>   core/pom.xml 6584af8123925e00f7b29d6b33201f4fc7dca231 
>   core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 96726daa800861c15e4a0d97514fbd45e31fa000 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2ActionExecutor.java b5b1bf90891fde1b13ca058e0b5f538b1b683128 
>   core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java c74e9e61c01a4b6430bbd5491c8cbaf0c4e90198 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 99e33445d68ac6a99a08e02ecc182f35103a53ce 
>   core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 69e1044e531a67c2d522f642376f4ad030423e53 
>   core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java 252f461267818c0da91c7ad59380b2f18a074f9f 
>   core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 6813a37d0ffc66da41636fe01e9cc7a37d6f6e79 
>   core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java 794e8258472df5fe18299eb3c62471cf9fecc186 
>   core/src/main/java/org/apache/oozie/util/ClasspathUtils.java PRE-CREATION 
>   core/src/main/resources/oozie-default.xml 6c2f7d82e933614ea6b434f9b035dca3751d1c7f 
>   core/src/test/java/org/apache/oozie/QueryServlet.java PRE-CREATION 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestCallbackNotifier.java PRE-CREATION 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 879bfebf58acc12ce787b151a67eaaa0716ff2bb 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestLauncherAM.java PRE-CREATION 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestPrepareActionsDriver.java df9e939cb236e8b65e7b3aa9ab683c33f5fa02b7 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestShellActionExecutor.java 6a962a1030a16e61a2af27a6316c4d71b17ea5c9 
>   core/src/test/java/org/apache/oozie/command/wf/HangServlet.java 3344cf97762c77f5660d94341e72df4243f76eb2 
>   core/src/test/java/org/apache/oozie/service/TestConfigurationService.java 2153bf122ddbdf81acd90c4922fa8d0f619d2d41 
>   core/src/test/java/org/apache/oozie/service/TestHadoopAccessorService.java 96faa4878111baafb81fdf0f3b38becba0c6f15e 
>   core/src/test/java/org/apache/oozie/test/XTestCase.java e3603692b2ae2ebb741adfa3a6ab5a37ea84eb96 
>   core/src/test/java/org/apache/oozie/util/TestClasspathUtils.java PRE-CREATION 
>   sharelib/distcp/pom.xml c8cc47cddc6d2922766e2d68ddddee96f5f3cfec 
>   sharelib/hcatalog/pom.xml 2b0c5042a6f295101f8c5fae0b08a4239b5e97d3 
>   sharelib/hive/pom.xml d10d7b85b70cbeb1906c83484739f34c09038134 
>   sharelib/hive2/pom.xml ce967c5575eb639335332dc6de7ce3f3c66a55ca 
>   sharelib/oozie/pom.xml dd95b4537c00b516b124341bdacf4fbd1fd351f9 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/CallbackNotifier.java PRE-CREATION 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java PRE-CREATION 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java f2cba13e89e8ddc16b59a04947309147ea6b34b7 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/PrepareActionsDriver.java 21ae456b768fb1cb5334a9e23e6c6082240f8dff 
>   sharelib/pig/pom.xml 562c5309cde72c7430fc03f4d342b6d30b5b4575 
>   sharelib/spark/pom.xml 46c6375314ac314fc6e702695303bbc8eda4f87b 
>   sharelib/sqoop/pom.xml d875c93ae599b990cdcbefdf2aa306eae41a9f0c 
>   sharelib/streaming/pom.xml fd795182f11f0f884177a56592e99aaea01d195d 
> 
> Diff: https://reviews.apache.org/r/49778/diff/
> 
> 
> Testing
> -------
> 
> - Tried a bunch of stuff against a Hadoop 2.4.0 cluster, including an "action" that would have different kinds of Exceptions, System.exit, etc.
> - Some new and old unit tests.
> 
> 
> Thanks,
> 
> Robert Kanter
> 
>

Re: Review Request 49778: OOZIE-2590: OYA: Create basic Oozie Launcher Application Master

Posted by Robert Kanter <rk...@cloudera.com>.


> On July 13, 2016, 10:51 p.m., Karthik Kambatla wrote:
> > core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java, line 1162
> > <https://reviews.apache.org/r/49778/diff/2/?file=1443093#file1443093line1162>
> >
> >     Isn't RunningJob an MR thing?

Yes.  We'll remove that eventually (there's a "TODO: OYA:" above the comments above it), but we need it for now in order to compile.  I need to look into what this recovery stuff is and how to deal with it in Yarn.


> On July 13, 2016, 10:51 p.m., Karthik Kambatla wrote:
> > core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java, line 1582
> > <https://reviews.apache.org/r/49778/diff/2/?file=1443093#file1443093line1582>
> >
> >     Looks like createYarnClient is called from multiple places. Can't it be created once and cached?

Hmm.  That's an interesting thought.  We actually do something like that for UserGroupInformations (UserGroupInformationService).  It might make sense to keep a cache of YarnClients for each user around.  Looking at UserGroupInformationService, it doesn't look like we ever remove anything from the cache except on shutdown though.

Would there be any overhead (on Oozie or Yarn) for keeping the YarnClients around?  Though this might get tricky if some code is using the YarnClient while/after it rolls off the cache and is closed.  That might be the reason the UGIs aren't ever removed.  Let's discuss this tomorrow in person.


> On July 13, 2016, 10:51 p.m., Karthik Kambatla wrote:
> > core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java, line 1636
> > <https://reviews.apache.org/r/49778/diff/2/?file=1443093#file1443093line1636>
> >
> >     Instead of the current approach of
> >     
> >     if (appStatus != null || fallback) {
> >       ...
> >       if (fallback) {
> >         ...
> >         appStatus = ...
> >       }
> >     }
> >     
> >     how about 
> >     
> >     if (fallback) {
> >       ...
> >       appStatus = 
> >     }
> >     
> >     if (appStatus != null) {
> >       ...
> >     }
> >     
> >     If avoiding creating actionData an extra time is bad, how about lazily creating it and reusing it the next time?

But there's actually a lot of overlap here between the two cases.  The inner ````if (fallback)```` does not have an else after it, so unless you get to the case where an exception is thrown, they do mostly the same thing.


> On July 13, 2016, 10:51 p.m., Karthik Kambatla wrote:
> > core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java, line 522
> > <https://reviews.apache.org/r/49778/diff/2/?file=1443097#file1443097line522>
> >
> >     Helper method to create YarnClient in Oozie?

Yes.  It handles the impersonation and creating it for the correct Yarn cluster.


> On July 13, 2016, 10:51 p.m., Karthik Kambatla wrote:
> > sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/CallbackNotifier.java, line 44
> > <https://reviews.apache.org/r/49778/diff/2/?file=1443116#file1443116line44>
> >
> >     Given there are two distinct callbacks here - one this and one the AMRMClientCallback, do you think there is a need to be explicit in naming this class?

It's in an Oozie package, but I guess it could be confusing.  I'll rename it.


> On July 13, 2016, 10:51 p.m., Karthik Kambatla wrote:
> > sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java, line 77
> > <https://reviews.apache.org/r/49778/diff/2/?file=1443117#file1443117line77>
> >
> >     Does Oozie still support Java versions older than Java7? If not, one can drop the types when calling the constructor.

Yes.  We're going to change that soon, but for now, we're doing Java 6.


> On July 13, 2016, 10:51 p.m., Karthik Kambatla wrote:
> > sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java, line 141
> > <https://reviews.apache.org/r/49778/diff/2/?file=1443117#file1443117line141>
> >
> >     +1. 
> >     
> >     It would help to break this up further. Especially, the try-finally blocks are hard to read. e.g. verifying registerWithRM has an associated unregister was not obvious.

Okay.  Everyone's been complaining about this.  I'll spend some more time playing around with this soon.


- Robert


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49778/#review142130
-----------------------------------------------------------


On July 13, 2016, 5:42 a.m., Robert Kanter wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/49778/
> -----------------------------------------------------------
> 
> (Updated July 13, 2016, 5:42 a.m.)
> 
> 
> Review request for oozie.
> 
> 
> Bugs: OOZIE-2590
>     https://issues.apache.org/jira/browse/OOZIE-2590
> 
> 
> Repository: oozie-git
> 
> 
> Description
> -------
> 
> See OOZIE-2590
> 
> 
> Diffs
> -----
> 
>   core/pom.xml 6584af8123925e00f7b29d6b33201f4fc7dca231 
>   core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 96726daa800861c15e4a0d97514fbd45e31fa000 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2ActionExecutor.java b5b1bf90891fde1b13ca058e0b5f538b1b683128 
>   core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java c74e9e61c01a4b6430bbd5491c8cbaf0c4e90198 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 99e33445d68ac6a99a08e02ecc182f35103a53ce 
>   core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 69e1044e531a67c2d522f642376f4ad030423e53 
>   core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java 252f461267818c0da91c7ad59380b2f18a074f9f 
>   core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 6813a37d0ffc66da41636fe01e9cc7a37d6f6e79 
>   core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java 794e8258472df5fe18299eb3c62471cf9fecc186 
>   core/src/main/java/org/apache/oozie/util/ClasspathUtils.java PRE-CREATION 
>   core/src/main/resources/oozie-default.xml 6c2f7d82e933614ea6b434f9b035dca3751d1c7f 
>   core/src/test/java/org/apache/oozie/QueryServlet.java PRE-CREATION 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestCallbackNotifier.java PRE-CREATION 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 879bfebf58acc12ce787b151a67eaaa0716ff2bb 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestLauncherAM.java PRE-CREATION 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestPrepareActionsDriver.java df9e939cb236e8b65e7b3aa9ab683c33f5fa02b7 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestShellActionExecutor.java 6a962a1030a16e61a2af27a6316c4d71b17ea5c9 
>   core/src/test/java/org/apache/oozie/command/wf/HangServlet.java 3344cf97762c77f5660d94341e72df4243f76eb2 
>   core/src/test/java/org/apache/oozie/service/TestConfigurationService.java 2153bf122ddbdf81acd90c4922fa8d0f619d2d41 
>   core/src/test/java/org/apache/oozie/service/TestHadoopAccessorService.java 96faa4878111baafb81fdf0f3b38becba0c6f15e 
>   core/src/test/java/org/apache/oozie/test/XTestCase.java e3603692b2ae2ebb741adfa3a6ab5a37ea84eb96 
>   core/src/test/java/org/apache/oozie/util/TestClasspathUtils.java PRE-CREATION 
>   sharelib/distcp/pom.xml c8cc47cddc6d2922766e2d68ddddee96f5f3cfec 
>   sharelib/hcatalog/pom.xml 2b0c5042a6f295101f8c5fae0b08a4239b5e97d3 
>   sharelib/hive/pom.xml d10d7b85b70cbeb1906c83484739f34c09038134 
>   sharelib/hive2/pom.xml ce967c5575eb639335332dc6de7ce3f3c66a55ca 
>   sharelib/oozie/pom.xml dd95b4537c00b516b124341bdacf4fbd1fd351f9 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/CallbackNotifier.java PRE-CREATION 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java PRE-CREATION 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java f2cba13e89e8ddc16b59a04947309147ea6b34b7 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/PrepareActionsDriver.java 21ae456b768fb1cb5334a9e23e6c6082240f8dff 
>   sharelib/pig/pom.xml 562c5309cde72c7430fc03f4d342b6d30b5b4575 
>   sharelib/spark/pom.xml 46c6375314ac314fc6e702695303bbc8eda4f87b 
>   sharelib/sqoop/pom.xml d875c93ae599b990cdcbefdf2aa306eae41a9f0c 
>   sharelib/streaming/pom.xml fd795182f11f0f884177a56592e99aaea01d195d 
> 
> Diff: https://reviews.apache.org/r/49778/diff/
> 
> 
> Testing
> -------
> 
> - Tried a bunch of stuff against a Hadoop 2.4.0 cluster, including an "action" that would have different kinds of Exceptions, System.exit, etc.
> - Some new and old unit tests.
> 
> 
> Thanks,
> 
> Robert Kanter
> 
>

Re: Review Request 49778: OOZIE-2590: OYA: Create basic Oozie Launcher Application Master

Posted by Robert Kanter <rk...@cloudera.com>.


> On July 13, 2016, 10:51 p.m., Karthik Kambatla wrote:
> > core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java, line 1582
> > <https://reviews.apache.org/r/49778/diff/2/?file=1443093#file1443093line1582>
> >
> >     Looks like createYarnClient is called from multiple places. Can't it be created once and cached?
> 
> Robert Kanter wrote:
>     Hmm.  That's an interesting thought.  We actually do something like that for UserGroupInformations (UserGroupInformationService).  It might make sense to keep a cache of YarnClients for each user around.  Looking at UserGroupInformationService, it doesn't look like we ever remove anything from the cache except on shutdown though.
>     
>     Would there be any overhead (on Oozie or Yarn) for keeping the YarnClients around?  Though this might get tricky if some code is using the YarnClient while/after it rolls off the cache and is closed.  That might be the reason the UGIs aren't ever removed.  Let's discuss this tomorrow in person.

Discussed this with Karthik.  Keeping the YarnClients around would put extra overhead on the RM to keep the connections open.  So we won't do that.

I double-checked, and there are two places (in non-test code) where we create a YarnClient.  One in HadoopAccessorService and one in LauncherAM, which can't use HadoopAccessorService and doesn't need the impersonation and cluster lookup anyway.


- Robert


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49778/#review142130
-----------------------------------------------------------


On July 13, 2016, 11:44 p.m., Robert Kanter wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/49778/
> -----------------------------------------------------------
> 
> (Updated July 13, 2016, 11:44 p.m.)
> 
> 
> Review request for oozie.
> 
> 
> Bugs: OOZIE-2590
>     https://issues.apache.org/jira/browse/OOZIE-2590
> 
> 
> Repository: oozie-git
> 
> 
> Description
> -------
> 
> See OOZIE-2590
> 
> 
> Diffs
> -----
> 
>   core/pom.xml 6584af8123925e00f7b29d6b33201f4fc7dca231 
>   core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 96726daa800861c15e4a0d97514fbd45e31fa000 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2ActionExecutor.java b5b1bf90891fde1b13ca058e0b5f538b1b683128 
>   core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java c74e9e61c01a4b6430bbd5491c8cbaf0c4e90198 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 99e33445d68ac6a99a08e02ecc182f35103a53ce 
>   core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 69e1044e531a67c2d522f642376f4ad030423e53 
>   core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java 252f461267818c0da91c7ad59380b2f18a074f9f 
>   core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 6813a37d0ffc66da41636fe01e9cc7a37d6f6e79 
>   core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java 794e8258472df5fe18299eb3c62471cf9fecc186 
>   core/src/main/java/org/apache/oozie/util/ClasspathUtils.java PRE-CREATION 
>   core/src/main/resources/oozie-default.xml 6c2f7d82e933614ea6b434f9b035dca3751d1c7f 
>   core/src/test/java/org/apache/oozie/QueryServlet.java PRE-CREATION 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 879bfebf58acc12ce787b151a67eaaa0716ff2bb 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestLauncherAM.java PRE-CREATION 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestLauncherAMCallbackNotifier.java PRE-CREATION 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestPrepareActionsDriver.java df9e939cb236e8b65e7b3aa9ab683c33f5fa02b7 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestShellActionExecutor.java 6a962a1030a16e61a2af27a6316c4d71b17ea5c9 
>   core/src/test/java/org/apache/oozie/command/wf/HangServlet.java 3344cf97762c77f5660d94341e72df4243f76eb2 
>   core/src/test/java/org/apache/oozie/service/TestConfigurationService.java 2153bf122ddbdf81acd90c4922fa8d0f619d2d41 
>   core/src/test/java/org/apache/oozie/service/TestHadoopAccessorService.java 96faa4878111baafb81fdf0f3b38becba0c6f15e 
>   core/src/test/java/org/apache/oozie/test/XTestCase.java e3603692b2ae2ebb741adfa3a6ab5a37ea84eb96 
>   core/src/test/java/org/apache/oozie/util/TestClasspathUtils.java PRE-CREATION 
>   sharelib/distcp/pom.xml c8cc47cddc6d2922766e2d68ddddee96f5f3cfec 
>   sharelib/hcatalog/pom.xml 2b0c5042a6f295101f8c5fae0b08a4239b5e97d3 
>   sharelib/hive/pom.xml d10d7b85b70cbeb1906c83484739f34c09038134 
>   sharelib/hive2/pom.xml ce967c5575eb639335332dc6de7ce3f3c66a55ca 
>   sharelib/oozie/pom.xml dd95b4537c00b516b124341bdacf4fbd1fd351f9 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java PRE-CREATION 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAMCallbackNotifier.java PRE-CREATION 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java f2cba13e89e8ddc16b59a04947309147ea6b34b7 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/PrepareActionsDriver.java 21ae456b768fb1cb5334a9e23e6c6082240f8dff 
>   sharelib/pig/pom.xml 562c5309cde72c7430fc03f4d342b6d30b5b4575 
>   sharelib/spark/pom.xml 46c6375314ac314fc6e702695303bbc8eda4f87b 
>   sharelib/sqoop/pom.xml d875c93ae599b990cdcbefdf2aa306eae41a9f0c 
>   sharelib/streaming/pom.xml fd795182f11f0f884177a56592e99aaea01d195d 
> 
> Diff: https://reviews.apache.org/r/49778/diff/
> 
> 
> Testing
> -------
> 
> - Tried a bunch of stuff against a Hadoop 2.4.0 cluster, including an "action" that would have different kinds of Exceptions, System.exit, etc.
> - Some new and old unit tests.
> 
> 
> Thanks,
> 
> Robert Kanter
> 
>

Re: Review Request 49778: OOZIE-2590: OYA: Create basic Oozie Launcher Application Master

Posted by Karthik Kambatla <ka...@cloudera.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49778/#review142130
-----------------------------------------------------------




core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java (line 1019)
<https://reviews.apache.org/r/49778/#comment207640>

    Isn't RunningJob an MR thing?



core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java (line 1089)
<https://reviews.apache.org/r/49778/#comment207643>

    Set yarnClient = null to future-proof?



core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java (line 1160)
<https://reviews.apache.org/r/49778/#comment207645>

    I suppose the TODO is to replace the false in the if. 
    
    Add it on the same line to be clearer?



core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java (line 1436)
<https://reviews.apache.org/r/49778/#comment207647>

    Looks like createYarnClient is called from multiple places. Can't it be created once and cached?



core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java (line 1489)
<https://reviews.apache.org/r/49778/#comment207649>

    Instead of the current approach of
    
    if (appStatus != null || fallback) {
      ...
      if (fallback) {
        ...
        appStatus = ...
      }
    }
    
    how about 
    
    if (fallback) {
      ...
      appStatus = 
    }
    
    if (appStatus != null) {
      ...
    }
    
    If avoiding creating actionData an extra time is bad, how about lazily creating it and reusing it the next time?



core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java (line 522)
<https://reviews.apache.org/r/49778/#comment207650>

    Helper method to create YarnClient in Oozie?



sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/CallbackNotifier.java (line 44)
<https://reviews.apache.org/r/49778/#comment207633>

    Given there are two distinct callbacks here - one this and one the AMRMClientCallback, do you think there is a need to be explicit in naming this class?



sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/CallbackNotifier.java (line 46)
<https://reviews.apache.org/r/49778/#comment207632>

    +1. Changing these properties to oozie makes total sense.



sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java (line 77)
<https://reviews.apache.org/r/49778/#comment207634>

    Does Oozie still support Java versions older than Java7? If not, one can drop the types when calling the constructor.



sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java (line 141)
<https://reviews.apache.org/r/49778/#comment207636>

    +1. 
    
    It would help to break this up further. Especially, the try-finally blocks are hard to read. e.g. verifying registerWithRM has an associated unregister was not obvious.



sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java (line 467)
<https://reviews.apache.org/r/49778/#comment207637>

    Should this be same as onError() to begin with?


- Karthik Kambatla


On July 13, 2016, 5:42 a.m., Robert Kanter wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/49778/
> -----------------------------------------------------------
> 
> (Updated July 13, 2016, 5:42 a.m.)
> 
> 
> Review request for oozie.
> 
> 
> Bugs: OOZIE-2590
>     https://issues.apache.org/jira/browse/OOZIE-2590
> 
> 
> Repository: oozie-git
> 
> 
> Description
> -------
> 
> See OOZIE-2590
> 
> 
> Diffs
> -----
> 
>   core/pom.xml 6584af8123925e00f7b29d6b33201f4fc7dca231 
>   core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 96726daa800861c15e4a0d97514fbd45e31fa000 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2ActionExecutor.java b5b1bf90891fde1b13ca058e0b5f538b1b683128 
>   core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java c74e9e61c01a4b6430bbd5491c8cbaf0c4e90198 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 99e33445d68ac6a99a08e02ecc182f35103a53ce 
>   core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 69e1044e531a67c2d522f642376f4ad030423e53 
>   core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java 252f461267818c0da91c7ad59380b2f18a074f9f 
>   core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 6813a37d0ffc66da41636fe01e9cc7a37d6f6e79 
>   core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java 794e8258472df5fe18299eb3c62471cf9fecc186 
>   core/src/main/java/org/apache/oozie/util/ClasspathUtils.java PRE-CREATION 
>   core/src/main/resources/oozie-default.xml 6c2f7d82e933614ea6b434f9b035dca3751d1c7f 
>   core/src/test/java/org/apache/oozie/QueryServlet.java PRE-CREATION 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestCallbackNotifier.java PRE-CREATION 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 879bfebf58acc12ce787b151a67eaaa0716ff2bb 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestLauncherAM.java PRE-CREATION 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestPrepareActionsDriver.java df9e939cb236e8b65e7b3aa9ab683c33f5fa02b7 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestShellActionExecutor.java 6a962a1030a16e61a2af27a6316c4d71b17ea5c9 
>   core/src/test/java/org/apache/oozie/command/wf/HangServlet.java 3344cf97762c77f5660d94341e72df4243f76eb2 
>   core/src/test/java/org/apache/oozie/service/TestConfigurationService.java 2153bf122ddbdf81acd90c4922fa8d0f619d2d41 
>   core/src/test/java/org/apache/oozie/service/TestHadoopAccessorService.java 96faa4878111baafb81fdf0f3b38becba0c6f15e 
>   core/src/test/java/org/apache/oozie/test/XTestCase.java e3603692b2ae2ebb741adfa3a6ab5a37ea84eb96 
>   core/src/test/java/org/apache/oozie/util/TestClasspathUtils.java PRE-CREATION 
>   sharelib/distcp/pom.xml c8cc47cddc6d2922766e2d68ddddee96f5f3cfec 
>   sharelib/hcatalog/pom.xml 2b0c5042a6f295101f8c5fae0b08a4239b5e97d3 
>   sharelib/hive/pom.xml d10d7b85b70cbeb1906c83484739f34c09038134 
>   sharelib/hive2/pom.xml ce967c5575eb639335332dc6de7ce3f3c66a55ca 
>   sharelib/oozie/pom.xml dd95b4537c00b516b124341bdacf4fbd1fd351f9 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/CallbackNotifier.java PRE-CREATION 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java PRE-CREATION 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java f2cba13e89e8ddc16b59a04947309147ea6b34b7 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/PrepareActionsDriver.java 21ae456b768fb1cb5334a9e23e6c6082240f8dff 
>   sharelib/pig/pom.xml 562c5309cde72c7430fc03f4d342b6d30b5b4575 
>   sharelib/spark/pom.xml 46c6375314ac314fc6e702695303bbc8eda4f87b 
>   sharelib/sqoop/pom.xml d875c93ae599b990cdcbefdf2aa306eae41a9f0c 
>   sharelib/streaming/pom.xml fd795182f11f0f884177a56592e99aaea01d195d 
> 
> Diff: https://reviews.apache.org/r/49778/diff/
> 
> 
> Testing
> -------
> 
> - Tried a bunch of stuff against a Hadoop 2.4.0 cluster, including an "action" that would have different kinds of Exceptions, System.exit, etc.
> - Some new and old unit tests.
> 
> 
> Thanks,
> 
> Robert Kanter
> 
>

Re: Review Request 49778: OOZIE-2590: OYA: Create basic Oozie Launcher Application Master

Posted by Peter Bacsko <pb...@cloudera.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49778/#review142723
-----------------------------------------------------------


Ship it!




Ship It!

- Peter Bacsko


On j�l. 19, 2016, 6:47 de, Robert Kanter wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/49778/
> -----------------------------------------------------------
> 
> (Updated j�l. 19, 2016, 6:47 de)
> 
> 
> Review request for oozie.
> 
> 
> Bugs: OOZIE-2590
>     https://issues.apache.org/jira/browse/OOZIE-2590
> 
> 
> Repository: oozie-git
> 
> 
> Description
> -------
> 
> See OOZIE-2590
> 
> 
> Diffs
> -----
> 
>   core/pom.xml 6584af8123925e00f7b29d6b33201f4fc7dca231 
>   core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 96726daa800861c15e4a0d97514fbd45e31fa000 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2ActionExecutor.java b5b1bf90891fde1b13ca058e0b5f538b1b683128 
>   core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java c74e9e61c01a4b6430bbd5491c8cbaf0c4e90198 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 99e33445d68ac6a99a08e02ecc182f35103a53ce 
>   core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 69e1044e531a67c2d522f642376f4ad030423e53 
>   core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java 252f461267818c0da91c7ad59380b2f18a074f9f 
>   core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 6813a37d0ffc66da41636fe01e9cc7a37d6f6e79 
>   core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java 794e8258472df5fe18299eb3c62471cf9fecc186 
>   core/src/main/java/org/apache/oozie/util/ClasspathUtils.java PRE-CREATION 
>   core/src/main/resources/oozie-default.xml 6c2f7d82e933614ea6b434f9b035dca3751d1c7f 
>   core/src/test/java/org/apache/oozie/QueryServlet.java PRE-CREATION 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 879bfebf58acc12ce787b151a67eaaa0716ff2bb 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestLauncherAM.java PRE-CREATION 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestLauncherAMCallbackNotifier.java PRE-CREATION 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestPrepareActionsDriver.java df9e939cb236e8b65e7b3aa9ab683c33f5fa02b7 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestShellActionExecutor.java 6a962a1030a16e61a2af27a6316c4d71b17ea5c9 
>   core/src/test/java/org/apache/oozie/command/wf/HangServlet.java 3344cf97762c77f5660d94341e72df4243f76eb2 
>   core/src/test/java/org/apache/oozie/service/TestConfigurationService.java 2153bf122ddbdf81acd90c4922fa8d0f619d2d41 
>   core/src/test/java/org/apache/oozie/service/TestHadoopAccessorService.java 96faa4878111baafb81fdf0f3b38becba0c6f15e 
>   core/src/test/java/org/apache/oozie/test/XTestCase.java e3603692b2ae2ebb741adfa3a6ab5a37ea84eb96 
>   core/src/test/java/org/apache/oozie/util/TestClasspathUtils.java PRE-CREATION 
>   sharelib/distcp/pom.xml c8cc47cddc6d2922766e2d68ddddee96f5f3cfec 
>   sharelib/hcatalog/pom.xml 2b0c5042a6f295101f8c5fae0b08a4239b5e97d3 
>   sharelib/hive/pom.xml d10d7b85b70cbeb1906c83484739f34c09038134 
>   sharelib/hive2/pom.xml ce967c5575eb639335332dc6de7ce3f3c66a55ca 
>   sharelib/oozie/pom.xml dd95b4537c00b516b124341bdacf4fbd1fd351f9 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java PRE-CREATION 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAMCallbackNotifier.java PRE-CREATION 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java f2cba13e89e8ddc16b59a04947309147ea6b34b7 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/PrepareActionsDriver.java 21ae456b768fb1cb5334a9e23e6c6082240f8dff 
>   sharelib/pig/pom.xml 562c5309cde72c7430fc03f4d342b6d30b5b4575 
>   sharelib/spark/pom.xml 46c6375314ac314fc6e702695303bbc8eda4f87b 
>   sharelib/sqoop/pom.xml d875c93ae599b990cdcbefdf2aa306eae41a9f0c 
>   sharelib/streaming/pom.xml fd795182f11f0f884177a56592e99aaea01d195d 
> 
> Diff: https://reviews.apache.org/r/49778/diff/
> 
> 
> Testing
> -------
> 
> - Tried a bunch of stuff against a Hadoop 2.4.0 cluster, including an "action" that would have different kinds of Exceptions, System.exit, etc.
> - Some new and old unit tests.
> 
> 
> Thanks,
> 
> Robert Kanter
> 
>

Re: Review Request 49778: OOZIE-2590: OYA: Create basic Oozie Launcher Application Master

Posted by Robert Kanter <rk...@cloudera.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49778/#review143164
-----------------------------------------------------------


Ship it!




Unless there's anymore comments here or on the JIRA, I'm planning on committing this Monday afternoon so we can unblock the rest of the work.

- Robert Kanter


On July 19, 2016, 6:47 a.m., Robert Kanter wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/49778/
> -----------------------------------------------------------
> 
> (Updated July 19, 2016, 6:47 a.m.)
> 
> 
> Review request for oozie.
> 
> 
> Bugs: OOZIE-2590
>     https://issues.apache.org/jira/browse/OOZIE-2590
> 
> 
> Repository: oozie-git
> 
> 
> Description
> -------
> 
> See OOZIE-2590
> 
> 
> Diffs
> -----
> 
>   core/pom.xml 6584af8123925e00f7b29d6b33201f4fc7dca231 
>   core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 96726daa800861c15e4a0d97514fbd45e31fa000 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2ActionExecutor.java b5b1bf90891fde1b13ca058e0b5f538b1b683128 
>   core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java c74e9e61c01a4b6430bbd5491c8cbaf0c4e90198 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 99e33445d68ac6a99a08e02ecc182f35103a53ce 
>   core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 69e1044e531a67c2d522f642376f4ad030423e53 
>   core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java 252f461267818c0da91c7ad59380b2f18a074f9f 
>   core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 6813a37d0ffc66da41636fe01e9cc7a37d6f6e79 
>   core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java 794e8258472df5fe18299eb3c62471cf9fecc186 
>   core/src/main/java/org/apache/oozie/util/ClasspathUtils.java PRE-CREATION 
>   core/src/main/resources/oozie-default.xml 6c2f7d82e933614ea6b434f9b035dca3751d1c7f 
>   core/src/test/java/org/apache/oozie/QueryServlet.java PRE-CREATION 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 879bfebf58acc12ce787b151a67eaaa0716ff2bb 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestLauncherAM.java PRE-CREATION 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestLauncherAMCallbackNotifier.java PRE-CREATION 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestPrepareActionsDriver.java df9e939cb236e8b65e7b3aa9ab683c33f5fa02b7 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestShellActionExecutor.java 6a962a1030a16e61a2af27a6316c4d71b17ea5c9 
>   core/src/test/java/org/apache/oozie/command/wf/HangServlet.java 3344cf97762c77f5660d94341e72df4243f76eb2 
>   core/src/test/java/org/apache/oozie/service/TestConfigurationService.java 2153bf122ddbdf81acd90c4922fa8d0f619d2d41 
>   core/src/test/java/org/apache/oozie/service/TestHadoopAccessorService.java 96faa4878111baafb81fdf0f3b38becba0c6f15e 
>   core/src/test/java/org/apache/oozie/test/XTestCase.java e3603692b2ae2ebb741adfa3a6ab5a37ea84eb96 
>   core/src/test/java/org/apache/oozie/util/TestClasspathUtils.java PRE-CREATION 
>   sharelib/distcp/pom.xml c8cc47cddc6d2922766e2d68ddddee96f5f3cfec 
>   sharelib/hcatalog/pom.xml 2b0c5042a6f295101f8c5fae0b08a4239b5e97d3 
>   sharelib/hive/pom.xml d10d7b85b70cbeb1906c83484739f34c09038134 
>   sharelib/hive2/pom.xml ce967c5575eb639335332dc6de7ce3f3c66a55ca 
>   sharelib/oozie/pom.xml dd95b4537c00b516b124341bdacf4fbd1fd351f9 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java PRE-CREATION 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAMCallbackNotifier.java PRE-CREATION 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java f2cba13e89e8ddc16b59a04947309147ea6b34b7 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/PrepareActionsDriver.java 21ae456b768fb1cb5334a9e23e6c6082240f8dff 
>   sharelib/pig/pom.xml 562c5309cde72c7430fc03f4d342b6d30b5b4575 
>   sharelib/spark/pom.xml 46c6375314ac314fc6e702695303bbc8eda4f87b 
>   sharelib/sqoop/pom.xml d875c93ae599b990cdcbefdf2aa306eae41a9f0c 
>   sharelib/streaming/pom.xml fd795182f11f0f884177a56592e99aaea01d195d 
> 
> Diff: https://reviews.apache.org/r/49778/diff/
> 
> 
> Testing
> -------
> 
> - Tried a bunch of stuff against a Hadoop 2.4.0 cluster, including an "action" that would have different kinds of Exceptions, System.exit, etc.
> - Some new and old unit tests.
> 
> 
> Thanks,
> 
> Robert Kanter
> 
>

Re: Review Request 49778: OOZIE-2590: OYA: Create basic Oozie Launcher Application Master

Posted by Robert Kanter <rk...@cloudera.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49778/
-----------------------------------------------------------

(Updated July 19, 2016, 6:47 a.m.)


Review request for oozie.


Changes
-------

The 005 patch address the remaining feedback.  If there's no other comments, I think we're ready to push this in to unblock the other OYA subtasks.


Bugs: OOZIE-2590
    https://issues.apache.org/jira/browse/OOZIE-2590


Repository: oozie-git


Description
-------

See OOZIE-2590


Diffs (updated)
-----

  core/pom.xml 6584af8123925e00f7b29d6b33201f4fc7dca231 
  core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 96726daa800861c15e4a0d97514fbd45e31fa000 
  core/src/main/java/org/apache/oozie/action/hadoop/Hive2ActionExecutor.java b5b1bf90891fde1b13ca058e0b5f538b1b683128 
  core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java c74e9e61c01a4b6430bbd5491c8cbaf0c4e90198 
  core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 99e33445d68ac6a99a08e02ecc182f35103a53ce 
  core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 69e1044e531a67c2d522f642376f4ad030423e53 
  core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java 252f461267818c0da91c7ad59380b2f18a074f9f 
  core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 6813a37d0ffc66da41636fe01e9cc7a37d6f6e79 
  core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java 794e8258472df5fe18299eb3c62471cf9fecc186 
  core/src/main/java/org/apache/oozie/util/ClasspathUtils.java PRE-CREATION 
  core/src/main/resources/oozie-default.xml 6c2f7d82e933614ea6b434f9b035dca3751d1c7f 
  core/src/test/java/org/apache/oozie/QueryServlet.java PRE-CREATION 
  core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 879bfebf58acc12ce787b151a67eaaa0716ff2bb 
  core/src/test/java/org/apache/oozie/action/hadoop/TestLauncherAM.java PRE-CREATION 
  core/src/test/java/org/apache/oozie/action/hadoop/TestLauncherAMCallbackNotifier.java PRE-CREATION 
  core/src/test/java/org/apache/oozie/action/hadoop/TestPrepareActionsDriver.java df9e939cb236e8b65e7b3aa9ab683c33f5fa02b7 
  core/src/test/java/org/apache/oozie/action/hadoop/TestShellActionExecutor.java 6a962a1030a16e61a2af27a6316c4d71b17ea5c9 
  core/src/test/java/org/apache/oozie/command/wf/HangServlet.java 3344cf97762c77f5660d94341e72df4243f76eb2 
  core/src/test/java/org/apache/oozie/service/TestConfigurationService.java 2153bf122ddbdf81acd90c4922fa8d0f619d2d41 
  core/src/test/java/org/apache/oozie/service/TestHadoopAccessorService.java 96faa4878111baafb81fdf0f3b38becba0c6f15e 
  core/src/test/java/org/apache/oozie/test/XTestCase.java e3603692b2ae2ebb741adfa3a6ab5a37ea84eb96 
  core/src/test/java/org/apache/oozie/util/TestClasspathUtils.java PRE-CREATION 
  sharelib/distcp/pom.xml c8cc47cddc6d2922766e2d68ddddee96f5f3cfec 
  sharelib/hcatalog/pom.xml 2b0c5042a6f295101f8c5fae0b08a4239b5e97d3 
  sharelib/hive/pom.xml d10d7b85b70cbeb1906c83484739f34c09038134 
  sharelib/hive2/pom.xml ce967c5575eb639335332dc6de7ce3f3c66a55ca 
  sharelib/oozie/pom.xml dd95b4537c00b516b124341bdacf4fbd1fd351f9 
  sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java PRE-CREATION 
  sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAMCallbackNotifier.java PRE-CREATION 
  sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java f2cba13e89e8ddc16b59a04947309147ea6b34b7 
  sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/PrepareActionsDriver.java 21ae456b768fb1cb5334a9e23e6c6082240f8dff 
  sharelib/pig/pom.xml 562c5309cde72c7430fc03f4d342b6d30b5b4575 
  sharelib/spark/pom.xml 46c6375314ac314fc6e702695303bbc8eda4f87b 
  sharelib/sqoop/pom.xml d875c93ae599b990cdcbefdf2aa306eae41a9f0c 
  sharelib/streaming/pom.xml fd795182f11f0f884177a56592e99aaea01d195d 

Diff: https://reviews.apache.org/r/49778/diff/


Testing
-------

- Tried a bunch of stuff against a Hadoop 2.4.0 cluster, including an "action" that would have different kinds of Exceptions, System.exit, etc.
- Some new and old unit tests.


Thanks,

Robert Kanter

Re: Review Request 49778: OOZIE-2590: OYA: Create basic Oozie Launcher Application Master

Posted by Robert Kanter <rk...@cloudera.com>.


> On July 15, 2016, 9:31 a.m., Attila Sasvari wrote:
> > sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAMCallbackNotifier.java, line 150
> > <https://reviews.apache.org/r/49778/diff/3-4/?file=1443561#file1443561line150>
> >
> >     Is it a valid case that user does not supply an endpoint?

This should normally be supplied by Oozie.  But there's no harm in writing defensive code.  If something funny does happen here, it's better to handle this with a nice message rather than a NPE.


- Robert


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49778/#review142351
-----------------------------------------------------------


On July 15, 2016, 1:22 a.m., Robert Kanter wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/49778/
> -----------------------------------------------------------
> 
> (Updated July 15, 2016, 1:22 a.m.)
> 
> 
> Review request for oozie.
> 
> 
> Bugs: OOZIE-2590
>     https://issues.apache.org/jira/browse/OOZIE-2590
> 
> 
> Repository: oozie-git
> 
> 
> Description
> -------
> 
> See OOZIE-2590
> 
> 
> Diffs
> -----
> 
>   core/pom.xml 6584af8123925e00f7b29d6b33201f4fc7dca231 
>   core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 96726daa800861c15e4a0d97514fbd45e31fa000 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2ActionExecutor.java b5b1bf90891fde1b13ca058e0b5f538b1b683128 
>   core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java c74e9e61c01a4b6430bbd5491c8cbaf0c4e90198 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 99e33445d68ac6a99a08e02ecc182f35103a53ce 
>   core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 69e1044e531a67c2d522f642376f4ad030423e53 
>   core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java 252f461267818c0da91c7ad59380b2f18a074f9f 
>   core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 6813a37d0ffc66da41636fe01e9cc7a37d6f6e79 
>   core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java 794e8258472df5fe18299eb3c62471cf9fecc186 
>   core/src/main/java/org/apache/oozie/util/ClasspathUtils.java PRE-CREATION 
>   core/src/main/resources/oozie-default.xml 6c2f7d82e933614ea6b434f9b035dca3751d1c7f 
>   core/src/test/java/org/apache/oozie/QueryServlet.java PRE-CREATION 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 879bfebf58acc12ce787b151a67eaaa0716ff2bb 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestLauncherAM.java PRE-CREATION 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestLauncherAMCallbackNotifier.java PRE-CREATION 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestPrepareActionsDriver.java df9e939cb236e8b65e7b3aa9ab683c33f5fa02b7 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestShellActionExecutor.java 6a962a1030a16e61a2af27a6316c4d71b17ea5c9 
>   core/src/test/java/org/apache/oozie/command/wf/HangServlet.java 3344cf97762c77f5660d94341e72df4243f76eb2 
>   core/src/test/java/org/apache/oozie/service/TestConfigurationService.java 2153bf122ddbdf81acd90c4922fa8d0f619d2d41 
>   core/src/test/java/org/apache/oozie/service/TestHadoopAccessorService.java 96faa4878111baafb81fdf0f3b38becba0c6f15e 
>   core/src/test/java/org/apache/oozie/test/XTestCase.java e3603692b2ae2ebb741adfa3a6ab5a37ea84eb96 
>   core/src/test/java/org/apache/oozie/util/TestClasspathUtils.java PRE-CREATION 
>   sharelib/distcp/pom.xml c8cc47cddc6d2922766e2d68ddddee96f5f3cfec 
>   sharelib/hcatalog/pom.xml 2b0c5042a6f295101f8c5fae0b08a4239b5e97d3 
>   sharelib/hive/pom.xml d10d7b85b70cbeb1906c83484739f34c09038134 
>   sharelib/hive2/pom.xml ce967c5575eb639335332dc6de7ce3f3c66a55ca 
>   sharelib/oozie/pom.xml dd95b4537c00b516b124341bdacf4fbd1fd351f9 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java PRE-CREATION 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAMCallbackNotifier.java PRE-CREATION 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java f2cba13e89e8ddc16b59a04947309147ea6b34b7 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/PrepareActionsDriver.java 21ae456b768fb1cb5334a9e23e6c6082240f8dff 
>   sharelib/pig/pom.xml 562c5309cde72c7430fc03f4d342b6d30b5b4575 
>   sharelib/spark/pom.xml 46c6375314ac314fc6e702695303bbc8eda4f87b 
>   sharelib/sqoop/pom.xml d875c93ae599b990cdcbefdf2aa306eae41a9f0c 
>   sharelib/streaming/pom.xml fd795182f11f0f884177a56592e99aaea01d195d 
> 
> Diff: https://reviews.apache.org/r/49778/diff/
> 
> 
> Testing
> -------
> 
> - Tried a bunch of stuff against a Hadoop 2.4.0 cluster, including an "action" that would have different kinds of Exceptions, System.exit, etc.
> - Some new and old unit tests.
> 
> 
> Thanks,
> 
> Robert Kanter
> 
>

Re: Review Request 49778: OOZIE-2590: OYA: Create basic Oozie Launcher Application Master

Posted by Attila Sasvari <as...@cloudera.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49778/#review142351
-----------------------------------------------------------




sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAMCallbackNotifier.java (line 141)
<https://reviews.apache.org/r/49778/#comment207934>

    Is it a valid case that user does not supply an endpoint?


- Attila Sasvari


On July 15, 2016, 1:22 a.m., Robert Kanter wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/49778/
> -----------------------------------------------------------
> 
> (Updated July 15, 2016, 1:22 a.m.)
> 
> 
> Review request for oozie.
> 
> 
> Bugs: OOZIE-2590
>     https://issues.apache.org/jira/browse/OOZIE-2590
> 
> 
> Repository: oozie-git
> 
> 
> Description
> -------
> 
> See OOZIE-2590
> 
> 
> Diffs
> -----
> 
>   core/pom.xml 6584af8123925e00f7b29d6b33201f4fc7dca231 
>   core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 96726daa800861c15e4a0d97514fbd45e31fa000 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2ActionExecutor.java b5b1bf90891fde1b13ca058e0b5f538b1b683128 
>   core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java c74e9e61c01a4b6430bbd5491c8cbaf0c4e90198 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 99e33445d68ac6a99a08e02ecc182f35103a53ce 
>   core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 69e1044e531a67c2d522f642376f4ad030423e53 
>   core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java 252f461267818c0da91c7ad59380b2f18a074f9f 
>   core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 6813a37d0ffc66da41636fe01e9cc7a37d6f6e79 
>   core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java 794e8258472df5fe18299eb3c62471cf9fecc186 
>   core/src/main/java/org/apache/oozie/util/ClasspathUtils.java PRE-CREATION 
>   core/src/main/resources/oozie-default.xml 6c2f7d82e933614ea6b434f9b035dca3751d1c7f 
>   core/src/test/java/org/apache/oozie/QueryServlet.java PRE-CREATION 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 879bfebf58acc12ce787b151a67eaaa0716ff2bb 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestLauncherAM.java PRE-CREATION 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestLauncherAMCallbackNotifier.java PRE-CREATION 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestPrepareActionsDriver.java df9e939cb236e8b65e7b3aa9ab683c33f5fa02b7 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestShellActionExecutor.java 6a962a1030a16e61a2af27a6316c4d71b17ea5c9 
>   core/src/test/java/org/apache/oozie/command/wf/HangServlet.java 3344cf97762c77f5660d94341e72df4243f76eb2 
>   core/src/test/java/org/apache/oozie/service/TestConfigurationService.java 2153bf122ddbdf81acd90c4922fa8d0f619d2d41 
>   core/src/test/java/org/apache/oozie/service/TestHadoopAccessorService.java 96faa4878111baafb81fdf0f3b38becba0c6f15e 
>   core/src/test/java/org/apache/oozie/test/XTestCase.java e3603692b2ae2ebb741adfa3a6ab5a37ea84eb96 
>   core/src/test/java/org/apache/oozie/util/TestClasspathUtils.java PRE-CREATION 
>   sharelib/distcp/pom.xml c8cc47cddc6d2922766e2d68ddddee96f5f3cfec 
>   sharelib/hcatalog/pom.xml 2b0c5042a6f295101f8c5fae0b08a4239b5e97d3 
>   sharelib/hive/pom.xml d10d7b85b70cbeb1906c83484739f34c09038134 
>   sharelib/hive2/pom.xml ce967c5575eb639335332dc6de7ce3f3c66a55ca 
>   sharelib/oozie/pom.xml dd95b4537c00b516b124341bdacf4fbd1fd351f9 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java PRE-CREATION 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAMCallbackNotifier.java PRE-CREATION 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java f2cba13e89e8ddc16b59a04947309147ea6b34b7 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/PrepareActionsDriver.java 21ae456b768fb1cb5334a9e23e6c6082240f8dff 
>   sharelib/pig/pom.xml 562c5309cde72c7430fc03f4d342b6d30b5b4575 
>   sharelib/spark/pom.xml 46c6375314ac314fc6e702695303bbc8eda4f87b 
>   sharelib/sqoop/pom.xml d875c93ae599b990cdcbefdf2aa306eae41a9f0c 
>   sharelib/streaming/pom.xml fd795182f11f0f884177a56592e99aaea01d195d 
> 
> Diff: https://reviews.apache.org/r/49778/diff/
> 
> 
> Testing
> -------
> 
> - Tried a bunch of stuff against a Hadoop 2.4.0 cluster, including an "action" that would have different kinds of Exceptions, System.exit, etc.
> - Some new and old unit tests.
> 
> 
> Thanks,
> 
> Robert Kanter
> 
>

Re: Review Request 49778: OOZIE-2590: OYA: Create basic Oozie Launcher Application Master

Posted by Robert Kanter <rk...@cloudera.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49778/
-----------------------------------------------------------

(Updated July 15, 2016, 1:22 a.m.)


Review request for oozie.


Changes
-------

The 004 patch addresses more feedback


Bugs: OOZIE-2590
    https://issues.apache.org/jira/browse/OOZIE-2590


Repository: oozie-git


Description
-------

See OOZIE-2590


Diffs (updated)
-----

  core/pom.xml 6584af8123925e00f7b29d6b33201f4fc7dca231 
  core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 96726daa800861c15e4a0d97514fbd45e31fa000 
  core/src/main/java/org/apache/oozie/action/hadoop/Hive2ActionExecutor.java b5b1bf90891fde1b13ca058e0b5f538b1b683128 
  core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java c74e9e61c01a4b6430bbd5491c8cbaf0c4e90198 
  core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 99e33445d68ac6a99a08e02ecc182f35103a53ce 
  core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 69e1044e531a67c2d522f642376f4ad030423e53 
  core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java 252f461267818c0da91c7ad59380b2f18a074f9f 
  core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 6813a37d0ffc66da41636fe01e9cc7a37d6f6e79 
  core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java 794e8258472df5fe18299eb3c62471cf9fecc186 
  core/src/main/java/org/apache/oozie/util/ClasspathUtils.java PRE-CREATION 
  core/src/main/resources/oozie-default.xml 6c2f7d82e933614ea6b434f9b035dca3751d1c7f 
  core/src/test/java/org/apache/oozie/QueryServlet.java PRE-CREATION 
  core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 879bfebf58acc12ce787b151a67eaaa0716ff2bb 
  core/src/test/java/org/apache/oozie/action/hadoop/TestLauncherAM.java PRE-CREATION 
  core/src/test/java/org/apache/oozie/action/hadoop/TestLauncherAMCallbackNotifier.java PRE-CREATION 
  core/src/test/java/org/apache/oozie/action/hadoop/TestPrepareActionsDriver.java df9e939cb236e8b65e7b3aa9ab683c33f5fa02b7 
  core/src/test/java/org/apache/oozie/action/hadoop/TestShellActionExecutor.java 6a962a1030a16e61a2af27a6316c4d71b17ea5c9 
  core/src/test/java/org/apache/oozie/command/wf/HangServlet.java 3344cf97762c77f5660d94341e72df4243f76eb2 
  core/src/test/java/org/apache/oozie/service/TestConfigurationService.java 2153bf122ddbdf81acd90c4922fa8d0f619d2d41 
  core/src/test/java/org/apache/oozie/service/TestHadoopAccessorService.java 96faa4878111baafb81fdf0f3b38becba0c6f15e 
  core/src/test/java/org/apache/oozie/test/XTestCase.java e3603692b2ae2ebb741adfa3a6ab5a37ea84eb96 
  core/src/test/java/org/apache/oozie/util/TestClasspathUtils.java PRE-CREATION 
  sharelib/distcp/pom.xml c8cc47cddc6d2922766e2d68ddddee96f5f3cfec 
  sharelib/hcatalog/pom.xml 2b0c5042a6f295101f8c5fae0b08a4239b5e97d3 
  sharelib/hive/pom.xml d10d7b85b70cbeb1906c83484739f34c09038134 
  sharelib/hive2/pom.xml ce967c5575eb639335332dc6de7ce3f3c66a55ca 
  sharelib/oozie/pom.xml dd95b4537c00b516b124341bdacf4fbd1fd351f9 
  sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java PRE-CREATION 
  sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAMCallbackNotifier.java PRE-CREATION 
  sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java f2cba13e89e8ddc16b59a04947309147ea6b34b7 
  sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/PrepareActionsDriver.java 21ae456b768fb1cb5334a9e23e6c6082240f8dff 
  sharelib/pig/pom.xml 562c5309cde72c7430fc03f4d342b6d30b5b4575 
  sharelib/spark/pom.xml 46c6375314ac314fc6e702695303bbc8eda4f87b 
  sharelib/sqoop/pom.xml d875c93ae599b990cdcbefdf2aa306eae41a9f0c 
  sharelib/streaming/pom.xml fd795182f11f0f884177a56592e99aaea01d195d 

Diff: https://reviews.apache.org/r/49778/diff/


Testing
-------

- Tried a bunch of stuff against a Hadoop 2.4.0 cluster, including an "action" that would have different kinds of Exceptions, System.exit, etc.
- Some new and old unit tests.


Thanks,

Robert Kanter

Re: Review Request 49778: OOZIE-2590: OYA: Create basic Oozie Launcher Application Master

Posted by Attila Sasvari <as...@cloudera.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49778/#review142230
-----------------------------------------------------------




sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java (line 389)
<https://reviews.apache.org/r/49778/#comment207779>

    actionData is never empty
    (see actionData.put(ACTION_DATA_FINAL_STATUS, finalStatus.toString()); before call)


- Attila Sasvari


On July 13, 2016, 11:44 p.m., Robert Kanter wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/49778/
> -----------------------------------------------------------
> 
> (Updated July 13, 2016, 11:44 p.m.)
> 
> 
> Review request for oozie.
> 
> 
> Bugs: OOZIE-2590
>     https://issues.apache.org/jira/browse/OOZIE-2590
> 
> 
> Repository: oozie-git
> 
> 
> Description
> -------
> 
> See OOZIE-2590
> 
> 
> Diffs
> -----
> 
>   core/pom.xml 6584af8123925e00f7b29d6b33201f4fc7dca231 
>   core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 96726daa800861c15e4a0d97514fbd45e31fa000 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2ActionExecutor.java b5b1bf90891fde1b13ca058e0b5f538b1b683128 
>   core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java c74e9e61c01a4b6430bbd5491c8cbaf0c4e90198 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 99e33445d68ac6a99a08e02ecc182f35103a53ce 
>   core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 69e1044e531a67c2d522f642376f4ad030423e53 
>   core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java 252f461267818c0da91c7ad59380b2f18a074f9f 
>   core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 6813a37d0ffc66da41636fe01e9cc7a37d6f6e79 
>   core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java 794e8258472df5fe18299eb3c62471cf9fecc186 
>   core/src/main/java/org/apache/oozie/util/ClasspathUtils.java PRE-CREATION 
>   core/src/main/resources/oozie-default.xml 6c2f7d82e933614ea6b434f9b035dca3751d1c7f 
>   core/src/test/java/org/apache/oozie/QueryServlet.java PRE-CREATION 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 879bfebf58acc12ce787b151a67eaaa0716ff2bb 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestLauncherAM.java PRE-CREATION 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestLauncherAMCallbackNotifier.java PRE-CREATION 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestPrepareActionsDriver.java df9e939cb236e8b65e7b3aa9ab683c33f5fa02b7 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestShellActionExecutor.java 6a962a1030a16e61a2af27a6316c4d71b17ea5c9 
>   core/src/test/java/org/apache/oozie/command/wf/HangServlet.java 3344cf97762c77f5660d94341e72df4243f76eb2 
>   core/src/test/java/org/apache/oozie/service/TestConfigurationService.java 2153bf122ddbdf81acd90c4922fa8d0f619d2d41 
>   core/src/test/java/org/apache/oozie/service/TestHadoopAccessorService.java 96faa4878111baafb81fdf0f3b38becba0c6f15e 
>   core/src/test/java/org/apache/oozie/test/XTestCase.java e3603692b2ae2ebb741adfa3a6ab5a37ea84eb96 
>   core/src/test/java/org/apache/oozie/util/TestClasspathUtils.java PRE-CREATION 
>   sharelib/distcp/pom.xml c8cc47cddc6d2922766e2d68ddddee96f5f3cfec 
>   sharelib/hcatalog/pom.xml 2b0c5042a6f295101f8c5fae0b08a4239b5e97d3 
>   sharelib/hive/pom.xml d10d7b85b70cbeb1906c83484739f34c09038134 
>   sharelib/hive2/pom.xml ce967c5575eb639335332dc6de7ce3f3c66a55ca 
>   sharelib/oozie/pom.xml dd95b4537c00b516b124341bdacf4fbd1fd351f9 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java PRE-CREATION 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAMCallbackNotifier.java PRE-CREATION 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java f2cba13e89e8ddc16b59a04947309147ea6b34b7 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/PrepareActionsDriver.java 21ae456b768fb1cb5334a9e23e6c6082240f8dff 
>   sharelib/pig/pom.xml 562c5309cde72c7430fc03f4d342b6d30b5b4575 
>   sharelib/spark/pom.xml 46c6375314ac314fc6e702695303bbc8eda4f87b 
>   sharelib/sqoop/pom.xml d875c93ae599b990cdcbefdf2aa306eae41a9f0c 
>   sharelib/streaming/pom.xml fd795182f11f0f884177a56592e99aaea01d195d 
> 
> Diff: https://reviews.apache.org/r/49778/diff/
> 
> 
> Testing
> -------
> 
> - Tried a bunch of stuff against a Hadoop 2.4.0 cluster, including an "action" that would have different kinds of Exceptions, System.exit, etc.
> - Some new and old unit tests.
> 
> 
> Thanks,
> 
> Robert Kanter
> 
>

Re: Review Request 49778: OOZIE-2590: OYA: Create basic Oozie Launcher Application Master

Posted by Robert Kanter <rk...@cloudera.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49778/#review142272
-----------------------------------------------------------




sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java (line 398)
<https://reviews.apache.org/r/49778/#comment207837>

    This is deprecated.  Let's use the newer method:
    https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/SequenceFile.java#L259


- Robert Kanter


On July 13, 2016, 11:44 p.m., Robert Kanter wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/49778/
> -----------------------------------------------------------
> 
> (Updated July 13, 2016, 11:44 p.m.)
> 
> 
> Review request for oozie.
> 
> 
> Bugs: OOZIE-2590
>     https://issues.apache.org/jira/browse/OOZIE-2590
> 
> 
> Repository: oozie-git
> 
> 
> Description
> -------
> 
> See OOZIE-2590
> 
> 
> Diffs
> -----
> 
>   core/pom.xml 6584af8123925e00f7b29d6b33201f4fc7dca231 
>   core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 96726daa800861c15e4a0d97514fbd45e31fa000 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2ActionExecutor.java b5b1bf90891fde1b13ca058e0b5f538b1b683128 
>   core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java c74e9e61c01a4b6430bbd5491c8cbaf0c4e90198 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 99e33445d68ac6a99a08e02ecc182f35103a53ce 
>   core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 69e1044e531a67c2d522f642376f4ad030423e53 
>   core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java 252f461267818c0da91c7ad59380b2f18a074f9f 
>   core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 6813a37d0ffc66da41636fe01e9cc7a37d6f6e79 
>   core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java 794e8258472df5fe18299eb3c62471cf9fecc186 
>   core/src/main/java/org/apache/oozie/util/ClasspathUtils.java PRE-CREATION 
>   core/src/main/resources/oozie-default.xml 6c2f7d82e933614ea6b434f9b035dca3751d1c7f 
>   core/src/test/java/org/apache/oozie/QueryServlet.java PRE-CREATION 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 879bfebf58acc12ce787b151a67eaaa0716ff2bb 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestLauncherAM.java PRE-CREATION 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestLauncherAMCallbackNotifier.java PRE-CREATION 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestPrepareActionsDriver.java df9e939cb236e8b65e7b3aa9ab683c33f5fa02b7 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestShellActionExecutor.java 6a962a1030a16e61a2af27a6316c4d71b17ea5c9 
>   core/src/test/java/org/apache/oozie/command/wf/HangServlet.java 3344cf97762c77f5660d94341e72df4243f76eb2 
>   core/src/test/java/org/apache/oozie/service/TestConfigurationService.java 2153bf122ddbdf81acd90c4922fa8d0f619d2d41 
>   core/src/test/java/org/apache/oozie/service/TestHadoopAccessorService.java 96faa4878111baafb81fdf0f3b38becba0c6f15e 
>   core/src/test/java/org/apache/oozie/test/XTestCase.java e3603692b2ae2ebb741adfa3a6ab5a37ea84eb96 
>   core/src/test/java/org/apache/oozie/util/TestClasspathUtils.java PRE-CREATION 
>   sharelib/distcp/pom.xml c8cc47cddc6d2922766e2d68ddddee96f5f3cfec 
>   sharelib/hcatalog/pom.xml 2b0c5042a6f295101f8c5fae0b08a4239b5e97d3 
>   sharelib/hive/pom.xml d10d7b85b70cbeb1906c83484739f34c09038134 
>   sharelib/hive2/pom.xml ce967c5575eb639335332dc6de7ce3f3c66a55ca 
>   sharelib/oozie/pom.xml dd95b4537c00b516b124341bdacf4fbd1fd351f9 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java PRE-CREATION 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAMCallbackNotifier.java PRE-CREATION 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java f2cba13e89e8ddc16b59a04947309147ea6b34b7 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/PrepareActionsDriver.java 21ae456b768fb1cb5334a9e23e6c6082240f8dff 
>   sharelib/pig/pom.xml 562c5309cde72c7430fc03f4d342b6d30b5b4575 
>   sharelib/spark/pom.xml 46c6375314ac314fc6e702695303bbc8eda4f87b 
>   sharelib/sqoop/pom.xml d875c93ae599b990cdcbefdf2aa306eae41a9f0c 
>   sharelib/streaming/pom.xml fd795182f11f0f884177a56592e99aaea01d195d 
> 
> Diff: https://reviews.apache.org/r/49778/diff/
> 
> 
> Testing
> -------
> 
> - Tried a bunch of stuff against a Hadoop 2.4.0 cluster, including an "action" that would have different kinds of Exceptions, System.exit, etc.
> - Some new and old unit tests.
> 
> 
> Thanks,
> 
> Robert Kanter
> 
>

Re: Review Request 49778: OOZIE-2590: OYA: Create basic Oozie Launcher Application Master

Posted by Peter Bacsko <pb...@cloudera.com>.


> On j�l. 14, 2016, 1:25 du, Peter Bacsko wrote:
> > core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java, line 1230
> > <https://reviews.apache.org/r/49778/diff/3/?file=1443537#file1443537line1230>
> >
> >     Minor: is it necessary to null out explicitly?

Ok, it was asked by Karthik.


- Peter


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49778/#review142234
-----------------------------------------------------------


On j�l. 13, 2016, 11:44 du, Robert Kanter wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/49778/
> -----------------------------------------------------------
> 
> (Updated j�l. 13, 2016, 11:44 du)
> 
> 
> Review request for oozie.
> 
> 
> Bugs: OOZIE-2590
>     https://issues.apache.org/jira/browse/OOZIE-2590
> 
> 
> Repository: oozie-git
> 
> 
> Description
> -------
> 
> See OOZIE-2590
> 
> 
> Diffs
> -----
> 
>   core/pom.xml 6584af8123925e00f7b29d6b33201f4fc7dca231 
>   core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 96726daa800861c15e4a0d97514fbd45e31fa000 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2ActionExecutor.java b5b1bf90891fde1b13ca058e0b5f538b1b683128 
>   core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java c74e9e61c01a4b6430bbd5491c8cbaf0c4e90198 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 99e33445d68ac6a99a08e02ecc182f35103a53ce 
>   core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 69e1044e531a67c2d522f642376f4ad030423e53 
>   core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java 252f461267818c0da91c7ad59380b2f18a074f9f 
>   core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 6813a37d0ffc66da41636fe01e9cc7a37d6f6e79 
>   core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java 794e8258472df5fe18299eb3c62471cf9fecc186 
>   core/src/main/java/org/apache/oozie/util/ClasspathUtils.java PRE-CREATION 
>   core/src/main/resources/oozie-default.xml 6c2f7d82e933614ea6b434f9b035dca3751d1c7f 
>   core/src/test/java/org/apache/oozie/QueryServlet.java PRE-CREATION 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 879bfebf58acc12ce787b151a67eaaa0716ff2bb 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestLauncherAM.java PRE-CREATION 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestLauncherAMCallbackNotifier.java PRE-CREATION 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestPrepareActionsDriver.java df9e939cb236e8b65e7b3aa9ab683c33f5fa02b7 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestShellActionExecutor.java 6a962a1030a16e61a2af27a6316c4d71b17ea5c9 
>   core/src/test/java/org/apache/oozie/command/wf/HangServlet.java 3344cf97762c77f5660d94341e72df4243f76eb2 
>   core/src/test/java/org/apache/oozie/service/TestConfigurationService.java 2153bf122ddbdf81acd90c4922fa8d0f619d2d41 
>   core/src/test/java/org/apache/oozie/service/TestHadoopAccessorService.java 96faa4878111baafb81fdf0f3b38becba0c6f15e 
>   core/src/test/java/org/apache/oozie/test/XTestCase.java e3603692b2ae2ebb741adfa3a6ab5a37ea84eb96 
>   core/src/test/java/org/apache/oozie/util/TestClasspathUtils.java PRE-CREATION 
>   sharelib/distcp/pom.xml c8cc47cddc6d2922766e2d68ddddee96f5f3cfec 
>   sharelib/hcatalog/pom.xml 2b0c5042a6f295101f8c5fae0b08a4239b5e97d3 
>   sharelib/hive/pom.xml d10d7b85b70cbeb1906c83484739f34c09038134 
>   sharelib/hive2/pom.xml ce967c5575eb639335332dc6de7ce3f3c66a55ca 
>   sharelib/oozie/pom.xml dd95b4537c00b516b124341bdacf4fbd1fd351f9 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java PRE-CREATION 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAMCallbackNotifier.java PRE-CREATION 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java f2cba13e89e8ddc16b59a04947309147ea6b34b7 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/PrepareActionsDriver.java 21ae456b768fb1cb5334a9e23e6c6082240f8dff 
>   sharelib/pig/pom.xml 562c5309cde72c7430fc03f4d342b6d30b5b4575 
>   sharelib/spark/pom.xml 46c6375314ac314fc6e702695303bbc8eda4f87b 
>   sharelib/sqoop/pom.xml d875c93ae599b990cdcbefdf2aa306eae41a9f0c 
>   sharelib/streaming/pom.xml fd795182f11f0f884177a56592e99aaea01d195d 
> 
> Diff: https://reviews.apache.org/r/49778/diff/
> 
> 
> Testing
> -------
> 
> - Tried a bunch of stuff against a Hadoop 2.4.0 cluster, including an "action" that would have different kinds of Exceptions, System.exit, etc.
> - Some new and old unit tests.
> 
> 
> Thanks,
> 
> Robert Kanter
> 
>

Re: Review Request 49778: OOZIE-2590: OYA: Create basic Oozie Launcher Application Master

Posted by Peter Bacsko <pb...@cloudera.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49778/#review142234
-----------------------------------------------------------




core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java (line 1087)
<https://reviews.apache.org/r/49778/#comment207782>

    Minor: is it necessary to null out explicitly?



core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java (line 1503)
<https://reviews.apache.org/r/49778/#comment207783>

    Another equals() (didn't spot this previously)


- Peter Bacsko


On j�l. 13, 2016, 11:44 du, Robert Kanter wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/49778/
> -----------------------------------------------------------
> 
> (Updated j�l. 13, 2016, 11:44 du)
> 
> 
> Review request for oozie.
> 
> 
> Bugs: OOZIE-2590
>     https://issues.apache.org/jira/browse/OOZIE-2590
> 
> 
> Repository: oozie-git
> 
> 
> Description
> -------
> 
> See OOZIE-2590
> 
> 
> Diffs
> -----
> 
>   core/pom.xml 6584af8123925e00f7b29d6b33201f4fc7dca231 
>   core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 96726daa800861c15e4a0d97514fbd45e31fa000 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2ActionExecutor.java b5b1bf90891fde1b13ca058e0b5f538b1b683128 
>   core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java c74e9e61c01a4b6430bbd5491c8cbaf0c4e90198 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 99e33445d68ac6a99a08e02ecc182f35103a53ce 
>   core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 69e1044e531a67c2d522f642376f4ad030423e53 
>   core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java 252f461267818c0da91c7ad59380b2f18a074f9f 
>   core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 6813a37d0ffc66da41636fe01e9cc7a37d6f6e79 
>   core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java 794e8258472df5fe18299eb3c62471cf9fecc186 
>   core/src/main/java/org/apache/oozie/util/ClasspathUtils.java PRE-CREATION 
>   core/src/main/resources/oozie-default.xml 6c2f7d82e933614ea6b434f9b035dca3751d1c7f 
>   core/src/test/java/org/apache/oozie/QueryServlet.java PRE-CREATION 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 879bfebf58acc12ce787b151a67eaaa0716ff2bb 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestLauncherAM.java PRE-CREATION 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestLauncherAMCallbackNotifier.java PRE-CREATION 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestPrepareActionsDriver.java df9e939cb236e8b65e7b3aa9ab683c33f5fa02b7 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestShellActionExecutor.java 6a962a1030a16e61a2af27a6316c4d71b17ea5c9 
>   core/src/test/java/org/apache/oozie/command/wf/HangServlet.java 3344cf97762c77f5660d94341e72df4243f76eb2 
>   core/src/test/java/org/apache/oozie/service/TestConfigurationService.java 2153bf122ddbdf81acd90c4922fa8d0f619d2d41 
>   core/src/test/java/org/apache/oozie/service/TestHadoopAccessorService.java 96faa4878111baafb81fdf0f3b38becba0c6f15e 
>   core/src/test/java/org/apache/oozie/test/XTestCase.java e3603692b2ae2ebb741adfa3a6ab5a37ea84eb96 
>   core/src/test/java/org/apache/oozie/util/TestClasspathUtils.java PRE-CREATION 
>   sharelib/distcp/pom.xml c8cc47cddc6d2922766e2d68ddddee96f5f3cfec 
>   sharelib/hcatalog/pom.xml 2b0c5042a6f295101f8c5fae0b08a4239b5e97d3 
>   sharelib/hive/pom.xml d10d7b85b70cbeb1906c83484739f34c09038134 
>   sharelib/hive2/pom.xml ce967c5575eb639335332dc6de7ce3f3c66a55ca 
>   sharelib/oozie/pom.xml dd95b4537c00b516b124341bdacf4fbd1fd351f9 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java PRE-CREATION 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAMCallbackNotifier.java PRE-CREATION 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java f2cba13e89e8ddc16b59a04947309147ea6b34b7 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/PrepareActionsDriver.java 21ae456b768fb1cb5334a9e23e6c6082240f8dff 
>   sharelib/pig/pom.xml 562c5309cde72c7430fc03f4d342b6d30b5b4575 
>   sharelib/spark/pom.xml 46c6375314ac314fc6e702695303bbc8eda4f87b 
>   sharelib/sqoop/pom.xml d875c93ae599b990cdcbefdf2aa306eae41a9f0c 
>   sharelib/streaming/pom.xml fd795182f11f0f884177a56592e99aaea01d195d 
> 
> Diff: https://reviews.apache.org/r/49778/diff/
> 
> 
> Testing
> -------
> 
> - Tried a bunch of stuff against a Hadoop 2.4.0 cluster, including an "action" that would have different kinds of Exceptions, System.exit, etc.
> - Some new and old unit tests.
> 
> 
> Thanks,
> 
> Robert Kanter
> 
>

Re: Review Request 49778: OOZIE-2590: OYA: Create basic Oozie Launcher Application Master

Posted by Robert Kanter <rk...@cloudera.com>.


> On July 14, 2016, 1:38 p.m., Peter Cseh wrote:
> > core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java, line 1262
> > <https://reviews.apache.org/r/49778/diff/2/?file=1443093#file1443093line1262>
> >
> >     This could be move to a separate class.

For a ~20 line misc utility-type method?


- Robert


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49778/#review142039
-----------------------------------------------------------


On July 13, 2016, 11:44 p.m., Robert Kanter wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/49778/
> -----------------------------------------------------------
> 
> (Updated July 13, 2016, 11:44 p.m.)
> 
> 
> Review request for oozie.
> 
> 
> Bugs: OOZIE-2590
>     https://issues.apache.org/jira/browse/OOZIE-2590
> 
> 
> Repository: oozie-git
> 
> 
> Description
> -------
> 
> See OOZIE-2590
> 
> 
> Diffs
> -----
> 
>   core/pom.xml 6584af8123925e00f7b29d6b33201f4fc7dca231 
>   core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 96726daa800861c15e4a0d97514fbd45e31fa000 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2ActionExecutor.java b5b1bf90891fde1b13ca058e0b5f538b1b683128 
>   core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java c74e9e61c01a4b6430bbd5491c8cbaf0c4e90198 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 99e33445d68ac6a99a08e02ecc182f35103a53ce 
>   core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 69e1044e531a67c2d522f642376f4ad030423e53 
>   core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java 252f461267818c0da91c7ad59380b2f18a074f9f 
>   core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 6813a37d0ffc66da41636fe01e9cc7a37d6f6e79 
>   core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java 794e8258472df5fe18299eb3c62471cf9fecc186 
>   core/src/main/java/org/apache/oozie/util/ClasspathUtils.java PRE-CREATION 
>   core/src/main/resources/oozie-default.xml 6c2f7d82e933614ea6b434f9b035dca3751d1c7f 
>   core/src/test/java/org/apache/oozie/QueryServlet.java PRE-CREATION 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 879bfebf58acc12ce787b151a67eaaa0716ff2bb 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestLauncherAM.java PRE-CREATION 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestLauncherAMCallbackNotifier.java PRE-CREATION 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestPrepareActionsDriver.java df9e939cb236e8b65e7b3aa9ab683c33f5fa02b7 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestShellActionExecutor.java 6a962a1030a16e61a2af27a6316c4d71b17ea5c9 
>   core/src/test/java/org/apache/oozie/command/wf/HangServlet.java 3344cf97762c77f5660d94341e72df4243f76eb2 
>   core/src/test/java/org/apache/oozie/service/TestConfigurationService.java 2153bf122ddbdf81acd90c4922fa8d0f619d2d41 
>   core/src/test/java/org/apache/oozie/service/TestHadoopAccessorService.java 96faa4878111baafb81fdf0f3b38becba0c6f15e 
>   core/src/test/java/org/apache/oozie/test/XTestCase.java e3603692b2ae2ebb741adfa3a6ab5a37ea84eb96 
>   core/src/test/java/org/apache/oozie/util/TestClasspathUtils.java PRE-CREATION 
>   sharelib/distcp/pom.xml c8cc47cddc6d2922766e2d68ddddee96f5f3cfec 
>   sharelib/hcatalog/pom.xml 2b0c5042a6f295101f8c5fae0b08a4239b5e97d3 
>   sharelib/hive/pom.xml d10d7b85b70cbeb1906c83484739f34c09038134 
>   sharelib/hive2/pom.xml ce967c5575eb639335332dc6de7ce3f3c66a55ca 
>   sharelib/oozie/pom.xml dd95b4537c00b516b124341bdacf4fbd1fd351f9 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java PRE-CREATION 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAMCallbackNotifier.java PRE-CREATION 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java f2cba13e89e8ddc16b59a04947309147ea6b34b7 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/PrepareActionsDriver.java 21ae456b768fb1cb5334a9e23e6c6082240f8dff 
>   sharelib/pig/pom.xml 562c5309cde72c7430fc03f4d342b6d30b5b4575 
>   sharelib/spark/pom.xml 46c6375314ac314fc6e702695303bbc8eda4f87b 
>   sharelib/sqoop/pom.xml d875c93ae599b990cdcbefdf2aa306eae41a9f0c 
>   sharelib/streaming/pom.xml fd795182f11f0f884177a56592e99aaea01d195d 
> 
> Diff: https://reviews.apache.org/r/49778/diff/
> 
> 
> Testing
> -------
> 
> - Tried a bunch of stuff against a Hadoop 2.4.0 cluster, including an "action" that would have different kinds of Exceptions, System.exit, etc.
> - Some new and old unit tests.
> 
> 
> Thanks,
> 
> Robert Kanter
> 
>

Re: Review Request 49778: OOZIE-2590: OYA: Create basic Oozie Launcher Application Master

Posted by Peter Cseh <ge...@cloudera.com>.


> On July 14, 2016, 1:38 p.m., Peter Cseh wrote:
> > core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java, line 1262
> > <https://reviews.apache.org/r/49778/diff/2/?file=1443093#file1443093line1262>
> >
> >     This could be move to a separate class.
> 
> Robert Kanter wrote:
>     For a ~20 line misc utility-type method?

A separate class might be an overkill. A better place for these createAppSubmissionContext and createLocalResourceForFile methods could ve HadoopAccessorService as createLocalResourceForFile uses it anyway.


- Peter


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49778/#review142039
-----------------------------------------------------------


On July 15, 2016, 1:22 a.m., Robert Kanter wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/49778/
> -----------------------------------------------------------
> 
> (Updated July 15, 2016, 1:22 a.m.)
> 
> 
> Review request for oozie.
> 
> 
> Bugs: OOZIE-2590
>     https://issues.apache.org/jira/browse/OOZIE-2590
> 
> 
> Repository: oozie-git
> 
> 
> Description
> -------
> 
> See OOZIE-2590
> 
> 
> Diffs
> -----
> 
>   core/pom.xml 6584af8123925e00f7b29d6b33201f4fc7dca231 
>   core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 96726daa800861c15e4a0d97514fbd45e31fa000 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2ActionExecutor.java b5b1bf90891fde1b13ca058e0b5f538b1b683128 
>   core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java c74e9e61c01a4b6430bbd5491c8cbaf0c4e90198 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 99e33445d68ac6a99a08e02ecc182f35103a53ce 
>   core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 69e1044e531a67c2d522f642376f4ad030423e53 
>   core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java 252f461267818c0da91c7ad59380b2f18a074f9f 
>   core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 6813a37d0ffc66da41636fe01e9cc7a37d6f6e79 
>   core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java 794e8258472df5fe18299eb3c62471cf9fecc186 
>   core/src/main/java/org/apache/oozie/util/ClasspathUtils.java PRE-CREATION 
>   core/src/main/resources/oozie-default.xml 6c2f7d82e933614ea6b434f9b035dca3751d1c7f 
>   core/src/test/java/org/apache/oozie/QueryServlet.java PRE-CREATION 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 879bfebf58acc12ce787b151a67eaaa0716ff2bb 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestLauncherAM.java PRE-CREATION 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestLauncherAMCallbackNotifier.java PRE-CREATION 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestPrepareActionsDriver.java df9e939cb236e8b65e7b3aa9ab683c33f5fa02b7 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestShellActionExecutor.java 6a962a1030a16e61a2af27a6316c4d71b17ea5c9 
>   core/src/test/java/org/apache/oozie/command/wf/HangServlet.java 3344cf97762c77f5660d94341e72df4243f76eb2 
>   core/src/test/java/org/apache/oozie/service/TestConfigurationService.java 2153bf122ddbdf81acd90c4922fa8d0f619d2d41 
>   core/src/test/java/org/apache/oozie/service/TestHadoopAccessorService.java 96faa4878111baafb81fdf0f3b38becba0c6f15e 
>   core/src/test/java/org/apache/oozie/test/XTestCase.java e3603692b2ae2ebb741adfa3a6ab5a37ea84eb96 
>   core/src/test/java/org/apache/oozie/util/TestClasspathUtils.java PRE-CREATION 
>   sharelib/distcp/pom.xml c8cc47cddc6d2922766e2d68ddddee96f5f3cfec 
>   sharelib/hcatalog/pom.xml 2b0c5042a6f295101f8c5fae0b08a4239b5e97d3 
>   sharelib/hive/pom.xml d10d7b85b70cbeb1906c83484739f34c09038134 
>   sharelib/hive2/pom.xml ce967c5575eb639335332dc6de7ce3f3c66a55ca 
>   sharelib/oozie/pom.xml dd95b4537c00b516b124341bdacf4fbd1fd351f9 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java PRE-CREATION 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAMCallbackNotifier.java PRE-CREATION 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java f2cba13e89e8ddc16b59a04947309147ea6b34b7 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/PrepareActionsDriver.java 21ae456b768fb1cb5334a9e23e6c6082240f8dff 
>   sharelib/pig/pom.xml 562c5309cde72c7430fc03f4d342b6d30b5b4575 
>   sharelib/spark/pom.xml 46c6375314ac314fc6e702695303bbc8eda4f87b 
>   sharelib/sqoop/pom.xml d875c93ae599b990cdcbefdf2aa306eae41a9f0c 
>   sharelib/streaming/pom.xml fd795182f11f0f884177a56592e99aaea01d195d 
> 
> Diff: https://reviews.apache.org/r/49778/diff/
> 
> 
> Testing
> -------
> 
> - Tried a bunch of stuff against a Hadoop 2.4.0 cluster, including an "action" that would have different kinds of Exceptions, System.exit, etc.
> - Some new and old unit tests.
> 
> 
> Thanks,
> 
> Robert Kanter
> 
>

Re: Review Request 49778: OOZIE-2590: OYA: Create basic Oozie Launcher Application Master

Posted by Robert Kanter <rk...@cloudera.com>.


> On July 14, 2016, 1:38 p.m., Peter Cseh wrote:
> > core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java, line 1262
> > <https://reviews.apache.org/r/49778/diff/2/?file=1443093#file1443093line1262>
> >
> >     This could be move to a separate class.
> 
> Robert Kanter wrote:
>     For a ~20 line misc utility-type method?
> 
> Peter Cseh wrote:
>     A separate class might be an overkill. A better place for these createAppSubmissionContext and createLocalResourceForFile methods could ve HadoopAccessorService as createLocalResourceForFile uses it anyway.

HadoopAcessorService is more for functions and code used for creating things to talk to Hadoop (e.g. create a YarnClient, get the FileSystem, load the Hadoop configs into a Configuration, etc), while JavaActionExecutor actually uses those.  So I think createAppSubmissionContext should stay in JavaActionExecutor (it also has a lot of things specific to Action code), but I'll move createLocalResourceForFile to HadoopAccessorService.


- Robert


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49778/#review142039
-----------------------------------------------------------


On July 15, 2016, 1:22 a.m., Robert Kanter wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/49778/
> -----------------------------------------------------------
> 
> (Updated July 15, 2016, 1:22 a.m.)
> 
> 
> Review request for oozie.
> 
> 
> Bugs: OOZIE-2590
>     https://issues.apache.org/jira/browse/OOZIE-2590
> 
> 
> Repository: oozie-git
> 
> 
> Description
> -------
> 
> See OOZIE-2590
> 
> 
> Diffs
> -----
> 
>   core/pom.xml 6584af8123925e00f7b29d6b33201f4fc7dca231 
>   core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 96726daa800861c15e4a0d97514fbd45e31fa000 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2ActionExecutor.java b5b1bf90891fde1b13ca058e0b5f538b1b683128 
>   core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java c74e9e61c01a4b6430bbd5491c8cbaf0c4e90198 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 99e33445d68ac6a99a08e02ecc182f35103a53ce 
>   core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 69e1044e531a67c2d522f642376f4ad030423e53 
>   core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java 252f461267818c0da91c7ad59380b2f18a074f9f 
>   core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 6813a37d0ffc66da41636fe01e9cc7a37d6f6e79 
>   core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java 794e8258472df5fe18299eb3c62471cf9fecc186 
>   core/src/main/java/org/apache/oozie/util/ClasspathUtils.java PRE-CREATION 
>   core/src/main/resources/oozie-default.xml 6c2f7d82e933614ea6b434f9b035dca3751d1c7f 
>   core/src/test/java/org/apache/oozie/QueryServlet.java PRE-CREATION 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 879bfebf58acc12ce787b151a67eaaa0716ff2bb 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestLauncherAM.java PRE-CREATION 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestLauncherAMCallbackNotifier.java PRE-CREATION 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestPrepareActionsDriver.java df9e939cb236e8b65e7b3aa9ab683c33f5fa02b7 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestShellActionExecutor.java 6a962a1030a16e61a2af27a6316c4d71b17ea5c9 
>   core/src/test/java/org/apache/oozie/command/wf/HangServlet.java 3344cf97762c77f5660d94341e72df4243f76eb2 
>   core/src/test/java/org/apache/oozie/service/TestConfigurationService.java 2153bf122ddbdf81acd90c4922fa8d0f619d2d41 
>   core/src/test/java/org/apache/oozie/service/TestHadoopAccessorService.java 96faa4878111baafb81fdf0f3b38becba0c6f15e 
>   core/src/test/java/org/apache/oozie/test/XTestCase.java e3603692b2ae2ebb741adfa3a6ab5a37ea84eb96 
>   core/src/test/java/org/apache/oozie/util/TestClasspathUtils.java PRE-CREATION 
>   sharelib/distcp/pom.xml c8cc47cddc6d2922766e2d68ddddee96f5f3cfec 
>   sharelib/hcatalog/pom.xml 2b0c5042a6f295101f8c5fae0b08a4239b5e97d3 
>   sharelib/hive/pom.xml d10d7b85b70cbeb1906c83484739f34c09038134 
>   sharelib/hive2/pom.xml ce967c5575eb639335332dc6de7ce3f3c66a55ca 
>   sharelib/oozie/pom.xml dd95b4537c00b516b124341bdacf4fbd1fd351f9 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java PRE-CREATION 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAMCallbackNotifier.java PRE-CREATION 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java f2cba13e89e8ddc16b59a04947309147ea6b34b7 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/PrepareActionsDriver.java 21ae456b768fb1cb5334a9e23e6c6082240f8dff 
>   sharelib/pig/pom.xml 562c5309cde72c7430fc03f4d342b6d30b5b4575 
>   sharelib/spark/pom.xml 46c6375314ac314fc6e702695303bbc8eda4f87b 
>   sharelib/sqoop/pom.xml d875c93ae599b990cdcbefdf2aa306eae41a9f0c 
>   sharelib/streaming/pom.xml fd795182f11f0f884177a56592e99aaea01d195d 
> 
> Diff: https://reviews.apache.org/r/49778/diff/
> 
> 
> Testing
> -------
> 
> - Tried a bunch of stuff against a Hadoop 2.4.0 cluster, including an "action" that would have different kinds of Exceptions, System.exit, etc.
> - Some new and old unit tests.
> 
> 
> Thanks,
> 
> Robert Kanter
> 
>

Re: Review Request 49778: OOZIE-2590: OYA: Create basic Oozie Launcher Application Master

Posted by Peter Cseh <ge...@cloudera.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49778/#review142039
-----------------------------------------------------------




core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java (line 1116)
<https://reviews.apache.org/r/49778/#comment207523>

    This could be move to a separate class.



sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java (lines 54 - 66)
<https://reviews.apache.org/r/49778/#comment207525>

    Constant and property names should be synced.


- Peter Cseh


On July 13, 2016, 11:44 p.m., Robert Kanter wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/49778/
> -----------------------------------------------------------
> 
> (Updated July 13, 2016, 11:44 p.m.)
> 
> 
> Review request for oozie.
> 
> 
> Bugs: OOZIE-2590
>     https://issues.apache.org/jira/browse/OOZIE-2590
> 
> 
> Repository: oozie-git
> 
> 
> Description
> -------
> 
> See OOZIE-2590
> 
> 
> Diffs
> -----
> 
>   core/pom.xml 6584af8123925e00f7b29d6b33201f4fc7dca231 
>   core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 96726daa800861c15e4a0d97514fbd45e31fa000 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2ActionExecutor.java b5b1bf90891fde1b13ca058e0b5f538b1b683128 
>   core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java c74e9e61c01a4b6430bbd5491c8cbaf0c4e90198 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 99e33445d68ac6a99a08e02ecc182f35103a53ce 
>   core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 69e1044e531a67c2d522f642376f4ad030423e53 
>   core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java 252f461267818c0da91c7ad59380b2f18a074f9f 
>   core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 6813a37d0ffc66da41636fe01e9cc7a37d6f6e79 
>   core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java 794e8258472df5fe18299eb3c62471cf9fecc186 
>   core/src/main/java/org/apache/oozie/util/ClasspathUtils.java PRE-CREATION 
>   core/src/main/resources/oozie-default.xml 6c2f7d82e933614ea6b434f9b035dca3751d1c7f 
>   core/src/test/java/org/apache/oozie/QueryServlet.java PRE-CREATION 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 879bfebf58acc12ce787b151a67eaaa0716ff2bb 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestLauncherAM.java PRE-CREATION 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestLauncherAMCallbackNotifier.java PRE-CREATION 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestPrepareActionsDriver.java df9e939cb236e8b65e7b3aa9ab683c33f5fa02b7 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestShellActionExecutor.java 6a962a1030a16e61a2af27a6316c4d71b17ea5c9 
>   core/src/test/java/org/apache/oozie/command/wf/HangServlet.java 3344cf97762c77f5660d94341e72df4243f76eb2 
>   core/src/test/java/org/apache/oozie/service/TestConfigurationService.java 2153bf122ddbdf81acd90c4922fa8d0f619d2d41 
>   core/src/test/java/org/apache/oozie/service/TestHadoopAccessorService.java 96faa4878111baafb81fdf0f3b38becba0c6f15e 
>   core/src/test/java/org/apache/oozie/test/XTestCase.java e3603692b2ae2ebb741adfa3a6ab5a37ea84eb96 
>   core/src/test/java/org/apache/oozie/util/TestClasspathUtils.java PRE-CREATION 
>   sharelib/distcp/pom.xml c8cc47cddc6d2922766e2d68ddddee96f5f3cfec 
>   sharelib/hcatalog/pom.xml 2b0c5042a6f295101f8c5fae0b08a4239b5e97d3 
>   sharelib/hive/pom.xml d10d7b85b70cbeb1906c83484739f34c09038134 
>   sharelib/hive2/pom.xml ce967c5575eb639335332dc6de7ce3f3c66a55ca 
>   sharelib/oozie/pom.xml dd95b4537c00b516b124341bdacf4fbd1fd351f9 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java PRE-CREATION 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAMCallbackNotifier.java PRE-CREATION 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java f2cba13e89e8ddc16b59a04947309147ea6b34b7 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/PrepareActionsDriver.java 21ae456b768fb1cb5334a9e23e6c6082240f8dff 
>   sharelib/pig/pom.xml 562c5309cde72c7430fc03f4d342b6d30b5b4575 
>   sharelib/spark/pom.xml 46c6375314ac314fc6e702695303bbc8eda4f87b 
>   sharelib/sqoop/pom.xml d875c93ae599b990cdcbefdf2aa306eae41a9f0c 
>   sharelib/streaming/pom.xml fd795182f11f0f884177a56592e99aaea01d195d 
> 
> Diff: https://reviews.apache.org/r/49778/diff/
> 
> 
> Testing
> -------
> 
> - Tried a bunch of stuff against a Hadoop 2.4.0 cluster, including an "action" that would have different kinds of Exceptions, System.exit, etc.
> - Some new and old unit tests.
> 
> 
> Thanks,
> 
> Robert Kanter
> 
>

Re: Review Request 49778: OOZIE-2590: OYA: Create basic Oozie Launcher Application Master

Posted by Robert Kanter <rk...@cloudera.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49778/
-----------------------------------------------------------

(Updated July 13, 2016, 11:44 p.m.)


Review request for oozie.


Changes
-------

The 002 patch addresses a bunch of review feedback from reviewboard.

I'll look more into splitting up the LauncherAM main method because everyone's been asking about it.


Bugs: OOZIE-2590
    https://issues.apache.org/jira/browse/OOZIE-2590


Repository: oozie-git


Description
-------

See OOZIE-2590


Diffs (updated)
-----

  core/pom.xml 6584af8123925e00f7b29d6b33201f4fc7dca231 
  core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 96726daa800861c15e4a0d97514fbd45e31fa000 
  core/src/main/java/org/apache/oozie/action/hadoop/Hive2ActionExecutor.java b5b1bf90891fde1b13ca058e0b5f538b1b683128 
  core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java c74e9e61c01a4b6430bbd5491c8cbaf0c4e90198 
  core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 99e33445d68ac6a99a08e02ecc182f35103a53ce 
  core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 69e1044e531a67c2d522f642376f4ad030423e53 
  core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java 252f461267818c0da91c7ad59380b2f18a074f9f 
  core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 6813a37d0ffc66da41636fe01e9cc7a37d6f6e79 
  core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java 794e8258472df5fe18299eb3c62471cf9fecc186 
  core/src/main/java/org/apache/oozie/util/ClasspathUtils.java PRE-CREATION 
  core/src/main/resources/oozie-default.xml 6c2f7d82e933614ea6b434f9b035dca3751d1c7f 
  core/src/test/java/org/apache/oozie/QueryServlet.java PRE-CREATION 
  core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 879bfebf58acc12ce787b151a67eaaa0716ff2bb 
  core/src/test/java/org/apache/oozie/action/hadoop/TestLauncherAM.java PRE-CREATION 
  core/src/test/java/org/apache/oozie/action/hadoop/TestLauncherAMCallbackNotifier.java PRE-CREATION 
  core/src/test/java/org/apache/oozie/action/hadoop/TestPrepareActionsDriver.java df9e939cb236e8b65e7b3aa9ab683c33f5fa02b7 
  core/src/test/java/org/apache/oozie/action/hadoop/TestShellActionExecutor.java 6a962a1030a16e61a2af27a6316c4d71b17ea5c9 
  core/src/test/java/org/apache/oozie/command/wf/HangServlet.java 3344cf97762c77f5660d94341e72df4243f76eb2 
  core/src/test/java/org/apache/oozie/service/TestConfigurationService.java 2153bf122ddbdf81acd90c4922fa8d0f619d2d41 
  core/src/test/java/org/apache/oozie/service/TestHadoopAccessorService.java 96faa4878111baafb81fdf0f3b38becba0c6f15e 
  core/src/test/java/org/apache/oozie/test/XTestCase.java e3603692b2ae2ebb741adfa3a6ab5a37ea84eb96 
  core/src/test/java/org/apache/oozie/util/TestClasspathUtils.java PRE-CREATION 
  sharelib/distcp/pom.xml c8cc47cddc6d2922766e2d68ddddee96f5f3cfec 
  sharelib/hcatalog/pom.xml 2b0c5042a6f295101f8c5fae0b08a4239b5e97d3 
  sharelib/hive/pom.xml d10d7b85b70cbeb1906c83484739f34c09038134 
  sharelib/hive2/pom.xml ce967c5575eb639335332dc6de7ce3f3c66a55ca 
  sharelib/oozie/pom.xml dd95b4537c00b516b124341bdacf4fbd1fd351f9 
  sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java PRE-CREATION 
  sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAMCallbackNotifier.java PRE-CREATION 
  sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java f2cba13e89e8ddc16b59a04947309147ea6b34b7 
  sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/PrepareActionsDriver.java 21ae456b768fb1cb5334a9e23e6c6082240f8dff 
  sharelib/pig/pom.xml 562c5309cde72c7430fc03f4d342b6d30b5b4575 
  sharelib/spark/pom.xml 46c6375314ac314fc6e702695303bbc8eda4f87b 
  sharelib/sqoop/pom.xml d875c93ae599b990cdcbefdf2aa306eae41a9f0c 
  sharelib/streaming/pom.xml fd795182f11f0f884177a56592e99aaea01d195d 

Diff: https://reviews.apache.org/r/49778/diff/


Testing
-------

- Tried a bunch of stuff against a Hadoop 2.4.0 cluster, including an "action" that would have different kinds of Exceptions, System.exit, etc.
- Some new and old unit tests.


Thanks,

Robert Kanter

Re: Review Request 49778: OOZIE-2590: OYA: Create basic Oozie Launcher Application Master

Posted by Peter Bacsko <pb...@cloudera.com>.


> On j�l. 13, 2016, 9:01 de, Attila Sasvari wrote:
> > core/src/main/java/org/apache/oozie/util/ClasspathUtils.java, line 56
> > <https://reviews.apache.org/r/49778/diff/2/?file=1443098#file1443098line56>
> >
> >     You could reduce the amount of code duplication by having a list of the needed classpath entries with something like
> >     
> >     ```
> >     List<String> classpathEntries = Arrays.asList(
> >                     ApplicationConstants.Environment.PWD.$(),
> >                     MRJobConfig.JOB_JAR + Path.SEPARATOR + MRJobConfig.JOB_JAR,
> >                     MRJobConfig.JOB_JAR + Path.SEPARATOR + "classes" + Path.SEPARATOR,
> >                     MRJobConfig.JOB_JAR + Path.SEPARATOR + "lib" + Path.SEPARATOR + "*",
> >                     ApplicationConstants.Environment.PWD.$() + Path.SEPARATOR + "*"
> >             );
> >     
> >             for (String c : classpathEntries) {
> >                 MRApps.addToEnvironment(env, ApplicationConstants.Environment.CLASSPATH.name(), c, conf);
> >             }
> >     ```

+1


- Peter


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49778/#review142033
-----------------------------------------------------------


On j�l. 13, 2016, 5:42 de, Robert Kanter wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/49778/
> -----------------------------------------------------------
> 
> (Updated j�l. 13, 2016, 5:42 de)
> 
> 
> Review request for oozie.
> 
> 
> Bugs: OOZIE-2590
>     https://issues.apache.org/jira/browse/OOZIE-2590
> 
> 
> Repository: oozie-git
> 
> 
> Description
> -------
> 
> See OOZIE-2590
> 
> 
> Diffs
> -----
> 
>   core/pom.xml 6584af8123925e00f7b29d6b33201f4fc7dca231 
>   core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 96726daa800861c15e4a0d97514fbd45e31fa000 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2ActionExecutor.java b5b1bf90891fde1b13ca058e0b5f538b1b683128 
>   core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java c74e9e61c01a4b6430bbd5491c8cbaf0c4e90198 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 99e33445d68ac6a99a08e02ecc182f35103a53ce 
>   core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 69e1044e531a67c2d522f642376f4ad030423e53 
>   core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java 252f461267818c0da91c7ad59380b2f18a074f9f 
>   core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 6813a37d0ffc66da41636fe01e9cc7a37d6f6e79 
>   core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java 794e8258472df5fe18299eb3c62471cf9fecc186 
>   core/src/main/java/org/apache/oozie/util/ClasspathUtils.java PRE-CREATION 
>   core/src/main/resources/oozie-default.xml 6c2f7d82e933614ea6b434f9b035dca3751d1c7f 
>   core/src/test/java/org/apache/oozie/QueryServlet.java PRE-CREATION 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestCallbackNotifier.java PRE-CREATION 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 879bfebf58acc12ce787b151a67eaaa0716ff2bb 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestLauncherAM.java PRE-CREATION 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestPrepareActionsDriver.java df9e939cb236e8b65e7b3aa9ab683c33f5fa02b7 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestShellActionExecutor.java 6a962a1030a16e61a2af27a6316c4d71b17ea5c9 
>   core/src/test/java/org/apache/oozie/command/wf/HangServlet.java 3344cf97762c77f5660d94341e72df4243f76eb2 
>   core/src/test/java/org/apache/oozie/service/TestConfigurationService.java 2153bf122ddbdf81acd90c4922fa8d0f619d2d41 
>   core/src/test/java/org/apache/oozie/service/TestHadoopAccessorService.java 96faa4878111baafb81fdf0f3b38becba0c6f15e 
>   core/src/test/java/org/apache/oozie/test/XTestCase.java e3603692b2ae2ebb741adfa3a6ab5a37ea84eb96 
>   core/src/test/java/org/apache/oozie/util/TestClasspathUtils.java PRE-CREATION 
>   sharelib/distcp/pom.xml c8cc47cddc6d2922766e2d68ddddee96f5f3cfec 
>   sharelib/hcatalog/pom.xml 2b0c5042a6f295101f8c5fae0b08a4239b5e97d3 
>   sharelib/hive/pom.xml d10d7b85b70cbeb1906c83484739f34c09038134 
>   sharelib/hive2/pom.xml ce967c5575eb639335332dc6de7ce3f3c66a55ca 
>   sharelib/oozie/pom.xml dd95b4537c00b516b124341bdacf4fbd1fd351f9 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/CallbackNotifier.java PRE-CREATION 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java PRE-CREATION 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java f2cba13e89e8ddc16b59a04947309147ea6b34b7 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/PrepareActionsDriver.java 21ae456b768fb1cb5334a9e23e6c6082240f8dff 
>   sharelib/pig/pom.xml 562c5309cde72c7430fc03f4d342b6d30b5b4575 
>   sharelib/spark/pom.xml 46c6375314ac314fc6e702695303bbc8eda4f87b 
>   sharelib/sqoop/pom.xml d875c93ae599b990cdcbefdf2aa306eae41a9f0c 
>   sharelib/streaming/pom.xml fd795182f11f0f884177a56592e99aaea01d195d 
> 
> Diff: https://reviews.apache.org/r/49778/diff/
> 
> 
> Testing
> -------
> 
> - Tried a bunch of stuff against a Hadoop 2.4.0 cluster, including an "action" that would have different kinds of Exceptions, System.exit, etc.
> - Some new and old unit tests.
> 
> 
> Thanks,
> 
> Robert Kanter
> 
>

Re: Review Request 49778: OOZIE-2590: OYA: Create basic Oozie Launcher Application Master

Posted by Robert Kanter <rk...@cloudera.com>.


> On July 13, 2016, 9:01 a.m., Attila Sasvari wrote:
> > core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java, line 371
> > <https://reviews.apache.org/r/49778/diff/2/?file=1443102#file1443102line371>
> >
> >     ActionExecutorException, HadoopAccessorException, IOException can only be thrown

Looks like it's only ActionExecutorException.


> On July 13, 2016, 9:01 a.m., Attila Sasvari wrote:
> > sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/CallbackNotifier.java, line 116
> > <https://reviews.apache.org/r/49778/diff/2/?file=1443116#file1443116line116>
> >
> >     network resources are not freed. connection should be closed after use

This gets executed in the AM, which shoudl exit shortly after the callback is sent out, so exiting the JVM would close it anyway, but it's safer to explicetly close it, so I'll do that.


> On July 13, 2016, 9:01 a.m., Attila Sasvari wrote:
> > sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java, line 141
> > <https://reviews.apache.org/r/49778/diff/2/?file=1443117#file1443117line141>
> >
> >     Multiple methods could extracted from this to make the method more readable.

As I said on earlier comments, I'm not sure that it would be.  Most of the code in there is error handling, and I think that splitting that between different methods will make it even more complicated and harder to read.


> On July 13, 2016, 9:01 a.m., Attila Sasvari wrote:
> > sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java, line 373
> > <https://reviews.apache.org/r/49778/diff/2/?file=1443117#file1443117line373>
> >
> >     - resource leak in case of an IOException 
> >     - is there any reason for not using BufferedReader?

No real reason.  I copied this from LauncherMapper, and was written with older stuff (like StringBuffer instead of StringBuilder).  I'll fix this and the resource leak.


- Robert


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49778/#review142033
-----------------------------------------------------------


On July 13, 2016, 5:42 a.m., Robert Kanter wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/49778/
> -----------------------------------------------------------
> 
> (Updated July 13, 2016, 5:42 a.m.)
> 
> 
> Review request for oozie.
> 
> 
> Bugs: OOZIE-2590
>     https://issues.apache.org/jira/browse/OOZIE-2590
> 
> 
> Repository: oozie-git
> 
> 
> Description
> -------
> 
> See OOZIE-2590
> 
> 
> Diffs
> -----
> 
>   core/pom.xml 6584af8123925e00f7b29d6b33201f4fc7dca231 
>   core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 96726daa800861c15e4a0d97514fbd45e31fa000 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2ActionExecutor.java b5b1bf90891fde1b13ca058e0b5f538b1b683128 
>   core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java c74e9e61c01a4b6430bbd5491c8cbaf0c4e90198 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 99e33445d68ac6a99a08e02ecc182f35103a53ce 
>   core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 69e1044e531a67c2d522f642376f4ad030423e53 
>   core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java 252f461267818c0da91c7ad59380b2f18a074f9f 
>   core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 6813a37d0ffc66da41636fe01e9cc7a37d6f6e79 
>   core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java 794e8258472df5fe18299eb3c62471cf9fecc186 
>   core/src/main/java/org/apache/oozie/util/ClasspathUtils.java PRE-CREATION 
>   core/src/main/resources/oozie-default.xml 6c2f7d82e933614ea6b434f9b035dca3751d1c7f 
>   core/src/test/java/org/apache/oozie/QueryServlet.java PRE-CREATION 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestCallbackNotifier.java PRE-CREATION 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 879bfebf58acc12ce787b151a67eaaa0716ff2bb 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestLauncherAM.java PRE-CREATION 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestPrepareActionsDriver.java df9e939cb236e8b65e7b3aa9ab683c33f5fa02b7 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestShellActionExecutor.java 6a962a1030a16e61a2af27a6316c4d71b17ea5c9 
>   core/src/test/java/org/apache/oozie/command/wf/HangServlet.java 3344cf97762c77f5660d94341e72df4243f76eb2 
>   core/src/test/java/org/apache/oozie/service/TestConfigurationService.java 2153bf122ddbdf81acd90c4922fa8d0f619d2d41 
>   core/src/test/java/org/apache/oozie/service/TestHadoopAccessorService.java 96faa4878111baafb81fdf0f3b38becba0c6f15e 
>   core/src/test/java/org/apache/oozie/test/XTestCase.java e3603692b2ae2ebb741adfa3a6ab5a37ea84eb96 
>   core/src/test/java/org/apache/oozie/util/TestClasspathUtils.java PRE-CREATION 
>   sharelib/distcp/pom.xml c8cc47cddc6d2922766e2d68ddddee96f5f3cfec 
>   sharelib/hcatalog/pom.xml 2b0c5042a6f295101f8c5fae0b08a4239b5e97d3 
>   sharelib/hive/pom.xml d10d7b85b70cbeb1906c83484739f34c09038134 
>   sharelib/hive2/pom.xml ce967c5575eb639335332dc6de7ce3f3c66a55ca 
>   sharelib/oozie/pom.xml dd95b4537c00b516b124341bdacf4fbd1fd351f9 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/CallbackNotifier.java PRE-CREATION 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java PRE-CREATION 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java f2cba13e89e8ddc16b59a04947309147ea6b34b7 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/PrepareActionsDriver.java 21ae456b768fb1cb5334a9e23e6c6082240f8dff 
>   sharelib/pig/pom.xml 562c5309cde72c7430fc03f4d342b6d30b5b4575 
>   sharelib/spark/pom.xml 46c6375314ac314fc6e702695303bbc8eda4f87b 
>   sharelib/sqoop/pom.xml d875c93ae599b990cdcbefdf2aa306eae41a9f0c 
>   sharelib/streaming/pom.xml fd795182f11f0f884177a56592e99aaea01d195d 
> 
> Diff: https://reviews.apache.org/r/49778/diff/
> 
> 
> Testing
> -------
> 
> - Tried a bunch of stuff against a Hadoop 2.4.0 cluster, including an "action" that would have different kinds of Exceptions, System.exit, etc.
> - Some new and old unit tests.
> 
> 
> Thanks,
> 
> Robert Kanter
> 
>

Re: Review Request 49778: OOZIE-2590: OYA: Create basic Oozie Launcher Application Master

Posted by Attila Sasvari <as...@cloudera.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49778/#review142033
-----------------------------------------------------------




core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java (line 1219)
<https://reviews.apache.org/r/49778/#comment207510>

    Should not we close fs after copying?



core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java (line 1598)
<https://reviews.apache.org/r/49778/#comment207511>

    Is this flag needed? You could LOG.error() in the first catch.



core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java (line 1603)
<https://reviews.apache.org/r/49778/#comment207512>

    ugi is never used



core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java (line 517)
<https://reviews.apache.org/r/49778/#comment207513>

    typo in javadoc - ugi should be user



core/src/main/java/org/apache/oozie/util/ClasspathUtils.java (line 56)
<https://reviews.apache.org/r/49778/#comment207514>

    You could reduce the amount of code duplication by having a list of the needed classpath entries with something like
    
    ```
    List<String> classpathEntries = Arrays.asList(
                    ApplicationConstants.Environment.PWD.$(),
                    MRJobConfig.JOB_JAR + Path.SEPARATOR + MRJobConfig.JOB_JAR,
                    MRJobConfig.JOB_JAR + Path.SEPARATOR + "classes" + Path.SEPARATOR,
                    MRJobConfig.JOB_JAR + Path.SEPARATOR + "lib" + Path.SEPARATOR + "*",
                    ApplicationConstants.Environment.PWD.$() + Path.SEPARATOR + "*"
            );
    
            for (String c : classpathEntries) {
                MRApps.addToEnvironment(env, ApplicationConstants.Environment.CLASSPATH.name(), c, conf);
            }
    ```



core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java (line 360)
<https://reviews.apache.org/r/49778/#comment207520>

    ActionExecutorException, HadoopAccessorException, IOException can only be thrown



sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/CallbackNotifier.java (line 116)
<https://reviews.apache.org/r/49778/#comment207519>

    network resources are not freed. connection should be closed after use



sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java (line 141)
<https://reviews.apache.org/r/49778/#comment207518>

    Multiple methods could extracted from this to make the method more readable.



sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java (line 291)
<https://reviews.apache.org/r/49778/#comment207517>

    unregisterApplicationMaster can only throw YarnException or IOException 
    
    Without multicatch we can
    ```
    catch (IOException ex) {
         logger.log(ex);
         throw ex;
    catch (SQLException ex) {
         logger.log(ex);
         throw ex;
    }
    ```
    Then the method's signature can be changed so that it  
    throws IOException, YarnException
    
    It would make the method more readable.



sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java (line 373)
<https://reviews.apache.org/r/49778/#comment207516>

    - resource leak in case of an IOException 
    - is there any reason for not using BufferedReader?


- Attila Sasvari


On July 13, 2016, 5:42 a.m., Robert Kanter wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/49778/
> -----------------------------------------------------------
> 
> (Updated July 13, 2016, 5:42 a.m.)
> 
> 
> Review request for oozie.
> 
> 
> Bugs: OOZIE-2590
>     https://issues.apache.org/jira/browse/OOZIE-2590
> 
> 
> Repository: oozie-git
> 
> 
> Description
> -------
> 
> See OOZIE-2590
> 
> 
> Diffs
> -----
> 
>   core/pom.xml 6584af8123925e00f7b29d6b33201f4fc7dca231 
>   core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 96726daa800861c15e4a0d97514fbd45e31fa000 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2ActionExecutor.java b5b1bf90891fde1b13ca058e0b5f538b1b683128 
>   core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java c74e9e61c01a4b6430bbd5491c8cbaf0c4e90198 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 99e33445d68ac6a99a08e02ecc182f35103a53ce 
>   core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 69e1044e531a67c2d522f642376f4ad030423e53 
>   core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java 252f461267818c0da91c7ad59380b2f18a074f9f 
>   core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 6813a37d0ffc66da41636fe01e9cc7a37d6f6e79 
>   core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java 794e8258472df5fe18299eb3c62471cf9fecc186 
>   core/src/main/java/org/apache/oozie/util/ClasspathUtils.java PRE-CREATION 
>   core/src/main/resources/oozie-default.xml 6c2f7d82e933614ea6b434f9b035dca3751d1c7f 
>   core/src/test/java/org/apache/oozie/QueryServlet.java PRE-CREATION 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestCallbackNotifier.java PRE-CREATION 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 879bfebf58acc12ce787b151a67eaaa0716ff2bb 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestLauncherAM.java PRE-CREATION 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestPrepareActionsDriver.java df9e939cb236e8b65e7b3aa9ab683c33f5fa02b7 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestShellActionExecutor.java 6a962a1030a16e61a2af27a6316c4d71b17ea5c9 
>   core/src/test/java/org/apache/oozie/command/wf/HangServlet.java 3344cf97762c77f5660d94341e72df4243f76eb2 
>   core/src/test/java/org/apache/oozie/service/TestConfigurationService.java 2153bf122ddbdf81acd90c4922fa8d0f619d2d41 
>   core/src/test/java/org/apache/oozie/service/TestHadoopAccessorService.java 96faa4878111baafb81fdf0f3b38becba0c6f15e 
>   core/src/test/java/org/apache/oozie/test/XTestCase.java e3603692b2ae2ebb741adfa3a6ab5a37ea84eb96 
>   core/src/test/java/org/apache/oozie/util/TestClasspathUtils.java PRE-CREATION 
>   sharelib/distcp/pom.xml c8cc47cddc6d2922766e2d68ddddee96f5f3cfec 
>   sharelib/hcatalog/pom.xml 2b0c5042a6f295101f8c5fae0b08a4239b5e97d3 
>   sharelib/hive/pom.xml d10d7b85b70cbeb1906c83484739f34c09038134 
>   sharelib/hive2/pom.xml ce967c5575eb639335332dc6de7ce3f3c66a55ca 
>   sharelib/oozie/pom.xml dd95b4537c00b516b124341bdacf4fbd1fd351f9 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/CallbackNotifier.java PRE-CREATION 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java PRE-CREATION 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java f2cba13e89e8ddc16b59a04947309147ea6b34b7 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/PrepareActionsDriver.java 21ae456b768fb1cb5334a9e23e6c6082240f8dff 
>   sharelib/pig/pom.xml 562c5309cde72c7430fc03f4d342b6d30b5b4575 
>   sharelib/spark/pom.xml 46c6375314ac314fc6e702695303bbc8eda4f87b 
>   sharelib/sqoop/pom.xml d875c93ae599b990cdcbefdf2aa306eae41a9f0c 
>   sharelib/streaming/pom.xml fd795182f11f0f884177a56592e99aaea01d195d 
> 
> Diff: https://reviews.apache.org/r/49778/diff/
> 
> 
> Testing
> -------
> 
> - Tried a bunch of stuff against a Hadoop 2.4.0 cluster, including an "action" that would have different kinds of Exceptions, System.exit, etc.
> - Some new and old unit tests.
> 
> 
> Thanks,
> 
> Robert Kanter
> 
>

Re: Review Request 49778: OOZIE-2590: OYA: Create basic Oozie Launcher Application Master

Posted by Robert Kanter <rk...@cloudera.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49778/
-----------------------------------------------------------

(Updated July 13, 2016, 5:42 a.m.)


Review request for oozie.


Changes
-------

The 002 patch addresses a bunch of review feedback


Bugs: OOZIE-2590
    https://issues.apache.org/jira/browse/OOZIE-2590


Repository: oozie-git


Description
-------

See OOZIE-2590


Diffs (updated)
-----

  core/pom.xml 6584af8123925e00f7b29d6b33201f4fc7dca231 
  core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 96726daa800861c15e4a0d97514fbd45e31fa000 
  core/src/main/java/org/apache/oozie/action/hadoop/Hive2ActionExecutor.java b5b1bf90891fde1b13ca058e0b5f538b1b683128 
  core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java c74e9e61c01a4b6430bbd5491c8cbaf0c4e90198 
  core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 99e33445d68ac6a99a08e02ecc182f35103a53ce 
  core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 69e1044e531a67c2d522f642376f4ad030423e53 
  core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java 252f461267818c0da91c7ad59380b2f18a074f9f 
  core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 6813a37d0ffc66da41636fe01e9cc7a37d6f6e79 
  core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java 794e8258472df5fe18299eb3c62471cf9fecc186 
  core/src/main/java/org/apache/oozie/util/ClasspathUtils.java PRE-CREATION 
  core/src/main/resources/oozie-default.xml 6c2f7d82e933614ea6b434f9b035dca3751d1c7f 
  core/src/test/java/org/apache/oozie/QueryServlet.java PRE-CREATION 
  core/src/test/java/org/apache/oozie/action/hadoop/TestCallbackNotifier.java PRE-CREATION 
  core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 879bfebf58acc12ce787b151a67eaaa0716ff2bb 
  core/src/test/java/org/apache/oozie/action/hadoop/TestLauncherAM.java PRE-CREATION 
  core/src/test/java/org/apache/oozie/action/hadoop/TestPrepareActionsDriver.java df9e939cb236e8b65e7b3aa9ab683c33f5fa02b7 
  core/src/test/java/org/apache/oozie/action/hadoop/TestShellActionExecutor.java 6a962a1030a16e61a2af27a6316c4d71b17ea5c9 
  core/src/test/java/org/apache/oozie/command/wf/HangServlet.java 3344cf97762c77f5660d94341e72df4243f76eb2 
  core/src/test/java/org/apache/oozie/service/TestConfigurationService.java 2153bf122ddbdf81acd90c4922fa8d0f619d2d41 
  core/src/test/java/org/apache/oozie/service/TestHadoopAccessorService.java 96faa4878111baafb81fdf0f3b38becba0c6f15e 
  core/src/test/java/org/apache/oozie/test/XTestCase.java e3603692b2ae2ebb741adfa3a6ab5a37ea84eb96 
  core/src/test/java/org/apache/oozie/util/TestClasspathUtils.java PRE-CREATION 
  sharelib/distcp/pom.xml c8cc47cddc6d2922766e2d68ddddee96f5f3cfec 
  sharelib/hcatalog/pom.xml 2b0c5042a6f295101f8c5fae0b08a4239b5e97d3 
  sharelib/hive/pom.xml d10d7b85b70cbeb1906c83484739f34c09038134 
  sharelib/hive2/pom.xml ce967c5575eb639335332dc6de7ce3f3c66a55ca 
  sharelib/oozie/pom.xml dd95b4537c00b516b124341bdacf4fbd1fd351f9 
  sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/CallbackNotifier.java PRE-CREATION 
  sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java PRE-CREATION 
  sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java f2cba13e89e8ddc16b59a04947309147ea6b34b7 
  sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/PrepareActionsDriver.java 21ae456b768fb1cb5334a9e23e6c6082240f8dff 
  sharelib/pig/pom.xml 562c5309cde72c7430fc03f4d342b6d30b5b4575 
  sharelib/spark/pom.xml 46c6375314ac314fc6e702695303bbc8eda4f87b 
  sharelib/sqoop/pom.xml d875c93ae599b990cdcbefdf2aa306eae41a9f0c 
  sharelib/streaming/pom.xml fd795182f11f0f884177a56592e99aaea01d195d 

Diff: https://reviews.apache.org/r/49778/diff/


Testing
-------

- Tried a bunch of stuff against a Hadoop 2.4.0 cluster, including an "action" that would have different kinds of Exceptions, System.exit, etc.
- Some new and old unit tests.


Thanks,

Robert Kanter

Re: Review Request 49778: OOZIE-2590: OYA: Create basic Oozie Launcher Application Master

Posted by Robert Kanter <rk...@cloudera.com>.


> On July 11, 2016, 12:09 p.m., Jaydeep Vishwakarma wrote:
> > core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java, line 537
> > <https://reviews.apache.org/r/49778/diff/1/?file=1439245#file1439245line537>
> >
> >     is this logging temporary for your debugging purpose? Otherwise this alone itself has no useful information during produciton logging.

I think that was there because I copied some example code that had it :)

I'll remove that, you're right that it's not helpful.


- Robert


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49778/#review141669
-----------------------------------------------------------


On July 8, 2016, 12:17 a.m., Robert Kanter wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/49778/
> -----------------------------------------------------------
> 
> (Updated July 8, 2016, 12:17 a.m.)
> 
> 
> Review request for oozie.
> 
> 
> Bugs: OOZIE-2590
>     https://issues.apache.org/jira/browse/OOZIE-2590
> 
> 
> Repository: oozie-git
> 
> 
> Description
> -------
> 
> See OOZIE-2590
> 
> 
> Diffs
> -----
> 
>   core/pom.xml 6584af8123925e00f7b29d6b33201f4fc7dca231 
>   core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 96726daa800861c15e4a0d97514fbd45e31fa000 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2ActionExecutor.java b5b1bf90891fde1b13ca058e0b5f538b1b683128 
>   core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java c74e9e61c01a4b6430bbd5491c8cbaf0c4e90198 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 99e33445d68ac6a99a08e02ecc182f35103a53ce 
>   core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 69e1044e531a67c2d522f642376f4ad030423e53 
>   core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java 252f461267818c0da91c7ad59380b2f18a074f9f 
>   core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 6813a37d0ffc66da41636fe01e9cc7a37d6f6e79 
>   core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java 794e8258472df5fe18299eb3c62471cf9fecc186 
>   core/src/main/java/org/apache/oozie/util/ClasspathUtils.java PRE-CREATION 
>   core/src/main/resources/oozie-default.xml 6c2f7d82e933614ea6b434f9b035dca3751d1c7f 
>   core/src/test/java/org/apache/oozie/QueryServlet.java PRE-CREATION 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestCallbackNotifier.java PRE-CREATION 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 879bfebf58acc12ce787b151a67eaaa0716ff2bb 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestLauncherAM.java PRE-CREATION 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestPrepareActionsDriver.java df9e939cb236e8b65e7b3aa9ab683c33f5fa02b7 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestShellActionExecutor.java 6a962a1030a16e61a2af27a6316c4d71b17ea5c9 
>   core/src/test/java/org/apache/oozie/command/wf/HangServlet.java 3344cf97762c77f5660d94341e72df4243f76eb2 
>   core/src/test/java/org/apache/oozie/service/TestConfigurationService.java 2153bf122ddbdf81acd90c4922fa8d0f619d2d41 
>   core/src/test/java/org/apache/oozie/service/TestHadoopAccessorService.java 96faa4878111baafb81fdf0f3b38becba0c6f15e 
>   core/src/test/java/org/apache/oozie/test/XTestCase.java e3603692b2ae2ebb741adfa3a6ab5a37ea84eb96 
>   core/src/test/java/org/apache/oozie/util/TestClasspathUtils.java PRE-CREATION 
>   sharelib/distcp/pom.xml c8cc47cddc6d2922766e2d68ddddee96f5f3cfec 
>   sharelib/hcatalog/pom.xml 2b0c5042a6f295101f8c5fae0b08a4239b5e97d3 
>   sharelib/hive/pom.xml d10d7b85b70cbeb1906c83484739f34c09038134 
>   sharelib/hive2/pom.xml ce967c5575eb639335332dc6de7ce3f3c66a55ca 
>   sharelib/oozie/pom.xml dd95b4537c00b516b124341bdacf4fbd1fd351f9 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/CallbackNotifier.java PRE-CREATION 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java PRE-CREATION 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java f2cba13e89e8ddc16b59a04947309147ea6b34b7 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/PrepareActionsDriver.java 21ae456b768fb1cb5334a9e23e6c6082240f8dff 
>   sharelib/pig/pom.xml 562c5309cde72c7430fc03f4d342b6d30b5b4575 
>   sharelib/spark/pom.xml 46c6375314ac314fc6e702695303bbc8eda4f87b 
>   sharelib/sqoop/pom.xml d875c93ae599b990cdcbefdf2aa306eae41a9f0c 
>   sharelib/streaming/pom.xml fd795182f11f0f884177a56592e99aaea01d195d 
> 
> Diff: https://reviews.apache.org/r/49778/diff/
> 
> 
> Testing
> -------
> 
> - Tried a bunch of stuff against a Hadoop 2.4.0 cluster, including an "action" that would have different kinds of Exceptions, System.exit, etc.
> - Some new and old unit tests.
> 
> 
> Thanks,
> 
> Robert Kanter
> 
>

Re: Review Request 49778: OOZIE-2590: OYA: Create basic Oozie Launcher Application Master

Posted by Jaydeep Vishwakarma <ja...@gmail.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49778/#review141669
-----------------------------------------------------------




core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java (line 537)
<https://reviews.apache.org/r/49778/#comment207045>

    is this logging temporary for your debugging purpose? Otherwise this alone itself has no useful information during produciton logging.


- Jaydeep Vishwakarma


On July 8, 2016, 12:17 a.m., Robert Kanter wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/49778/
> -----------------------------------------------------------
> 
> (Updated July 8, 2016, 12:17 a.m.)
> 
> 
> Review request for oozie.
> 
> 
> Bugs: OOZIE-2590
>     https://issues.apache.org/jira/browse/OOZIE-2590
> 
> 
> Repository: oozie-git
> 
> 
> Description
> -------
> 
> See OOZIE-2590
> 
> 
> Diffs
> -----
> 
>   core/pom.xml 6584af8123925e00f7b29d6b33201f4fc7dca231 
>   core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 96726daa800861c15e4a0d97514fbd45e31fa000 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2ActionExecutor.java b5b1bf90891fde1b13ca058e0b5f538b1b683128 
>   core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java c74e9e61c01a4b6430bbd5491c8cbaf0c4e90198 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 99e33445d68ac6a99a08e02ecc182f35103a53ce 
>   core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 69e1044e531a67c2d522f642376f4ad030423e53 
>   core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java 252f461267818c0da91c7ad59380b2f18a074f9f 
>   core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 6813a37d0ffc66da41636fe01e9cc7a37d6f6e79 
>   core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java 794e8258472df5fe18299eb3c62471cf9fecc186 
>   core/src/main/java/org/apache/oozie/util/ClasspathUtils.java PRE-CREATION 
>   core/src/main/resources/oozie-default.xml 6c2f7d82e933614ea6b434f9b035dca3751d1c7f 
>   core/src/test/java/org/apache/oozie/QueryServlet.java PRE-CREATION 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestCallbackNotifier.java PRE-CREATION 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 879bfebf58acc12ce787b151a67eaaa0716ff2bb 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestLauncherAM.java PRE-CREATION 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestPrepareActionsDriver.java df9e939cb236e8b65e7b3aa9ab683c33f5fa02b7 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestShellActionExecutor.java 6a962a1030a16e61a2af27a6316c4d71b17ea5c9 
>   core/src/test/java/org/apache/oozie/command/wf/HangServlet.java 3344cf97762c77f5660d94341e72df4243f76eb2 
>   core/src/test/java/org/apache/oozie/service/TestConfigurationService.java 2153bf122ddbdf81acd90c4922fa8d0f619d2d41 
>   core/src/test/java/org/apache/oozie/service/TestHadoopAccessorService.java 96faa4878111baafb81fdf0f3b38becba0c6f15e 
>   core/src/test/java/org/apache/oozie/test/XTestCase.java e3603692b2ae2ebb741adfa3a6ab5a37ea84eb96 
>   core/src/test/java/org/apache/oozie/util/TestClasspathUtils.java PRE-CREATION 
>   sharelib/distcp/pom.xml c8cc47cddc6d2922766e2d68ddddee96f5f3cfec 
>   sharelib/hcatalog/pom.xml 2b0c5042a6f295101f8c5fae0b08a4239b5e97d3 
>   sharelib/hive/pom.xml d10d7b85b70cbeb1906c83484739f34c09038134 
>   sharelib/hive2/pom.xml ce967c5575eb639335332dc6de7ce3f3c66a55ca 
>   sharelib/oozie/pom.xml dd95b4537c00b516b124341bdacf4fbd1fd351f9 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/CallbackNotifier.java PRE-CREATION 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java PRE-CREATION 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java f2cba13e89e8ddc16b59a04947309147ea6b34b7 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/PrepareActionsDriver.java 21ae456b768fb1cb5334a9e23e6c6082240f8dff 
>   sharelib/pig/pom.xml 562c5309cde72c7430fc03f4d342b6d30b5b4575 
>   sharelib/spark/pom.xml 46c6375314ac314fc6e702695303bbc8eda4f87b 
>   sharelib/sqoop/pom.xml d875c93ae599b990cdcbefdf2aa306eae41a9f0c 
>   sharelib/streaming/pom.xml fd795182f11f0f884177a56592e99aaea01d195d 
> 
> Diff: https://reviews.apache.org/r/49778/diff/
> 
> 
> Testing
> -------
> 
> - Tried a bunch of stuff against a Hadoop 2.4.0 cluster, including an "action" that would have different kinds of Exceptions, System.exit, etc.
> - Some new and old unit tests.
> 
> 
> Thanks,
> 
> Robert Kanter
> 
>

Re: Review Request 49778: OOZIE-2590: OYA: Create basic Oozie Launcher Application Master

Posted by Peter Bacsko <pb...@cloudera.com>.


> On j�l. 8, 2016, 1:28 du, Peter Bacsko wrote:
> > sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java, line 342
> > <https://reviews.apache.org/r/49778/diff/1/?file=1439265#file1439265line342>
> >
> >     Shorter:
> >     
> >     IOUtils.sizeOf()
> >     IOUtils.readFileToString()

Fix: not IOUtils, but FileUtils (commons-io)


- Peter


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49778/#review141263
-----------------------------------------------------------


On j�l. 8, 2016, 12:17 de, Robert Kanter wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/49778/
> -----------------------------------------------------------
> 
> (Updated j�l. 8, 2016, 12:17 de)
> 
> 
> Review request for oozie.
> 
> 
> Bugs: OOZIE-2590
>     https://issues.apache.org/jira/browse/OOZIE-2590
> 
> 
> Repository: oozie-git
> 
> 
> Description
> -------
> 
> See OOZIE-2590
> 
> 
> Diffs
> -----
> 
>   core/pom.xml 6584af8123925e00f7b29d6b33201f4fc7dca231 
>   core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 96726daa800861c15e4a0d97514fbd45e31fa000 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2ActionExecutor.java b5b1bf90891fde1b13ca058e0b5f538b1b683128 
>   core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java c74e9e61c01a4b6430bbd5491c8cbaf0c4e90198 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 99e33445d68ac6a99a08e02ecc182f35103a53ce 
>   core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 69e1044e531a67c2d522f642376f4ad030423e53 
>   core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java 252f461267818c0da91c7ad59380b2f18a074f9f 
>   core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 6813a37d0ffc66da41636fe01e9cc7a37d6f6e79 
>   core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java 794e8258472df5fe18299eb3c62471cf9fecc186 
>   core/src/main/java/org/apache/oozie/util/ClasspathUtils.java PRE-CREATION 
>   core/src/main/resources/oozie-default.xml 6c2f7d82e933614ea6b434f9b035dca3751d1c7f 
>   core/src/test/java/org/apache/oozie/QueryServlet.java PRE-CREATION 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestCallbackNotifier.java PRE-CREATION 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 879bfebf58acc12ce787b151a67eaaa0716ff2bb 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestLauncherAM.java PRE-CREATION 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestPrepareActionsDriver.java df9e939cb236e8b65e7b3aa9ab683c33f5fa02b7 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestShellActionExecutor.java 6a962a1030a16e61a2af27a6316c4d71b17ea5c9 
>   core/src/test/java/org/apache/oozie/command/wf/HangServlet.java 3344cf97762c77f5660d94341e72df4243f76eb2 
>   core/src/test/java/org/apache/oozie/service/TestConfigurationService.java 2153bf122ddbdf81acd90c4922fa8d0f619d2d41 
>   core/src/test/java/org/apache/oozie/service/TestHadoopAccessorService.java 96faa4878111baafb81fdf0f3b38becba0c6f15e 
>   core/src/test/java/org/apache/oozie/test/XTestCase.java e3603692b2ae2ebb741adfa3a6ab5a37ea84eb96 
>   core/src/test/java/org/apache/oozie/util/TestClasspathUtils.java PRE-CREATION 
>   sharelib/distcp/pom.xml c8cc47cddc6d2922766e2d68ddddee96f5f3cfec 
>   sharelib/hcatalog/pom.xml 2b0c5042a6f295101f8c5fae0b08a4239b5e97d3 
>   sharelib/hive/pom.xml d10d7b85b70cbeb1906c83484739f34c09038134 
>   sharelib/hive2/pom.xml ce967c5575eb639335332dc6de7ce3f3c66a55ca 
>   sharelib/oozie/pom.xml dd95b4537c00b516b124341bdacf4fbd1fd351f9 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/CallbackNotifier.java PRE-CREATION 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java PRE-CREATION 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java f2cba13e89e8ddc16b59a04947309147ea6b34b7 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/PrepareActionsDriver.java 21ae456b768fb1cb5334a9e23e6c6082240f8dff 
>   sharelib/pig/pom.xml 562c5309cde72c7430fc03f4d342b6d30b5b4575 
>   sharelib/spark/pom.xml 46c6375314ac314fc6e702695303bbc8eda4f87b 
>   sharelib/sqoop/pom.xml d875c93ae599b990cdcbefdf2aa306eae41a9f0c 
>   sharelib/streaming/pom.xml fd795182f11f0f884177a56592e99aaea01d195d 
> 
> Diff: https://reviews.apache.org/r/49778/diff/
> 
> 
> Testing
> -------
> 
> - Tried a bunch of stuff against a Hadoop 2.4.0 cluster, including an "action" that would have different kinds of Exceptions, System.exit, etc.
> - Some new and old unit tests.
> 
> 
> Thanks,
> 
> Robert Kanter
> 
>

Re: Review Request 49778: OOZIE-2590: OYA: Create basic Oozie Launcher Application Master

Posted by Peter Bacsko <pb...@cloudera.com>.


> On j�l. 8, 2016, 1:28 du, Peter Bacsko wrote:
> > core/src/test/java/org/apache/oozie/QueryServlet.java, line 31
> > <https://reviews.apache.org/r/49778/diff/1/?file=1439248#file1439248line31>
> >
> >     I suppose this class is just temporary, right?
> 
> Robert Kanter wrote:
>     No, it's used for some unit tests in TestCallbackNotifier.  We have a number of misc tiny servlets doing various things that are used for tests.  It probably would be a good idea to better organize them (they're currently all over the place), but that would be another JIRA.

Just said because I wasn't sure about the purpose of this servlet (seemed like a temporary hack). I'd add a comment here to just to clarify this.


> On j�l. 8, 2016, 1:28 du, Peter Bacsko wrote:
> > core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java, line 380
> > <https://reviews.apache.org/r/49778/diff/1/?file=1439250#file1439250line380>
> >
> >     Why array?
> 
> Robert Kanter wrote:
>     This is a trick to get around the fact that yarnClient needs to be final (because of the inner class), but I'm assigning it in the try block, and I need it to start out as null for the finally check to close it.  It's an unfortunate trick that's sometimes needed for this sort of situation.  Try removing the array stuff and see what happens :)

I see, in this case it's indeed necessary. However it still looks confusing. IMO it's worth an extra comment on this and perhaps using MutableObject from commons-lang would look nicer (just an idea).


> On j�l. 8, 2016, 1:28 du, Peter Bacsko wrote:
> > sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java, line 286
> > <https://reviews.apache.org/r/49778/diff/1/?file=1439265#file1439265line286>
> >
> >     StringUtils.isEmpty ?
> 
> Robert Kanter wrote:
>     Given that this is the AM, I don't want to add the entirety of Commons just for this method.  But I'll change the equals empty string if statement to check the length instead (which is what isEmpty does).

Hm, didn't see that it's sharelib, OK.


> On j�l. 8, 2016, 1:28 du, Peter Bacsko wrote:
> > sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java, line 342
> > <https://reviews.apache.org/r/49778/diff/1/?file=1439265#file1439265line342>
> >
> >     Shorter:
> >     
> >     IOUtils.sizeOf()
> >     IOUtils.readFileToString()
> 
> Peter Bacsko wrote:
>     Fix: not IOUtils, but FileUtils (commons-io)
> 
> Robert Kanter wrote:
>     We'd have to add commons-io, which might be worth it for a more complicated function like this.  However, we enforce a limit on the data size, which we can't do with FileUtils.  I think our current code here is fine, it's been around since 2013 in LauncherMapper.  Though I will replace StringBuffer with StringBuilder.

Mmm, sharelib again :) Then forget about it.


> On j�l. 8, 2016, 1:28 du, Peter Bacsko wrote:
> > core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java, line 1614
> > <https://reviews.apache.org/r/49778/diff/1/?file=1439241#file1439241line1614>
> >
> >     This method is too long
> 
> Robert Kanter wrote:
>     It's actually shorter than the MR version of this method.  And I think splitting this out might make it much harder to read.

Maybe I was just fooled by the amount of diff. Let's drop this for now and maybe address it later.


> On j�l. 8, 2016, 1:28 du, Peter Bacsko wrote:
> > core/src/main/java/org/apache/oozie/util/ClasspathUtils.java, line 49
> > <https://reviews.apache.org/r/49778/diff/1/?file=1439246#file1439246line49>
> >
> >     I would pass this as a parameter to avoid storing state in the class.
> 
> Robert Kanter wrote:
>     If we do that, then JavaActionExecutor has to know if it's in a minicluster or not.  The way it is now, XTestCase sets this to true for tests, and no other code has to be aware of it.  I don't like putting test-related code into production code, but this is needed here, and it's not too invasive.
>     
>     By the way, the equivalent code in Hadoop does this via a lookup in the Configuration, which is more expensive than the simple boolean I used.

OK, sounds reasonable.


- Peter


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49778/#review141263
-----------------------------------------------------------


On j�l. 13, 2016, 5:42 de, Robert Kanter wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/49778/
> -----------------------------------------------------------
> 
> (Updated j�l. 13, 2016, 5:42 de)
> 
> 
> Review request for oozie.
> 
> 
> Bugs: OOZIE-2590
>     https://issues.apache.org/jira/browse/OOZIE-2590
> 
> 
> Repository: oozie-git
> 
> 
> Description
> -------
> 
> See OOZIE-2590
> 
> 
> Diffs
> -----
> 
>   core/pom.xml 6584af8123925e00f7b29d6b33201f4fc7dca231 
>   core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 96726daa800861c15e4a0d97514fbd45e31fa000 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2ActionExecutor.java b5b1bf90891fde1b13ca058e0b5f538b1b683128 
>   core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java c74e9e61c01a4b6430bbd5491c8cbaf0c4e90198 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 99e33445d68ac6a99a08e02ecc182f35103a53ce 
>   core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 69e1044e531a67c2d522f642376f4ad030423e53 
>   core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java 252f461267818c0da91c7ad59380b2f18a074f9f 
>   core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 6813a37d0ffc66da41636fe01e9cc7a37d6f6e79 
>   core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java 794e8258472df5fe18299eb3c62471cf9fecc186 
>   core/src/main/java/org/apache/oozie/util/ClasspathUtils.java PRE-CREATION 
>   core/src/main/resources/oozie-default.xml 6c2f7d82e933614ea6b434f9b035dca3751d1c7f 
>   core/src/test/java/org/apache/oozie/QueryServlet.java PRE-CREATION 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestCallbackNotifier.java PRE-CREATION 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 879bfebf58acc12ce787b151a67eaaa0716ff2bb 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestLauncherAM.java PRE-CREATION 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestPrepareActionsDriver.java df9e939cb236e8b65e7b3aa9ab683c33f5fa02b7 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestShellActionExecutor.java 6a962a1030a16e61a2af27a6316c4d71b17ea5c9 
>   core/src/test/java/org/apache/oozie/command/wf/HangServlet.java 3344cf97762c77f5660d94341e72df4243f76eb2 
>   core/src/test/java/org/apache/oozie/service/TestConfigurationService.java 2153bf122ddbdf81acd90c4922fa8d0f619d2d41 
>   core/src/test/java/org/apache/oozie/service/TestHadoopAccessorService.java 96faa4878111baafb81fdf0f3b38becba0c6f15e 
>   core/src/test/java/org/apache/oozie/test/XTestCase.java e3603692b2ae2ebb741adfa3a6ab5a37ea84eb96 
>   core/src/test/java/org/apache/oozie/util/TestClasspathUtils.java PRE-CREATION 
>   sharelib/distcp/pom.xml c8cc47cddc6d2922766e2d68ddddee96f5f3cfec 
>   sharelib/hcatalog/pom.xml 2b0c5042a6f295101f8c5fae0b08a4239b5e97d3 
>   sharelib/hive/pom.xml d10d7b85b70cbeb1906c83484739f34c09038134 
>   sharelib/hive2/pom.xml ce967c5575eb639335332dc6de7ce3f3c66a55ca 
>   sharelib/oozie/pom.xml dd95b4537c00b516b124341bdacf4fbd1fd351f9 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/CallbackNotifier.java PRE-CREATION 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java PRE-CREATION 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java f2cba13e89e8ddc16b59a04947309147ea6b34b7 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/PrepareActionsDriver.java 21ae456b768fb1cb5334a9e23e6c6082240f8dff 
>   sharelib/pig/pom.xml 562c5309cde72c7430fc03f4d342b6d30b5b4575 
>   sharelib/spark/pom.xml 46c6375314ac314fc6e702695303bbc8eda4f87b 
>   sharelib/sqoop/pom.xml d875c93ae599b990cdcbefdf2aa306eae41a9f0c 
>   sharelib/streaming/pom.xml fd795182f11f0f884177a56592e99aaea01d195d 
> 
> Diff: https://reviews.apache.org/r/49778/diff/
> 
> 
> Testing
> -------
> 
> - Tried a bunch of stuff against a Hadoop 2.4.0 cluster, including an "action" that would have different kinds of Exceptions, System.exit, etc.
> - Some new and old unit tests.
> 
> 
> Thanks,
> 
> Robert Kanter
> 
>

Re: Review Request 49778: OOZIE-2590: OYA: Create basic Oozie Launcher Application Master

Posted by Robert Kanter <rk...@cloudera.com>.


> On July 8, 2016, 1:28 p.m., Peter Bacsko wrote:
> > core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java, line 380
> > <https://reviews.apache.org/r/49778/diff/1/?file=1439250#file1439250line380>
> >
> >     Why array?
> 
> Robert Kanter wrote:
>     This is a trick to get around the fact that yarnClient needs to be final (because of the inner class), but I'm assigning it in the try block, and I need it to start out as null for the finally check to close it.  It's an unfortunate trick that's sometimes needed for this sort of situation.  Try removing the array stuff and see what happens :)
> 
> Peter Bacsko wrote:
>     I see, in this case it's indeed necessary. However it still looks confusing. IMO it's worth an extra comment on this and perhaps using MutableObject from commons-lang would look nicer (just an idea).
> 
> Robert Kanter wrote:
>     MutableObject looks like a nice way around this, except that it doesn't support Generics, so you have to cast, which is also ugly.  I'll add a comment.
> 
> Peter Bacsko wrote:
>     I was actually looking at MutableObject from commons-lang3. That class supports generic. I think it's available on the classpath.

Oh, you're right.  I'll use that.  It'll be in the next patch update.


- Robert


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49778/#review141263
-----------------------------------------------------------


On July 13, 2016, 11:44 p.m., Robert Kanter wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/49778/
> -----------------------------------------------------------
> 
> (Updated July 13, 2016, 11:44 p.m.)
> 
> 
> Review request for oozie.
> 
> 
> Bugs: OOZIE-2590
>     https://issues.apache.org/jira/browse/OOZIE-2590
> 
> 
> Repository: oozie-git
> 
> 
> Description
> -------
> 
> See OOZIE-2590
> 
> 
> Diffs
> -----
> 
>   core/pom.xml 6584af8123925e00f7b29d6b33201f4fc7dca231 
>   core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 96726daa800861c15e4a0d97514fbd45e31fa000 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2ActionExecutor.java b5b1bf90891fde1b13ca058e0b5f538b1b683128 
>   core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java c74e9e61c01a4b6430bbd5491c8cbaf0c4e90198 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 99e33445d68ac6a99a08e02ecc182f35103a53ce 
>   core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 69e1044e531a67c2d522f642376f4ad030423e53 
>   core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java 252f461267818c0da91c7ad59380b2f18a074f9f 
>   core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 6813a37d0ffc66da41636fe01e9cc7a37d6f6e79 
>   core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java 794e8258472df5fe18299eb3c62471cf9fecc186 
>   core/src/main/java/org/apache/oozie/util/ClasspathUtils.java PRE-CREATION 
>   core/src/main/resources/oozie-default.xml 6c2f7d82e933614ea6b434f9b035dca3751d1c7f 
>   core/src/test/java/org/apache/oozie/QueryServlet.java PRE-CREATION 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 879bfebf58acc12ce787b151a67eaaa0716ff2bb 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestLauncherAM.java PRE-CREATION 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestLauncherAMCallbackNotifier.java PRE-CREATION 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestPrepareActionsDriver.java df9e939cb236e8b65e7b3aa9ab683c33f5fa02b7 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestShellActionExecutor.java 6a962a1030a16e61a2af27a6316c4d71b17ea5c9 
>   core/src/test/java/org/apache/oozie/command/wf/HangServlet.java 3344cf97762c77f5660d94341e72df4243f76eb2 
>   core/src/test/java/org/apache/oozie/service/TestConfigurationService.java 2153bf122ddbdf81acd90c4922fa8d0f619d2d41 
>   core/src/test/java/org/apache/oozie/service/TestHadoopAccessorService.java 96faa4878111baafb81fdf0f3b38becba0c6f15e 
>   core/src/test/java/org/apache/oozie/test/XTestCase.java e3603692b2ae2ebb741adfa3a6ab5a37ea84eb96 
>   core/src/test/java/org/apache/oozie/util/TestClasspathUtils.java PRE-CREATION 
>   sharelib/distcp/pom.xml c8cc47cddc6d2922766e2d68ddddee96f5f3cfec 
>   sharelib/hcatalog/pom.xml 2b0c5042a6f295101f8c5fae0b08a4239b5e97d3 
>   sharelib/hive/pom.xml d10d7b85b70cbeb1906c83484739f34c09038134 
>   sharelib/hive2/pom.xml ce967c5575eb639335332dc6de7ce3f3c66a55ca 
>   sharelib/oozie/pom.xml dd95b4537c00b516b124341bdacf4fbd1fd351f9 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java PRE-CREATION 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAMCallbackNotifier.java PRE-CREATION 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java f2cba13e89e8ddc16b59a04947309147ea6b34b7 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/PrepareActionsDriver.java 21ae456b768fb1cb5334a9e23e6c6082240f8dff 
>   sharelib/pig/pom.xml 562c5309cde72c7430fc03f4d342b6d30b5b4575 
>   sharelib/spark/pom.xml 46c6375314ac314fc6e702695303bbc8eda4f87b 
>   sharelib/sqoop/pom.xml d875c93ae599b990cdcbefdf2aa306eae41a9f0c 
>   sharelib/streaming/pom.xml fd795182f11f0f884177a56592e99aaea01d195d 
> 
> Diff: https://reviews.apache.org/r/49778/diff/
> 
> 
> Testing
> -------
> 
> - Tried a bunch of stuff against a Hadoop 2.4.0 cluster, including an "action" that would have different kinds of Exceptions, System.exit, etc.
> - Some new and old unit tests.
> 
> 
> Thanks,
> 
> Robert Kanter
> 
>

Re: Review Request 49778: OOZIE-2590: OYA: Create basic Oozie Launcher Application Master

Posted by Robert Kanter <rk...@cloudera.com>.


> On July 8, 2016, 1:28 p.m., Peter Bacsko wrote:
> > core/src/test/java/org/apache/oozie/QueryServlet.java, line 31
> > <https://reviews.apache.org/r/49778/diff/1/?file=1439248#file1439248line31>
> >
> >     I suppose this class is just temporary, right?
> 
> Robert Kanter wrote:
>     No, it's used for some unit tests in TestCallbackNotifier.  We have a number of misc tiny servlets doing various things that are used for tests.  It probably would be a good idea to better organize them (they're currently all over the place), but that would be another JIRA.
> 
> Peter Bacsko wrote:
>     Just said because I wasn't sure about the purpose of this servlet (seemed like a temporary hack). I'd add a comment here to just to clarify this.

It's in the test directory (core/src/test/...).  I've created OOZIE-2607 to reorganize all of the test servlets.


> On July 8, 2016, 1:28 p.m., Peter Bacsko wrote:
> > core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java, line 380
> > <https://reviews.apache.org/r/49778/diff/1/?file=1439250#file1439250line380>
> >
> >     Why array?
> 
> Robert Kanter wrote:
>     This is a trick to get around the fact that yarnClient needs to be final (because of the inner class), but I'm assigning it in the try block, and I need it to start out as null for the finally check to close it.  It's an unfortunate trick that's sometimes needed for this sort of situation.  Try removing the array stuff and see what happens :)
> 
> Peter Bacsko wrote:
>     I see, in this case it's indeed necessary. However it still looks confusing. IMO it's worth an extra comment on this and perhaps using MutableObject from commons-lang would look nicer (just an idea).

MutableObject looks like a nice way around this, except that it doesn't support Generics, so you have to cast, which is also ugly.  I'll add a comment.


> On July 8, 2016, 1:28 p.m., Peter Bacsko wrote:
> > sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java, line 121
> > <https://reviews.apache.org/r/49778/diff/1/?file=1439265#file1439265line121>
> >
> >     This method is extremely long, certain parts should be factored out to private methods.
> >     
> >     Especially because it has nested try-finally stuff inside.
> 
> Robert Kanter wrote:
>     I did simplify (and improve a few things) over the MapperLauncher version, but I can do some more.

I looked at this again.  I don't think there's a way to really split things out anymore than they already are.  I know it's a bit complicated, but I think trying to split this out any further than I already have will make it even more complicated and harder to read.


- Robert


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49778/#review141263
-----------------------------------------------------------


On July 13, 2016, 5:42 a.m., Robert Kanter wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/49778/
> -----------------------------------------------------------
> 
> (Updated July 13, 2016, 5:42 a.m.)
> 
> 
> Review request for oozie.
> 
> 
> Bugs: OOZIE-2590
>     https://issues.apache.org/jira/browse/OOZIE-2590
> 
> 
> Repository: oozie-git
> 
> 
> Description
> -------
> 
> See OOZIE-2590
> 
> 
> Diffs
> -----
> 
>   core/pom.xml 6584af8123925e00f7b29d6b33201f4fc7dca231 
>   core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 96726daa800861c15e4a0d97514fbd45e31fa000 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2ActionExecutor.java b5b1bf90891fde1b13ca058e0b5f538b1b683128 
>   core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java c74e9e61c01a4b6430bbd5491c8cbaf0c4e90198 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 99e33445d68ac6a99a08e02ecc182f35103a53ce 
>   core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 69e1044e531a67c2d522f642376f4ad030423e53 
>   core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java 252f461267818c0da91c7ad59380b2f18a074f9f 
>   core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 6813a37d0ffc66da41636fe01e9cc7a37d6f6e79 
>   core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java 794e8258472df5fe18299eb3c62471cf9fecc186 
>   core/src/main/java/org/apache/oozie/util/ClasspathUtils.java PRE-CREATION 
>   core/src/main/resources/oozie-default.xml 6c2f7d82e933614ea6b434f9b035dca3751d1c7f 
>   core/src/test/java/org/apache/oozie/QueryServlet.java PRE-CREATION 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestCallbackNotifier.java PRE-CREATION 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 879bfebf58acc12ce787b151a67eaaa0716ff2bb 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestLauncherAM.java PRE-CREATION 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestPrepareActionsDriver.java df9e939cb236e8b65e7b3aa9ab683c33f5fa02b7 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestShellActionExecutor.java 6a962a1030a16e61a2af27a6316c4d71b17ea5c9 
>   core/src/test/java/org/apache/oozie/command/wf/HangServlet.java 3344cf97762c77f5660d94341e72df4243f76eb2 
>   core/src/test/java/org/apache/oozie/service/TestConfigurationService.java 2153bf122ddbdf81acd90c4922fa8d0f619d2d41 
>   core/src/test/java/org/apache/oozie/service/TestHadoopAccessorService.java 96faa4878111baafb81fdf0f3b38becba0c6f15e 
>   core/src/test/java/org/apache/oozie/test/XTestCase.java e3603692b2ae2ebb741adfa3a6ab5a37ea84eb96 
>   core/src/test/java/org/apache/oozie/util/TestClasspathUtils.java PRE-CREATION 
>   sharelib/distcp/pom.xml c8cc47cddc6d2922766e2d68ddddee96f5f3cfec 
>   sharelib/hcatalog/pom.xml 2b0c5042a6f295101f8c5fae0b08a4239b5e97d3 
>   sharelib/hive/pom.xml d10d7b85b70cbeb1906c83484739f34c09038134 
>   sharelib/hive2/pom.xml ce967c5575eb639335332dc6de7ce3f3c66a55ca 
>   sharelib/oozie/pom.xml dd95b4537c00b516b124341bdacf4fbd1fd351f9 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/CallbackNotifier.java PRE-CREATION 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java PRE-CREATION 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java f2cba13e89e8ddc16b59a04947309147ea6b34b7 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/PrepareActionsDriver.java 21ae456b768fb1cb5334a9e23e6c6082240f8dff 
>   sharelib/pig/pom.xml 562c5309cde72c7430fc03f4d342b6d30b5b4575 
>   sharelib/spark/pom.xml 46c6375314ac314fc6e702695303bbc8eda4f87b 
>   sharelib/sqoop/pom.xml d875c93ae599b990cdcbefdf2aa306eae41a9f0c 
>   sharelib/streaming/pom.xml fd795182f11f0f884177a56592e99aaea01d195d 
> 
> Diff: https://reviews.apache.org/r/49778/diff/
> 
> 
> Testing
> -------
> 
> - Tried a bunch of stuff against a Hadoop 2.4.0 cluster, including an "action" that would have different kinds of Exceptions, System.exit, etc.
> - Some new and old unit tests.
> 
> 
> Thanks,
> 
> Robert Kanter
> 
>

Re: Review Request 49778: OOZIE-2590: OYA: Create basic Oozie Launcher Application Master

Posted by Peter Bacsko <pb...@cloudera.com>.


> On j�l. 8, 2016, 1:28 du, Peter Bacsko wrote:
> > core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java, line 380
> > <https://reviews.apache.org/r/49778/diff/1/?file=1439250#file1439250line380>
> >
> >     Why array?
> 
> Robert Kanter wrote:
>     This is a trick to get around the fact that yarnClient needs to be final (because of the inner class), but I'm assigning it in the try block, and I need it to start out as null for the finally check to close it.  It's an unfortunate trick that's sometimes needed for this sort of situation.  Try removing the array stuff and see what happens :)
> 
> Peter Bacsko wrote:
>     I see, in this case it's indeed necessary. However it still looks confusing. IMO it's worth an extra comment on this and perhaps using MutableObject from commons-lang would look nicer (just an idea).
> 
> Robert Kanter wrote:
>     MutableObject looks like a nice way around this, except that it doesn't support Generics, so you have to cast, which is also ugly.  I'll add a comment.

I was actually looking at MutableObject from commons-lang3. That class supports generic. I think it's available on the classpath.


- Peter


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49778/#review141263
-----------------------------------------------------------


On j�l. 13, 2016, 11:44 du, Robert Kanter wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/49778/
> -----------------------------------------------------------
> 
> (Updated j�l. 13, 2016, 11:44 du)
> 
> 
> Review request for oozie.
> 
> 
> Bugs: OOZIE-2590
>     https://issues.apache.org/jira/browse/OOZIE-2590
> 
> 
> Repository: oozie-git
> 
> 
> Description
> -------
> 
> See OOZIE-2590
> 
> 
> Diffs
> -----
> 
>   core/pom.xml 6584af8123925e00f7b29d6b33201f4fc7dca231 
>   core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 96726daa800861c15e4a0d97514fbd45e31fa000 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2ActionExecutor.java b5b1bf90891fde1b13ca058e0b5f538b1b683128 
>   core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java c74e9e61c01a4b6430bbd5491c8cbaf0c4e90198 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 99e33445d68ac6a99a08e02ecc182f35103a53ce 
>   core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 69e1044e531a67c2d522f642376f4ad030423e53 
>   core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java 252f461267818c0da91c7ad59380b2f18a074f9f 
>   core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 6813a37d0ffc66da41636fe01e9cc7a37d6f6e79 
>   core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java 794e8258472df5fe18299eb3c62471cf9fecc186 
>   core/src/main/java/org/apache/oozie/util/ClasspathUtils.java PRE-CREATION 
>   core/src/main/resources/oozie-default.xml 6c2f7d82e933614ea6b434f9b035dca3751d1c7f 
>   core/src/test/java/org/apache/oozie/QueryServlet.java PRE-CREATION 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 879bfebf58acc12ce787b151a67eaaa0716ff2bb 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestLauncherAM.java PRE-CREATION 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestLauncherAMCallbackNotifier.java PRE-CREATION 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestPrepareActionsDriver.java df9e939cb236e8b65e7b3aa9ab683c33f5fa02b7 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestShellActionExecutor.java 6a962a1030a16e61a2af27a6316c4d71b17ea5c9 
>   core/src/test/java/org/apache/oozie/command/wf/HangServlet.java 3344cf97762c77f5660d94341e72df4243f76eb2 
>   core/src/test/java/org/apache/oozie/service/TestConfigurationService.java 2153bf122ddbdf81acd90c4922fa8d0f619d2d41 
>   core/src/test/java/org/apache/oozie/service/TestHadoopAccessorService.java 96faa4878111baafb81fdf0f3b38becba0c6f15e 
>   core/src/test/java/org/apache/oozie/test/XTestCase.java e3603692b2ae2ebb741adfa3a6ab5a37ea84eb96 
>   core/src/test/java/org/apache/oozie/util/TestClasspathUtils.java PRE-CREATION 
>   sharelib/distcp/pom.xml c8cc47cddc6d2922766e2d68ddddee96f5f3cfec 
>   sharelib/hcatalog/pom.xml 2b0c5042a6f295101f8c5fae0b08a4239b5e97d3 
>   sharelib/hive/pom.xml d10d7b85b70cbeb1906c83484739f34c09038134 
>   sharelib/hive2/pom.xml ce967c5575eb639335332dc6de7ce3f3c66a55ca 
>   sharelib/oozie/pom.xml dd95b4537c00b516b124341bdacf4fbd1fd351f9 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java PRE-CREATION 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAMCallbackNotifier.java PRE-CREATION 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java f2cba13e89e8ddc16b59a04947309147ea6b34b7 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/PrepareActionsDriver.java 21ae456b768fb1cb5334a9e23e6c6082240f8dff 
>   sharelib/pig/pom.xml 562c5309cde72c7430fc03f4d342b6d30b5b4575 
>   sharelib/spark/pom.xml 46c6375314ac314fc6e702695303bbc8eda4f87b 
>   sharelib/sqoop/pom.xml d875c93ae599b990cdcbefdf2aa306eae41a9f0c 
>   sharelib/streaming/pom.xml fd795182f11f0f884177a56592e99aaea01d195d 
> 
> Diff: https://reviews.apache.org/r/49778/diff/
> 
> 
> Testing
> -------
> 
> - Tried a bunch of stuff against a Hadoop 2.4.0 cluster, including an "action" that would have different kinds of Exceptions, System.exit, etc.
> - Some new and old unit tests.
> 
> 
> Thanks,
> 
> Robert Kanter
> 
>

Re: Review Request 49778: OOZIE-2590: OYA: Create basic Oozie Launcher Application Master

Posted by Robert Kanter <rk...@cloudera.com>.


> On July 8, 2016, 1:28 p.m., Peter Bacsko wrote:
> > core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java, line 1295
> > <https://reviews.apache.org/r/49778/diff/1/?file=1439241#file1439241line1295>
> >
> >     2048 is a magic number - purpose?

2GB.  It's the amount of RAM we're going to allocate to the AM.  There's a "TODO: OYA:" here to choose good defaults and make it configurable.  I'll make the comment more clear what this does.


> On July 8, 2016, 1:28 p.m., Peter Bacsko wrote:
> > core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java, line 1299
> > <https://reviews.apache.org/r/49778/diff/1/?file=1439241#file1439241line1299>
> >
> >     JavaDoc says this call is blocking. So waitUntilAccepted is probably not necessary.

Great!  I'll remove the wait thing.


> On July 8, 2016, 1:28 p.m., Peter Bacsko wrote:
> > core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java, line 1614
> > <https://reviews.apache.org/r/49778/diff/1/?file=1439241#file1439241line1614>
> >
> >     This method is too long

It's actually shorter than the MR version of this method.  And I think splitting this out might make it much harder to read.


> On July 8, 2016, 1:28 p.m., Peter Bacsko wrote:
> > core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java, line 1640
> > <https://reviews.apache.org/r/49778/diff/1/?file=1439241#file1439241line1640>
> >
> >     What if appStatus is null and fallback is false?

Then we go to the else statement because the App is still RUNNING.


> On July 8, 2016, 1:28 p.m., Peter Bacsko wrote:
> > core/src/main/java/org/apache/oozie/util/ClasspathUtils.java, line 49
> > <https://reviews.apache.org/r/49778/diff/1/?file=1439246#file1439246line49>
> >
> >     I would pass this as a parameter to avoid storing state in the class.

If we do that, then JavaActionExecutor has to know if it's in a minicluster or not.  The way it is now, XTestCase sets this to true for tests, and no other code has to be aware of it.  I don't like putting test-related code into production code, but this is needed here, and it's not too invasive.

By the way, the equivalent code in Hadoop does this via a lookup in the Configuration, which is more expensive than the simple boolean I used.


> On July 8, 2016, 1:28 p.m., Peter Bacsko wrote:
> > core/src/test/java/org/apache/oozie/QueryServlet.java, line 31
> > <https://reviews.apache.org/r/49778/diff/1/?file=1439248#file1439248line31>
> >
> >     I suppose this class is just temporary, right?

No, it's used for some unit tests in TestCallbackNotifier.  We have a number of misc tiny servlets doing various things that are used for tests.  It probably would be a good idea to better organize them (they're currently all over the place), but that would be another JIRA.


> On July 8, 2016, 1:28 p.m., Peter Bacsko wrote:
> > core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java, line 380
> > <https://reviews.apache.org/r/49778/diff/1/?file=1439250#file1439250line380>
> >
> >     Why array?

This is a trick to get around the fact that yarnClient needs to be final (because of the inner class), but I'm assigning it in the try block, and I need it to start out as null for the finally check to close it.  It's an unfortunate trick that's sometimes needed for this sort of situation.  Try removing the array stuff and see what happens :)


> On July 8, 2016, 1:28 p.m., Peter Bacsko wrote:
> > sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/CallbackNotifier.java, line 55
> > <https://reviews.apache.org/r/49778/diff/1/?file=1439264#file1439264line55>
> >
> >     I would make these final if they don't change

All but one of them changes.


> On July 8, 2016, 1:28 p.m., Peter Bacsko wrote:
> > sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java, line 121
> > <https://reviews.apache.org/r/49778/diff/1/?file=1439265#file1439265line121>
> >
> >     This method is extremely long, certain parts should be factored out to private methods.
> >     
> >     Especially because it has nested try-finally stuff inside.

I did simplify (and improve a few things) over the MapperLauncher version, but I can do some more.


> On July 8, 2016, 1:28 p.m., Peter Bacsko wrote:
> > sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java, line 286
> > <https://reviews.apache.org/r/49778/diff/1/?file=1439265#file1439265line286>
> >
> >     StringUtils.isEmpty ?

Given that this is the AM, I don't want to add the entirety of Commons just for this method.  But I'll change the equals empty string if statement to check the length instead (which is what isEmpty does).


> On July 8, 2016, 1:28 p.m., Peter Bacsko wrote:
> > sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java, line 342
> > <https://reviews.apache.org/r/49778/diff/1/?file=1439265#file1439265line342>
> >
> >     Shorter:
> >     
> >     IOUtils.sizeOf()
> >     IOUtils.readFileToString()
> 
> Peter Bacsko wrote:
>     Fix: not IOUtils, but FileUtils (commons-io)

We'd have to add commons-io, which might be worth it for a more complicated function like this.  However, we enforce a limit on the data size, which we can't do with FileUtils.  I think our current code here is fine, it's been around since 2013 in LauncherMapper.  Though I will replace StringBuffer with StringBuilder.


- Robert


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49778/#review141263
-----------------------------------------------------------


On July 8, 2016, 12:17 a.m., Robert Kanter wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/49778/
> -----------------------------------------------------------
> 
> (Updated July 8, 2016, 12:17 a.m.)
> 
> 
> Review request for oozie.
> 
> 
> Bugs: OOZIE-2590
>     https://issues.apache.org/jira/browse/OOZIE-2590
> 
> 
> Repository: oozie-git
> 
> 
> Description
> -------
> 
> See OOZIE-2590
> 
> 
> Diffs
> -----
> 
>   core/pom.xml 6584af8123925e00f7b29d6b33201f4fc7dca231 
>   core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 96726daa800861c15e4a0d97514fbd45e31fa000 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2ActionExecutor.java b5b1bf90891fde1b13ca058e0b5f538b1b683128 
>   core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java c74e9e61c01a4b6430bbd5491c8cbaf0c4e90198 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 99e33445d68ac6a99a08e02ecc182f35103a53ce 
>   core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 69e1044e531a67c2d522f642376f4ad030423e53 
>   core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java 252f461267818c0da91c7ad59380b2f18a074f9f 
>   core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 6813a37d0ffc66da41636fe01e9cc7a37d6f6e79 
>   core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java 794e8258472df5fe18299eb3c62471cf9fecc186 
>   core/src/main/java/org/apache/oozie/util/ClasspathUtils.java PRE-CREATION 
>   core/src/main/resources/oozie-default.xml 6c2f7d82e933614ea6b434f9b035dca3751d1c7f 
>   core/src/test/java/org/apache/oozie/QueryServlet.java PRE-CREATION 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestCallbackNotifier.java PRE-CREATION 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 879bfebf58acc12ce787b151a67eaaa0716ff2bb 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestLauncherAM.java PRE-CREATION 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestPrepareActionsDriver.java df9e939cb236e8b65e7b3aa9ab683c33f5fa02b7 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestShellActionExecutor.java 6a962a1030a16e61a2af27a6316c4d71b17ea5c9 
>   core/src/test/java/org/apache/oozie/command/wf/HangServlet.java 3344cf97762c77f5660d94341e72df4243f76eb2 
>   core/src/test/java/org/apache/oozie/service/TestConfigurationService.java 2153bf122ddbdf81acd90c4922fa8d0f619d2d41 
>   core/src/test/java/org/apache/oozie/service/TestHadoopAccessorService.java 96faa4878111baafb81fdf0f3b38becba0c6f15e 
>   core/src/test/java/org/apache/oozie/test/XTestCase.java e3603692b2ae2ebb741adfa3a6ab5a37ea84eb96 
>   core/src/test/java/org/apache/oozie/util/TestClasspathUtils.java PRE-CREATION 
>   sharelib/distcp/pom.xml c8cc47cddc6d2922766e2d68ddddee96f5f3cfec 
>   sharelib/hcatalog/pom.xml 2b0c5042a6f295101f8c5fae0b08a4239b5e97d3 
>   sharelib/hive/pom.xml d10d7b85b70cbeb1906c83484739f34c09038134 
>   sharelib/hive2/pom.xml ce967c5575eb639335332dc6de7ce3f3c66a55ca 
>   sharelib/oozie/pom.xml dd95b4537c00b516b124341bdacf4fbd1fd351f9 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/CallbackNotifier.java PRE-CREATION 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java PRE-CREATION 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java f2cba13e89e8ddc16b59a04947309147ea6b34b7 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/PrepareActionsDriver.java 21ae456b768fb1cb5334a9e23e6c6082240f8dff 
>   sharelib/pig/pom.xml 562c5309cde72c7430fc03f4d342b6d30b5b4575 
>   sharelib/spark/pom.xml 46c6375314ac314fc6e702695303bbc8eda4f87b 
>   sharelib/sqoop/pom.xml d875c93ae599b990cdcbefdf2aa306eae41a9f0c 
>   sharelib/streaming/pom.xml fd795182f11f0f884177a56592e99aaea01d195d 
> 
> Diff: https://reviews.apache.org/r/49778/diff/
> 
> 
> Testing
> -------
> 
> - Tried a bunch of stuff against a Hadoop 2.4.0 cluster, including an "action" that would have different kinds of Exceptions, System.exit, etc.
> - Some new and old unit tests.
> 
> 
> Thanks,
> 
> Robert Kanter
> 
>

Re: Review Request 49778: OOZIE-2590: OYA: Create basic Oozie Launcher Application Master

Posted by Peter Bacsko <pb...@cloudera.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49778/#review141263
-----------------------------------------------------------




core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java (line 954)
<https://reviews.apache.org/r/49778/#comment206699>

    I knopw this is a very early version, but this method is extremely long. Should be split up or extract common parts to a separate class.



core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java (line 1070)
<https://reviews.apache.org/r/49778/#comment206700>

    I think this whole try block should be a separate class sth like YarnApplicationSubmitter.



core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java (line 1153)
<https://reviews.apache.org/r/49778/#comment206743>

    2048 is a magic number - purpose?



core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java (line 1157)
<https://reviews.apache.org/r/49778/#comment206703>

    JavaDoc says this call is blocking. So waitUntilAccepted is probably not necessary.



core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java (line 1469)
<https://reviews.apache.org/r/49778/#comment206744>

    This method is too long



core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java (line 1494)
<https://reviews.apache.org/r/49778/#comment206727>

    What if appStatus is null and fallback is false?



core/src/main/java/org/apache/oozie/util/ClasspathUtils.java (line 49)
<https://reviews.apache.org/r/49778/#comment206737>

    I would pass this as a parameter to avoid storing state in the class.



core/src/test/java/org/apache/oozie/QueryServlet.java (line 31)
<https://reviews.apache.org/r/49778/#comment206745>

    I suppose this class is just temporary, right?



core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java (line 369)
<https://reviews.apache.org/r/49778/#comment206702>

    Why array?



sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/CallbackNotifier.java (line 55)
<https://reviews.apache.org/r/49778/#comment206738>

    I would make these final if they don't change



sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/CallbackNotifier.java (line 142)
<https://reviews.apache.org/r/49778/#comment206742>

    Minor: probably it's worth calling notifyURL() or similar to avoid confusion with Object.notify()



sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java (line 121)
<https://reviews.apache.org/r/49778/#comment206739>

    This method is extremely long, certain parts should be factored out to private methods.
    
    Especially because it has nested try-finally stuff inside.



sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java (line 286)
<https://reviews.apache.org/r/49778/#comment206740>

    StringUtils.isEmpty ?



sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java (line 342)
<https://reviews.apache.org/r/49778/#comment206741>

    Shorter:
    
    IOUtils.sizeOf()
    IOUtils.readFileToString()


- Peter Bacsko


On j�l. 8, 2016, 12:17 de, Robert Kanter wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/49778/
> -----------------------------------------------------------
> 
> (Updated j�l. 8, 2016, 12:17 de)
> 
> 
> Review request for oozie.
> 
> 
> Bugs: OOZIE-2590
>     https://issues.apache.org/jira/browse/OOZIE-2590
> 
> 
> Repository: oozie-git
> 
> 
> Description
> -------
> 
> See OOZIE-2590
> 
> 
> Diffs
> -----
> 
>   core/pom.xml 6584af8123925e00f7b29d6b33201f4fc7dca231 
>   core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 96726daa800861c15e4a0d97514fbd45e31fa000 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2ActionExecutor.java b5b1bf90891fde1b13ca058e0b5f538b1b683128 
>   core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java c74e9e61c01a4b6430bbd5491c8cbaf0c4e90198 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 99e33445d68ac6a99a08e02ecc182f35103a53ce 
>   core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 69e1044e531a67c2d522f642376f4ad030423e53 
>   core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java 252f461267818c0da91c7ad59380b2f18a074f9f 
>   core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 6813a37d0ffc66da41636fe01e9cc7a37d6f6e79 
>   core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java 794e8258472df5fe18299eb3c62471cf9fecc186 
>   core/src/main/java/org/apache/oozie/util/ClasspathUtils.java PRE-CREATION 
>   core/src/main/resources/oozie-default.xml 6c2f7d82e933614ea6b434f9b035dca3751d1c7f 
>   core/src/test/java/org/apache/oozie/QueryServlet.java PRE-CREATION 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestCallbackNotifier.java PRE-CREATION 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 879bfebf58acc12ce787b151a67eaaa0716ff2bb 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestLauncherAM.java PRE-CREATION 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestPrepareActionsDriver.java df9e939cb236e8b65e7b3aa9ab683c33f5fa02b7 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestShellActionExecutor.java 6a962a1030a16e61a2af27a6316c4d71b17ea5c9 
>   core/src/test/java/org/apache/oozie/command/wf/HangServlet.java 3344cf97762c77f5660d94341e72df4243f76eb2 
>   core/src/test/java/org/apache/oozie/service/TestConfigurationService.java 2153bf122ddbdf81acd90c4922fa8d0f619d2d41 
>   core/src/test/java/org/apache/oozie/service/TestHadoopAccessorService.java 96faa4878111baafb81fdf0f3b38becba0c6f15e 
>   core/src/test/java/org/apache/oozie/test/XTestCase.java e3603692b2ae2ebb741adfa3a6ab5a37ea84eb96 
>   core/src/test/java/org/apache/oozie/util/TestClasspathUtils.java PRE-CREATION 
>   sharelib/distcp/pom.xml c8cc47cddc6d2922766e2d68ddddee96f5f3cfec 
>   sharelib/hcatalog/pom.xml 2b0c5042a6f295101f8c5fae0b08a4239b5e97d3 
>   sharelib/hive/pom.xml d10d7b85b70cbeb1906c83484739f34c09038134 
>   sharelib/hive2/pom.xml ce967c5575eb639335332dc6de7ce3f3c66a55ca 
>   sharelib/oozie/pom.xml dd95b4537c00b516b124341bdacf4fbd1fd351f9 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/CallbackNotifier.java PRE-CREATION 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java PRE-CREATION 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java f2cba13e89e8ddc16b59a04947309147ea6b34b7 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/PrepareActionsDriver.java 21ae456b768fb1cb5334a9e23e6c6082240f8dff 
>   sharelib/pig/pom.xml 562c5309cde72c7430fc03f4d342b6d30b5b4575 
>   sharelib/spark/pom.xml 46c6375314ac314fc6e702695303bbc8eda4f87b 
>   sharelib/sqoop/pom.xml d875c93ae599b990cdcbefdf2aa306eae41a9f0c 
>   sharelib/streaming/pom.xml fd795182f11f0f884177a56592e99aaea01d195d 
> 
> Diff: https://reviews.apache.org/r/49778/diff/
> 
> 
> Testing
> -------
> 
> - Tried a bunch of stuff against a Hadoop 2.4.0 cluster, including an "action" that would have different kinds of Exceptions, System.exit, etc.
> - Some new and old unit tests.
> 
> 
> Thanks,
> 
> Robert Kanter
> 
>

Re: Review Request 49778: OOZIE-2590: OYA: Create basic Oozie Launcher Application Master

Posted by Robert Kanter <rk...@cloudera.com>.


> On July 8, 2016, 12:06 p.m., Jaydeep Vishwakarma wrote:
> > core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java, line 1158
> > <https://reviews.apache.org/r/49778/diff/1/?file=1439241#file1439241line1158>
> >
> >     I am thinking, could it be the case when launcher submitted to the cluster and server shutdown before updating the status?

It looks like that could be it, but it's going to take some more digging to figure out how it works and how to adapt it for YARN (if we even still need it).  I've created OOZIE-2609 to follow up on this so we can move forward with OOZIE-2590 so that we can unblock all of the other OYA subtasks.


- Robert


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49778/#review141265
-----------------------------------------------------------


On July 15, 2016, 1:22 a.m., Robert Kanter wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/49778/
> -----------------------------------------------------------
> 
> (Updated July 15, 2016, 1:22 a.m.)
> 
> 
> Review request for oozie.
> 
> 
> Bugs: OOZIE-2590
>     https://issues.apache.org/jira/browse/OOZIE-2590
> 
> 
> Repository: oozie-git
> 
> 
> Description
> -------
> 
> See OOZIE-2590
> 
> 
> Diffs
> -----
> 
>   core/pom.xml 6584af8123925e00f7b29d6b33201f4fc7dca231 
>   core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 96726daa800861c15e4a0d97514fbd45e31fa000 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2ActionExecutor.java b5b1bf90891fde1b13ca058e0b5f538b1b683128 
>   core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java c74e9e61c01a4b6430bbd5491c8cbaf0c4e90198 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 99e33445d68ac6a99a08e02ecc182f35103a53ce 
>   core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 69e1044e531a67c2d522f642376f4ad030423e53 
>   core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java 252f461267818c0da91c7ad59380b2f18a074f9f 
>   core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 6813a37d0ffc66da41636fe01e9cc7a37d6f6e79 
>   core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java 794e8258472df5fe18299eb3c62471cf9fecc186 
>   core/src/main/java/org/apache/oozie/util/ClasspathUtils.java PRE-CREATION 
>   core/src/main/resources/oozie-default.xml 6c2f7d82e933614ea6b434f9b035dca3751d1c7f 
>   core/src/test/java/org/apache/oozie/QueryServlet.java PRE-CREATION 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 879bfebf58acc12ce787b151a67eaaa0716ff2bb 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestLauncherAM.java PRE-CREATION 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestLauncherAMCallbackNotifier.java PRE-CREATION 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestPrepareActionsDriver.java df9e939cb236e8b65e7b3aa9ab683c33f5fa02b7 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestShellActionExecutor.java 6a962a1030a16e61a2af27a6316c4d71b17ea5c9 
>   core/src/test/java/org/apache/oozie/command/wf/HangServlet.java 3344cf97762c77f5660d94341e72df4243f76eb2 
>   core/src/test/java/org/apache/oozie/service/TestConfigurationService.java 2153bf122ddbdf81acd90c4922fa8d0f619d2d41 
>   core/src/test/java/org/apache/oozie/service/TestHadoopAccessorService.java 96faa4878111baafb81fdf0f3b38becba0c6f15e 
>   core/src/test/java/org/apache/oozie/test/XTestCase.java e3603692b2ae2ebb741adfa3a6ab5a37ea84eb96 
>   core/src/test/java/org/apache/oozie/util/TestClasspathUtils.java PRE-CREATION 
>   sharelib/distcp/pom.xml c8cc47cddc6d2922766e2d68ddddee96f5f3cfec 
>   sharelib/hcatalog/pom.xml 2b0c5042a6f295101f8c5fae0b08a4239b5e97d3 
>   sharelib/hive/pom.xml d10d7b85b70cbeb1906c83484739f34c09038134 
>   sharelib/hive2/pom.xml ce967c5575eb639335332dc6de7ce3f3c66a55ca 
>   sharelib/oozie/pom.xml dd95b4537c00b516b124341bdacf4fbd1fd351f9 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java PRE-CREATION 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAMCallbackNotifier.java PRE-CREATION 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java f2cba13e89e8ddc16b59a04947309147ea6b34b7 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/PrepareActionsDriver.java 21ae456b768fb1cb5334a9e23e6c6082240f8dff 
>   sharelib/pig/pom.xml 562c5309cde72c7430fc03f4d342b6d30b5b4575 
>   sharelib/spark/pom.xml 46c6375314ac314fc6e702695303bbc8eda4f87b 
>   sharelib/sqoop/pom.xml d875c93ae599b990cdcbefdf2aa306eae41a9f0c 
>   sharelib/streaming/pom.xml fd795182f11f0f884177a56592e99aaea01d195d 
> 
> Diff: https://reviews.apache.org/r/49778/diff/
> 
> 
> Testing
> -------
> 
> - Tried a bunch of stuff against a Hadoop 2.4.0 cluster, including an "action" that would have different kinds of Exceptions, System.exit, etc.
> - Some new and old unit tests.
> 
> 
> Thanks,
> 
> Robert Kanter
> 
>

Re: Review Request 49778: OOZIE-2590: OYA: Create basic Oozie Launcher Application Master

Posted by Robert Kanter <rk...@cloudera.com>.


> On July 8, 2016, 12:06 p.m., Jaydeep Vishwakarma wrote:
> > core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java, line 169
> > <https://reviews.apache.org/r/49778/diff/1/?file=1439241#file1439241line169>
> >
> >     we dont need this right?

We don't if I copy the necessary constants from LauncherMapper to LauncherAM, which I'll do.


> On July 8, 2016, 12:06 p.m., Jaydeep Vishwakarma wrote:
> > core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java, line 174
> > <https://reviews.apache.org/r/49778/diff/1/?file=1439241#file1439241line174>
> >
> >     you missed to add callbackNotifier.java

We add the classes here, not the Java files.


> On July 8, 2016, 12:06 p.m., Jaydeep Vishwakarma wrote:
> > core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java, line 1299
> > <https://reviews.apache.org/r/49778/diff/1/?file=1439241#file1439241line1299>
> >
> >     what if job fail to lauch

You get a YarnException.


> On July 8, 2016, 12:06 p.m., Jaydeep Vishwakarma wrote:
> > core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java, line 1300
> > <https://reviews.apache.org/r/49778/diff/1/?file=1439241#file1439241line1300>
> >
> >     should'nt we capture the failure?

Peter pointed out that ````yarnClient.submitApplication(appContext);```` is blocking, so I'm going to remove this.  We can always add something like this later if we run into problems.


- Robert


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49778/#review141265
-----------------------------------------------------------


On July 8, 2016, 12:17 a.m., Robert Kanter wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/49778/
> -----------------------------------------------------------
> 
> (Updated July 8, 2016, 12:17 a.m.)
> 
> 
> Review request for oozie.
> 
> 
> Bugs: OOZIE-2590
>     https://issues.apache.org/jira/browse/OOZIE-2590
> 
> 
> Repository: oozie-git
> 
> 
> Description
> -------
> 
> See OOZIE-2590
> 
> 
> Diffs
> -----
> 
>   core/pom.xml 6584af8123925e00f7b29d6b33201f4fc7dca231 
>   core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 96726daa800861c15e4a0d97514fbd45e31fa000 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2ActionExecutor.java b5b1bf90891fde1b13ca058e0b5f538b1b683128 
>   core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java c74e9e61c01a4b6430bbd5491c8cbaf0c4e90198 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 99e33445d68ac6a99a08e02ecc182f35103a53ce 
>   core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 69e1044e531a67c2d522f642376f4ad030423e53 
>   core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java 252f461267818c0da91c7ad59380b2f18a074f9f 
>   core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 6813a37d0ffc66da41636fe01e9cc7a37d6f6e79 
>   core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java 794e8258472df5fe18299eb3c62471cf9fecc186 
>   core/src/main/java/org/apache/oozie/util/ClasspathUtils.java PRE-CREATION 
>   core/src/main/resources/oozie-default.xml 6c2f7d82e933614ea6b434f9b035dca3751d1c7f 
>   core/src/test/java/org/apache/oozie/QueryServlet.java PRE-CREATION 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestCallbackNotifier.java PRE-CREATION 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 879bfebf58acc12ce787b151a67eaaa0716ff2bb 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestLauncherAM.java PRE-CREATION 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestPrepareActionsDriver.java df9e939cb236e8b65e7b3aa9ab683c33f5fa02b7 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestShellActionExecutor.java 6a962a1030a16e61a2af27a6316c4d71b17ea5c9 
>   core/src/test/java/org/apache/oozie/command/wf/HangServlet.java 3344cf97762c77f5660d94341e72df4243f76eb2 
>   core/src/test/java/org/apache/oozie/service/TestConfigurationService.java 2153bf122ddbdf81acd90c4922fa8d0f619d2d41 
>   core/src/test/java/org/apache/oozie/service/TestHadoopAccessorService.java 96faa4878111baafb81fdf0f3b38becba0c6f15e 
>   core/src/test/java/org/apache/oozie/test/XTestCase.java e3603692b2ae2ebb741adfa3a6ab5a37ea84eb96 
>   core/src/test/java/org/apache/oozie/util/TestClasspathUtils.java PRE-CREATION 
>   sharelib/distcp/pom.xml c8cc47cddc6d2922766e2d68ddddee96f5f3cfec 
>   sharelib/hcatalog/pom.xml 2b0c5042a6f295101f8c5fae0b08a4239b5e97d3 
>   sharelib/hive/pom.xml d10d7b85b70cbeb1906c83484739f34c09038134 
>   sharelib/hive2/pom.xml ce967c5575eb639335332dc6de7ce3f3c66a55ca 
>   sharelib/oozie/pom.xml dd95b4537c00b516b124341bdacf4fbd1fd351f9 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/CallbackNotifier.java PRE-CREATION 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java PRE-CREATION 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java f2cba13e89e8ddc16b59a04947309147ea6b34b7 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/PrepareActionsDriver.java 21ae456b768fb1cb5334a9e23e6c6082240f8dff 
>   sharelib/pig/pom.xml 562c5309cde72c7430fc03f4d342b6d30b5b4575 
>   sharelib/spark/pom.xml 46c6375314ac314fc6e702695303bbc8eda4f87b 
>   sharelib/sqoop/pom.xml d875c93ae599b990cdcbefdf2aa306eae41a9f0c 
>   sharelib/streaming/pom.xml fd795182f11f0f884177a56592e99aaea01d195d 
> 
> Diff: https://reviews.apache.org/r/49778/diff/
> 
> 
> Testing
> -------
> 
> - Tried a bunch of stuff against a Hadoop 2.4.0 cluster, including an "action" that would have different kinds of Exceptions, System.exit, etc.
> - Some new and old unit tests.
> 
> 
> Thanks,
> 
> Robert Kanter
> 
>

Re: Review Request 49778: OOZIE-2590: OYA: Create basic Oozie Launcher Application Master

Posted by Jaydeep Vishwakarma <ja...@gmail.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49778/#review141265
-----------------------------------------------------------




core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java (line 167)
<https://reviews.apache.org/r/49778/#comment206705>

    we dont need this right?



core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java (line 172)
<https://reviews.apache.org/r/49778/#comment206725>

    you missed to add callbackNotifier.java



core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java (line 1016)
<https://reviews.apache.org/r/49778/#comment206711>

    I am thinking, could it be the case when launcher submitted to the cluster and server shutdown before updating the status?



core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java (line 1073)
<https://reviews.apache.org/r/49778/#comment206724>

    I think this method is too long, Can you devided further?



core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java (line 1157)
<https://reviews.apache.org/r/49778/#comment206728>

    what if job fail to lauch



core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java (line 1158)
<https://reviews.apache.org/r/49778/#comment206723>

    should'nt we capture the failure?


- Jaydeep Vishwakarma


On July 8, 2016, 12:17 a.m., Robert Kanter wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/49778/
> -----------------------------------------------------------
> 
> (Updated July 8, 2016, 12:17 a.m.)
> 
> 
> Review request for oozie.
> 
> 
> Bugs: OOZIE-2590
>     https://issues.apache.org/jira/browse/OOZIE-2590
> 
> 
> Repository: oozie-git
> 
> 
> Description
> -------
> 
> See OOZIE-2590
> 
> 
> Diffs
> -----
> 
>   core/pom.xml 6584af8123925e00f7b29d6b33201f4fc7dca231 
>   core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 96726daa800861c15e4a0d97514fbd45e31fa000 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2ActionExecutor.java b5b1bf90891fde1b13ca058e0b5f538b1b683128 
>   core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java c74e9e61c01a4b6430bbd5491c8cbaf0c4e90198 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 99e33445d68ac6a99a08e02ecc182f35103a53ce 
>   core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 69e1044e531a67c2d522f642376f4ad030423e53 
>   core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java 252f461267818c0da91c7ad59380b2f18a074f9f 
>   core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 6813a37d0ffc66da41636fe01e9cc7a37d6f6e79 
>   core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java 794e8258472df5fe18299eb3c62471cf9fecc186 
>   core/src/main/java/org/apache/oozie/util/ClasspathUtils.java PRE-CREATION 
>   core/src/main/resources/oozie-default.xml 6c2f7d82e933614ea6b434f9b035dca3751d1c7f 
>   core/src/test/java/org/apache/oozie/QueryServlet.java PRE-CREATION 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestCallbackNotifier.java PRE-CREATION 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 879bfebf58acc12ce787b151a67eaaa0716ff2bb 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestLauncherAM.java PRE-CREATION 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestPrepareActionsDriver.java df9e939cb236e8b65e7b3aa9ab683c33f5fa02b7 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestShellActionExecutor.java 6a962a1030a16e61a2af27a6316c4d71b17ea5c9 
>   core/src/test/java/org/apache/oozie/command/wf/HangServlet.java 3344cf97762c77f5660d94341e72df4243f76eb2 
>   core/src/test/java/org/apache/oozie/service/TestConfigurationService.java 2153bf122ddbdf81acd90c4922fa8d0f619d2d41 
>   core/src/test/java/org/apache/oozie/service/TestHadoopAccessorService.java 96faa4878111baafb81fdf0f3b38becba0c6f15e 
>   core/src/test/java/org/apache/oozie/test/XTestCase.java e3603692b2ae2ebb741adfa3a6ab5a37ea84eb96 
>   core/src/test/java/org/apache/oozie/util/TestClasspathUtils.java PRE-CREATION 
>   sharelib/distcp/pom.xml c8cc47cddc6d2922766e2d68ddddee96f5f3cfec 
>   sharelib/hcatalog/pom.xml 2b0c5042a6f295101f8c5fae0b08a4239b5e97d3 
>   sharelib/hive/pom.xml d10d7b85b70cbeb1906c83484739f34c09038134 
>   sharelib/hive2/pom.xml ce967c5575eb639335332dc6de7ce3f3c66a55ca 
>   sharelib/oozie/pom.xml dd95b4537c00b516b124341bdacf4fbd1fd351f9 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/CallbackNotifier.java PRE-CREATION 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java PRE-CREATION 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java f2cba13e89e8ddc16b59a04947309147ea6b34b7 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/PrepareActionsDriver.java 21ae456b768fb1cb5334a9e23e6c6082240f8dff 
>   sharelib/pig/pom.xml 562c5309cde72c7430fc03f4d342b6d30b5b4575 
>   sharelib/spark/pom.xml 46c6375314ac314fc6e702695303bbc8eda4f87b 
>   sharelib/sqoop/pom.xml d875c93ae599b990cdcbefdf2aa306eae41a9f0c 
>   sharelib/streaming/pom.xml fd795182f11f0f884177a56592e99aaea01d195d 
> 
> Diff: https://reviews.apache.org/r/49778/diff/
> 
> 
> Testing
> -------
> 
> - Tried a bunch of stuff against a Hadoop 2.4.0 cluster, including an "action" that would have different kinds of Exceptions, System.exit, etc.
> - Some new and old unit tests.
> 
> 
> Thanks,
> 
> Robert Kanter
> 
>