You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@oozie.apache.org by Attila Sasvari <as...@cloudera.com> on 2017/09/05 15:52:46 UTC

Re: Review Request 61529: OOZIE-2687 Create XML schema for launcher configurations

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61529/
-----------------------------------------------------------

(Updated Sept. 5, 2017, 3:52 p.m.)


Review request for oozie and Peter Cseh.


Changes
-------

- new schemas are now 1.0
- validation tests for new schemas
- added helper classes to validate schemas with import / include statements 
- refactored JavaActionExecutor


Repository: oozie-git


Description
-------

New workflow xsd to allow changing launcher configuration (e.g. setting java system properties)


Diffs (updated)
-----

  client/src/main/resources/hive-action-1.0.xsd PRE-CREATION 
  client/src/main/resources/hive2-action-1.0.xsd PRE-CREATION 
  client/src/main/resources/oozie-common-1.0.xsd PRE-CREATION 
  client/src/main/resources/oozie-workflow-1.0.xsd PRE-CREATION 
  client/src/main/resources/shell-action-1.0.xsd PRE-CREATION 
  client/src/main/resources/spark-action-1.0.xsd PRE-CREATION 
  client/src/main/resources/sqoop-action-1.0.xsd PRE-CREATION 
  core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java bca79aa052521ea4f6f16e76bd69f84fb16be790 
  core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 8fdc50cce15271c13b20d1aaab2ffb95cb8fa711 
  core/src/main/java/org/apache/oozie/action/ssh/SshActionExecutor.java 7e3348550e2ef44ae4bd6c5a7a417052bf9c108a 
  core/src/main/java/org/apache/oozie/jms/MessageReceiver.java 47bfd2bcf9b53eed0882c51cdb5c530e0f6ccfe7 
  core/src/main/java/org/apache/oozie/service/LiteWorkflowStoreService.java ffc29af2f834da6d0890ea7215c5a62fd7cd693e 
  core/src/main/java/org/apache/oozie/service/SchemaService.java 137e2c0b4840c6d3858683db1aa38f54bd55be92 
  core/src/main/java/org/apache/oozie/util/WritableUtils.java aa027e37ba23d4f481698e0bcd93c26c763a0b1f 
  core/src/main/java/org/apache/oozie/util/schema/Input.java PRE-CREATION 
  core/src/main/java/org/apache/oozie/util/schema/ResourceResolver.java PRE-CREATION 
  core/src/main/java/org/apache/oozie/workflow/lite/LauncherConfigHandler.java PRE-CREATION 
  core/src/main/java/org/apache/oozie/workflow/lite/LiteWorkflowAppParser.java a74e5c759fe3e336e7e98c61b8f5ac52efa4100c 
  core/src/main/java/org/apache/oozie/workflow/lite/LiteWorkflowLib.java 23df0867e5984c518e8424d79365997bec28f619 
  core/src/main/resources/oozie-default.xml 0d174b5069aff36ca3a9d48722c5bba69623a046 
  core/src/test/java/org/apache/oozie/action/hadoop/ActionExecutorTestCase.java d74160a09ee237f4067e00ded526e7ec94528a6b 
  core/src/test/java/org/apache/oozie/action/hadoop/LauncherMainTester.java 6cee7a8948ca7f2c7aaf44d4f172d18a1308a482 
  core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java ce674adf9cffa20690083e6298875a6f9047a109 
  core/src/test/java/org/apache/oozie/service/TestSchemaService.java 88a10dbead32879334c3953a90a411a3a18fc0b3 
  core/src/test/java/org/apache/oozie/util/TestMetricsInstrumentation.java a882c82b25154338f89e8a90f831b6f0300fbfe7 
  core/src/test/java/org/apache/oozie/workflow/lite/TestLiteWorkflowAppParser.java 21332404bea79fe0a3cea045d57b3e52a1f15cae 
  core/src/test/resources/wf-schema-global-launcherconf-override.xml PRE-CREATION 
  core/src/test/resources/wf-schema-global-launcherconf.xml PRE-CREATION 
  sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java 6a98d6ef3a09a7a75272f3f0f9a9c2fc5472e76d 
  sharelib/pig/src/test/java/org/apache/oozie/action/hadoop/UDFTester.java ff999ff2cd4875fbf3ad5e1a7eacf9972e848630 


Diff: https://reviews.apache.org/r/61529/diff/3/

Changes: https://reviews.apache.org/r/61529/diff/2-3/


Testing
-------

- TestJavaActionExecutor, TestLiteWorkflowAppParser tests passed
- java-main example on pseudo hadoop succeeded, modified workflow xml so that verbose class loading information is printed out of stdout of the launcher job


Thanks,

Attila Sasvari

Re: Review Request 61529: OOZIE-2687 Create XML schema for launcher configurations

Posted by Peter Bacsko <pb...@cloudera.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61529/#review184649
-----------------------------------------------------------




client/src/main/resources/oozie-common-1.0.xsd
Lines 24 (patched)
<https://reviews.apache.org/r/61529/#comment260825>

    Do we want a sequence here? As Robert mentioned, an xs:choice could be better.



core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
Lines 26 (patched)
<https://reviews.apache.org/r/61529/#comment260826>

    Avoid *



core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
Lines 33 (patched)
<https://reviews.apache.org/r/61529/#comment260827>

    Avoid *



core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
Lines 865 (patched)
<https://reviews.apache.org/r/61529/#comment260828>

    Identation seems odd



core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
Lines 981 (patched)
<https://reviews.apache.org/r/61529/#comment260829>

    This should not be logged on an INFO level



core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
Line 1238 (original), 1269 (patched)
<https://reviews.apache.org/r/61529/#comment260830>

    In theory we don't have to handle the case when we have both java-opt and java-opts, because the schema validator should take care of it.
    
    Still, I'd be more confident if we explicitly checked this condition (that is, one or the another). We can add a comment that it's just a safety net.



core/src/main/java/org/apache/oozie/util/WritableUtils.java
Lines 191 (patched)
<https://reviews.apache.org/r/61529/#comment260831>

    I remember creating this method for serializing something, but later abandoned that approach.
    
    If it's not necessary, please remote this,



core/src/main/java/org/apache/oozie/util/WritableUtils.java
Lines 245 (patched)
<https://reviews.apache.org/r/61529/#comment260832>

    Same thing here.


- Peter Bacsko


On szept. 5, 2017, 3:52 du, Attila Sasvari wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61529/
> -----------------------------------------------------------
> 
> (Updated szept. 5, 2017, 3:52 du)
> 
> 
> Review request for oozie and Peter Cseh.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> -------
> 
> New workflow xsd to allow changing launcher configuration (e.g. setting java system properties)
> 
> 
> Diffs
> -----
> 
>   client/src/main/resources/hive-action-1.0.xsd PRE-CREATION 
>   client/src/main/resources/hive2-action-1.0.xsd PRE-CREATION 
>   client/src/main/resources/oozie-common-1.0.xsd PRE-CREATION 
>   client/src/main/resources/oozie-workflow-1.0.xsd PRE-CREATION 
>   client/src/main/resources/shell-action-1.0.xsd PRE-CREATION 
>   client/src/main/resources/spark-action-1.0.xsd PRE-CREATION 
>   client/src/main/resources/sqoop-action-1.0.xsd PRE-CREATION 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java bca79aa052521ea4f6f16e76bd69f84fb16be790 
>   core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 8fdc50cce15271c13b20d1aaab2ffb95cb8fa711 
>   core/src/main/java/org/apache/oozie/action/ssh/SshActionExecutor.java 7e3348550e2ef44ae4bd6c5a7a417052bf9c108a 
>   core/src/main/java/org/apache/oozie/jms/MessageReceiver.java 47bfd2bcf9b53eed0882c51cdb5c530e0f6ccfe7 
>   core/src/main/java/org/apache/oozie/service/LiteWorkflowStoreService.java ffc29af2f834da6d0890ea7215c5a62fd7cd693e 
>   core/src/main/java/org/apache/oozie/service/SchemaService.java 137e2c0b4840c6d3858683db1aa38f54bd55be92 
>   core/src/main/java/org/apache/oozie/util/WritableUtils.java aa027e37ba23d4f481698e0bcd93c26c763a0b1f 
>   core/src/main/java/org/apache/oozie/util/schema/Input.java PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/schema/ResourceResolver.java PRE-CREATION 
>   core/src/main/java/org/apache/oozie/workflow/lite/LauncherConfigHandler.java PRE-CREATION 
>   core/src/main/java/org/apache/oozie/workflow/lite/LiteWorkflowAppParser.java a74e5c759fe3e336e7e98c61b8f5ac52efa4100c 
>   core/src/main/java/org/apache/oozie/workflow/lite/LiteWorkflowLib.java 23df0867e5984c518e8424d79365997bec28f619 
>   core/src/main/resources/oozie-default.xml 0d174b5069aff36ca3a9d48722c5bba69623a046 
>   core/src/test/java/org/apache/oozie/action/hadoop/ActionExecutorTestCase.java d74160a09ee237f4067e00ded526e7ec94528a6b 
>   core/src/test/java/org/apache/oozie/action/hadoop/LauncherMainTester.java 6cee7a8948ca7f2c7aaf44d4f172d18a1308a482 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java ce674adf9cffa20690083e6298875a6f9047a109 
>   core/src/test/java/org/apache/oozie/service/TestSchemaService.java 88a10dbead32879334c3953a90a411a3a18fc0b3 
>   core/src/test/java/org/apache/oozie/util/TestMetricsInstrumentation.java a882c82b25154338f89e8a90f831b6f0300fbfe7 
>   core/src/test/java/org/apache/oozie/workflow/lite/TestLiteWorkflowAppParser.java 21332404bea79fe0a3cea045d57b3e52a1f15cae 
>   core/src/test/resources/wf-schema-global-launcherconf-override.xml PRE-CREATION 
>   core/src/test/resources/wf-schema-global-launcherconf.xml PRE-CREATION 
>   release-log.txt e2311eab6392ca08ad855da887e391f3f14a7408 
>   sharelib/distcp/src/main/java/org/apache/oozie/action/hadoop/DistcpMain.java d17d5beb76754655720a0e29079daf676c507841 
>   sharelib/hive/src/main/java/org/apache/oozie/action/hadoop/HiveMain.java fb3fabf027a1dda537d2bd86d574abaf95a685db 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java 6a98d6ef3a09a7a75272f3f0f9a9c2fc5472e76d 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMain.java f1f52c6e876fae56c90979baef3d31a0f5ba2396 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/MapReduceMain.java 77aaa62e23185e6df053f01fd1c4db1f2e972f6b 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/PasswordMasker.java 6a73bf9b8b1d0c5cbb008594c75c9e779298a29d 
>   sharelib/oozie/src/test/java/org/apache/oozie/action/hadoop/TestLauncherMain.java a793df1330d10828101abf9107d565db7972a95f 
>   sharelib/pig/src/main/java/org/apache/oozie/action/hadoop/PigMain.java 13ee04f4482471b61ee363f9cb98f04436c9e580 
>   sharelib/pig/src/test/java/org/apache/oozie/action/hadoop/UDFTester.java ff999ff2cd4875fbf3ad5e1a7eacf9972e848630 
>   sharelib/spark/src/main/java/org/apache/oozie/action/hadoop/SparkMain.java 8aa8343cfcfa136aedef0bf9d53fb4a11800e07c 
>   sharelib/sqoop/src/main/java/org/apache/oozie/action/hadoop/SqoopMain.java e889ef9f857e8417353f28ccb8870f9d843a9d33 
> 
> 
> Diff: https://reviews.apache.org/r/61529/diff/4/
> 
> 
> Testing
> -------
> 
> - TestJavaActionExecutor, TestLiteWorkflowAppParser tests passed
> - java-main example on pseudo hadoop succeeded, modified workflow xml so that verbose class loading information is printed out of stdout of the launcher job
> 
> 
> Thanks,
> 
> Attila Sasvari
> 
>

Re: Review Request 61529: OOZIE-2687 Create XML schema for launcher configurations

Posted by Peter Cseh <ge...@cloudera.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61529/#review184643
-----------------------------------------------------------




core/src/main/java/org/apache/oozie/workflow/lite/LauncherConfigHandler.java
Lines 59 (patched)
<https://reviews.apache.org/r/61529/#comment260821>

    In oozie-default.xml you state that this is in Mb. Would it be better to call this memory.mb or just leave it like this?


- Peter Cseh


On Sept. 5, 2017, 3:52 p.m., Attila Sasvari wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61529/
> -----------------------------------------------------------
> 
> (Updated Sept. 5, 2017, 3:52 p.m.)
> 
> 
> Review request for oozie and Peter Cseh.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> -------
> 
> New workflow xsd to allow changing launcher configuration (e.g. setting java system properties)
> 
> 
> Diffs
> -----
> 
>   client/src/main/resources/hive-action-1.0.xsd PRE-CREATION 
>   client/src/main/resources/hive2-action-1.0.xsd PRE-CREATION 
>   client/src/main/resources/oozie-common-1.0.xsd PRE-CREATION 
>   client/src/main/resources/oozie-workflow-1.0.xsd PRE-CREATION 
>   client/src/main/resources/shell-action-1.0.xsd PRE-CREATION 
>   client/src/main/resources/spark-action-1.0.xsd PRE-CREATION 
>   client/src/main/resources/sqoop-action-1.0.xsd PRE-CREATION 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java bca79aa052521ea4f6f16e76bd69f84fb16be790 
>   core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 8fdc50cce15271c13b20d1aaab2ffb95cb8fa711 
>   core/src/main/java/org/apache/oozie/action/ssh/SshActionExecutor.java 7e3348550e2ef44ae4bd6c5a7a417052bf9c108a 
>   core/src/main/java/org/apache/oozie/jms/MessageReceiver.java 47bfd2bcf9b53eed0882c51cdb5c530e0f6ccfe7 
>   core/src/main/java/org/apache/oozie/service/LiteWorkflowStoreService.java ffc29af2f834da6d0890ea7215c5a62fd7cd693e 
>   core/src/main/java/org/apache/oozie/service/SchemaService.java 137e2c0b4840c6d3858683db1aa38f54bd55be92 
>   core/src/main/java/org/apache/oozie/util/WritableUtils.java aa027e37ba23d4f481698e0bcd93c26c763a0b1f 
>   core/src/main/java/org/apache/oozie/util/schema/Input.java PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/schema/ResourceResolver.java PRE-CREATION 
>   core/src/main/java/org/apache/oozie/workflow/lite/LauncherConfigHandler.java PRE-CREATION 
>   core/src/main/java/org/apache/oozie/workflow/lite/LiteWorkflowAppParser.java a74e5c759fe3e336e7e98c61b8f5ac52efa4100c 
>   core/src/main/java/org/apache/oozie/workflow/lite/LiteWorkflowLib.java 23df0867e5984c518e8424d79365997bec28f619 
>   core/src/main/resources/oozie-default.xml 0d174b5069aff36ca3a9d48722c5bba69623a046 
>   core/src/test/java/org/apache/oozie/action/hadoop/ActionExecutorTestCase.java d74160a09ee237f4067e00ded526e7ec94528a6b 
>   core/src/test/java/org/apache/oozie/action/hadoop/LauncherMainTester.java 6cee7a8948ca7f2c7aaf44d4f172d18a1308a482 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java ce674adf9cffa20690083e6298875a6f9047a109 
>   core/src/test/java/org/apache/oozie/service/TestSchemaService.java 88a10dbead32879334c3953a90a411a3a18fc0b3 
>   core/src/test/java/org/apache/oozie/util/TestMetricsInstrumentation.java a882c82b25154338f89e8a90f831b6f0300fbfe7 
>   core/src/test/java/org/apache/oozie/workflow/lite/TestLiteWorkflowAppParser.java 21332404bea79fe0a3cea045d57b3e52a1f15cae 
>   core/src/test/resources/wf-schema-global-launcherconf-override.xml PRE-CREATION 
>   core/src/test/resources/wf-schema-global-launcherconf.xml PRE-CREATION 
>   release-log.txt e2311eab6392ca08ad855da887e391f3f14a7408 
>   sharelib/distcp/src/main/java/org/apache/oozie/action/hadoop/DistcpMain.java d17d5beb76754655720a0e29079daf676c507841 
>   sharelib/hive/src/main/java/org/apache/oozie/action/hadoop/HiveMain.java fb3fabf027a1dda537d2bd86d574abaf95a685db 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java 6a98d6ef3a09a7a75272f3f0f9a9c2fc5472e76d 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMain.java f1f52c6e876fae56c90979baef3d31a0f5ba2396 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/MapReduceMain.java 77aaa62e23185e6df053f01fd1c4db1f2e972f6b 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/PasswordMasker.java 6a73bf9b8b1d0c5cbb008594c75c9e779298a29d 
>   sharelib/oozie/src/test/java/org/apache/oozie/action/hadoop/TestLauncherMain.java a793df1330d10828101abf9107d565db7972a95f 
>   sharelib/pig/src/main/java/org/apache/oozie/action/hadoop/PigMain.java 13ee04f4482471b61ee363f9cb98f04436c9e580 
>   sharelib/pig/src/test/java/org/apache/oozie/action/hadoop/UDFTester.java ff999ff2cd4875fbf3ad5e1a7eacf9972e848630 
>   sharelib/spark/src/main/java/org/apache/oozie/action/hadoop/SparkMain.java 8aa8343cfcfa136aedef0bf9d53fb4a11800e07c 
>   sharelib/sqoop/src/main/java/org/apache/oozie/action/hadoop/SqoopMain.java e889ef9f857e8417353f28ccb8870f9d843a9d33 
> 
> 
> Diff: https://reviews.apache.org/r/61529/diff/4/
> 
> 
> Testing
> -------
> 
> - TestJavaActionExecutor, TestLiteWorkflowAppParser tests passed
> - java-main example on pseudo hadoop succeeded, modified workflow xml so that verbose class loading information is printed out of stdout of the launcher job
> 
> 
> Thanks,
> 
> Attila Sasvari
> 
>

Re: Review Request 61529: OOZIE-2687 Create XML schema for launcher configurations

Posted by Peter Cseh <ge...@cloudera.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61529/#review184668
-----------------------------------------------------------




core/src/test/java/org/apache/oozie/service/TestSchemaService.java
Lines 143-148 (patched)
<https://reviews.apache.org/r/61529/#comment260860>

    I don't like the fact that we have to use the workflow: prefix here for launcher properties. If anything I'd use the launcher: prefix, but ideally there should be no need for prefixes.


- Peter Cseh


On Sept. 5, 2017, 3:52 p.m., Attila Sasvari wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61529/
> -----------------------------------------------------------
> 
> (Updated Sept. 5, 2017, 3:52 p.m.)
> 
> 
> Review request for oozie and Peter Cseh.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> -------
> 
> New workflow xsd to allow changing launcher configuration (e.g. setting java system properties)
> 
> 
> Diffs
> -----
> 
>   client/src/main/resources/hive-action-1.0.xsd PRE-CREATION 
>   client/src/main/resources/hive2-action-1.0.xsd PRE-CREATION 
>   client/src/main/resources/oozie-common-1.0.xsd PRE-CREATION 
>   client/src/main/resources/oozie-workflow-1.0.xsd PRE-CREATION 
>   client/src/main/resources/shell-action-1.0.xsd PRE-CREATION 
>   client/src/main/resources/spark-action-1.0.xsd PRE-CREATION 
>   client/src/main/resources/sqoop-action-1.0.xsd PRE-CREATION 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java bca79aa052521ea4f6f16e76bd69f84fb16be790 
>   core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 8fdc50cce15271c13b20d1aaab2ffb95cb8fa711 
>   core/src/main/java/org/apache/oozie/action/ssh/SshActionExecutor.java 7e3348550e2ef44ae4bd6c5a7a417052bf9c108a 
>   core/src/main/java/org/apache/oozie/jms/MessageReceiver.java 47bfd2bcf9b53eed0882c51cdb5c530e0f6ccfe7 
>   core/src/main/java/org/apache/oozie/service/LiteWorkflowStoreService.java ffc29af2f834da6d0890ea7215c5a62fd7cd693e 
>   core/src/main/java/org/apache/oozie/service/SchemaService.java 137e2c0b4840c6d3858683db1aa38f54bd55be92 
>   core/src/main/java/org/apache/oozie/util/WritableUtils.java aa027e37ba23d4f481698e0bcd93c26c763a0b1f 
>   core/src/main/java/org/apache/oozie/util/schema/Input.java PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/schema/ResourceResolver.java PRE-CREATION 
>   core/src/main/java/org/apache/oozie/workflow/lite/LauncherConfigHandler.java PRE-CREATION 
>   core/src/main/java/org/apache/oozie/workflow/lite/LiteWorkflowAppParser.java a74e5c759fe3e336e7e98c61b8f5ac52efa4100c 
>   core/src/main/java/org/apache/oozie/workflow/lite/LiteWorkflowLib.java 23df0867e5984c518e8424d79365997bec28f619 
>   core/src/main/resources/oozie-default.xml 0d174b5069aff36ca3a9d48722c5bba69623a046 
>   core/src/test/java/org/apache/oozie/action/hadoop/ActionExecutorTestCase.java d74160a09ee237f4067e00ded526e7ec94528a6b 
>   core/src/test/java/org/apache/oozie/action/hadoop/LauncherMainTester.java 6cee7a8948ca7f2c7aaf44d4f172d18a1308a482 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java ce674adf9cffa20690083e6298875a6f9047a109 
>   core/src/test/java/org/apache/oozie/service/TestSchemaService.java 88a10dbead32879334c3953a90a411a3a18fc0b3 
>   core/src/test/java/org/apache/oozie/util/TestMetricsInstrumentation.java a882c82b25154338f89e8a90f831b6f0300fbfe7 
>   core/src/test/java/org/apache/oozie/workflow/lite/TestLiteWorkflowAppParser.java 21332404bea79fe0a3cea045d57b3e52a1f15cae 
>   core/src/test/resources/wf-schema-global-launcherconf-override.xml PRE-CREATION 
>   core/src/test/resources/wf-schema-global-launcherconf.xml PRE-CREATION 
>   release-log.txt e2311eab6392ca08ad855da887e391f3f14a7408 
>   sharelib/distcp/src/main/java/org/apache/oozie/action/hadoop/DistcpMain.java d17d5beb76754655720a0e29079daf676c507841 
>   sharelib/hive/src/main/java/org/apache/oozie/action/hadoop/HiveMain.java fb3fabf027a1dda537d2bd86d574abaf95a685db 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java 6a98d6ef3a09a7a75272f3f0f9a9c2fc5472e76d 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMain.java f1f52c6e876fae56c90979baef3d31a0f5ba2396 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/MapReduceMain.java 77aaa62e23185e6df053f01fd1c4db1f2e972f6b 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/PasswordMasker.java 6a73bf9b8b1d0c5cbb008594c75c9e779298a29d 
>   sharelib/oozie/src/test/java/org/apache/oozie/action/hadoop/TestLauncherMain.java a793df1330d10828101abf9107d565db7972a95f 
>   sharelib/pig/src/main/java/org/apache/oozie/action/hadoop/PigMain.java 13ee04f4482471b61ee363f9cb98f04436c9e580 
>   sharelib/pig/src/test/java/org/apache/oozie/action/hadoop/UDFTester.java ff999ff2cd4875fbf3ad5e1a7eacf9972e848630 
>   sharelib/spark/src/main/java/org/apache/oozie/action/hadoop/SparkMain.java 8aa8343cfcfa136aedef0bf9d53fb4a11800e07c 
>   sharelib/sqoop/src/main/java/org/apache/oozie/action/hadoop/SqoopMain.java e889ef9f857e8417353f28ccb8870f9d843a9d33 
> 
> 
> Diff: https://reviews.apache.org/r/61529/diff/4/
> 
> 
> Testing
> -------
> 
> - TestJavaActionExecutor, TestLiteWorkflowAppParser tests passed
> - java-main example on pseudo hadoop succeeded, modified workflow xml so that verbose class loading information is printed out of stdout of the launcher job
> 
> 
> Thanks,
> 
> Attila Sasvari
> 
>

Re: Review Request 61529: OOZIE-2687 Create XML schema for launcher configurations

Posted by Peter Bacsko <pb...@cloudera.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61529/#review184655
-----------------------------------------------------------




core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
Lines 1296 (patched)
<https://reviews.apache.org/r/61529/#comment260844>

    Unnecessary comment



core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
Lines 1363 (patched)
<https://reviews.apache.org/r/61529/#comment260846>

    I haven't seen a testcase which covers this path, would be useful to have one



core/src/main/java/org/apache/oozie/action/ssh/SshActionExecutor.java
Line 179 (original), 179 (patched)
<https://reviews.apache.org/r/61529/#comment260847>

    Could you explain what this rename is about? Now We have "processSettings" instead of "processw everywhere.



core/src/main/resources/oozie-default.xml
Lines 3087 (patched)
<https://reviews.apache.org/r/61529/#comment260843>

    Perhaps this property could be called
    
    oozie.launcher.default.memory.mb
    
    so users won't accidentally define it in different units.


- Peter Bacsko


On szept. 5, 2017, 3:52 du, Attila Sasvari wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61529/
> -----------------------------------------------------------
> 
> (Updated szept. 5, 2017, 3:52 du)
> 
> 
> Review request for oozie and Peter Cseh.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> -------
> 
> New workflow xsd to allow changing launcher configuration (e.g. setting java system properties)
> 
> 
> Diffs
> -----
> 
>   client/src/main/resources/hive-action-1.0.xsd PRE-CREATION 
>   client/src/main/resources/hive2-action-1.0.xsd PRE-CREATION 
>   client/src/main/resources/oozie-common-1.0.xsd PRE-CREATION 
>   client/src/main/resources/oozie-workflow-1.0.xsd PRE-CREATION 
>   client/src/main/resources/shell-action-1.0.xsd PRE-CREATION 
>   client/src/main/resources/spark-action-1.0.xsd PRE-CREATION 
>   client/src/main/resources/sqoop-action-1.0.xsd PRE-CREATION 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java bca79aa052521ea4f6f16e76bd69f84fb16be790 
>   core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 8fdc50cce15271c13b20d1aaab2ffb95cb8fa711 
>   core/src/main/java/org/apache/oozie/action/ssh/SshActionExecutor.java 7e3348550e2ef44ae4bd6c5a7a417052bf9c108a 
>   core/src/main/java/org/apache/oozie/jms/MessageReceiver.java 47bfd2bcf9b53eed0882c51cdb5c530e0f6ccfe7 
>   core/src/main/java/org/apache/oozie/service/LiteWorkflowStoreService.java ffc29af2f834da6d0890ea7215c5a62fd7cd693e 
>   core/src/main/java/org/apache/oozie/service/SchemaService.java 137e2c0b4840c6d3858683db1aa38f54bd55be92 
>   core/src/main/java/org/apache/oozie/util/WritableUtils.java aa027e37ba23d4f481698e0bcd93c26c763a0b1f 
>   core/src/main/java/org/apache/oozie/util/schema/Input.java PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/schema/ResourceResolver.java PRE-CREATION 
>   core/src/main/java/org/apache/oozie/workflow/lite/LauncherConfigHandler.java PRE-CREATION 
>   core/src/main/java/org/apache/oozie/workflow/lite/LiteWorkflowAppParser.java a74e5c759fe3e336e7e98c61b8f5ac52efa4100c 
>   core/src/main/java/org/apache/oozie/workflow/lite/LiteWorkflowLib.java 23df0867e5984c518e8424d79365997bec28f619 
>   core/src/main/resources/oozie-default.xml 0d174b5069aff36ca3a9d48722c5bba69623a046 
>   core/src/test/java/org/apache/oozie/action/hadoop/ActionExecutorTestCase.java d74160a09ee237f4067e00ded526e7ec94528a6b 
>   core/src/test/java/org/apache/oozie/action/hadoop/LauncherMainTester.java 6cee7a8948ca7f2c7aaf44d4f172d18a1308a482 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java ce674adf9cffa20690083e6298875a6f9047a109 
>   core/src/test/java/org/apache/oozie/service/TestSchemaService.java 88a10dbead32879334c3953a90a411a3a18fc0b3 
>   core/src/test/java/org/apache/oozie/util/TestMetricsInstrumentation.java a882c82b25154338f89e8a90f831b6f0300fbfe7 
>   core/src/test/java/org/apache/oozie/workflow/lite/TestLiteWorkflowAppParser.java 21332404bea79fe0a3cea045d57b3e52a1f15cae 
>   core/src/test/resources/wf-schema-global-launcherconf-override.xml PRE-CREATION 
>   core/src/test/resources/wf-schema-global-launcherconf.xml PRE-CREATION 
>   release-log.txt e2311eab6392ca08ad855da887e391f3f14a7408 
>   sharelib/distcp/src/main/java/org/apache/oozie/action/hadoop/DistcpMain.java d17d5beb76754655720a0e29079daf676c507841 
>   sharelib/hive/src/main/java/org/apache/oozie/action/hadoop/HiveMain.java fb3fabf027a1dda537d2bd86d574abaf95a685db 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java 6a98d6ef3a09a7a75272f3f0f9a9c2fc5472e76d 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMain.java f1f52c6e876fae56c90979baef3d31a0f5ba2396 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/MapReduceMain.java 77aaa62e23185e6df053f01fd1c4db1f2e972f6b 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/PasswordMasker.java 6a73bf9b8b1d0c5cbb008594c75c9e779298a29d 
>   sharelib/oozie/src/test/java/org/apache/oozie/action/hadoop/TestLauncherMain.java a793df1330d10828101abf9107d565db7972a95f 
>   sharelib/pig/src/main/java/org/apache/oozie/action/hadoop/PigMain.java 13ee04f4482471b61ee363f9cb98f04436c9e580 
>   sharelib/pig/src/test/java/org/apache/oozie/action/hadoop/UDFTester.java ff999ff2cd4875fbf3ad5e1a7eacf9972e848630 
>   sharelib/spark/src/main/java/org/apache/oozie/action/hadoop/SparkMain.java 8aa8343cfcfa136aedef0bf9d53fb4a11800e07c 
>   sharelib/sqoop/src/main/java/org/apache/oozie/action/hadoop/SqoopMain.java e889ef9f857e8417353f28ccb8870f9d843a9d33 
> 
> 
> Diff: https://reviews.apache.org/r/61529/diff/4/
> 
> 
> Testing
> -------
> 
> - TestJavaActionExecutor, TestLiteWorkflowAppParser tests passed
> - java-main example on pseudo hadoop succeeded, modified workflow xml so that verbose class loading information is printed out of stdout of the launcher job
> 
> 
> Thanks,
> 
> Attila Sasvari
> 
>

Re: Review Request 61529: OOZIE-2687 Create XML schema for launcher configurations

Posted by Peter Bacsko <pb...@cloudera.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61529/#review184650
-----------------------------------------------------------




core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java
Lines 2268 (patched)
<https://reviews.apache.org/r/61529/#comment260833>

    Have you considered creating new tests?
    
    Is this coverage enough?



core/src/test/java/org/apache/oozie/service/TestSchemaService.java
Lines 101 (patched)
<https://reviews.apache.org/r/61529/#comment260834>

    Wouldn't it be better to store all these dummy workflows in an external file?


- Peter Bacsko


On szept. 5, 2017, 3:52 du, Attila Sasvari wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61529/
> -----------------------------------------------------------
> 
> (Updated szept. 5, 2017, 3:52 du)
> 
> 
> Review request for oozie and Peter Cseh.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> -------
> 
> New workflow xsd to allow changing launcher configuration (e.g. setting java system properties)
> 
> 
> Diffs
> -----
> 
>   client/src/main/resources/hive-action-1.0.xsd PRE-CREATION 
>   client/src/main/resources/hive2-action-1.0.xsd PRE-CREATION 
>   client/src/main/resources/oozie-common-1.0.xsd PRE-CREATION 
>   client/src/main/resources/oozie-workflow-1.0.xsd PRE-CREATION 
>   client/src/main/resources/shell-action-1.0.xsd PRE-CREATION 
>   client/src/main/resources/spark-action-1.0.xsd PRE-CREATION 
>   client/src/main/resources/sqoop-action-1.0.xsd PRE-CREATION 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java bca79aa052521ea4f6f16e76bd69f84fb16be790 
>   core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 8fdc50cce15271c13b20d1aaab2ffb95cb8fa711 
>   core/src/main/java/org/apache/oozie/action/ssh/SshActionExecutor.java 7e3348550e2ef44ae4bd6c5a7a417052bf9c108a 
>   core/src/main/java/org/apache/oozie/jms/MessageReceiver.java 47bfd2bcf9b53eed0882c51cdb5c530e0f6ccfe7 
>   core/src/main/java/org/apache/oozie/service/LiteWorkflowStoreService.java ffc29af2f834da6d0890ea7215c5a62fd7cd693e 
>   core/src/main/java/org/apache/oozie/service/SchemaService.java 137e2c0b4840c6d3858683db1aa38f54bd55be92 
>   core/src/main/java/org/apache/oozie/util/WritableUtils.java aa027e37ba23d4f481698e0bcd93c26c763a0b1f 
>   core/src/main/java/org/apache/oozie/util/schema/Input.java PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/schema/ResourceResolver.java PRE-CREATION 
>   core/src/main/java/org/apache/oozie/workflow/lite/LauncherConfigHandler.java PRE-CREATION 
>   core/src/main/java/org/apache/oozie/workflow/lite/LiteWorkflowAppParser.java a74e5c759fe3e336e7e98c61b8f5ac52efa4100c 
>   core/src/main/java/org/apache/oozie/workflow/lite/LiteWorkflowLib.java 23df0867e5984c518e8424d79365997bec28f619 
>   core/src/main/resources/oozie-default.xml 0d174b5069aff36ca3a9d48722c5bba69623a046 
>   core/src/test/java/org/apache/oozie/action/hadoop/ActionExecutorTestCase.java d74160a09ee237f4067e00ded526e7ec94528a6b 
>   core/src/test/java/org/apache/oozie/action/hadoop/LauncherMainTester.java 6cee7a8948ca7f2c7aaf44d4f172d18a1308a482 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java ce674adf9cffa20690083e6298875a6f9047a109 
>   core/src/test/java/org/apache/oozie/service/TestSchemaService.java 88a10dbead32879334c3953a90a411a3a18fc0b3 
>   core/src/test/java/org/apache/oozie/util/TestMetricsInstrumentation.java a882c82b25154338f89e8a90f831b6f0300fbfe7 
>   core/src/test/java/org/apache/oozie/workflow/lite/TestLiteWorkflowAppParser.java 21332404bea79fe0a3cea045d57b3e52a1f15cae 
>   core/src/test/resources/wf-schema-global-launcherconf-override.xml PRE-CREATION 
>   core/src/test/resources/wf-schema-global-launcherconf.xml PRE-CREATION 
>   release-log.txt e2311eab6392ca08ad855da887e391f3f14a7408 
>   sharelib/distcp/src/main/java/org/apache/oozie/action/hadoop/DistcpMain.java d17d5beb76754655720a0e29079daf676c507841 
>   sharelib/hive/src/main/java/org/apache/oozie/action/hadoop/HiveMain.java fb3fabf027a1dda537d2bd86d574abaf95a685db 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java 6a98d6ef3a09a7a75272f3f0f9a9c2fc5472e76d 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMain.java f1f52c6e876fae56c90979baef3d31a0f5ba2396 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/MapReduceMain.java 77aaa62e23185e6df053f01fd1c4db1f2e972f6b 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/PasswordMasker.java 6a73bf9b8b1d0c5cbb008594c75c9e779298a29d 
>   sharelib/oozie/src/test/java/org/apache/oozie/action/hadoop/TestLauncherMain.java a793df1330d10828101abf9107d565db7972a95f 
>   sharelib/pig/src/main/java/org/apache/oozie/action/hadoop/PigMain.java 13ee04f4482471b61ee363f9cb98f04436c9e580 
>   sharelib/pig/src/test/java/org/apache/oozie/action/hadoop/UDFTester.java ff999ff2cd4875fbf3ad5e1a7eacf9972e848630 
>   sharelib/spark/src/main/java/org/apache/oozie/action/hadoop/SparkMain.java 8aa8343cfcfa136aedef0bf9d53fb4a11800e07c 
>   sharelib/sqoop/src/main/java/org/apache/oozie/action/hadoop/SqoopMain.java e889ef9f857e8417353f28ccb8870f9d843a9d33 
> 
> 
> Diff: https://reviews.apache.org/r/61529/diff/4/
> 
> 
> Testing
> -------
> 
> - TestJavaActionExecutor, TestLiteWorkflowAppParser tests passed
> - java-main example on pseudo hadoop succeeded, modified workflow xml so that verbose class loading information is printed out of stdout of the launcher job
> 
> 
> Thanks,
> 
> Attila Sasvari
> 
>

Re: Review Request 61529: OOZIE-2687 Create XML schema for launcher configurations

Posted by Peter Cseh <ge...@cloudera.com>.


> On Sept. 11, 2017, 11:36 p.m., Robert Kanter wrote:
> > client/src/main/resources/oozie-workflow-1.0.xsd
> > Line 234 (original), 234 (patched)
> > <https://reviews.apache.org/r/61529/diff/8-9/?file=1818281#file1818281line234>
> >
> >     Should we think about moving the Java, MR, Pig, etc actions out of the workflow XSD and into their own XSDs to be more consistent with the other action types?  The 1.0 XSD schemas is a good opportunity.
> >     
> >     The downside would be backwards compatibility where you can't simply update your workflow schema version on an existing workflow like you typically could in the past - you'd also have to add the xmlns string for these actions.  
> >     
> >     I think it would be okay to force people to add the xmlns.  No other changes should be needed so it's pretty minimal.  Plus, Oozie still accepts the 0.x schemas, so you don't *have* to upgrade the schemas to 1.0 anyway.
> >     
> >     Attila, Geza, what do you think?
> 
> Attila Sasvari wrote:
>     For this purpose I created https://issues.apache.org/jira/browse/OOZIE-3033 (this patch is already quite big).

As this is a breaking change I'd rather fix this in a later jira. I think we can get away using 1.0 as a version and doing big changes like that in follow-up steps.


- Peter


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61529/#review185128
-----------------------------------------------------------


On Sept. 13, 2017, 1:45 p.m., Attila Sasvari wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61529/
> -----------------------------------------------------------
> 
> (Updated Sept. 13, 2017, 1:45 p.m.)
> 
> 
> Review request for oozie and Peter Cseh.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> -------
> 
> New workflow xsd to allow changing launcher configuration (e.g. setting java system properties)
> 
> 
> Diffs
> -----
> 
>   client/src/main/resources/hive-action-1.0.xsd PRE-CREATION 
>   client/src/main/resources/hive2-action-1.0.xsd PRE-CREATION 
>   client/src/main/resources/oozie-common-1.0.xsd PRE-CREATION 
>   client/src/main/resources/oozie-workflow-1.0.xsd PRE-CREATION 
>   client/src/main/resources/shell-action-1.0.xsd PRE-CREATION 
>   client/src/main/resources/spark-action-1.0.xsd PRE-CREATION 
>   client/src/main/resources/sqoop-action-1.0.xsd PRE-CREATION 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 49fd4b8560b21aef725c3268f0ec0e7f46a46ee9 
>   core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 8fdc50cce15271c13b20d1aaab2ffb95cb8fa711 
>   core/src/main/java/org/apache/oozie/action/ssh/SshActionExecutor.java 7e3348550e2ef44ae4bd6c5a7a417052bf9c108a 
>   core/src/main/java/org/apache/oozie/jms/MessageReceiver.java 47bfd2bcf9b53eed0882c51cdb5c530e0f6ccfe7 
>   core/src/main/java/org/apache/oozie/service/LiteWorkflowStoreService.java ffc29af2f834da6d0890ea7215c5a62fd7cd693e 
>   core/src/main/java/org/apache/oozie/service/SchemaService.java 137e2c0b4840c6d3858683db1aa38f54bd55be92 
>   core/src/main/java/org/apache/oozie/util/WritableUtils.java aa027e37ba23d4f481698e0bcd93c26c763a0b1f 
>   core/src/main/java/org/apache/oozie/util/schema/Input.java PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/schema/ResourceResolver.java PRE-CREATION 
>   core/src/main/java/org/apache/oozie/workflow/lite/LauncherConfigHandler.java PRE-CREATION 
>   core/src/main/java/org/apache/oozie/workflow/lite/LiteWorkflowAppParser.java a74e5c759fe3e336e7e98c61b8f5ac52efa4100c 
>   core/src/main/java/org/apache/oozie/workflow/lite/LiteWorkflowLib.java 23df0867e5984c518e8424d79365997bec28f619 
>   core/src/main/resources/oozie-default.xml 0d174b5069aff36ca3a9d48722c5bba69623a046 
>   core/src/test/java/org/apache/oozie/action/hadoop/ActionExecutorTestCase.java d74160a09ee237f4067e00ded526e7ec94528a6b 
>   core/src/test/java/org/apache/oozie/action/hadoop/LauncherMainTester.java 6cee7a8948ca7f2c7aaf44d4f172d18a1308a482 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java d1d78fd527a425b076edd7bb23ba56ebdba62664 
>   core/src/test/java/org/apache/oozie/service/TestSchemaService.java 88a10dbead32879334c3953a90a411a3a18fc0b3 
>   core/src/test/java/org/apache/oozie/util/TestMetricsInstrumentation.java a882c82b25154338f89e8a90f831b6f0300fbfe7 
>   core/src/test/java/org/apache/oozie/workflow/lite/TestLiteWorkflowAppParser.java 21332404bea79fe0a3cea045d57b3e52a1f15cae 
>   core/src/test/resources/wf-schema-global-launcherconf-override.xml PRE-CREATION 
>   core/src/test/resources/wf-schema-global-launcherconf.xml PRE-CREATION 
>   pom.xml db18f30814b9b6a73ba872c2cd7946692d0b876b 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java 6a98d6ef3a09a7a75272f3f0f9a9c2fc5472e76d 
>   sharelib/pig/src/test/java/org/apache/oozie/action/hadoop/UDFTester.java ff999ff2cd4875fbf3ad5e1a7eacf9972e848630 
> 
> 
> Diff: https://reviews.apache.org/r/61529/diff/11/
> 
> 
> Testing
> -------
> 
> - TestJavaActionExecutor, TestLiteWorkflowAppParser tests passed
> - java-main example on pseudo hadoop succeeded, modified workflow xml so that verbose class loading information is printed out of stdout of the launcher job
> 
> 
> Thanks,
> 
> Attila Sasvari
> 
>

Re: Review Request 61529: OOZIE-2687 Create XML schema for launcher configurations

Posted by Robert Kanter <rk...@cloudera.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61529/#review185128
-----------------------------------------------------------




client/src/main/resources/hive-action-1.0.xsd
Line 38 (original), 37 (patched)
<https://reviews.apache.org/r/61529/#comment261357>

    I only just realized this, but I think we should make the order:
    - launcher
    - job-xml
    - configuration
    
    The reasoning being that launcher is for the launcher job, while job-xml and configuration are (mostly) for the child job(s).  Plus, job-xml and configuration are effectively the same thing, while launcher is not.



client/src/main/resources/oozie-common-1.0.xsd
Line 21 (original), 21 (patched)
<https://reviews.apache.org/r/61529/#comment261354>

    I don't think we actually need this "launcher" element here.  The LAUNCHER type is defined below and used in the other XSDs.



client/src/main/resources/oozie-workflow-1.0.xsd
Line 234 (original), 234 (patched)
<https://reviews.apache.org/r/61529/#comment261356>

    Should we think about moving the Java, MR, Pig, etc actions out of the workflow XSD and into their own XSDs to be more consistent with the other action types?  The 1.0 XSD schemas is a good opportunity.
    
    The downside would be backwards compatibility where you can't simply update your workflow schema version on an existing workflow like you typically could in the past - you'd also have to add the xmlns string for these actions.  
    
    I think it would be okay to force people to add the xmlns.  No other changes should be needed so it's pretty minimal.  Plus, Oozie still accepts the 0.x schemas, so you don't *have* to upgrade the schemas to 1.0 anyway.
    
    Attila, Geza, what do you think?



client/src/main/resources/oozie-workflow-1.0.xsd
Lines 310-335 (original), 278-303 (patched)
<https://reviews.apache.org/r/61529/#comment261355>

    Should we move these into the common XSD too?  Perhaps in the future we'll support these in the prepare block.


- Robert Kanter


On Sept. 11, 2017, 10:50 p.m., Attila Sasvari wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61529/
> -----------------------------------------------------------
> 
> (Updated Sept. 11, 2017, 10:50 p.m.)
> 
> 
> Review request for oozie and Peter Cseh.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> -------
> 
> New workflow xsd to allow changing launcher configuration (e.g. setting java system properties)
> 
> 
> Diffs
> -----
> 
>   client/src/main/resources/hive-action-1.0.xsd PRE-CREATION 
>   client/src/main/resources/hive2-action-1.0.xsd PRE-CREATION 
>   client/src/main/resources/oozie-common-1.0.xsd PRE-CREATION 
>   client/src/main/resources/oozie-workflow-1.0.xsd PRE-CREATION 
>   client/src/main/resources/shell-action-1.0.xsd PRE-CREATION 
>   client/src/main/resources/spark-action-1.0.xsd PRE-CREATION 
>   client/src/main/resources/sqoop-action-1.0.xsd PRE-CREATION 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 49fd4b8560b21aef725c3268f0ec0e7f46a46ee9 
>   core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 8fdc50cce15271c13b20d1aaab2ffb95cb8fa711 
>   core/src/main/java/org/apache/oozie/action/ssh/SshActionExecutor.java 7e3348550e2ef44ae4bd6c5a7a417052bf9c108a 
>   core/src/main/java/org/apache/oozie/jms/MessageReceiver.java 47bfd2bcf9b53eed0882c51cdb5c530e0f6ccfe7 
>   core/src/main/java/org/apache/oozie/service/LiteWorkflowStoreService.java ffc29af2f834da6d0890ea7215c5a62fd7cd693e 
>   core/src/main/java/org/apache/oozie/service/SchemaService.java 137e2c0b4840c6d3858683db1aa38f54bd55be92 
>   core/src/main/java/org/apache/oozie/util/WritableUtils.java aa027e37ba23d4f481698e0bcd93c26c763a0b1f 
>   core/src/main/java/org/apache/oozie/util/schema/Input.java PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/schema/ResourceResolver.java PRE-CREATION 
>   core/src/main/java/org/apache/oozie/workflow/lite/LauncherConfigHandler.java PRE-CREATION 
>   core/src/main/java/org/apache/oozie/workflow/lite/LiteWorkflowAppParser.java a74e5c759fe3e336e7e98c61b8f5ac52efa4100c 
>   core/src/main/java/org/apache/oozie/workflow/lite/LiteWorkflowLib.java 23df0867e5984c518e8424d79365997bec28f619 
>   core/src/main/resources/oozie-default.xml 0d174b5069aff36ca3a9d48722c5bba69623a046 
>   core/src/test/java/org/apache/oozie/action/hadoop/ActionExecutorTestCase.java d74160a09ee237f4067e00ded526e7ec94528a6b 
>   core/src/test/java/org/apache/oozie/action/hadoop/LauncherMainTester.java 6cee7a8948ca7f2c7aaf44d4f172d18a1308a482 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java d1d78fd527a425b076edd7bb23ba56ebdba62664 
>   core/src/test/java/org/apache/oozie/service/TestSchemaService.java 88a10dbead32879334c3953a90a411a3a18fc0b3 
>   core/src/test/java/org/apache/oozie/util/TestMetricsInstrumentation.java a882c82b25154338f89e8a90f831b6f0300fbfe7 
>   core/src/test/java/org/apache/oozie/workflow/lite/TestLiteWorkflowAppParser.java 21332404bea79fe0a3cea045d57b3e52a1f15cae 
>   core/src/test/resources/wf-schema-global-launcherconf-override.xml PRE-CREATION 
>   core/src/test/resources/wf-schema-global-launcherconf.xml PRE-CREATION 
>   pom.xml db18f30814b9b6a73ba872c2cd7946692d0b876b 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java 6a98d6ef3a09a7a75272f3f0f9a9c2fc5472e76d 
>   sharelib/pig/src/test/java/org/apache/oozie/action/hadoop/UDFTester.java ff999ff2cd4875fbf3ad5e1a7eacf9972e848630 
> 
> 
> Diff: https://reviews.apache.org/r/61529/diff/9/
> 
> 
> Testing
> -------
> 
> - TestJavaActionExecutor, TestLiteWorkflowAppParser tests passed
> - java-main example on pseudo hadoop succeeded, modified workflow xml so that verbose class loading information is printed out of stdout of the launcher job
> 
> 
> Thanks,
> 
> Attila Sasvari
> 
>

Re: Review Request 61529: OOZIE-2687 Create XML schema for launcher configurations

Posted by Robert Kanter <rk...@cloudera.com>.


> On Sept. 14, 2017, 12:16 a.m., Robert Kanter wrote:
> > Ship It!

Oh, we need to update the docs.  That can be a followup JIRA.


- Robert


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61529/#review185352
-----------------------------------------------------------


On Sept. 13, 2017, 1:45 p.m., Attila Sasvari wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61529/
> -----------------------------------------------------------
> 
> (Updated Sept. 13, 2017, 1:45 p.m.)
> 
> 
> Review request for oozie and Peter Cseh.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> -------
> 
> New workflow xsd to allow changing launcher configuration (e.g. setting java system properties)
> 
> 
> Diffs
> -----
> 
>   client/src/main/resources/hive-action-1.0.xsd PRE-CREATION 
>   client/src/main/resources/hive2-action-1.0.xsd PRE-CREATION 
>   client/src/main/resources/oozie-common-1.0.xsd PRE-CREATION 
>   client/src/main/resources/oozie-workflow-1.0.xsd PRE-CREATION 
>   client/src/main/resources/shell-action-1.0.xsd PRE-CREATION 
>   client/src/main/resources/spark-action-1.0.xsd PRE-CREATION 
>   client/src/main/resources/sqoop-action-1.0.xsd PRE-CREATION 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 49fd4b8560b21aef725c3268f0ec0e7f46a46ee9 
>   core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 8fdc50cce15271c13b20d1aaab2ffb95cb8fa711 
>   core/src/main/java/org/apache/oozie/action/ssh/SshActionExecutor.java 7e3348550e2ef44ae4bd6c5a7a417052bf9c108a 
>   core/src/main/java/org/apache/oozie/jms/MessageReceiver.java 47bfd2bcf9b53eed0882c51cdb5c530e0f6ccfe7 
>   core/src/main/java/org/apache/oozie/service/LiteWorkflowStoreService.java ffc29af2f834da6d0890ea7215c5a62fd7cd693e 
>   core/src/main/java/org/apache/oozie/service/SchemaService.java 137e2c0b4840c6d3858683db1aa38f54bd55be92 
>   core/src/main/java/org/apache/oozie/util/WritableUtils.java aa027e37ba23d4f481698e0bcd93c26c763a0b1f 
>   core/src/main/java/org/apache/oozie/util/schema/Input.java PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/schema/ResourceResolver.java PRE-CREATION 
>   core/src/main/java/org/apache/oozie/workflow/lite/LauncherConfigHandler.java PRE-CREATION 
>   core/src/main/java/org/apache/oozie/workflow/lite/LiteWorkflowAppParser.java a74e5c759fe3e336e7e98c61b8f5ac52efa4100c 
>   core/src/main/java/org/apache/oozie/workflow/lite/LiteWorkflowLib.java 23df0867e5984c518e8424d79365997bec28f619 
>   core/src/main/resources/oozie-default.xml 0d174b5069aff36ca3a9d48722c5bba69623a046 
>   core/src/test/java/org/apache/oozie/action/hadoop/ActionExecutorTestCase.java d74160a09ee237f4067e00ded526e7ec94528a6b 
>   core/src/test/java/org/apache/oozie/action/hadoop/LauncherMainTester.java 6cee7a8948ca7f2c7aaf44d4f172d18a1308a482 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java d1d78fd527a425b076edd7bb23ba56ebdba62664 
>   core/src/test/java/org/apache/oozie/service/TestSchemaService.java 88a10dbead32879334c3953a90a411a3a18fc0b3 
>   core/src/test/java/org/apache/oozie/util/TestMetricsInstrumentation.java a882c82b25154338f89e8a90f831b6f0300fbfe7 
>   core/src/test/java/org/apache/oozie/workflow/lite/TestLiteWorkflowAppParser.java 21332404bea79fe0a3cea045d57b3e52a1f15cae 
>   core/src/test/resources/wf-schema-global-launcherconf-override.xml PRE-CREATION 
>   core/src/test/resources/wf-schema-global-launcherconf.xml PRE-CREATION 
>   pom.xml db18f30814b9b6a73ba872c2cd7946692d0b876b 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java 6a98d6ef3a09a7a75272f3f0f9a9c2fc5472e76d 
>   sharelib/pig/src/test/java/org/apache/oozie/action/hadoop/UDFTester.java ff999ff2cd4875fbf3ad5e1a7eacf9972e848630 
> 
> 
> Diff: https://reviews.apache.org/r/61529/diff/11/
> 
> 
> Testing
> -------
> 
> - TestJavaActionExecutor, TestLiteWorkflowAppParser tests passed
> - java-main example on pseudo hadoop succeeded, modified workflow xml so that verbose class loading information is printed out of stdout of the launcher job
> 
> 
> Thanks,
> 
> Attila Sasvari
> 
>