You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@oozie.apache.org by Attila Sasvari <as...@cloudera.com> on 2017/08/10 09:40:56 UTC
Re: Review Request 61529: OOZIE-2687 Create XML schema for launcher
configurations
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61529/
-----------------------------------------------------------
(Updated Aug. 10, 2017, 9:40 a.m.)
Review request for oozie and Peter Cseh.
Repository: oozie-git
Description
-------
New workflow xsd to allow changing launcher configuration (e.g. setting java system properties)
Diffs
-----
client/src/main/resources/hive-action-0.7.xsd PRE-CREATION
client/src/main/resources/hive2-action-0.3.xsd PRE-CREATION
client/src/main/resources/oozie-common-0.1.xsd PRE-CREATION
client/src/main/resources/oozie-workflow-0.6.xsd PRE-CREATION
client/src/main/resources/shell-action-0.4.xsd PRE-CREATION
client/src/main/resources/spark-action-0.3.xsd PRE-CREATION
client/src/main/resources/sqoop-action-0.5.xsd PRE-CREATION
core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java dc17950418ada9b4848ed5597702a29e7124ab38
core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 8fdc50cce15271c13b20d1aaab2ffb95cb8fa711
core/src/main/java/org/apache/oozie/service/LiteWorkflowStoreService.java ffc29af2f834da6d0890ea7215c5a62fd7cd693e
core/src/main/java/org/apache/oozie/util/WritableUtils.java aa027e37ba23d4f481698e0bcd93c26c763a0b1f
core/src/main/java/org/apache/oozie/workflow/lite/LiteWorkflowAppParser.java a74e5c759fe3e336e7e98c61b8f5ac52efa4100c
core/src/main/java/org/apache/oozie/workflow/lite/LiteWorkflowLib.java 23df0867e5984c518e8424d79365997bec28f619
core/src/main/resources/oozie-default.xml 0d174b5069aff36ca3a9d48722c5bba69623a046
core/src/test/java/org/apache/oozie/action/hadoop/ActionExecutorTestCase.java d74160a09ee237f4067e00ded526e7ec94528a6b
core/src/test/java/org/apache/oozie/action/hadoop/LauncherMainTester.java 6cee7a8948ca7f2c7aaf44d4f172d18a1308a482
core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java c51c64a18c74c73aa9907027992033a9975bf4d7
core/src/test/java/org/apache/oozie/workflow/lite/TestLiteWorkflowAppParser.java 21332404bea79fe0a3cea045d57b3e52a1f15cae
core/src/test/resources/wf-schema-global-launcherconf-override.xml PRE-CREATION
core/src/test/resources/wf-schema-global-launcherconf.xml PRE-CREATION
sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java 6a98d6ef3a09a7a75272f3f0f9a9c2fc5472e76d
Diff: https://reviews.apache.org/r/61529/diff/1/
Testing
-------
- TestJavaActionExecutor, TestLiteWorkflowAppParser tests passed
- java-main example on pseudo hadoop succeeded, modified workflow xml so that verbose class loading information is printed out of stdout of the launcher job
Thanks,
Attila Sasvari
Re: Review Request 61529: OOZIE-2687 Create XML schema for launcher
configurations
Posted by Peter Cseh <ge...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61529/#review184643
-----------------------------------------------------------
core/src/main/java/org/apache/oozie/workflow/lite/LauncherConfigHandler.java
Lines 59 (patched)
<https://reviews.apache.org/r/61529/#comment260821>
In oozie-default.xml you state that this is in Mb. Would it be better to call this memory.mb or just leave it like this?
- Peter Cseh
On Sept. 5, 2017, 3:52 p.m., Attila Sasvari wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61529/
> -----------------------------------------------------------
>
> (Updated Sept. 5, 2017, 3:52 p.m.)
>
>
> Review request for oozie and Peter Cseh.
>
>
> Repository: oozie-git
>
>
> Description
> -------
>
> New workflow xsd to allow changing launcher configuration (e.g. setting java system properties)
>
>
> Diffs
> -----
>
> client/src/main/resources/hive-action-1.0.xsd PRE-CREATION
> client/src/main/resources/hive2-action-1.0.xsd PRE-CREATION
> client/src/main/resources/oozie-common-1.0.xsd PRE-CREATION
> client/src/main/resources/oozie-workflow-1.0.xsd PRE-CREATION
> client/src/main/resources/shell-action-1.0.xsd PRE-CREATION
> client/src/main/resources/spark-action-1.0.xsd PRE-CREATION
> client/src/main/resources/sqoop-action-1.0.xsd PRE-CREATION
> core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java bca79aa052521ea4f6f16e76bd69f84fb16be790
> core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 8fdc50cce15271c13b20d1aaab2ffb95cb8fa711
> core/src/main/java/org/apache/oozie/action/ssh/SshActionExecutor.java 7e3348550e2ef44ae4bd6c5a7a417052bf9c108a
> core/src/main/java/org/apache/oozie/jms/MessageReceiver.java 47bfd2bcf9b53eed0882c51cdb5c530e0f6ccfe7
> core/src/main/java/org/apache/oozie/service/LiteWorkflowStoreService.java ffc29af2f834da6d0890ea7215c5a62fd7cd693e
> core/src/main/java/org/apache/oozie/service/SchemaService.java 137e2c0b4840c6d3858683db1aa38f54bd55be92
> core/src/main/java/org/apache/oozie/util/WritableUtils.java aa027e37ba23d4f481698e0bcd93c26c763a0b1f
> core/src/main/java/org/apache/oozie/util/schema/Input.java PRE-CREATION
> core/src/main/java/org/apache/oozie/util/schema/ResourceResolver.java PRE-CREATION
> core/src/main/java/org/apache/oozie/workflow/lite/LauncherConfigHandler.java PRE-CREATION
> core/src/main/java/org/apache/oozie/workflow/lite/LiteWorkflowAppParser.java a74e5c759fe3e336e7e98c61b8f5ac52efa4100c
> core/src/main/java/org/apache/oozie/workflow/lite/LiteWorkflowLib.java 23df0867e5984c518e8424d79365997bec28f619
> core/src/main/resources/oozie-default.xml 0d174b5069aff36ca3a9d48722c5bba69623a046
> core/src/test/java/org/apache/oozie/action/hadoop/ActionExecutorTestCase.java d74160a09ee237f4067e00ded526e7ec94528a6b
> core/src/test/java/org/apache/oozie/action/hadoop/LauncherMainTester.java 6cee7a8948ca7f2c7aaf44d4f172d18a1308a482
> core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java ce674adf9cffa20690083e6298875a6f9047a109
> core/src/test/java/org/apache/oozie/service/TestSchemaService.java 88a10dbead32879334c3953a90a411a3a18fc0b3
> core/src/test/java/org/apache/oozie/util/TestMetricsInstrumentation.java a882c82b25154338f89e8a90f831b6f0300fbfe7
> core/src/test/java/org/apache/oozie/workflow/lite/TestLiteWorkflowAppParser.java 21332404bea79fe0a3cea045d57b3e52a1f15cae
> core/src/test/resources/wf-schema-global-launcherconf-override.xml PRE-CREATION
> core/src/test/resources/wf-schema-global-launcherconf.xml PRE-CREATION
> release-log.txt e2311eab6392ca08ad855da887e391f3f14a7408
> sharelib/distcp/src/main/java/org/apache/oozie/action/hadoop/DistcpMain.java d17d5beb76754655720a0e29079daf676c507841
> sharelib/hive/src/main/java/org/apache/oozie/action/hadoop/HiveMain.java fb3fabf027a1dda537d2bd86d574abaf95a685db
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java 6a98d6ef3a09a7a75272f3f0f9a9c2fc5472e76d
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMain.java f1f52c6e876fae56c90979baef3d31a0f5ba2396
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/MapReduceMain.java 77aaa62e23185e6df053f01fd1c4db1f2e972f6b
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/PasswordMasker.java 6a73bf9b8b1d0c5cbb008594c75c9e779298a29d
> sharelib/oozie/src/test/java/org/apache/oozie/action/hadoop/TestLauncherMain.java a793df1330d10828101abf9107d565db7972a95f
> sharelib/pig/src/main/java/org/apache/oozie/action/hadoop/PigMain.java 13ee04f4482471b61ee363f9cb98f04436c9e580
> sharelib/pig/src/test/java/org/apache/oozie/action/hadoop/UDFTester.java ff999ff2cd4875fbf3ad5e1a7eacf9972e848630
> sharelib/spark/src/main/java/org/apache/oozie/action/hadoop/SparkMain.java 8aa8343cfcfa136aedef0bf9d53fb4a11800e07c
> sharelib/sqoop/src/main/java/org/apache/oozie/action/hadoop/SqoopMain.java e889ef9f857e8417353f28ccb8870f9d843a9d33
>
>
> Diff: https://reviews.apache.org/r/61529/diff/4/
>
>
> Testing
> -------
>
> - TestJavaActionExecutor, TestLiteWorkflowAppParser tests passed
> - java-main example on pseudo hadoop succeeded, modified workflow xml so that verbose class loading information is printed out of stdout of the launcher job
>
>
> Thanks,
>
> Attila Sasvari
>
>
Re: Review Request 61529: OOZIE-2687 Create XML schema for launcher
configurations
Posted by Peter Bacsko <pb...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61529/#review184649
-----------------------------------------------------------
client/src/main/resources/oozie-common-1.0.xsd
Lines 24 (patched)
<https://reviews.apache.org/r/61529/#comment260825>
Do we want a sequence here? As Robert mentioned, an xs:choice could be better.
core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
Lines 26 (patched)
<https://reviews.apache.org/r/61529/#comment260826>
Avoid *
core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
Lines 33 (patched)
<https://reviews.apache.org/r/61529/#comment260827>
Avoid *
core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
Lines 865 (patched)
<https://reviews.apache.org/r/61529/#comment260828>
Identation seems odd
core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
Lines 981 (patched)
<https://reviews.apache.org/r/61529/#comment260829>
This should not be logged on an INFO level
core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
Line 1238 (original), 1269 (patched)
<https://reviews.apache.org/r/61529/#comment260830>
In theory we don't have to handle the case when we have both java-opt and java-opts, because the schema validator should take care of it.
Still, I'd be more confident if we explicitly checked this condition (that is, one or the another). We can add a comment that it's just a safety net.
core/src/main/java/org/apache/oozie/util/WritableUtils.java
Lines 191 (patched)
<https://reviews.apache.org/r/61529/#comment260831>
I remember creating this method for serializing something, but later abandoned that approach.
If it's not necessary, please remote this,
core/src/main/java/org/apache/oozie/util/WritableUtils.java
Lines 245 (patched)
<https://reviews.apache.org/r/61529/#comment260832>
Same thing here.
- Peter Bacsko
On szept. 5, 2017, 3:52 du, Attila Sasvari wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61529/
> -----------------------------------------------------------
>
> (Updated szept. 5, 2017, 3:52 du)
>
>
> Review request for oozie and Peter Cseh.
>
>
> Repository: oozie-git
>
>
> Description
> -------
>
> New workflow xsd to allow changing launcher configuration (e.g. setting java system properties)
>
>
> Diffs
> -----
>
> client/src/main/resources/hive-action-1.0.xsd PRE-CREATION
> client/src/main/resources/hive2-action-1.0.xsd PRE-CREATION
> client/src/main/resources/oozie-common-1.0.xsd PRE-CREATION
> client/src/main/resources/oozie-workflow-1.0.xsd PRE-CREATION
> client/src/main/resources/shell-action-1.0.xsd PRE-CREATION
> client/src/main/resources/spark-action-1.0.xsd PRE-CREATION
> client/src/main/resources/sqoop-action-1.0.xsd PRE-CREATION
> core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java bca79aa052521ea4f6f16e76bd69f84fb16be790
> core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 8fdc50cce15271c13b20d1aaab2ffb95cb8fa711
> core/src/main/java/org/apache/oozie/action/ssh/SshActionExecutor.java 7e3348550e2ef44ae4bd6c5a7a417052bf9c108a
> core/src/main/java/org/apache/oozie/jms/MessageReceiver.java 47bfd2bcf9b53eed0882c51cdb5c530e0f6ccfe7
> core/src/main/java/org/apache/oozie/service/LiteWorkflowStoreService.java ffc29af2f834da6d0890ea7215c5a62fd7cd693e
> core/src/main/java/org/apache/oozie/service/SchemaService.java 137e2c0b4840c6d3858683db1aa38f54bd55be92
> core/src/main/java/org/apache/oozie/util/WritableUtils.java aa027e37ba23d4f481698e0bcd93c26c763a0b1f
> core/src/main/java/org/apache/oozie/util/schema/Input.java PRE-CREATION
> core/src/main/java/org/apache/oozie/util/schema/ResourceResolver.java PRE-CREATION
> core/src/main/java/org/apache/oozie/workflow/lite/LauncherConfigHandler.java PRE-CREATION
> core/src/main/java/org/apache/oozie/workflow/lite/LiteWorkflowAppParser.java a74e5c759fe3e336e7e98c61b8f5ac52efa4100c
> core/src/main/java/org/apache/oozie/workflow/lite/LiteWorkflowLib.java 23df0867e5984c518e8424d79365997bec28f619
> core/src/main/resources/oozie-default.xml 0d174b5069aff36ca3a9d48722c5bba69623a046
> core/src/test/java/org/apache/oozie/action/hadoop/ActionExecutorTestCase.java d74160a09ee237f4067e00ded526e7ec94528a6b
> core/src/test/java/org/apache/oozie/action/hadoop/LauncherMainTester.java 6cee7a8948ca7f2c7aaf44d4f172d18a1308a482
> core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java ce674adf9cffa20690083e6298875a6f9047a109
> core/src/test/java/org/apache/oozie/service/TestSchemaService.java 88a10dbead32879334c3953a90a411a3a18fc0b3
> core/src/test/java/org/apache/oozie/util/TestMetricsInstrumentation.java a882c82b25154338f89e8a90f831b6f0300fbfe7
> core/src/test/java/org/apache/oozie/workflow/lite/TestLiteWorkflowAppParser.java 21332404bea79fe0a3cea045d57b3e52a1f15cae
> core/src/test/resources/wf-schema-global-launcherconf-override.xml PRE-CREATION
> core/src/test/resources/wf-schema-global-launcherconf.xml PRE-CREATION
> release-log.txt e2311eab6392ca08ad855da887e391f3f14a7408
> sharelib/distcp/src/main/java/org/apache/oozie/action/hadoop/DistcpMain.java d17d5beb76754655720a0e29079daf676c507841
> sharelib/hive/src/main/java/org/apache/oozie/action/hadoop/HiveMain.java fb3fabf027a1dda537d2bd86d574abaf95a685db
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java 6a98d6ef3a09a7a75272f3f0f9a9c2fc5472e76d
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMain.java f1f52c6e876fae56c90979baef3d31a0f5ba2396
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/MapReduceMain.java 77aaa62e23185e6df053f01fd1c4db1f2e972f6b
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/PasswordMasker.java 6a73bf9b8b1d0c5cbb008594c75c9e779298a29d
> sharelib/oozie/src/test/java/org/apache/oozie/action/hadoop/TestLauncherMain.java a793df1330d10828101abf9107d565db7972a95f
> sharelib/pig/src/main/java/org/apache/oozie/action/hadoop/PigMain.java 13ee04f4482471b61ee363f9cb98f04436c9e580
> sharelib/pig/src/test/java/org/apache/oozie/action/hadoop/UDFTester.java ff999ff2cd4875fbf3ad5e1a7eacf9972e848630
> sharelib/spark/src/main/java/org/apache/oozie/action/hadoop/SparkMain.java 8aa8343cfcfa136aedef0bf9d53fb4a11800e07c
> sharelib/sqoop/src/main/java/org/apache/oozie/action/hadoop/SqoopMain.java e889ef9f857e8417353f28ccb8870f9d843a9d33
>
>
> Diff: https://reviews.apache.org/r/61529/diff/4/
>
>
> Testing
> -------
>
> - TestJavaActionExecutor, TestLiteWorkflowAppParser tests passed
> - java-main example on pseudo hadoop succeeded, modified workflow xml so that verbose class loading information is printed out of stdout of the launcher job
>
>
> Thanks,
>
> Attila Sasvari
>
>
Re: Review Request 61529: OOZIE-2687 Create XML schema for launcher
configurations
Posted by Peter Cseh <ge...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61529/#review184668
-----------------------------------------------------------
core/src/test/java/org/apache/oozie/service/TestSchemaService.java
Lines 143-148 (patched)
<https://reviews.apache.org/r/61529/#comment260860>
I don't like the fact that we have to use the workflow: prefix here for launcher properties. If anything I'd use the launcher: prefix, but ideally there should be no need for prefixes.
- Peter Cseh
On Sept. 5, 2017, 3:52 p.m., Attila Sasvari wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61529/
> -----------------------------------------------------------
>
> (Updated Sept. 5, 2017, 3:52 p.m.)
>
>
> Review request for oozie and Peter Cseh.
>
>
> Repository: oozie-git
>
>
> Description
> -------
>
> New workflow xsd to allow changing launcher configuration (e.g. setting java system properties)
>
>
> Diffs
> -----
>
> client/src/main/resources/hive-action-1.0.xsd PRE-CREATION
> client/src/main/resources/hive2-action-1.0.xsd PRE-CREATION
> client/src/main/resources/oozie-common-1.0.xsd PRE-CREATION
> client/src/main/resources/oozie-workflow-1.0.xsd PRE-CREATION
> client/src/main/resources/shell-action-1.0.xsd PRE-CREATION
> client/src/main/resources/spark-action-1.0.xsd PRE-CREATION
> client/src/main/resources/sqoop-action-1.0.xsd PRE-CREATION
> core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java bca79aa052521ea4f6f16e76bd69f84fb16be790
> core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 8fdc50cce15271c13b20d1aaab2ffb95cb8fa711
> core/src/main/java/org/apache/oozie/action/ssh/SshActionExecutor.java 7e3348550e2ef44ae4bd6c5a7a417052bf9c108a
> core/src/main/java/org/apache/oozie/jms/MessageReceiver.java 47bfd2bcf9b53eed0882c51cdb5c530e0f6ccfe7
> core/src/main/java/org/apache/oozie/service/LiteWorkflowStoreService.java ffc29af2f834da6d0890ea7215c5a62fd7cd693e
> core/src/main/java/org/apache/oozie/service/SchemaService.java 137e2c0b4840c6d3858683db1aa38f54bd55be92
> core/src/main/java/org/apache/oozie/util/WritableUtils.java aa027e37ba23d4f481698e0bcd93c26c763a0b1f
> core/src/main/java/org/apache/oozie/util/schema/Input.java PRE-CREATION
> core/src/main/java/org/apache/oozie/util/schema/ResourceResolver.java PRE-CREATION
> core/src/main/java/org/apache/oozie/workflow/lite/LauncherConfigHandler.java PRE-CREATION
> core/src/main/java/org/apache/oozie/workflow/lite/LiteWorkflowAppParser.java a74e5c759fe3e336e7e98c61b8f5ac52efa4100c
> core/src/main/java/org/apache/oozie/workflow/lite/LiteWorkflowLib.java 23df0867e5984c518e8424d79365997bec28f619
> core/src/main/resources/oozie-default.xml 0d174b5069aff36ca3a9d48722c5bba69623a046
> core/src/test/java/org/apache/oozie/action/hadoop/ActionExecutorTestCase.java d74160a09ee237f4067e00ded526e7ec94528a6b
> core/src/test/java/org/apache/oozie/action/hadoop/LauncherMainTester.java 6cee7a8948ca7f2c7aaf44d4f172d18a1308a482
> core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java ce674adf9cffa20690083e6298875a6f9047a109
> core/src/test/java/org/apache/oozie/service/TestSchemaService.java 88a10dbead32879334c3953a90a411a3a18fc0b3
> core/src/test/java/org/apache/oozie/util/TestMetricsInstrumentation.java a882c82b25154338f89e8a90f831b6f0300fbfe7
> core/src/test/java/org/apache/oozie/workflow/lite/TestLiteWorkflowAppParser.java 21332404bea79fe0a3cea045d57b3e52a1f15cae
> core/src/test/resources/wf-schema-global-launcherconf-override.xml PRE-CREATION
> core/src/test/resources/wf-schema-global-launcherconf.xml PRE-CREATION
> release-log.txt e2311eab6392ca08ad855da887e391f3f14a7408
> sharelib/distcp/src/main/java/org/apache/oozie/action/hadoop/DistcpMain.java d17d5beb76754655720a0e29079daf676c507841
> sharelib/hive/src/main/java/org/apache/oozie/action/hadoop/HiveMain.java fb3fabf027a1dda537d2bd86d574abaf95a685db
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java 6a98d6ef3a09a7a75272f3f0f9a9c2fc5472e76d
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMain.java f1f52c6e876fae56c90979baef3d31a0f5ba2396
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/MapReduceMain.java 77aaa62e23185e6df053f01fd1c4db1f2e972f6b
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/PasswordMasker.java 6a73bf9b8b1d0c5cbb008594c75c9e779298a29d
> sharelib/oozie/src/test/java/org/apache/oozie/action/hadoop/TestLauncherMain.java a793df1330d10828101abf9107d565db7972a95f
> sharelib/pig/src/main/java/org/apache/oozie/action/hadoop/PigMain.java 13ee04f4482471b61ee363f9cb98f04436c9e580
> sharelib/pig/src/test/java/org/apache/oozie/action/hadoop/UDFTester.java ff999ff2cd4875fbf3ad5e1a7eacf9972e848630
> sharelib/spark/src/main/java/org/apache/oozie/action/hadoop/SparkMain.java 8aa8343cfcfa136aedef0bf9d53fb4a11800e07c
> sharelib/sqoop/src/main/java/org/apache/oozie/action/hadoop/SqoopMain.java e889ef9f857e8417353f28ccb8870f9d843a9d33
>
>
> Diff: https://reviews.apache.org/r/61529/diff/4/
>
>
> Testing
> -------
>
> - TestJavaActionExecutor, TestLiteWorkflowAppParser tests passed
> - java-main example on pseudo hadoop succeeded, modified workflow xml so that verbose class loading information is printed out of stdout of the launcher job
>
>
> Thanks,
>
> Attila Sasvari
>
>
Re: Review Request 61529: OOZIE-2687 Create XML schema for launcher
configurations
Posted by Peter Bacsko <pb...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61529/#review184655
-----------------------------------------------------------
core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
Lines 1296 (patched)
<https://reviews.apache.org/r/61529/#comment260844>
Unnecessary comment
core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
Lines 1363 (patched)
<https://reviews.apache.org/r/61529/#comment260846>
I haven't seen a testcase which covers this path, would be useful to have one
core/src/main/java/org/apache/oozie/action/ssh/SshActionExecutor.java
Line 179 (original), 179 (patched)
<https://reviews.apache.org/r/61529/#comment260847>
Could you explain what this rename is about? Now We have "processSettings" instead of "processw everywhere.
core/src/main/resources/oozie-default.xml
Lines 3087 (patched)
<https://reviews.apache.org/r/61529/#comment260843>
Perhaps this property could be called
oozie.launcher.default.memory.mb
so users won't accidentally define it in different units.
- Peter Bacsko
On szept. 5, 2017, 3:52 du, Attila Sasvari wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61529/
> -----------------------------------------------------------
>
> (Updated szept. 5, 2017, 3:52 du)
>
>
> Review request for oozie and Peter Cseh.
>
>
> Repository: oozie-git
>
>
> Description
> -------
>
> New workflow xsd to allow changing launcher configuration (e.g. setting java system properties)
>
>
> Diffs
> -----
>
> client/src/main/resources/hive-action-1.0.xsd PRE-CREATION
> client/src/main/resources/hive2-action-1.0.xsd PRE-CREATION
> client/src/main/resources/oozie-common-1.0.xsd PRE-CREATION
> client/src/main/resources/oozie-workflow-1.0.xsd PRE-CREATION
> client/src/main/resources/shell-action-1.0.xsd PRE-CREATION
> client/src/main/resources/spark-action-1.0.xsd PRE-CREATION
> client/src/main/resources/sqoop-action-1.0.xsd PRE-CREATION
> core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java bca79aa052521ea4f6f16e76bd69f84fb16be790
> core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 8fdc50cce15271c13b20d1aaab2ffb95cb8fa711
> core/src/main/java/org/apache/oozie/action/ssh/SshActionExecutor.java 7e3348550e2ef44ae4bd6c5a7a417052bf9c108a
> core/src/main/java/org/apache/oozie/jms/MessageReceiver.java 47bfd2bcf9b53eed0882c51cdb5c530e0f6ccfe7
> core/src/main/java/org/apache/oozie/service/LiteWorkflowStoreService.java ffc29af2f834da6d0890ea7215c5a62fd7cd693e
> core/src/main/java/org/apache/oozie/service/SchemaService.java 137e2c0b4840c6d3858683db1aa38f54bd55be92
> core/src/main/java/org/apache/oozie/util/WritableUtils.java aa027e37ba23d4f481698e0bcd93c26c763a0b1f
> core/src/main/java/org/apache/oozie/util/schema/Input.java PRE-CREATION
> core/src/main/java/org/apache/oozie/util/schema/ResourceResolver.java PRE-CREATION
> core/src/main/java/org/apache/oozie/workflow/lite/LauncherConfigHandler.java PRE-CREATION
> core/src/main/java/org/apache/oozie/workflow/lite/LiteWorkflowAppParser.java a74e5c759fe3e336e7e98c61b8f5ac52efa4100c
> core/src/main/java/org/apache/oozie/workflow/lite/LiteWorkflowLib.java 23df0867e5984c518e8424d79365997bec28f619
> core/src/main/resources/oozie-default.xml 0d174b5069aff36ca3a9d48722c5bba69623a046
> core/src/test/java/org/apache/oozie/action/hadoop/ActionExecutorTestCase.java d74160a09ee237f4067e00ded526e7ec94528a6b
> core/src/test/java/org/apache/oozie/action/hadoop/LauncherMainTester.java 6cee7a8948ca7f2c7aaf44d4f172d18a1308a482
> core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java ce674adf9cffa20690083e6298875a6f9047a109
> core/src/test/java/org/apache/oozie/service/TestSchemaService.java 88a10dbead32879334c3953a90a411a3a18fc0b3
> core/src/test/java/org/apache/oozie/util/TestMetricsInstrumentation.java a882c82b25154338f89e8a90f831b6f0300fbfe7
> core/src/test/java/org/apache/oozie/workflow/lite/TestLiteWorkflowAppParser.java 21332404bea79fe0a3cea045d57b3e52a1f15cae
> core/src/test/resources/wf-schema-global-launcherconf-override.xml PRE-CREATION
> core/src/test/resources/wf-schema-global-launcherconf.xml PRE-CREATION
> release-log.txt e2311eab6392ca08ad855da887e391f3f14a7408
> sharelib/distcp/src/main/java/org/apache/oozie/action/hadoop/DistcpMain.java d17d5beb76754655720a0e29079daf676c507841
> sharelib/hive/src/main/java/org/apache/oozie/action/hadoop/HiveMain.java fb3fabf027a1dda537d2bd86d574abaf95a685db
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java 6a98d6ef3a09a7a75272f3f0f9a9c2fc5472e76d
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMain.java f1f52c6e876fae56c90979baef3d31a0f5ba2396
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/MapReduceMain.java 77aaa62e23185e6df053f01fd1c4db1f2e972f6b
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/PasswordMasker.java 6a73bf9b8b1d0c5cbb008594c75c9e779298a29d
> sharelib/oozie/src/test/java/org/apache/oozie/action/hadoop/TestLauncherMain.java a793df1330d10828101abf9107d565db7972a95f
> sharelib/pig/src/main/java/org/apache/oozie/action/hadoop/PigMain.java 13ee04f4482471b61ee363f9cb98f04436c9e580
> sharelib/pig/src/test/java/org/apache/oozie/action/hadoop/UDFTester.java ff999ff2cd4875fbf3ad5e1a7eacf9972e848630
> sharelib/spark/src/main/java/org/apache/oozie/action/hadoop/SparkMain.java 8aa8343cfcfa136aedef0bf9d53fb4a11800e07c
> sharelib/sqoop/src/main/java/org/apache/oozie/action/hadoop/SqoopMain.java e889ef9f857e8417353f28ccb8870f9d843a9d33
>
>
> Diff: https://reviews.apache.org/r/61529/diff/4/
>
>
> Testing
> -------
>
> - TestJavaActionExecutor, TestLiteWorkflowAppParser tests passed
> - java-main example on pseudo hadoop succeeded, modified workflow xml so that verbose class loading information is printed out of stdout of the launcher job
>
>
> Thanks,
>
> Attila Sasvari
>
>
Re: Review Request 61529: OOZIE-2687 Create XML schema for launcher
configurations
Posted by Peter Bacsko <pb...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61529/#review184650
-----------------------------------------------------------
core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java
Lines 2268 (patched)
<https://reviews.apache.org/r/61529/#comment260833>
Have you considered creating new tests?
Is this coverage enough?
core/src/test/java/org/apache/oozie/service/TestSchemaService.java
Lines 101 (patched)
<https://reviews.apache.org/r/61529/#comment260834>
Wouldn't it be better to store all these dummy workflows in an external file?
- Peter Bacsko
On szept. 5, 2017, 3:52 du, Attila Sasvari wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61529/
> -----------------------------------------------------------
>
> (Updated szept. 5, 2017, 3:52 du)
>
>
> Review request for oozie and Peter Cseh.
>
>
> Repository: oozie-git
>
>
> Description
> -------
>
> New workflow xsd to allow changing launcher configuration (e.g. setting java system properties)
>
>
> Diffs
> -----
>
> client/src/main/resources/hive-action-1.0.xsd PRE-CREATION
> client/src/main/resources/hive2-action-1.0.xsd PRE-CREATION
> client/src/main/resources/oozie-common-1.0.xsd PRE-CREATION
> client/src/main/resources/oozie-workflow-1.0.xsd PRE-CREATION
> client/src/main/resources/shell-action-1.0.xsd PRE-CREATION
> client/src/main/resources/spark-action-1.0.xsd PRE-CREATION
> client/src/main/resources/sqoop-action-1.0.xsd PRE-CREATION
> core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java bca79aa052521ea4f6f16e76bd69f84fb16be790
> core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 8fdc50cce15271c13b20d1aaab2ffb95cb8fa711
> core/src/main/java/org/apache/oozie/action/ssh/SshActionExecutor.java 7e3348550e2ef44ae4bd6c5a7a417052bf9c108a
> core/src/main/java/org/apache/oozie/jms/MessageReceiver.java 47bfd2bcf9b53eed0882c51cdb5c530e0f6ccfe7
> core/src/main/java/org/apache/oozie/service/LiteWorkflowStoreService.java ffc29af2f834da6d0890ea7215c5a62fd7cd693e
> core/src/main/java/org/apache/oozie/service/SchemaService.java 137e2c0b4840c6d3858683db1aa38f54bd55be92
> core/src/main/java/org/apache/oozie/util/WritableUtils.java aa027e37ba23d4f481698e0bcd93c26c763a0b1f
> core/src/main/java/org/apache/oozie/util/schema/Input.java PRE-CREATION
> core/src/main/java/org/apache/oozie/util/schema/ResourceResolver.java PRE-CREATION
> core/src/main/java/org/apache/oozie/workflow/lite/LauncherConfigHandler.java PRE-CREATION
> core/src/main/java/org/apache/oozie/workflow/lite/LiteWorkflowAppParser.java a74e5c759fe3e336e7e98c61b8f5ac52efa4100c
> core/src/main/java/org/apache/oozie/workflow/lite/LiteWorkflowLib.java 23df0867e5984c518e8424d79365997bec28f619
> core/src/main/resources/oozie-default.xml 0d174b5069aff36ca3a9d48722c5bba69623a046
> core/src/test/java/org/apache/oozie/action/hadoop/ActionExecutorTestCase.java d74160a09ee237f4067e00ded526e7ec94528a6b
> core/src/test/java/org/apache/oozie/action/hadoop/LauncherMainTester.java 6cee7a8948ca7f2c7aaf44d4f172d18a1308a482
> core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java ce674adf9cffa20690083e6298875a6f9047a109
> core/src/test/java/org/apache/oozie/service/TestSchemaService.java 88a10dbead32879334c3953a90a411a3a18fc0b3
> core/src/test/java/org/apache/oozie/util/TestMetricsInstrumentation.java a882c82b25154338f89e8a90f831b6f0300fbfe7
> core/src/test/java/org/apache/oozie/workflow/lite/TestLiteWorkflowAppParser.java 21332404bea79fe0a3cea045d57b3e52a1f15cae
> core/src/test/resources/wf-schema-global-launcherconf-override.xml PRE-CREATION
> core/src/test/resources/wf-schema-global-launcherconf.xml PRE-CREATION
> release-log.txt e2311eab6392ca08ad855da887e391f3f14a7408
> sharelib/distcp/src/main/java/org/apache/oozie/action/hadoop/DistcpMain.java d17d5beb76754655720a0e29079daf676c507841
> sharelib/hive/src/main/java/org/apache/oozie/action/hadoop/HiveMain.java fb3fabf027a1dda537d2bd86d574abaf95a685db
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java 6a98d6ef3a09a7a75272f3f0f9a9c2fc5472e76d
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMain.java f1f52c6e876fae56c90979baef3d31a0f5ba2396
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/MapReduceMain.java 77aaa62e23185e6df053f01fd1c4db1f2e972f6b
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/PasswordMasker.java 6a73bf9b8b1d0c5cbb008594c75c9e779298a29d
> sharelib/oozie/src/test/java/org/apache/oozie/action/hadoop/TestLauncherMain.java a793df1330d10828101abf9107d565db7972a95f
> sharelib/pig/src/main/java/org/apache/oozie/action/hadoop/PigMain.java 13ee04f4482471b61ee363f9cb98f04436c9e580
> sharelib/pig/src/test/java/org/apache/oozie/action/hadoop/UDFTester.java ff999ff2cd4875fbf3ad5e1a7eacf9972e848630
> sharelib/spark/src/main/java/org/apache/oozie/action/hadoop/SparkMain.java 8aa8343cfcfa136aedef0bf9d53fb4a11800e07c
> sharelib/sqoop/src/main/java/org/apache/oozie/action/hadoop/SqoopMain.java e889ef9f857e8417353f28ccb8870f9d843a9d33
>
>
> Diff: https://reviews.apache.org/r/61529/diff/4/
>
>
> Testing
> -------
>
> - TestJavaActionExecutor, TestLiteWorkflowAppParser tests passed
> - java-main example on pseudo hadoop succeeded, modified workflow xml so that verbose class loading information is printed out of stdout of the launcher job
>
>
> Thanks,
>
> Attila Sasvari
>
>
Re: Review Request 61529: OOZIE-2687 Create XML schema for launcher
configurations
Posted by Peter Cseh <ge...@cloudera.com>.
> On Sept. 11, 2017, 11:36 p.m., Robert Kanter wrote:
> > client/src/main/resources/oozie-workflow-1.0.xsd
> > Line 234 (original), 234 (patched)
> > <https://reviews.apache.org/r/61529/diff/8-9/?file=1818281#file1818281line234>
> >
> > Should we think about moving the Java, MR, Pig, etc actions out of the workflow XSD and into their own XSDs to be more consistent with the other action types? The 1.0 XSD schemas is a good opportunity.
> >
> > The downside would be backwards compatibility where you can't simply update your workflow schema version on an existing workflow like you typically could in the past - you'd also have to add the xmlns string for these actions.
> >
> > I think it would be okay to force people to add the xmlns. No other changes should be needed so it's pretty minimal. Plus, Oozie still accepts the 0.x schemas, so you don't *have* to upgrade the schemas to 1.0 anyway.
> >
> > Attila, Geza, what do you think?
>
> Attila Sasvari wrote:
> For this purpose I created https://issues.apache.org/jira/browse/OOZIE-3033 (this patch is already quite big).
As this is a breaking change I'd rather fix this in a later jira. I think we can get away using 1.0 as a version and doing big changes like that in follow-up steps.
- Peter
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61529/#review185128
-----------------------------------------------------------
On Sept. 13, 2017, 1:45 p.m., Attila Sasvari wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61529/
> -----------------------------------------------------------
>
> (Updated Sept. 13, 2017, 1:45 p.m.)
>
>
> Review request for oozie and Peter Cseh.
>
>
> Repository: oozie-git
>
>
> Description
> -------
>
> New workflow xsd to allow changing launcher configuration (e.g. setting java system properties)
>
>
> Diffs
> -----
>
> client/src/main/resources/hive-action-1.0.xsd PRE-CREATION
> client/src/main/resources/hive2-action-1.0.xsd PRE-CREATION
> client/src/main/resources/oozie-common-1.0.xsd PRE-CREATION
> client/src/main/resources/oozie-workflow-1.0.xsd PRE-CREATION
> client/src/main/resources/shell-action-1.0.xsd PRE-CREATION
> client/src/main/resources/spark-action-1.0.xsd PRE-CREATION
> client/src/main/resources/sqoop-action-1.0.xsd PRE-CREATION
> core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 49fd4b8560b21aef725c3268f0ec0e7f46a46ee9
> core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 8fdc50cce15271c13b20d1aaab2ffb95cb8fa711
> core/src/main/java/org/apache/oozie/action/ssh/SshActionExecutor.java 7e3348550e2ef44ae4bd6c5a7a417052bf9c108a
> core/src/main/java/org/apache/oozie/jms/MessageReceiver.java 47bfd2bcf9b53eed0882c51cdb5c530e0f6ccfe7
> core/src/main/java/org/apache/oozie/service/LiteWorkflowStoreService.java ffc29af2f834da6d0890ea7215c5a62fd7cd693e
> core/src/main/java/org/apache/oozie/service/SchemaService.java 137e2c0b4840c6d3858683db1aa38f54bd55be92
> core/src/main/java/org/apache/oozie/util/WritableUtils.java aa027e37ba23d4f481698e0bcd93c26c763a0b1f
> core/src/main/java/org/apache/oozie/util/schema/Input.java PRE-CREATION
> core/src/main/java/org/apache/oozie/util/schema/ResourceResolver.java PRE-CREATION
> core/src/main/java/org/apache/oozie/workflow/lite/LauncherConfigHandler.java PRE-CREATION
> core/src/main/java/org/apache/oozie/workflow/lite/LiteWorkflowAppParser.java a74e5c759fe3e336e7e98c61b8f5ac52efa4100c
> core/src/main/java/org/apache/oozie/workflow/lite/LiteWorkflowLib.java 23df0867e5984c518e8424d79365997bec28f619
> core/src/main/resources/oozie-default.xml 0d174b5069aff36ca3a9d48722c5bba69623a046
> core/src/test/java/org/apache/oozie/action/hadoop/ActionExecutorTestCase.java d74160a09ee237f4067e00ded526e7ec94528a6b
> core/src/test/java/org/apache/oozie/action/hadoop/LauncherMainTester.java 6cee7a8948ca7f2c7aaf44d4f172d18a1308a482
> core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java d1d78fd527a425b076edd7bb23ba56ebdba62664
> core/src/test/java/org/apache/oozie/service/TestSchemaService.java 88a10dbead32879334c3953a90a411a3a18fc0b3
> core/src/test/java/org/apache/oozie/util/TestMetricsInstrumentation.java a882c82b25154338f89e8a90f831b6f0300fbfe7
> core/src/test/java/org/apache/oozie/workflow/lite/TestLiteWorkflowAppParser.java 21332404bea79fe0a3cea045d57b3e52a1f15cae
> core/src/test/resources/wf-schema-global-launcherconf-override.xml PRE-CREATION
> core/src/test/resources/wf-schema-global-launcherconf.xml PRE-CREATION
> pom.xml db18f30814b9b6a73ba872c2cd7946692d0b876b
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java 6a98d6ef3a09a7a75272f3f0f9a9c2fc5472e76d
> sharelib/pig/src/test/java/org/apache/oozie/action/hadoop/UDFTester.java ff999ff2cd4875fbf3ad5e1a7eacf9972e848630
>
>
> Diff: https://reviews.apache.org/r/61529/diff/11/
>
>
> Testing
> -------
>
> - TestJavaActionExecutor, TestLiteWorkflowAppParser tests passed
> - java-main example on pseudo hadoop succeeded, modified workflow xml so that verbose class loading information is printed out of stdout of the launcher job
>
>
> Thanks,
>
> Attila Sasvari
>
>
Re: Review Request 61529: OOZIE-2687 Create XML schema for launcher
configurations
Posted by Robert Kanter <rk...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61529/#review185128
-----------------------------------------------------------
client/src/main/resources/hive-action-1.0.xsd
Line 38 (original), 37 (patched)
<https://reviews.apache.org/r/61529/#comment261357>
I only just realized this, but I think we should make the order:
- launcher
- job-xml
- configuration
The reasoning being that launcher is for the launcher job, while job-xml and configuration are (mostly) for the child job(s). Plus, job-xml and configuration are effectively the same thing, while launcher is not.
client/src/main/resources/oozie-common-1.0.xsd
Line 21 (original), 21 (patched)
<https://reviews.apache.org/r/61529/#comment261354>
I don't think we actually need this "launcher" element here. The LAUNCHER type is defined below and used in the other XSDs.
client/src/main/resources/oozie-workflow-1.0.xsd
Line 234 (original), 234 (patched)
<https://reviews.apache.org/r/61529/#comment261356>
Should we think about moving the Java, MR, Pig, etc actions out of the workflow XSD and into their own XSDs to be more consistent with the other action types? The 1.0 XSD schemas is a good opportunity.
The downside would be backwards compatibility where you can't simply update your workflow schema version on an existing workflow like you typically could in the past - you'd also have to add the xmlns string for these actions.
I think it would be okay to force people to add the xmlns. No other changes should be needed so it's pretty minimal. Plus, Oozie still accepts the 0.x schemas, so you don't *have* to upgrade the schemas to 1.0 anyway.
Attila, Geza, what do you think?
client/src/main/resources/oozie-workflow-1.0.xsd
Lines 310-335 (original), 278-303 (patched)
<https://reviews.apache.org/r/61529/#comment261355>
Should we move these into the common XSD too? Perhaps in the future we'll support these in the prepare block.
- Robert Kanter
On Sept. 11, 2017, 10:50 p.m., Attila Sasvari wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61529/
> -----------------------------------------------------------
>
> (Updated Sept. 11, 2017, 10:50 p.m.)
>
>
> Review request for oozie and Peter Cseh.
>
>
> Repository: oozie-git
>
>
> Description
> -------
>
> New workflow xsd to allow changing launcher configuration (e.g. setting java system properties)
>
>
> Diffs
> -----
>
> client/src/main/resources/hive-action-1.0.xsd PRE-CREATION
> client/src/main/resources/hive2-action-1.0.xsd PRE-CREATION
> client/src/main/resources/oozie-common-1.0.xsd PRE-CREATION
> client/src/main/resources/oozie-workflow-1.0.xsd PRE-CREATION
> client/src/main/resources/shell-action-1.0.xsd PRE-CREATION
> client/src/main/resources/spark-action-1.0.xsd PRE-CREATION
> client/src/main/resources/sqoop-action-1.0.xsd PRE-CREATION
> core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 49fd4b8560b21aef725c3268f0ec0e7f46a46ee9
> core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 8fdc50cce15271c13b20d1aaab2ffb95cb8fa711
> core/src/main/java/org/apache/oozie/action/ssh/SshActionExecutor.java 7e3348550e2ef44ae4bd6c5a7a417052bf9c108a
> core/src/main/java/org/apache/oozie/jms/MessageReceiver.java 47bfd2bcf9b53eed0882c51cdb5c530e0f6ccfe7
> core/src/main/java/org/apache/oozie/service/LiteWorkflowStoreService.java ffc29af2f834da6d0890ea7215c5a62fd7cd693e
> core/src/main/java/org/apache/oozie/service/SchemaService.java 137e2c0b4840c6d3858683db1aa38f54bd55be92
> core/src/main/java/org/apache/oozie/util/WritableUtils.java aa027e37ba23d4f481698e0bcd93c26c763a0b1f
> core/src/main/java/org/apache/oozie/util/schema/Input.java PRE-CREATION
> core/src/main/java/org/apache/oozie/util/schema/ResourceResolver.java PRE-CREATION
> core/src/main/java/org/apache/oozie/workflow/lite/LauncherConfigHandler.java PRE-CREATION
> core/src/main/java/org/apache/oozie/workflow/lite/LiteWorkflowAppParser.java a74e5c759fe3e336e7e98c61b8f5ac52efa4100c
> core/src/main/java/org/apache/oozie/workflow/lite/LiteWorkflowLib.java 23df0867e5984c518e8424d79365997bec28f619
> core/src/main/resources/oozie-default.xml 0d174b5069aff36ca3a9d48722c5bba69623a046
> core/src/test/java/org/apache/oozie/action/hadoop/ActionExecutorTestCase.java d74160a09ee237f4067e00ded526e7ec94528a6b
> core/src/test/java/org/apache/oozie/action/hadoop/LauncherMainTester.java 6cee7a8948ca7f2c7aaf44d4f172d18a1308a482
> core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java d1d78fd527a425b076edd7bb23ba56ebdba62664
> core/src/test/java/org/apache/oozie/service/TestSchemaService.java 88a10dbead32879334c3953a90a411a3a18fc0b3
> core/src/test/java/org/apache/oozie/util/TestMetricsInstrumentation.java a882c82b25154338f89e8a90f831b6f0300fbfe7
> core/src/test/java/org/apache/oozie/workflow/lite/TestLiteWorkflowAppParser.java 21332404bea79fe0a3cea045d57b3e52a1f15cae
> core/src/test/resources/wf-schema-global-launcherconf-override.xml PRE-CREATION
> core/src/test/resources/wf-schema-global-launcherconf.xml PRE-CREATION
> pom.xml db18f30814b9b6a73ba872c2cd7946692d0b876b
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java 6a98d6ef3a09a7a75272f3f0f9a9c2fc5472e76d
> sharelib/pig/src/test/java/org/apache/oozie/action/hadoop/UDFTester.java ff999ff2cd4875fbf3ad5e1a7eacf9972e848630
>
>
> Diff: https://reviews.apache.org/r/61529/diff/9/
>
>
> Testing
> -------
>
> - TestJavaActionExecutor, TestLiteWorkflowAppParser tests passed
> - java-main example on pseudo hadoop succeeded, modified workflow xml so that verbose class loading information is printed out of stdout of the launcher job
>
>
> Thanks,
>
> Attila Sasvari
>
>
Re: Review Request 61529: OOZIE-2687 Create XML schema for launcher
configurations
Posted by Robert Kanter <rk...@cloudera.com>.
> On Sept. 14, 2017, 12:16 a.m., Robert Kanter wrote:
> > Ship It!
Oh, we need to update the docs. That can be a followup JIRA.
- Robert
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61529/#review185352
-----------------------------------------------------------
On Sept. 13, 2017, 1:45 p.m., Attila Sasvari wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61529/
> -----------------------------------------------------------
>
> (Updated Sept. 13, 2017, 1:45 p.m.)
>
>
> Review request for oozie and Peter Cseh.
>
>
> Repository: oozie-git
>
>
> Description
> -------
>
> New workflow xsd to allow changing launcher configuration (e.g. setting java system properties)
>
>
> Diffs
> -----
>
> client/src/main/resources/hive-action-1.0.xsd PRE-CREATION
> client/src/main/resources/hive2-action-1.0.xsd PRE-CREATION
> client/src/main/resources/oozie-common-1.0.xsd PRE-CREATION
> client/src/main/resources/oozie-workflow-1.0.xsd PRE-CREATION
> client/src/main/resources/shell-action-1.0.xsd PRE-CREATION
> client/src/main/resources/spark-action-1.0.xsd PRE-CREATION
> client/src/main/resources/sqoop-action-1.0.xsd PRE-CREATION
> core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 49fd4b8560b21aef725c3268f0ec0e7f46a46ee9
> core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 8fdc50cce15271c13b20d1aaab2ffb95cb8fa711
> core/src/main/java/org/apache/oozie/action/ssh/SshActionExecutor.java 7e3348550e2ef44ae4bd6c5a7a417052bf9c108a
> core/src/main/java/org/apache/oozie/jms/MessageReceiver.java 47bfd2bcf9b53eed0882c51cdb5c530e0f6ccfe7
> core/src/main/java/org/apache/oozie/service/LiteWorkflowStoreService.java ffc29af2f834da6d0890ea7215c5a62fd7cd693e
> core/src/main/java/org/apache/oozie/service/SchemaService.java 137e2c0b4840c6d3858683db1aa38f54bd55be92
> core/src/main/java/org/apache/oozie/util/WritableUtils.java aa027e37ba23d4f481698e0bcd93c26c763a0b1f
> core/src/main/java/org/apache/oozie/util/schema/Input.java PRE-CREATION
> core/src/main/java/org/apache/oozie/util/schema/ResourceResolver.java PRE-CREATION
> core/src/main/java/org/apache/oozie/workflow/lite/LauncherConfigHandler.java PRE-CREATION
> core/src/main/java/org/apache/oozie/workflow/lite/LiteWorkflowAppParser.java a74e5c759fe3e336e7e98c61b8f5ac52efa4100c
> core/src/main/java/org/apache/oozie/workflow/lite/LiteWorkflowLib.java 23df0867e5984c518e8424d79365997bec28f619
> core/src/main/resources/oozie-default.xml 0d174b5069aff36ca3a9d48722c5bba69623a046
> core/src/test/java/org/apache/oozie/action/hadoop/ActionExecutorTestCase.java d74160a09ee237f4067e00ded526e7ec94528a6b
> core/src/test/java/org/apache/oozie/action/hadoop/LauncherMainTester.java 6cee7a8948ca7f2c7aaf44d4f172d18a1308a482
> core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java d1d78fd527a425b076edd7bb23ba56ebdba62664
> core/src/test/java/org/apache/oozie/service/TestSchemaService.java 88a10dbead32879334c3953a90a411a3a18fc0b3
> core/src/test/java/org/apache/oozie/util/TestMetricsInstrumentation.java a882c82b25154338f89e8a90f831b6f0300fbfe7
> core/src/test/java/org/apache/oozie/workflow/lite/TestLiteWorkflowAppParser.java 21332404bea79fe0a3cea045d57b3e52a1f15cae
> core/src/test/resources/wf-schema-global-launcherconf-override.xml PRE-CREATION
> core/src/test/resources/wf-schema-global-launcherconf.xml PRE-CREATION
> pom.xml db18f30814b9b6a73ba872c2cd7946692d0b876b
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java 6a98d6ef3a09a7a75272f3f0f9a9c2fc5472e76d
> sharelib/pig/src/test/java/org/apache/oozie/action/hadoop/UDFTester.java ff999ff2cd4875fbf3ad5e1a7eacf9972e848630
>
>
> Diff: https://reviews.apache.org/r/61529/diff/11/
>
>
> Testing
> -------
>
> - TestJavaActionExecutor, TestLiteWorkflowAppParser tests passed
> - java-main example on pseudo hadoop succeeded, modified workflow xml so that verbose class loading information is printed out of stdout of the launcher job
>
>
> Thanks,
>
> Attila Sasvari
>
>
Re: Review Request 61529: OOZIE-2687 Create XML schema for launcher
configurations
Posted by Robert Kanter <rk...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61529/#review185352
-----------------------------------------------------------
Ship it!
Ship It!
- Robert Kanter
On Sept. 13, 2017, 1:45 p.m., Attila Sasvari wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61529/
> -----------------------------------------------------------
>
> (Updated Sept. 13, 2017, 1:45 p.m.)
>
>
> Review request for oozie and Peter Cseh.
>
>
> Repository: oozie-git
>
>
> Description
> -------
>
> New workflow xsd to allow changing launcher configuration (e.g. setting java system properties)
>
>
> Diffs
> -----
>
> client/src/main/resources/hive-action-1.0.xsd PRE-CREATION
> client/src/main/resources/hive2-action-1.0.xsd PRE-CREATION
> client/src/main/resources/oozie-common-1.0.xsd PRE-CREATION
> client/src/main/resources/oozie-workflow-1.0.xsd PRE-CREATION
> client/src/main/resources/shell-action-1.0.xsd PRE-CREATION
> client/src/main/resources/spark-action-1.0.xsd PRE-CREATION
> client/src/main/resources/sqoop-action-1.0.xsd PRE-CREATION
> core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 49fd4b8560b21aef725c3268f0ec0e7f46a46ee9
> core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 8fdc50cce15271c13b20d1aaab2ffb95cb8fa711
> core/src/main/java/org/apache/oozie/action/ssh/SshActionExecutor.java 7e3348550e2ef44ae4bd6c5a7a417052bf9c108a
> core/src/main/java/org/apache/oozie/jms/MessageReceiver.java 47bfd2bcf9b53eed0882c51cdb5c530e0f6ccfe7
> core/src/main/java/org/apache/oozie/service/LiteWorkflowStoreService.java ffc29af2f834da6d0890ea7215c5a62fd7cd693e
> core/src/main/java/org/apache/oozie/service/SchemaService.java 137e2c0b4840c6d3858683db1aa38f54bd55be92
> core/src/main/java/org/apache/oozie/util/WritableUtils.java aa027e37ba23d4f481698e0bcd93c26c763a0b1f
> core/src/main/java/org/apache/oozie/util/schema/Input.java PRE-CREATION
> core/src/main/java/org/apache/oozie/util/schema/ResourceResolver.java PRE-CREATION
> core/src/main/java/org/apache/oozie/workflow/lite/LauncherConfigHandler.java PRE-CREATION
> core/src/main/java/org/apache/oozie/workflow/lite/LiteWorkflowAppParser.java a74e5c759fe3e336e7e98c61b8f5ac52efa4100c
> core/src/main/java/org/apache/oozie/workflow/lite/LiteWorkflowLib.java 23df0867e5984c518e8424d79365997bec28f619
> core/src/main/resources/oozie-default.xml 0d174b5069aff36ca3a9d48722c5bba69623a046
> core/src/test/java/org/apache/oozie/action/hadoop/ActionExecutorTestCase.java d74160a09ee237f4067e00ded526e7ec94528a6b
> core/src/test/java/org/apache/oozie/action/hadoop/LauncherMainTester.java 6cee7a8948ca7f2c7aaf44d4f172d18a1308a482
> core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java d1d78fd527a425b076edd7bb23ba56ebdba62664
> core/src/test/java/org/apache/oozie/service/TestSchemaService.java 88a10dbead32879334c3953a90a411a3a18fc0b3
> core/src/test/java/org/apache/oozie/util/TestMetricsInstrumentation.java a882c82b25154338f89e8a90f831b6f0300fbfe7
> core/src/test/java/org/apache/oozie/workflow/lite/TestLiteWorkflowAppParser.java 21332404bea79fe0a3cea045d57b3e52a1f15cae
> core/src/test/resources/wf-schema-global-launcherconf-override.xml PRE-CREATION
> core/src/test/resources/wf-schema-global-launcherconf.xml PRE-CREATION
> pom.xml db18f30814b9b6a73ba872c2cd7946692d0b876b
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java 6a98d6ef3a09a7a75272f3f0f9a9c2fc5472e76d
> sharelib/pig/src/test/java/org/apache/oozie/action/hadoop/UDFTester.java ff999ff2cd4875fbf3ad5e1a7eacf9972e848630
>
>
> Diff: https://reviews.apache.org/r/61529/diff/11/
>
>
> Testing
> -------
>
> - TestJavaActionExecutor, TestLiteWorkflowAppParser tests passed
> - java-main example on pseudo hadoop succeeded, modified workflow xml so that verbose class loading information is printed out of stdout of the launcher job
>
>
> Thanks,
>
> Attila Sasvari
>
>
Re: Review Request 61529: OOZIE-2687 Create XML schema for launcher
configurations
Posted by Attila Sasvari <as...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61529/
-----------------------------------------------------------
(Updated Sept. 13, 2017, 1:45 p.m.)
Review request for oozie and Peter Cseh.
Changes
-------
- addressed review comments: schema elements' order
- moved complex types to common
Repository: oozie-git
Description
-------
New workflow xsd to allow changing launcher configuration (e.g. setting java system properties)
Diffs (updated)
-----
client/src/main/resources/hive-action-1.0.xsd PRE-CREATION
client/src/main/resources/hive2-action-1.0.xsd PRE-CREATION
client/src/main/resources/oozie-common-1.0.xsd PRE-CREATION
client/src/main/resources/oozie-workflow-1.0.xsd PRE-CREATION
client/src/main/resources/shell-action-1.0.xsd PRE-CREATION
client/src/main/resources/spark-action-1.0.xsd PRE-CREATION
client/src/main/resources/sqoop-action-1.0.xsd PRE-CREATION
core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 49fd4b8560b21aef725c3268f0ec0e7f46a46ee9
core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 8fdc50cce15271c13b20d1aaab2ffb95cb8fa711
core/src/main/java/org/apache/oozie/action/ssh/SshActionExecutor.java 7e3348550e2ef44ae4bd6c5a7a417052bf9c108a
core/src/main/java/org/apache/oozie/jms/MessageReceiver.java 47bfd2bcf9b53eed0882c51cdb5c530e0f6ccfe7
core/src/main/java/org/apache/oozie/service/LiteWorkflowStoreService.java ffc29af2f834da6d0890ea7215c5a62fd7cd693e
core/src/main/java/org/apache/oozie/service/SchemaService.java 137e2c0b4840c6d3858683db1aa38f54bd55be92
core/src/main/java/org/apache/oozie/util/WritableUtils.java aa027e37ba23d4f481698e0bcd93c26c763a0b1f
core/src/main/java/org/apache/oozie/util/schema/Input.java PRE-CREATION
core/src/main/java/org/apache/oozie/util/schema/ResourceResolver.java PRE-CREATION
core/src/main/java/org/apache/oozie/workflow/lite/LauncherConfigHandler.java PRE-CREATION
core/src/main/java/org/apache/oozie/workflow/lite/LiteWorkflowAppParser.java a74e5c759fe3e336e7e98c61b8f5ac52efa4100c
core/src/main/java/org/apache/oozie/workflow/lite/LiteWorkflowLib.java 23df0867e5984c518e8424d79365997bec28f619
core/src/main/resources/oozie-default.xml 0d174b5069aff36ca3a9d48722c5bba69623a046
core/src/test/java/org/apache/oozie/action/hadoop/ActionExecutorTestCase.java d74160a09ee237f4067e00ded526e7ec94528a6b
core/src/test/java/org/apache/oozie/action/hadoop/LauncherMainTester.java 6cee7a8948ca7f2c7aaf44d4f172d18a1308a482
core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java d1d78fd527a425b076edd7bb23ba56ebdba62664
core/src/test/java/org/apache/oozie/service/TestSchemaService.java 88a10dbead32879334c3953a90a411a3a18fc0b3
core/src/test/java/org/apache/oozie/util/TestMetricsInstrumentation.java a882c82b25154338f89e8a90f831b6f0300fbfe7
core/src/test/java/org/apache/oozie/workflow/lite/TestLiteWorkflowAppParser.java 21332404bea79fe0a3cea045d57b3e52a1f15cae
core/src/test/resources/wf-schema-global-launcherconf-override.xml PRE-CREATION
core/src/test/resources/wf-schema-global-launcherconf.xml PRE-CREATION
pom.xml db18f30814b9b6a73ba872c2cd7946692d0b876b
sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java 6a98d6ef3a09a7a75272f3f0f9a9c2fc5472e76d
sharelib/pig/src/test/java/org/apache/oozie/action/hadoop/UDFTester.java ff999ff2cd4875fbf3ad5e1a7eacf9972e848630
Diff: https://reviews.apache.org/r/61529/diff/10/
Changes: https://reviews.apache.org/r/61529/diff/9-10/
Testing
-------
- TestJavaActionExecutor, TestLiteWorkflowAppParser tests passed
- java-main example on pseudo hadoop succeeded, modified workflow xml so that verbose class loading information is printed out of stdout of the launcher job
Thanks,
Attila Sasvari
Re: Review Request 61529: OOZIE-2687 Create XML schema for launcher
configurations
Posted by Attila Sasvari <as...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61529/
-----------------------------------------------------------
(Updated Sept. 11, 2017, 10:50 p.m.)
Review request for oozie and Peter Cseh.
Changes
-------
- chameleon schema validation
- extracted additional complex types to common schema
Repository: oozie-git
Description
-------
New workflow xsd to allow changing launcher configuration (e.g. setting java system properties)
Diffs (updated)
-----
client/src/main/resources/hive-action-1.0.xsd PRE-CREATION
client/src/main/resources/hive2-action-1.0.xsd PRE-CREATION
client/src/main/resources/oozie-common-1.0.xsd PRE-CREATION
client/src/main/resources/oozie-workflow-1.0.xsd PRE-CREATION
client/src/main/resources/shell-action-1.0.xsd PRE-CREATION
client/src/main/resources/spark-action-1.0.xsd PRE-CREATION
client/src/main/resources/sqoop-action-1.0.xsd PRE-CREATION
core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 49fd4b8560b21aef725c3268f0ec0e7f46a46ee9
core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 8fdc50cce15271c13b20d1aaab2ffb95cb8fa711
core/src/main/java/org/apache/oozie/action/ssh/SshActionExecutor.java 7e3348550e2ef44ae4bd6c5a7a417052bf9c108a
core/src/main/java/org/apache/oozie/jms/MessageReceiver.java 47bfd2bcf9b53eed0882c51cdb5c530e0f6ccfe7
core/src/main/java/org/apache/oozie/service/LiteWorkflowStoreService.java ffc29af2f834da6d0890ea7215c5a62fd7cd693e
core/src/main/java/org/apache/oozie/service/SchemaService.java 137e2c0b4840c6d3858683db1aa38f54bd55be92
core/src/main/java/org/apache/oozie/util/WritableUtils.java aa027e37ba23d4f481698e0bcd93c26c763a0b1f
core/src/main/java/org/apache/oozie/util/schema/Input.java PRE-CREATION
core/src/main/java/org/apache/oozie/util/schema/ResourceResolver.java PRE-CREATION
core/src/main/java/org/apache/oozie/workflow/lite/LauncherConfigHandler.java PRE-CREATION
core/src/main/java/org/apache/oozie/workflow/lite/LiteWorkflowAppParser.java a74e5c759fe3e336e7e98c61b8f5ac52efa4100c
core/src/main/java/org/apache/oozie/workflow/lite/LiteWorkflowLib.java 23df0867e5984c518e8424d79365997bec28f619
core/src/main/resources/oozie-default.xml 0d174b5069aff36ca3a9d48722c5bba69623a046
core/src/test/java/org/apache/oozie/action/hadoop/ActionExecutorTestCase.java d74160a09ee237f4067e00ded526e7ec94528a6b
core/src/test/java/org/apache/oozie/action/hadoop/LauncherMainTester.java 6cee7a8948ca7f2c7aaf44d4f172d18a1308a482
core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java d1d78fd527a425b076edd7bb23ba56ebdba62664
core/src/test/java/org/apache/oozie/service/TestSchemaService.java 88a10dbead32879334c3953a90a411a3a18fc0b3
core/src/test/java/org/apache/oozie/util/TestMetricsInstrumentation.java a882c82b25154338f89e8a90f831b6f0300fbfe7
core/src/test/java/org/apache/oozie/workflow/lite/TestLiteWorkflowAppParser.java 21332404bea79fe0a3cea045d57b3e52a1f15cae
core/src/test/resources/wf-schema-global-launcherconf-override.xml PRE-CREATION
core/src/test/resources/wf-schema-global-launcherconf.xml PRE-CREATION
pom.xml db18f30814b9b6a73ba872c2cd7946692d0b876b
sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java 6a98d6ef3a09a7a75272f3f0f9a9c2fc5472e76d
sharelib/pig/src/test/java/org/apache/oozie/action/hadoop/UDFTester.java ff999ff2cd4875fbf3ad5e1a7eacf9972e848630
Diff: https://reviews.apache.org/r/61529/diff/9/
Changes: https://reviews.apache.org/r/61529/diff/8-9/
Testing
-------
- TestJavaActionExecutor, TestLiteWorkflowAppParser tests passed
- java-main example on pseudo hadoop succeeded, modified workflow xml so that verbose class loading information is printed out of stdout of the launcher job
Thanks,
Attila Sasvari
Re: Review Request 61529: OOZIE-2687 Create XML schema for launcher
configurations
Posted by Attila Sasvari <as...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61529/
-----------------------------------------------------------
(Updated Sept. 8, 2017, 10:04 a.m.)
Review request for oozie and Peter Cseh.
Changes
-------
using ``ref`` inside workflow actions to reference launcher element
Repository: oozie-git
Description
-------
New workflow xsd to allow changing launcher configuration (e.g. setting java system properties)
Diffs (updated)
-----
client/src/main/resources/hive-action-1.0.xsd PRE-CREATION
client/src/main/resources/hive2-action-1.0.xsd PRE-CREATION
client/src/main/resources/oozie-common-1.0.xsd PRE-CREATION
client/src/main/resources/oozie-workflow-1.0.xsd PRE-CREATION
client/src/main/resources/shell-action-1.0.xsd PRE-CREATION
client/src/main/resources/spark-action-1.0.xsd PRE-CREATION
client/src/main/resources/sqoop-action-1.0.xsd PRE-CREATION
core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java bca79aa052521ea4f6f16e76bd69f84fb16be790
core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 8fdc50cce15271c13b20d1aaab2ffb95cb8fa711
core/src/main/java/org/apache/oozie/action/ssh/SshActionExecutor.java 7e3348550e2ef44ae4bd6c5a7a417052bf9c108a
core/src/main/java/org/apache/oozie/jms/MessageReceiver.java 47bfd2bcf9b53eed0882c51cdb5c530e0f6ccfe7
core/src/main/java/org/apache/oozie/service/LiteWorkflowStoreService.java ffc29af2f834da6d0890ea7215c5a62fd7cd693e
core/src/main/java/org/apache/oozie/service/SchemaService.java 137e2c0b4840c6d3858683db1aa38f54bd55be92
core/src/main/java/org/apache/oozie/util/WritableUtils.java aa027e37ba23d4f481698e0bcd93c26c763a0b1f
core/src/main/java/org/apache/oozie/util/schema/Input.java PRE-CREATION
core/src/main/java/org/apache/oozie/util/schema/ResourceResolver.java PRE-CREATION
core/src/main/java/org/apache/oozie/workflow/lite/LauncherConfigHandler.java PRE-CREATION
core/src/main/java/org/apache/oozie/workflow/lite/LiteWorkflowAppParser.java a74e5c759fe3e336e7e98c61b8f5ac52efa4100c
core/src/main/java/org/apache/oozie/workflow/lite/LiteWorkflowLib.java 23df0867e5984c518e8424d79365997bec28f619
core/src/main/resources/oozie-default.xml 0d174b5069aff36ca3a9d48722c5bba69623a046
core/src/test/java/org/apache/oozie/action/hadoop/ActionExecutorTestCase.java d74160a09ee237f4067e00ded526e7ec94528a6b
core/src/test/java/org/apache/oozie/action/hadoop/LauncherMainTester.java 6cee7a8948ca7f2c7aaf44d4f172d18a1308a482
core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java ce674adf9cffa20690083e6298875a6f9047a109
core/src/test/java/org/apache/oozie/service/TestSchemaService.java 88a10dbead32879334c3953a90a411a3a18fc0b3
core/src/test/java/org/apache/oozie/util/TestMetricsInstrumentation.java a882c82b25154338f89e8a90f831b6f0300fbfe7
core/src/test/java/org/apache/oozie/workflow/lite/TestLiteWorkflowAppParser.java 21332404bea79fe0a3cea045d57b3e52a1f15cae
core/src/test/resources/wf-schema-global-launcherconf-override.xml PRE-CREATION
core/src/test/resources/wf-schema-global-launcherconf.xml PRE-CREATION
sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java 6a98d6ef3a09a7a75272f3f0f9a9c2fc5472e76d
sharelib/pig/src/test/java/org/apache/oozie/action/hadoop/UDFTester.java ff999ff2cd4875fbf3ad5e1a7eacf9972e848630
Diff: https://reviews.apache.org/r/61529/diff/8/
Changes: https://reviews.apache.org/r/61529/diff/7-8/
Testing
-------
- TestJavaActionExecutor, TestLiteWorkflowAppParser tests passed
- java-main example on pseudo hadoop succeeded, modified workflow xml so that verbose class loading information is printed out of stdout of the launcher job
Thanks,
Attila Sasvari
Re: Review Request 61529: OOZIE-2687 Create XML schema for launcher
configurations
Posted by Attila Sasvari <as...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61529/
-----------------------------------------------------------
(Updated Sept. 7, 2017, 4:33 p.m.)
Review request for oozie and Peter Cseh.
Repository: oozie-git
Description
-------
New workflow xsd to allow changing launcher configuration (e.g. setting java system properties)
Diffs (updated)
-----
client/src/main/resources/hive-action-1.0.xsd PRE-CREATION
client/src/main/resources/hive2-action-1.0.xsd PRE-CREATION
client/src/main/resources/oozie-common-1.0.xsd PRE-CREATION
client/src/main/resources/oozie-workflow-1.0.xsd PRE-CREATION
client/src/main/resources/shell-action-1.0.xsd PRE-CREATION
client/src/main/resources/spark-action-1.0.xsd PRE-CREATION
client/src/main/resources/sqoop-action-1.0.xsd PRE-CREATION
core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java bca79aa052521ea4f6f16e76bd69f84fb16be790
core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 8fdc50cce15271c13b20d1aaab2ffb95cb8fa711
core/src/main/java/org/apache/oozie/action/ssh/SshActionExecutor.java 7e3348550e2ef44ae4bd6c5a7a417052bf9c108a
core/src/main/java/org/apache/oozie/jms/MessageReceiver.java 47bfd2bcf9b53eed0882c51cdb5c530e0f6ccfe7
core/src/main/java/org/apache/oozie/service/LiteWorkflowStoreService.java ffc29af2f834da6d0890ea7215c5a62fd7cd693e
core/src/main/java/org/apache/oozie/service/SchemaService.java 137e2c0b4840c6d3858683db1aa38f54bd55be92
core/src/main/java/org/apache/oozie/util/WritableUtils.java aa027e37ba23d4f481698e0bcd93c26c763a0b1f
core/src/main/java/org/apache/oozie/util/schema/Input.java PRE-CREATION
core/src/main/java/org/apache/oozie/util/schema/ResourceResolver.java PRE-CREATION
core/src/main/java/org/apache/oozie/workflow/lite/LauncherConfigHandler.java PRE-CREATION
core/src/main/java/org/apache/oozie/workflow/lite/LiteWorkflowAppParser.java a74e5c759fe3e336e7e98c61b8f5ac52efa4100c
core/src/main/java/org/apache/oozie/workflow/lite/LiteWorkflowLib.java 23df0867e5984c518e8424d79365997bec28f619
core/src/main/resources/oozie-default.xml 0d174b5069aff36ca3a9d48722c5bba69623a046
core/src/test/java/org/apache/oozie/action/hadoop/ActionExecutorTestCase.java d74160a09ee237f4067e00ded526e7ec94528a6b
core/src/test/java/org/apache/oozie/action/hadoop/LauncherMainTester.java 6cee7a8948ca7f2c7aaf44d4f172d18a1308a482
core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java ce674adf9cffa20690083e6298875a6f9047a109
core/src/test/java/org/apache/oozie/service/TestSchemaService.java 88a10dbead32879334c3953a90a411a3a18fc0b3
core/src/test/java/org/apache/oozie/util/TestMetricsInstrumentation.java a882c82b25154338f89e8a90f831b6f0300fbfe7
core/src/test/java/org/apache/oozie/workflow/lite/TestLiteWorkflowAppParser.java 21332404bea79fe0a3cea045d57b3e52a1f15cae
core/src/test/resources/wf-schema-global-launcherconf-override.xml PRE-CREATION
core/src/test/resources/wf-schema-global-launcherconf.xml PRE-CREATION
sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java 6a98d6ef3a09a7a75272f3f0f9a9c2fc5472e76d
sharelib/pig/src/test/java/org/apache/oozie/action/hadoop/UDFTester.java ff999ff2cd4875fbf3ad5e1a7eacf9972e848630
Diff: https://reviews.apache.org/r/61529/diff/7/
Changes: https://reviews.apache.org/r/61529/diff/6-7/
Testing
-------
- TestJavaActionExecutor, TestLiteWorkflowAppParser tests passed
- java-main example on pseudo hadoop succeeded, modified workflow xml so that verbose class loading information is printed out of stdout of the launcher job
Thanks,
Attila Sasvari
Re: Review Request 61529: OOZIE-2687 Create XML schema for launcher
configurations
Posted by Attila Sasvari <as...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61529/
-----------------------------------------------------------
(Updated Sept. 7, 2017, 1:17 p.m.)
Review request for oozie and Peter Cseh.
Changes
-------
- New tests for launcher.env
Repository: oozie-git
Description
-------
New workflow xsd to allow changing launcher configuration (e.g. setting java system properties)
Diffs (updated)
-----
client/src/main/resources/hive-action-1.0.xsd PRE-CREATION
client/src/main/resources/hive2-action-1.0.xsd PRE-CREATION
client/src/main/resources/oozie-common-1.0.xsd PRE-CREATION
client/src/main/resources/oozie-workflow-1.0.xsd PRE-CREATION
client/src/main/resources/shell-action-1.0.xsd PRE-CREATION
client/src/main/resources/spark-action-1.0.xsd PRE-CREATION
client/src/main/resources/sqoop-action-1.0.xsd PRE-CREATION
core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java bca79aa052521ea4f6f16e76bd69f84fb16be790
core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 8fdc50cce15271c13b20d1aaab2ffb95cb8fa711
core/src/main/java/org/apache/oozie/action/ssh/SshActionExecutor.java 7e3348550e2ef44ae4bd6c5a7a417052bf9c108a
core/src/main/java/org/apache/oozie/jms/MessageReceiver.java 47bfd2bcf9b53eed0882c51cdb5c530e0f6ccfe7
core/src/main/java/org/apache/oozie/service/LiteWorkflowStoreService.java ffc29af2f834da6d0890ea7215c5a62fd7cd693e
core/src/main/java/org/apache/oozie/service/SchemaService.java 137e2c0b4840c6d3858683db1aa38f54bd55be92
core/src/main/java/org/apache/oozie/util/WritableUtils.java aa027e37ba23d4f481698e0bcd93c26c763a0b1f
core/src/main/java/org/apache/oozie/util/schema/Input.java PRE-CREATION
core/src/main/java/org/apache/oozie/util/schema/ResourceResolver.java PRE-CREATION
core/src/main/java/org/apache/oozie/workflow/lite/LauncherConfigHandler.java PRE-CREATION
core/src/main/java/org/apache/oozie/workflow/lite/LiteWorkflowAppParser.java a74e5c759fe3e336e7e98c61b8f5ac52efa4100c
core/src/main/java/org/apache/oozie/workflow/lite/LiteWorkflowLib.java 23df0867e5984c518e8424d79365997bec28f619
core/src/main/resources/oozie-default.xml 0d174b5069aff36ca3a9d48722c5bba69623a046
core/src/test/java/org/apache/oozie/action/hadoop/ActionExecutorTestCase.java d74160a09ee237f4067e00ded526e7ec94528a6b
core/src/test/java/org/apache/oozie/action/hadoop/LauncherMainTester.java 6cee7a8948ca7f2c7aaf44d4f172d18a1308a482
core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java ce674adf9cffa20690083e6298875a6f9047a109
core/src/test/java/org/apache/oozie/service/TestSchemaService.java 88a10dbead32879334c3953a90a411a3a18fc0b3
core/src/test/java/org/apache/oozie/util/TestMetricsInstrumentation.java a882c82b25154338f89e8a90f831b6f0300fbfe7
core/src/test/java/org/apache/oozie/workflow/lite/TestLiteWorkflowAppParser.java 21332404bea79fe0a3cea045d57b3e52a1f15cae
core/src/test/resources/wf-schema-global-launcherconf-override.xml PRE-CREATION
core/src/test/resources/wf-schema-global-launcherconf.xml PRE-CREATION
sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java 6a98d6ef3a09a7a75272f3f0f9a9c2fc5472e76d
sharelib/pig/src/test/java/org/apache/oozie/action/hadoop/UDFTester.java ff999ff2cd4875fbf3ad5e1a7eacf9972e848630
Diff: https://reviews.apache.org/r/61529/diff/6/
Changes: https://reviews.apache.org/r/61529/diff/5-6/
Testing
-------
- TestJavaActionExecutor, TestLiteWorkflowAppParser tests passed
- java-main example on pseudo hadoop succeeded, modified workflow xml so that verbose class loading information is printed out of stdout of the launcher job
Thanks,
Attila Sasvari
Re: Review Request 61529: OOZIE-2687 Create XML schema for launcher
configurations
Posted by Attila Sasvari <as...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61529/
-----------------------------------------------------------
(Updated Sept. 5, 2017, 3:52 p.m.)
Review request for oozie and Peter Cseh.
Changes
-------
- new schemas are now 1.0
- validation tests for new schemas
- added helper classes to validate schemas with import / include statements
- refactored JavaActionExecutor
Repository: oozie-git
Description
-------
New workflow xsd to allow changing launcher configuration (e.g. setting java system properties)
Diffs (updated)
-----
client/src/main/resources/hive-action-1.0.xsd PRE-CREATION
client/src/main/resources/hive2-action-1.0.xsd PRE-CREATION
client/src/main/resources/oozie-common-1.0.xsd PRE-CREATION
client/src/main/resources/oozie-workflow-1.0.xsd PRE-CREATION
client/src/main/resources/shell-action-1.0.xsd PRE-CREATION
client/src/main/resources/spark-action-1.0.xsd PRE-CREATION
client/src/main/resources/sqoop-action-1.0.xsd PRE-CREATION
core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java bca79aa052521ea4f6f16e76bd69f84fb16be790
core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 8fdc50cce15271c13b20d1aaab2ffb95cb8fa711
core/src/main/java/org/apache/oozie/action/ssh/SshActionExecutor.java 7e3348550e2ef44ae4bd6c5a7a417052bf9c108a
core/src/main/java/org/apache/oozie/jms/MessageReceiver.java 47bfd2bcf9b53eed0882c51cdb5c530e0f6ccfe7
core/src/main/java/org/apache/oozie/service/LiteWorkflowStoreService.java ffc29af2f834da6d0890ea7215c5a62fd7cd693e
core/src/main/java/org/apache/oozie/service/SchemaService.java 137e2c0b4840c6d3858683db1aa38f54bd55be92
core/src/main/java/org/apache/oozie/util/WritableUtils.java aa027e37ba23d4f481698e0bcd93c26c763a0b1f
core/src/main/java/org/apache/oozie/util/schema/Input.java PRE-CREATION
core/src/main/java/org/apache/oozie/util/schema/ResourceResolver.java PRE-CREATION
core/src/main/java/org/apache/oozie/workflow/lite/LauncherConfigHandler.java PRE-CREATION
core/src/main/java/org/apache/oozie/workflow/lite/LiteWorkflowAppParser.java a74e5c759fe3e336e7e98c61b8f5ac52efa4100c
core/src/main/java/org/apache/oozie/workflow/lite/LiteWorkflowLib.java 23df0867e5984c518e8424d79365997bec28f619
core/src/main/resources/oozie-default.xml 0d174b5069aff36ca3a9d48722c5bba69623a046
core/src/test/java/org/apache/oozie/action/hadoop/ActionExecutorTestCase.java d74160a09ee237f4067e00ded526e7ec94528a6b
core/src/test/java/org/apache/oozie/action/hadoop/LauncherMainTester.java 6cee7a8948ca7f2c7aaf44d4f172d18a1308a482
core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java ce674adf9cffa20690083e6298875a6f9047a109
core/src/test/java/org/apache/oozie/service/TestSchemaService.java 88a10dbead32879334c3953a90a411a3a18fc0b3
core/src/test/java/org/apache/oozie/util/TestMetricsInstrumentation.java a882c82b25154338f89e8a90f831b6f0300fbfe7
core/src/test/java/org/apache/oozie/workflow/lite/TestLiteWorkflowAppParser.java 21332404bea79fe0a3cea045d57b3e52a1f15cae
core/src/test/resources/wf-schema-global-launcherconf-override.xml PRE-CREATION
core/src/test/resources/wf-schema-global-launcherconf.xml PRE-CREATION
sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java 6a98d6ef3a09a7a75272f3f0f9a9c2fc5472e76d
sharelib/pig/src/test/java/org/apache/oozie/action/hadoop/UDFTester.java ff999ff2cd4875fbf3ad5e1a7eacf9972e848630
Diff: https://reviews.apache.org/r/61529/diff/3/
Changes: https://reviews.apache.org/r/61529/diff/2-3/
Testing
-------
- TestJavaActionExecutor, TestLiteWorkflowAppParser tests passed
- java-main example on pseudo hadoop succeeded, modified workflow xml so that verbose class loading information is printed out of stdout of the launcher job
Thanks,
Attila Sasvari
Re: Review Request 61529: OOZIE-2687 Create XML schema for launcher
configurations
Posted by Robert Kanter <rk...@cloudera.com>.
> On Aug. 16, 2017, 8:33 p.m., Robert Kanter wrote:
> > client/src/main/resources/hive-action-0.7.xsd
> > Lines 49 (patched)
> > <https://reviews.apache.org/r/61529/diff/2/?file=1796924#file1796924line49>
> >
> > Can we put CONFIGURATION into oozie-common?
> >
> > Same with PREPARE, DELETE, and MKDIR?
>
> Attila Sasvari wrote:
> Right now I changed oozie-common so that its targetNamespace is targetNamespace="uri:oozie:workflow:1.0"
> This way common prefix is not needed if ``<launcher>`` is present in the global config (include statement can be used).
>
> ```
> <global>
> <launcher>
> <workflow:memory>1024</workflow:memory>
> <workflow:vcores>2</workflow:vcores>
> <workflow:java-opts>-verbose:class</workflow:java-opts>
> <workflow:env>dummyEnv=42</workflow:env>
> <workflow:queue>dummyQueue</workflow:queue>
> </launcher>
> </global>
> ```
>
> Other actions use the import statement and still need the ``workflow`` prefix for the vcore,memory, etc. if they include the ``<launcher>`` element. They also need to define workflow namespace with ``xmlns:workflow``.
>
> ```
> <workflow-app xmlns="uri:oozie:workflow:1.0" xmlns:workflow="uri:oozie:workflow:1.0" name="shell-wf">
> <start to="shell-node"/>
> <action name="shell-node">
> <shell xmlns="uri:oozie:shell-action:1.0">
> <job-tracker>${jobTracker}</job-tracker>
> <name-node>${nameNode}</name-node>
> <launcher>
> <workflow:memory>1024</workflow:memory>
> <workflow:vcores>2</workflow:vcores>
> <workflow:java-opts>-verbose:class</workflow:java-opts>
> <workflow:env>dummyEnv=42</workflow:env>
> <workflow:queue>dummyQueue</workflow:queue>
> <workflow:sharelib>a,b,c</workflow:sharelib>
> </launcher>
> ```
>
> ``xs:any`` would require the prefix (like the sla complex type). Using ``<xs:any namespace="##other" minOccurs="1" maxOccurs="1"/>`` could work, but is too permissive - for example you could add any element (e.g. a hive action) instead of a ``launcher``.
>
> I am trying to find a better way, but please share if you have some ideas.
>
> Attila Sasvari wrote:
> If I use ``ref`` like ``<xs:element ref="workflow:launcher" minOccurs="0" maxOccurs="1"/>`` for example in the hive action, it becomes a bit more clean to define a ``launcher`` element:
>
> ```
> <workflow-app xmlns="uri:oozie:workflow:1.0" name="hive-wf">
> ...
> <hive xmlns="uri:oozie:workflow:hive-action:1.0">
> ...
> <launcher xmlns="uri:oozie:workflow:1.0">
> <memory>1024</memory>
> </launcher>"
> ```
>
> This way it is not needed to define the workflow namespace in the workflow element and use it as prefix.
>
> I was also considering to add other elements to the common workflow namespace. Because of the namespace differences of actions, it would not be easy to include say ``configuration`` - we would need to define workflow namespace and prefix.
>
> Peter Cseh wrote:
> I like the idea of setting xmlns for launcher inside the launcher tag if we don't have to use prefixes after that.
>
> Robert Kanter wrote:
> Have you tried using include instead of import? From some googling, it sounds like if you (a) use include, (b) don't set a target namespace, and (c) don't have a namespace in the included xsd, then it will work the way we want.
>
> https://stackoverflow.com/questions/2357943/whats-the-difference-between-xsdinclude-and-xsdimport
> https://stackoverflow.com/questions/914020/xsd-with-imports-and-namespaces
> https://stackoverflow.com/questions/9805359/how-to-handle-multiple-namespaces-with-different-uri-in-xsd
>
> This link talks about this case from an XSD spec point of view, but it's hard to understand:
> https://www.w3.org/TR/xmlschema-1/#compound-schema
>
>
> If that works, then I'd vote we go for that. Otherwise, I'd vote for whatever is easiest for the user. If that means we have to duplicate XSD code like we do today, that's ok.
>
> Attila Sasvari wrote:
> Thanks for the ideas. I have already read some of those entries. Anyway, I tried again the suggestion: I had to use ``import`` otherwise referencing did not work:
> - Using ``<xs:include schemaLocation="oozie-common-1.0.xsd"/>`` and ``<xs:element ref="launcher" minOccurs="0" maxOccurs="1"/>`` :
> ```
> Caused by: org.xml.sax.SAXParseException; systemId: file:///Users/asasvari/workspace/apache/oozie_dup/core/oozie-workflow-1.0.xsd; lineNumber: 71; columnNumber: 69; src-resolve.4.1: Error resolving component 'launcher'. It was detected that 'launcher' has no namespace, but components with no target namespace are not referenceable from schema document 'file:///Users/asasvari/workspace/apache/oozie_dup/core/oozie-workflow-1.0.xsd'. If 'launcher' is intended to have a namespace, perhaps a prefix needs to be provided. If it is intended that 'launcher' has no namespace, then an 'import' without a "namespace" attribute should be added to 'file:///Users/asasvari/workspace/apache/oozie_dup/core/oozie-workflow-1.0.xsd'.
>
> ```
> - Using ``<xs:element name="launcher" type="LAUNCHER" minOccurs="0" maxOccurs="1"/>`` also resulted in:
> ```
> Caused by: org.xml.sax.SAXParseException; systemId: file:///Users/asasvari/workspace/apache/oozie_dup/core/oozie-workflow-1.0.xsd; lineNumber: 71; columnNumber: 86; src-resolve.4.1: Error resolving component 'LAUNCHER'. It was detected that 'LAUNCHER' has no namespace, but components with no target namespace are not referenceable from schema document 'file:///Users/asasvari/workspace/apache/oozie_dup/core/oozie-workflow-1.0.xsd'. If 'LAUNCHER' is intended to have a namespace, perhaps a prefix needs to be provided. If it is intended that 'LAUNCHER' has no namespace, then an 'import' without a "namespace" attribute should be added to 'file:///Users/asasvari/workspace/apache/oozie_dup/core/oozie-workflow-1.0.xsd'.
>
> ```
>
> When I switched to ``<xs:import schemaLocation="oozie-common-1.0.xsd"/>`` and ``<xs:element ref="launcher" minOccurs="0" maxOccurs="1"/>``, the following exception was thrown by Xerces:
> ```
> org.xml.sax.SAXParseException; lineNumber: 4; columnNumber: 24; cvc-complex-type.2.4.a: Invalid content was found starting with element 'memory.mb'. One of '{memory.mb, vcores, java-opts, env, queue, sharelib}' is expected.
> ```
>
> Running the sqoop action validation test was also strange:
> ```
> org.xml.sax.SAXParseException; lineNumber: 3; columnNumber: 19; cvc-complex-type.2.4.a: Invalid content was found starting with element 'launcher'. One of '{"uri:oozie:workflow:1.0":job-tracker, "uri:oozie:workflow:1.0":resource-manager, "uri:oozie:workflow:1.0":name-node, "uri:oozie:workflow:1.0":job-xml, launcher, "uri:oozie:workflow:1.0":configuration}' is expected.
> ```
>
> It is strange that referencing an element with a complex type from a default namespace fails to validate.
I tried playing around with this, and looked especially at the third stackoverflow link, and I think I got it working. Here's a proof of concept patch that goes on top of your patch. It changes the oozie-common xsd to not have a namespace, and the hive xsd to use the LAUNCHER type from there without a prefix. I ran TestSchemaService#testHiveActionLauncherConfig to verify. Assuming this isn't a fluke, we'd just need to extend this to the other schemas and move other common things to oozie-common like \<configuration>. Please take a look and let me know what you think.
````
diff --git a/client/src/main/resources/hive-action-1.0.xsd b/client/src/main/resources/hive-action-1.0.xsd
index 6fc921ae51..de6e7c1529 100644
--- a/client/src/main/resources/hive-action-1.0.xsd
+++ b/client/src/main/resources/hive-action-1.0.xsd
@@ -18,11 +18,10 @@
-->
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
xmlns:hive="uri:oozie:hive-action:1.0"
- xmlns:workflow="uri:oozie:workflow:1.0"
elementFormDefault="qualified"
targetNamespace="uri:oozie:hive-action:1.0">
- <xs:import namespace="uri:oozie:workflow:1.0"/>
+ <xs:include schemaLocation="oozie-common-1.0.xsd"/>
<xs:element name="hive" type="hive:ACTION"/>
@@ -35,7 +34,7 @@
<xs:element name="name-node" type="xs:string" minOccurs="0" maxOccurs="1"/>
<xs:element name="prepare" type="hive:PREPARE" minOccurs="0" maxOccurs="1"/>
<xs:element name="job-xml" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
- <xs:element ref="workflow:launcher" minOccurs="0" maxOccurs="1"/>
+ <xs:element name="launcher" type="hive:LAUNCHER" minOccurs="0" maxOccurs="1"/>
<xs:element name="configuration" type="hive:CONFIGURATION" minOccurs="0" maxOccurs="1"/>
<xs:choice minOccurs="1" maxOccurs="1">
<xs:element name="script" type="xs:string" minOccurs="1" maxOccurs="1"/>
diff --git a/client/src/main/resources/oozie-common-1.0.xsd b/client/src/main/resources/oozie-common-1.0.xsd
index e149f6d4b6..beb9d4d0ca 100644
--- a/client/src/main/resources/oozie-common-1.0.xsd
+++ b/client/src/main/resources/oozie-common-1.0.xsd
@@ -17,8 +17,8 @@
limitations under the License.
-->
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:workflow="uri:oozie:workflow:1.0"
- elementFormDefault="qualified" targetNamespace="uri:oozie:workflow:1.0">
- <xs:element name="launcher" type="workflow:LAUNCHER"/>
+ elementFormDefault="qualified">
+ <xs:element name="launcher" type="LAUNCHER"/>
<xs:complexType name="LAUNCHER">
<xs:choice maxOccurs="unbounded">
diff --git a/core/src/test/java/org/apache/oozie/service/TestSchemaService.java b/core/src/test/java/org/apache/oozie/service/TestSchemaService.java
index 79c688e331..d7465cade4 100644
--- a/core/src/test/java/org/apache/oozie/service/TestSchemaService.java
+++ b/core/src/test/java/org/apache/oozie/service/TestSchemaService.java
@@ -128,7 +128,7 @@ public class TestSchemaService extends XTestCase {
"</workflow-app>\n";
private static final String HIVE_ACTION_LAUNCHER_CONF =
- "<workflow-app xmlns:workflow=\"uri:oozie:workflow:1.0\" xmlns=\"uri:oozie:workflow:1.0\" name=\"hive-wf\">\n" +
+ "<workflow-app xmlns=\"uri:oozie:workflow:1.0\" name=\"hive-wf\">\n" +
" <start to=\"hive-node\"/>\n" +
"\n" +
" <action name=\"hive-node\">\n" +
@@ -139,7 +139,7 @@ public class TestSchemaService extends XTestCase {
" <delete path=\"${nameNode}/user/${wf:user()}/${examplesRoot}/output-data/hive\"/>\n" +
" <mkdir path=\"${nameNode}/user/${wf:user()}/${examplesRoot}/output-data\"/>\n" +
" </prepare>\n" +
- " <launcher xmlns=\"uri:oozie:workflow:1.0\">\n" +
+ " <launcher>\n" +
" <memory.mb>1024</memory.mb>\n" +
" <vcores>2</vcores>\n" +
" <java-opts>dummyJavaOpts</java-opts>\n" +
````
- Robert
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61529/#review183065
-----------------------------------------------------------
On Sept. 8, 2017, 10:04 a.m., Attila Sasvari wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61529/
> -----------------------------------------------------------
>
> (Updated Sept. 8, 2017, 10:04 a.m.)
>
>
> Review request for oozie and Peter Cseh.
>
>
> Repository: oozie-git
>
>
> Description
> -------
>
> New workflow xsd to allow changing launcher configuration (e.g. setting java system properties)
>
>
> Diffs
> -----
>
> client/src/main/resources/hive-action-1.0.xsd PRE-CREATION
> client/src/main/resources/hive2-action-1.0.xsd PRE-CREATION
> client/src/main/resources/oozie-common-1.0.xsd PRE-CREATION
> client/src/main/resources/oozie-workflow-1.0.xsd PRE-CREATION
> client/src/main/resources/shell-action-1.0.xsd PRE-CREATION
> client/src/main/resources/spark-action-1.0.xsd PRE-CREATION
> client/src/main/resources/sqoop-action-1.0.xsd PRE-CREATION
> core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java bca79aa052521ea4f6f16e76bd69f84fb16be790
> core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 8fdc50cce15271c13b20d1aaab2ffb95cb8fa711
> core/src/main/java/org/apache/oozie/action/ssh/SshActionExecutor.java 7e3348550e2ef44ae4bd6c5a7a417052bf9c108a
> core/src/main/java/org/apache/oozie/jms/MessageReceiver.java 47bfd2bcf9b53eed0882c51cdb5c530e0f6ccfe7
> core/src/main/java/org/apache/oozie/service/LiteWorkflowStoreService.java ffc29af2f834da6d0890ea7215c5a62fd7cd693e
> core/src/main/java/org/apache/oozie/service/SchemaService.java 137e2c0b4840c6d3858683db1aa38f54bd55be92
> core/src/main/java/org/apache/oozie/util/WritableUtils.java aa027e37ba23d4f481698e0bcd93c26c763a0b1f
> core/src/main/java/org/apache/oozie/util/schema/Input.java PRE-CREATION
> core/src/main/java/org/apache/oozie/util/schema/ResourceResolver.java PRE-CREATION
> core/src/main/java/org/apache/oozie/workflow/lite/LauncherConfigHandler.java PRE-CREATION
> core/src/main/java/org/apache/oozie/workflow/lite/LiteWorkflowAppParser.java a74e5c759fe3e336e7e98c61b8f5ac52efa4100c
> core/src/main/java/org/apache/oozie/workflow/lite/LiteWorkflowLib.java 23df0867e5984c518e8424d79365997bec28f619
> core/src/main/resources/oozie-default.xml 0d174b5069aff36ca3a9d48722c5bba69623a046
> core/src/test/java/org/apache/oozie/action/hadoop/ActionExecutorTestCase.java d74160a09ee237f4067e00ded526e7ec94528a6b
> core/src/test/java/org/apache/oozie/action/hadoop/LauncherMainTester.java 6cee7a8948ca7f2c7aaf44d4f172d18a1308a482
> core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java ce674adf9cffa20690083e6298875a6f9047a109
> core/src/test/java/org/apache/oozie/service/TestSchemaService.java 88a10dbead32879334c3953a90a411a3a18fc0b3
> core/src/test/java/org/apache/oozie/util/TestMetricsInstrumentation.java a882c82b25154338f89e8a90f831b6f0300fbfe7
> core/src/test/java/org/apache/oozie/workflow/lite/TestLiteWorkflowAppParser.java 21332404bea79fe0a3cea045d57b3e52a1f15cae
> core/src/test/resources/wf-schema-global-launcherconf-override.xml PRE-CREATION
> core/src/test/resources/wf-schema-global-launcherconf.xml PRE-CREATION
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java 6a98d6ef3a09a7a75272f3f0f9a9c2fc5472e76d
> sharelib/pig/src/test/java/org/apache/oozie/action/hadoop/UDFTester.java ff999ff2cd4875fbf3ad5e1a7eacf9972e848630
>
>
> Diff: https://reviews.apache.org/r/61529/diff/8/
>
>
> Testing
> -------
>
> - TestJavaActionExecutor, TestLiteWorkflowAppParser tests passed
> - java-main example on pseudo hadoop succeeded, modified workflow xml so that verbose class loading information is printed out of stdout of the launcher job
>
>
> Thanks,
>
> Attila Sasvari
>
>
Re: Review Request 61529: OOZIE-2687 Create XML schema for launcher
configurations
Posted by Attila Sasvari <as...@cloudera.com>.
> On Aug. 16, 2017, 8:33 p.m., Robert Kanter wrote:
> > client/src/main/resources/hive-action-0.7.xsd
> > Lines 49 (patched)
> > <https://reviews.apache.org/r/61529/diff/2/?file=1796924#file1796924line49>
> >
> > Can we put CONFIGURATION into oozie-common?
> >
> > Same with PREPARE, DELETE, and MKDIR?
>
> Attila Sasvari wrote:
> Right now I changed oozie-common so that its targetNamespace is targetNamespace="uri:oozie:workflow:1.0"
> This way common prefix is not needed if ``<launcher>`` is present in the global config (include statement can be used).
>
> ```
> <global>
> <launcher>
> <workflow:memory>1024</workflow:memory>
> <workflow:vcores>2</workflow:vcores>
> <workflow:java-opts>-verbose:class</workflow:java-opts>
> <workflow:env>dummyEnv=42</workflow:env>
> <workflow:queue>dummyQueue</workflow:queue>
> </launcher>
> </global>
> ```
>
> Other actions use the import statement and still need the ``workflow`` prefix for the vcore,memory, etc. if they include the ``<launcher>`` element. They also need to define workflow namespace with ``xmlns:workflow``.
>
> ```
> <workflow-app xmlns="uri:oozie:workflow:1.0" xmlns:workflow="uri:oozie:workflow:1.0" name="shell-wf">
> <start to="shell-node"/>
> <action name="shell-node">
> <shell xmlns="uri:oozie:shell-action:1.0">
> <job-tracker>${jobTracker}</job-tracker>
> <name-node>${nameNode}</name-node>
> <launcher>
> <workflow:memory>1024</workflow:memory>
> <workflow:vcores>2</workflow:vcores>
> <workflow:java-opts>-verbose:class</workflow:java-opts>
> <workflow:env>dummyEnv=42</workflow:env>
> <workflow:queue>dummyQueue</workflow:queue>
> <workflow:sharelib>a,b,c</workflow:sharelib>
> </launcher>
> ```
>
> ``xs:any`` would require the prefix (like the sla complex type). Using ``<xs:any namespace="##other" minOccurs="1" maxOccurs="1"/>`` could work, but is too permissive - for example you could add any element (e.g. a hive action) instead of a ``launcher``.
>
> I am trying to find a better way, but please share if you have some ideas.
>
> Attila Sasvari wrote:
> If I use ``ref`` like ``<xs:element ref="workflow:launcher" minOccurs="0" maxOccurs="1"/>`` for example in the hive action, it becomes a bit more clean to define a ``launcher`` element:
>
> ```
> <workflow-app xmlns="uri:oozie:workflow:1.0" name="hive-wf">
> ...
> <hive xmlns="uri:oozie:workflow:hive-action:1.0">
> ...
> <launcher xmlns="uri:oozie:workflow:1.0">
> <memory>1024</memory>
> </launcher>"
> ```
>
> This way it is not needed to define the workflow namespace in the workflow element and use it as prefix.
>
> I was also considering to add other elements to the common workflow namespace. Because of the namespace differences of actions, it would not be easy to include say ``configuration`` - we would need to define workflow namespace and prefix.
>
> Peter Cseh wrote:
> I like the idea of setting xmlns for launcher inside the launcher tag if we don't have to use prefixes after that.
>
> Robert Kanter wrote:
> Have you tried using include instead of import? From some googling, it sounds like if you (a) use include, (b) don't set a target namespace, and (c) don't have a namespace in the included xsd, then it will work the way we want.
>
> https://stackoverflow.com/questions/2357943/whats-the-difference-between-xsdinclude-and-xsdimport
> https://stackoverflow.com/questions/914020/xsd-with-imports-and-namespaces
> https://stackoverflow.com/questions/9805359/how-to-handle-multiple-namespaces-with-different-uri-in-xsd
>
> This link talks about this case from an XSD spec point of view, but it's hard to understand:
> https://www.w3.org/TR/xmlschema-1/#compound-schema
>
>
> If that works, then I'd vote we go for that. Otherwise, I'd vote for whatever is easiest for the user. If that means we have to duplicate XSD code like we do today, that's ok.
Thanks for the ideas. I have already read some of those entries. Anyway, I tried again the suggestion: I had to use ``import`` otherwise referencing did not work:
- Using ``<xs:include schemaLocation="oozie-common-1.0.xsd"/>`` and ``<xs:element ref="launcher" minOccurs="0" maxOccurs="1"/>`` :
```
Caused by: org.xml.sax.SAXParseException; systemId: file:///Users/asasvari/workspace/apache/oozie_dup/core/oozie-workflow-1.0.xsd; lineNumber: 71; columnNumber: 69; src-resolve.4.1: Error resolving component 'launcher'. It was detected that 'launcher' has no namespace, but components with no target namespace are not referenceable from schema document 'file:///Users/asasvari/workspace/apache/oozie_dup/core/oozie-workflow-1.0.xsd'. If 'launcher' is intended to have a namespace, perhaps a prefix needs to be provided. If it is intended that 'launcher' has no namespace, then an 'import' without a "namespace" attribute should be added to 'file:///Users/asasvari/workspace/apache/oozie_dup/core/oozie-workflow-1.0.xsd'.
```
- Using ``<xs:element name="launcher" type="LAUNCHER" minOccurs="0" maxOccurs="1"/>`` also resulted in:
```
Caused by: org.xml.sax.SAXParseException; systemId: file:///Users/asasvari/workspace/apache/oozie_dup/core/oozie-workflow-1.0.xsd; lineNumber: 71; columnNumber: 86; src-resolve.4.1: Error resolving component 'LAUNCHER'. It was detected that 'LAUNCHER' has no namespace, but components with no target namespace are not referenceable from schema document 'file:///Users/asasvari/workspace/apache/oozie_dup/core/oozie-workflow-1.0.xsd'. If 'LAUNCHER' is intended to have a namespace, perhaps a prefix needs to be provided. If it is intended that 'LAUNCHER' has no namespace, then an 'import' without a "namespace" attribute should be added to 'file:///Users/asasvari/workspace/apache/oozie_dup/core/oozie-workflow-1.0.xsd'.
```
When I switched to ``<xs:import schemaLocation="oozie-common-1.0.xsd"/>`` and ``<xs:element ref="launcher" minOccurs="0" maxOccurs="1"/>``, the following exception was thrown by Xerces:
```
org.xml.sax.SAXParseException; lineNumber: 4; columnNumber: 24; cvc-complex-type.2.4.a: Invalid content was found starting with element 'memory.mb'. One of '{memory.mb, vcores, java-opts, env, queue, sharelib}' is expected.
```
Running the sqoop action validation test was also strange:
```
org.xml.sax.SAXParseException; lineNumber: 3; columnNumber: 19; cvc-complex-type.2.4.a: Invalid content was found starting with element 'launcher'. One of '{"uri:oozie:workflow:1.0":job-tracker, "uri:oozie:workflow:1.0":resource-manager, "uri:oozie:workflow:1.0":name-node, "uri:oozie:workflow:1.0":job-xml, launcher, "uri:oozie:workflow:1.0":configuration}' is expected.
```
It is strange that referencing an element with a complex type from a default namespace fails to validate.
- Attila
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61529/#review183065
-----------------------------------------------------------
On Sept. 8, 2017, 10:04 a.m., Attila Sasvari wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61529/
> -----------------------------------------------------------
>
> (Updated Sept. 8, 2017, 10:04 a.m.)
>
>
> Review request for oozie and Peter Cseh.
>
>
> Repository: oozie-git
>
>
> Description
> -------
>
> New workflow xsd to allow changing launcher configuration (e.g. setting java system properties)
>
>
> Diffs
> -----
>
> client/src/main/resources/hive-action-1.0.xsd PRE-CREATION
> client/src/main/resources/hive2-action-1.0.xsd PRE-CREATION
> client/src/main/resources/oozie-common-1.0.xsd PRE-CREATION
> client/src/main/resources/oozie-workflow-1.0.xsd PRE-CREATION
> client/src/main/resources/shell-action-1.0.xsd PRE-CREATION
> client/src/main/resources/spark-action-1.0.xsd PRE-CREATION
> client/src/main/resources/sqoop-action-1.0.xsd PRE-CREATION
> core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java bca79aa052521ea4f6f16e76bd69f84fb16be790
> core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 8fdc50cce15271c13b20d1aaab2ffb95cb8fa711
> core/src/main/java/org/apache/oozie/action/ssh/SshActionExecutor.java 7e3348550e2ef44ae4bd6c5a7a417052bf9c108a
> core/src/main/java/org/apache/oozie/jms/MessageReceiver.java 47bfd2bcf9b53eed0882c51cdb5c530e0f6ccfe7
> core/src/main/java/org/apache/oozie/service/LiteWorkflowStoreService.java ffc29af2f834da6d0890ea7215c5a62fd7cd693e
> core/src/main/java/org/apache/oozie/service/SchemaService.java 137e2c0b4840c6d3858683db1aa38f54bd55be92
> core/src/main/java/org/apache/oozie/util/WritableUtils.java aa027e37ba23d4f481698e0bcd93c26c763a0b1f
> core/src/main/java/org/apache/oozie/util/schema/Input.java PRE-CREATION
> core/src/main/java/org/apache/oozie/util/schema/ResourceResolver.java PRE-CREATION
> core/src/main/java/org/apache/oozie/workflow/lite/LauncherConfigHandler.java PRE-CREATION
> core/src/main/java/org/apache/oozie/workflow/lite/LiteWorkflowAppParser.java a74e5c759fe3e336e7e98c61b8f5ac52efa4100c
> core/src/main/java/org/apache/oozie/workflow/lite/LiteWorkflowLib.java 23df0867e5984c518e8424d79365997bec28f619
> core/src/main/resources/oozie-default.xml 0d174b5069aff36ca3a9d48722c5bba69623a046
> core/src/test/java/org/apache/oozie/action/hadoop/ActionExecutorTestCase.java d74160a09ee237f4067e00ded526e7ec94528a6b
> core/src/test/java/org/apache/oozie/action/hadoop/LauncherMainTester.java 6cee7a8948ca7f2c7aaf44d4f172d18a1308a482
> core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java ce674adf9cffa20690083e6298875a6f9047a109
> core/src/test/java/org/apache/oozie/service/TestSchemaService.java 88a10dbead32879334c3953a90a411a3a18fc0b3
> core/src/test/java/org/apache/oozie/util/TestMetricsInstrumentation.java a882c82b25154338f89e8a90f831b6f0300fbfe7
> core/src/test/java/org/apache/oozie/workflow/lite/TestLiteWorkflowAppParser.java 21332404bea79fe0a3cea045d57b3e52a1f15cae
> core/src/test/resources/wf-schema-global-launcherconf-override.xml PRE-CREATION
> core/src/test/resources/wf-schema-global-launcherconf.xml PRE-CREATION
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java 6a98d6ef3a09a7a75272f3f0f9a9c2fc5472e76d
> sharelib/pig/src/test/java/org/apache/oozie/action/hadoop/UDFTester.java ff999ff2cd4875fbf3ad5e1a7eacf9972e848630
>
>
> Diff: https://reviews.apache.org/r/61529/diff/8/
>
>
> Testing
> -------
>
> - TestJavaActionExecutor, TestLiteWorkflowAppParser tests passed
> - java-main example on pseudo hadoop succeeded, modified workflow xml so that verbose class loading information is printed out of stdout of the launcher job
>
>
> Thanks,
>
> Attila Sasvari
>
>
Re: Review Request 61529: OOZIE-2687 Create XML schema for launcher
configurations
Posted by Attila Sasvari <as...@cloudera.com>.
> On Aug. 16, 2017, 8:33 p.m., Robert Kanter wrote:
> > client/src/main/resources/hive-action-0.7.xsd
> > Lines 49 (patched)
> > <https://reviews.apache.org/r/61529/diff/2/?file=1796924#file1796924line49>
> >
> > Can we put CONFIGURATION into oozie-common?
> >
> > Same with PREPARE, DELETE, and MKDIR?
Right now I changed oozie-common so that its targetNamespace is targetNamespace="uri:oozie:workflow:1.0"
This way common prefix is not needed if ``<launcher>`` is present in the global config (include statement can be used).
```
<global>
<launcher>
<workflow:memory>1024</workflow:memory>
<workflow:vcores>2</workflow:vcores>
<workflow:java-opts>-verbose:class</workflow:java-opts>
<workflow:env>dummyEnv=42</workflow:env>
<workflow:queue>dummyQueue</workflow:queue>
</launcher>
</global>
```
Other actions use the import statement and still need the ``workflow`` prefix for the vcore,memory, etc. if they include the ``<launcher>`` element. They also need to define workflow namespace with ``xmlns:workflow``.
```
<workflow-app xmlns="uri:oozie:workflow:1.0" xmlns:workflow="uri:oozie:workflow:1.0" name="shell-wf">
<start to="shell-node"/>
<action name="shell-node">
<shell xmlns="uri:oozie:shell-action:1.0">
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<launcher>
<workflow:memory>1024</workflow:memory>
<workflow:vcores>2</workflow:vcores>
<workflow:java-opts>-verbose:class</workflow:java-opts>
<workflow:env>dummyEnv=42</workflow:env>
<workflow:queue>dummyQueue</workflow:queue>
<workflow:sharelib>a,b,c</workflow:sharelib>
</launcher>
```
``xs:any`` would require the prefix (like the sla complex type). Using ``<xs:any namespace="##other" minOccurs="1" maxOccurs="1"/>`` could work, but is too permissive - for example you could add any element (e.g. a hive action) instead of a ``launcher``.
I am trying to find a better way, but please share if you have some ideas.
- Attila
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61529/#review183065
-----------------------------------------------------------
On Sept. 5, 2017, 3:52 p.m., Attila Sasvari wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61529/
> -----------------------------------------------------------
>
> (Updated Sept. 5, 2017, 3:52 p.m.)
>
>
> Review request for oozie and Peter Cseh.
>
>
> Repository: oozie-git
>
>
> Description
> -------
>
> New workflow xsd to allow changing launcher configuration (e.g. setting java system properties)
>
>
> Diffs
> -----
>
> client/src/main/resources/hive-action-1.0.xsd PRE-CREATION
> client/src/main/resources/hive2-action-1.0.xsd PRE-CREATION
> client/src/main/resources/oozie-common-1.0.xsd PRE-CREATION
> client/src/main/resources/oozie-workflow-1.0.xsd PRE-CREATION
> client/src/main/resources/shell-action-1.0.xsd PRE-CREATION
> client/src/main/resources/spark-action-1.0.xsd PRE-CREATION
> client/src/main/resources/sqoop-action-1.0.xsd PRE-CREATION
> core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java bca79aa052521ea4f6f16e76bd69f84fb16be790
> core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 8fdc50cce15271c13b20d1aaab2ffb95cb8fa711
> core/src/main/java/org/apache/oozie/action/ssh/SshActionExecutor.java 7e3348550e2ef44ae4bd6c5a7a417052bf9c108a
> core/src/main/java/org/apache/oozie/jms/MessageReceiver.java 47bfd2bcf9b53eed0882c51cdb5c530e0f6ccfe7
> core/src/main/java/org/apache/oozie/service/LiteWorkflowStoreService.java ffc29af2f834da6d0890ea7215c5a62fd7cd693e
> core/src/main/java/org/apache/oozie/service/SchemaService.java 137e2c0b4840c6d3858683db1aa38f54bd55be92
> core/src/main/java/org/apache/oozie/util/WritableUtils.java aa027e37ba23d4f481698e0bcd93c26c763a0b1f
> core/src/main/java/org/apache/oozie/util/schema/Input.java PRE-CREATION
> core/src/main/java/org/apache/oozie/util/schema/ResourceResolver.java PRE-CREATION
> core/src/main/java/org/apache/oozie/workflow/lite/LauncherConfigHandler.java PRE-CREATION
> core/src/main/java/org/apache/oozie/workflow/lite/LiteWorkflowAppParser.java a74e5c759fe3e336e7e98c61b8f5ac52efa4100c
> core/src/main/java/org/apache/oozie/workflow/lite/LiteWorkflowLib.java 23df0867e5984c518e8424d79365997bec28f619
> core/src/main/resources/oozie-default.xml 0d174b5069aff36ca3a9d48722c5bba69623a046
> core/src/test/java/org/apache/oozie/action/hadoop/ActionExecutorTestCase.java d74160a09ee237f4067e00ded526e7ec94528a6b
> core/src/test/java/org/apache/oozie/action/hadoop/LauncherMainTester.java 6cee7a8948ca7f2c7aaf44d4f172d18a1308a482
> core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java ce674adf9cffa20690083e6298875a6f9047a109
> core/src/test/java/org/apache/oozie/service/TestSchemaService.java 88a10dbead32879334c3953a90a411a3a18fc0b3
> core/src/test/java/org/apache/oozie/util/TestMetricsInstrumentation.java a882c82b25154338f89e8a90f831b6f0300fbfe7
> core/src/test/java/org/apache/oozie/workflow/lite/TestLiteWorkflowAppParser.java 21332404bea79fe0a3cea045d57b3e52a1f15cae
> core/src/test/resources/wf-schema-global-launcherconf-override.xml PRE-CREATION
> core/src/test/resources/wf-schema-global-launcherconf.xml PRE-CREATION
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java 6a98d6ef3a09a7a75272f3f0f9a9c2fc5472e76d
> sharelib/pig/src/test/java/org/apache/oozie/action/hadoop/UDFTester.java ff999ff2cd4875fbf3ad5e1a7eacf9972e848630
>
>
> Diff: https://reviews.apache.org/r/61529/diff/3/
>
>
> Testing
> -------
>
> - TestJavaActionExecutor, TestLiteWorkflowAppParser tests passed
> - java-main example on pseudo hadoop succeeded, modified workflow xml so that verbose class loading information is printed out of stdout of the launcher job
>
>
> Thanks,
>
> Attila Sasvari
>
>
Re: Review Request 61529: OOZIE-2687 Create XML schema for launcher
configurations
Posted by Robert Kanter <rk...@cloudera.com>.
> On Aug. 16, 2017, 8:33 p.m., Robert Kanter wrote:
> > client/src/main/resources/hive-action-0.7.xsd
> > Lines 49 (patched)
> > <https://reviews.apache.org/r/61529/diff/2/?file=1796924#file1796924line49>
> >
> > Can we put CONFIGURATION into oozie-common?
> >
> > Same with PREPARE, DELETE, and MKDIR?
>
> Attila Sasvari wrote:
> Right now I changed oozie-common so that its targetNamespace is targetNamespace="uri:oozie:workflow:1.0"
> This way common prefix is not needed if ``<launcher>`` is present in the global config (include statement can be used).
>
> ```
> <global>
> <launcher>
> <workflow:memory>1024</workflow:memory>
> <workflow:vcores>2</workflow:vcores>
> <workflow:java-opts>-verbose:class</workflow:java-opts>
> <workflow:env>dummyEnv=42</workflow:env>
> <workflow:queue>dummyQueue</workflow:queue>
> </launcher>
> </global>
> ```
>
> Other actions use the import statement and still need the ``workflow`` prefix for the vcore,memory, etc. if they include the ``<launcher>`` element. They also need to define workflow namespace with ``xmlns:workflow``.
>
> ```
> <workflow-app xmlns="uri:oozie:workflow:1.0" xmlns:workflow="uri:oozie:workflow:1.0" name="shell-wf">
> <start to="shell-node"/>
> <action name="shell-node">
> <shell xmlns="uri:oozie:shell-action:1.0">
> <job-tracker>${jobTracker}</job-tracker>
> <name-node>${nameNode}</name-node>
> <launcher>
> <workflow:memory>1024</workflow:memory>
> <workflow:vcores>2</workflow:vcores>
> <workflow:java-opts>-verbose:class</workflow:java-opts>
> <workflow:env>dummyEnv=42</workflow:env>
> <workflow:queue>dummyQueue</workflow:queue>
> <workflow:sharelib>a,b,c</workflow:sharelib>
> </launcher>
> ```
>
> ``xs:any`` would require the prefix (like the sla complex type). Using ``<xs:any namespace="##other" minOccurs="1" maxOccurs="1"/>`` could work, but is too permissive - for example you could add any element (e.g. a hive action) instead of a ``launcher``.
>
> I am trying to find a better way, but please share if you have some ideas.
>
> Attila Sasvari wrote:
> If I use ``ref`` like ``<xs:element ref="workflow:launcher" minOccurs="0" maxOccurs="1"/>`` for example in the hive action, it becomes a bit more clean to define a ``launcher`` element:
>
> ```
> <workflow-app xmlns="uri:oozie:workflow:1.0" name="hive-wf">
> ...
> <hive xmlns="uri:oozie:workflow:hive-action:1.0">
> ...
> <launcher xmlns="uri:oozie:workflow:1.0">
> <memory>1024</memory>
> </launcher>"
> ```
>
> This way it is not needed to define the workflow namespace in the workflow element and use it as prefix.
>
> I was also considering to add other elements to the common workflow namespace. Because of the namespace differences of actions, it would not be easy to include say ``configuration`` - we would need to define workflow namespace and prefix.
>
> Peter Cseh wrote:
> I like the idea of setting xmlns for launcher inside the launcher tag if we don't have to use prefixes after that.
Have you tried using include instead of import? From some googling, it sounds like if you (a) use include, (b) don't set a target namespace, and (c) don't have a namespace in the included xsd, then it will work the way we want.
https://stackoverflow.com/questions/2357943/whats-the-difference-between-xsdinclude-and-xsdimport
https://stackoverflow.com/questions/914020/xsd-with-imports-and-namespaces
https://stackoverflow.com/questions/9805359/how-to-handle-multiple-namespaces-with-different-uri-in-xsd
This link talks about this case from an XSD spec point of view, but it's hard to understand:
https://www.w3.org/TR/xmlschema-1/#compound-schema
If that works, then I'd vote we go for that. Otherwise, I'd vote for whatever is easiest for the user. If that means we have to duplicate XSD code like we do today, that's ok.
- Robert
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61529/#review183065
-----------------------------------------------------------
On Sept. 7, 2017, 4:33 p.m., Attila Sasvari wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61529/
> -----------------------------------------------------------
>
> (Updated Sept. 7, 2017, 4:33 p.m.)
>
>
> Review request for oozie and Peter Cseh.
>
>
> Repository: oozie-git
>
>
> Description
> -------
>
> New workflow xsd to allow changing launcher configuration (e.g. setting java system properties)
>
>
> Diffs
> -----
>
> client/src/main/resources/hive-action-1.0.xsd PRE-CREATION
> client/src/main/resources/hive2-action-1.0.xsd PRE-CREATION
> client/src/main/resources/oozie-common-1.0.xsd PRE-CREATION
> client/src/main/resources/oozie-workflow-1.0.xsd PRE-CREATION
> client/src/main/resources/shell-action-1.0.xsd PRE-CREATION
> client/src/main/resources/spark-action-1.0.xsd PRE-CREATION
> client/src/main/resources/sqoop-action-1.0.xsd PRE-CREATION
> core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java bca79aa052521ea4f6f16e76bd69f84fb16be790
> core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 8fdc50cce15271c13b20d1aaab2ffb95cb8fa711
> core/src/main/java/org/apache/oozie/action/ssh/SshActionExecutor.java 7e3348550e2ef44ae4bd6c5a7a417052bf9c108a
> core/src/main/java/org/apache/oozie/jms/MessageReceiver.java 47bfd2bcf9b53eed0882c51cdb5c530e0f6ccfe7
> core/src/main/java/org/apache/oozie/service/LiteWorkflowStoreService.java ffc29af2f834da6d0890ea7215c5a62fd7cd693e
> core/src/main/java/org/apache/oozie/service/SchemaService.java 137e2c0b4840c6d3858683db1aa38f54bd55be92
> core/src/main/java/org/apache/oozie/util/WritableUtils.java aa027e37ba23d4f481698e0bcd93c26c763a0b1f
> core/src/main/java/org/apache/oozie/util/schema/Input.java PRE-CREATION
> core/src/main/java/org/apache/oozie/util/schema/ResourceResolver.java PRE-CREATION
> core/src/main/java/org/apache/oozie/workflow/lite/LauncherConfigHandler.java PRE-CREATION
> core/src/main/java/org/apache/oozie/workflow/lite/LiteWorkflowAppParser.java a74e5c759fe3e336e7e98c61b8f5ac52efa4100c
> core/src/main/java/org/apache/oozie/workflow/lite/LiteWorkflowLib.java 23df0867e5984c518e8424d79365997bec28f619
> core/src/main/resources/oozie-default.xml 0d174b5069aff36ca3a9d48722c5bba69623a046
> core/src/test/java/org/apache/oozie/action/hadoop/ActionExecutorTestCase.java d74160a09ee237f4067e00ded526e7ec94528a6b
> core/src/test/java/org/apache/oozie/action/hadoop/LauncherMainTester.java 6cee7a8948ca7f2c7aaf44d4f172d18a1308a482
> core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java ce674adf9cffa20690083e6298875a6f9047a109
> core/src/test/java/org/apache/oozie/service/TestSchemaService.java 88a10dbead32879334c3953a90a411a3a18fc0b3
> core/src/test/java/org/apache/oozie/util/TestMetricsInstrumentation.java a882c82b25154338f89e8a90f831b6f0300fbfe7
> core/src/test/java/org/apache/oozie/workflow/lite/TestLiteWorkflowAppParser.java 21332404bea79fe0a3cea045d57b3e52a1f15cae
> core/src/test/resources/wf-schema-global-launcherconf-override.xml PRE-CREATION
> core/src/test/resources/wf-schema-global-launcherconf.xml PRE-CREATION
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java 6a98d6ef3a09a7a75272f3f0f9a9c2fc5472e76d
> sharelib/pig/src/test/java/org/apache/oozie/action/hadoop/UDFTester.java ff999ff2cd4875fbf3ad5e1a7eacf9972e848630
>
>
> Diff: https://reviews.apache.org/r/61529/diff/7/
>
>
> Testing
> -------
>
> - TestJavaActionExecutor, TestLiteWorkflowAppParser tests passed
> - java-main example on pseudo hadoop succeeded, modified workflow xml so that verbose class loading information is printed out of stdout of the launcher job
>
>
> Thanks,
>
> Attila Sasvari
>
>
Re: Review Request 61529: OOZIE-2687 Create XML schema for launcher
configurations
Posted by Peter Cseh <ge...@cloudera.com>.
> On Aug. 16, 2017, 8:33 p.m., Robert Kanter wrote:
> > client/src/main/resources/hive-action-0.7.xsd
> > Lines 49 (patched)
> > <https://reviews.apache.org/r/61529/diff/2/?file=1796924#file1796924line49>
> >
> > Can we put CONFIGURATION into oozie-common?
> >
> > Same with PREPARE, DELETE, and MKDIR?
>
> Attila Sasvari wrote:
> Right now I changed oozie-common so that its targetNamespace is targetNamespace="uri:oozie:workflow:1.0"
> This way common prefix is not needed if ``<launcher>`` is present in the global config (include statement can be used).
>
> ```
> <global>
> <launcher>
> <workflow:memory>1024</workflow:memory>
> <workflow:vcores>2</workflow:vcores>
> <workflow:java-opts>-verbose:class</workflow:java-opts>
> <workflow:env>dummyEnv=42</workflow:env>
> <workflow:queue>dummyQueue</workflow:queue>
> </launcher>
> </global>
> ```
>
> Other actions use the import statement and still need the ``workflow`` prefix for the vcore,memory, etc. if they include the ``<launcher>`` element. They also need to define workflow namespace with ``xmlns:workflow``.
>
> ```
> <workflow-app xmlns="uri:oozie:workflow:1.0" xmlns:workflow="uri:oozie:workflow:1.0" name="shell-wf">
> <start to="shell-node"/>
> <action name="shell-node">
> <shell xmlns="uri:oozie:shell-action:1.0">
> <job-tracker>${jobTracker}</job-tracker>
> <name-node>${nameNode}</name-node>
> <launcher>
> <workflow:memory>1024</workflow:memory>
> <workflow:vcores>2</workflow:vcores>
> <workflow:java-opts>-verbose:class</workflow:java-opts>
> <workflow:env>dummyEnv=42</workflow:env>
> <workflow:queue>dummyQueue</workflow:queue>
> <workflow:sharelib>a,b,c</workflow:sharelib>
> </launcher>
> ```
>
> ``xs:any`` would require the prefix (like the sla complex type). Using ``<xs:any namespace="##other" minOccurs="1" maxOccurs="1"/>`` could work, but is too permissive - for example you could add any element (e.g. a hive action) instead of a ``launcher``.
>
> I am trying to find a better way, but please share if you have some ideas.
>
> Attila Sasvari wrote:
> If I use ``ref`` like ``<xs:element ref="workflow:launcher" minOccurs="0" maxOccurs="1"/>`` for example in the hive action, it becomes a bit more clean to define a ``launcher`` element:
>
> ```
> <workflow-app xmlns="uri:oozie:workflow:1.0" name="hive-wf">
> ...
> <hive xmlns="uri:oozie:workflow:hive-action:1.0">
> ...
> <launcher xmlns="uri:oozie:workflow:1.0">
> <memory>1024</memory>
> </launcher>"
> ```
>
> This way it is not needed to define the workflow namespace in the workflow element and use it as prefix.
>
> I was also considering to add other elements to the common workflow namespace. Because of the namespace differences of actions, it would not be easy to include say ``configuration`` - we would need to define workflow namespace and prefix.
I like the idea of setting xmlns for launcher inside the launcher tag if we don't have to use prefixes after that.
- Peter
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61529/#review183065
-----------------------------------------------------------
On Sept. 7, 2017, 4:33 p.m., Attila Sasvari wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61529/
> -----------------------------------------------------------
>
> (Updated Sept. 7, 2017, 4:33 p.m.)
>
>
> Review request for oozie and Peter Cseh.
>
>
> Repository: oozie-git
>
>
> Description
> -------
>
> New workflow xsd to allow changing launcher configuration (e.g. setting java system properties)
>
>
> Diffs
> -----
>
> client/src/main/resources/hive-action-1.0.xsd PRE-CREATION
> client/src/main/resources/hive2-action-1.0.xsd PRE-CREATION
> client/src/main/resources/oozie-common-1.0.xsd PRE-CREATION
> client/src/main/resources/oozie-workflow-1.0.xsd PRE-CREATION
> client/src/main/resources/shell-action-1.0.xsd PRE-CREATION
> client/src/main/resources/spark-action-1.0.xsd PRE-CREATION
> client/src/main/resources/sqoop-action-1.0.xsd PRE-CREATION
> core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java bca79aa052521ea4f6f16e76bd69f84fb16be790
> core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 8fdc50cce15271c13b20d1aaab2ffb95cb8fa711
> core/src/main/java/org/apache/oozie/action/ssh/SshActionExecutor.java 7e3348550e2ef44ae4bd6c5a7a417052bf9c108a
> core/src/main/java/org/apache/oozie/jms/MessageReceiver.java 47bfd2bcf9b53eed0882c51cdb5c530e0f6ccfe7
> core/src/main/java/org/apache/oozie/service/LiteWorkflowStoreService.java ffc29af2f834da6d0890ea7215c5a62fd7cd693e
> core/src/main/java/org/apache/oozie/service/SchemaService.java 137e2c0b4840c6d3858683db1aa38f54bd55be92
> core/src/main/java/org/apache/oozie/util/WritableUtils.java aa027e37ba23d4f481698e0bcd93c26c763a0b1f
> core/src/main/java/org/apache/oozie/util/schema/Input.java PRE-CREATION
> core/src/main/java/org/apache/oozie/util/schema/ResourceResolver.java PRE-CREATION
> core/src/main/java/org/apache/oozie/workflow/lite/LauncherConfigHandler.java PRE-CREATION
> core/src/main/java/org/apache/oozie/workflow/lite/LiteWorkflowAppParser.java a74e5c759fe3e336e7e98c61b8f5ac52efa4100c
> core/src/main/java/org/apache/oozie/workflow/lite/LiteWorkflowLib.java 23df0867e5984c518e8424d79365997bec28f619
> core/src/main/resources/oozie-default.xml 0d174b5069aff36ca3a9d48722c5bba69623a046
> core/src/test/java/org/apache/oozie/action/hadoop/ActionExecutorTestCase.java d74160a09ee237f4067e00ded526e7ec94528a6b
> core/src/test/java/org/apache/oozie/action/hadoop/LauncherMainTester.java 6cee7a8948ca7f2c7aaf44d4f172d18a1308a482
> core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java ce674adf9cffa20690083e6298875a6f9047a109
> core/src/test/java/org/apache/oozie/service/TestSchemaService.java 88a10dbead32879334c3953a90a411a3a18fc0b3
> core/src/test/java/org/apache/oozie/util/TestMetricsInstrumentation.java a882c82b25154338f89e8a90f831b6f0300fbfe7
> core/src/test/java/org/apache/oozie/workflow/lite/TestLiteWorkflowAppParser.java 21332404bea79fe0a3cea045d57b3e52a1f15cae
> core/src/test/resources/wf-schema-global-launcherconf-override.xml PRE-CREATION
> core/src/test/resources/wf-schema-global-launcherconf.xml PRE-CREATION
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java 6a98d6ef3a09a7a75272f3f0f9a9c2fc5472e76d
> sharelib/pig/src/test/java/org/apache/oozie/action/hadoop/UDFTester.java ff999ff2cd4875fbf3ad5e1a7eacf9972e848630
>
>
> Diff: https://reviews.apache.org/r/61529/diff/7/
>
>
> Testing
> -------
>
> - TestJavaActionExecutor, TestLiteWorkflowAppParser tests passed
> - java-main example on pseudo hadoop succeeded, modified workflow xml so that verbose class loading information is printed out of stdout of the launcher job
>
>
> Thanks,
>
> Attila Sasvari
>
>
Re: Review Request 61529: OOZIE-2687 Create XML schema for launcher
configurations
Posted by Robert Kanter <rk...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61529/#review183065
-----------------------------------------------------------
client/src/main/resources/hive-action-0.7.xsd
Lines 31 (patched)
<https://reviews.apache.org/r/61529/#comment259056>
Should we do "resource-manager" to be consistent with "job-tracker", "name-node", "job-xml", "capture-output", etc?
client/src/main/resources/hive-action-0.7.xsd
Lines 49 (patched)
<https://reviews.apache.org/r/61529/#comment259057>
Can we put CONFIGURATION into oozie-common?
Same with PREPARE, DELETE, and MKDIR?
client/src/main/resources/oozie-common-0.1.xsd
Lines 25-26 (patched)
<https://reviews.apache.org/r/61529/#comment259058>
Something to think about: YARN is currently working on supporting custom resource types. This allows admins to add other things like GPUs or really anything. It's probably fine to not worry about that for now, but it's something we should think about how to support in the future.
core/src/main/resources/oozie-default.xml
Lines 1612-1616 (original), 1613-1617 (patched)
<https://reviews.apache.org/r/61529/#comment259060>
Don't you need to add all of the other new schemas?
- Robert Kanter
On Aug. 10, 2017, 9:40 a.m., Attila Sasvari wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61529/
> -----------------------------------------------------------
>
> (Updated Aug. 10, 2017, 9:40 a.m.)
>
>
> Review request for oozie and Peter Cseh.
>
>
> Repository: oozie-git
>
>
> Description
> -------
>
> New workflow xsd to allow changing launcher configuration (e.g. setting java system properties)
>
>
> Diffs
> -----
>
> client/src/main/resources/hive-action-0.7.xsd PRE-CREATION
> client/src/main/resources/hive2-action-0.3.xsd PRE-CREATION
> client/src/main/resources/oozie-common-0.1.xsd PRE-CREATION
> client/src/main/resources/oozie-workflow-0.6.xsd PRE-CREATION
> client/src/main/resources/shell-action-0.4.xsd PRE-CREATION
> client/src/main/resources/spark-action-0.3.xsd PRE-CREATION
> client/src/main/resources/sqoop-action-0.5.xsd PRE-CREATION
> core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java dc17950418ada9b4848ed5597702a29e7124ab38
> core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 8fdc50cce15271c13b20d1aaab2ffb95cb8fa711
> core/src/main/java/org/apache/oozie/service/LiteWorkflowStoreService.java ffc29af2f834da6d0890ea7215c5a62fd7cd693e
> core/src/main/java/org/apache/oozie/util/WritableUtils.java aa027e37ba23d4f481698e0bcd93c26c763a0b1f
> core/src/main/java/org/apache/oozie/workflow/lite/LiteWorkflowAppParser.java a74e5c759fe3e336e7e98c61b8f5ac52efa4100c
> core/src/main/java/org/apache/oozie/workflow/lite/LiteWorkflowLib.java 23df0867e5984c518e8424d79365997bec28f619
> core/src/main/resources/oozie-default.xml 0d174b5069aff36ca3a9d48722c5bba69623a046
> core/src/test/java/org/apache/oozie/action/hadoop/ActionExecutorTestCase.java d74160a09ee237f4067e00ded526e7ec94528a6b
> core/src/test/java/org/apache/oozie/action/hadoop/LauncherMainTester.java 6cee7a8948ca7f2c7aaf44d4f172d18a1308a482
> core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java c51c64a18c74c73aa9907027992033a9975bf4d7
> core/src/test/java/org/apache/oozie/workflow/lite/TestLiteWorkflowAppParser.java 21332404bea79fe0a3cea045d57b3e52a1f15cae
> core/src/test/resources/wf-schema-global-launcherconf-override.xml PRE-CREATION
> core/src/test/resources/wf-schema-global-launcherconf.xml PRE-CREATION
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java 6a98d6ef3a09a7a75272f3f0f9a9c2fc5472e76d
>
>
> Diff: https://reviews.apache.org/r/61529/diff/2/
>
>
> Testing
> -------
>
> - TestJavaActionExecutor, TestLiteWorkflowAppParser tests passed
> - java-main example on pseudo hadoop succeeded, modified workflow xml so that verbose class loading information is printed out of stdout of the launcher job
>
>
> Thanks,
>
> Attila Sasvari
>
>
Re: Review Request 61529: OOZIE-2687 Create XML schema for launcher
configurations
Posted by Attila Sasvari <as...@cloudera.com>.
> On Aug. 10, 2017, 10:07 a.m., Peter Cseh wrote:
> > core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
> > Line 1376 (original), 1510 (patched)
> > <https://reviews.apache.org/r/61529/diff/1/?file=1794152#file1794152line1515>
> >
> > Have we decided to use a special <mr-jars> tag or createa a "map-reduce" sharelib that can be referenced?
Not yet. I reverted this change.
> On Aug. 10, 2017, 10:07 a.m., Peter Cseh wrote:
> > core/src/main/resources/oozie-default.xml
> > Line 3077 (original), 3077-3106 (patched)
> > <https://reviews.apache.org/r/61529/diff/1/?file=1794158#file1794158line3077>
> >
> > Please file a follow-up jira to allow setting default core/memory/queue for launcher per action type
Created OOZIE-3034, feel free to add more details to the description
- Attila
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61529/#review182566
-----------------------------------------------------------
On Aug. 10, 2017, 9:40 a.m., Attila Sasvari wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61529/
> -----------------------------------------------------------
>
> (Updated Aug. 10, 2017, 9:40 a.m.)
>
>
> Review request for oozie and Peter Cseh.
>
>
> Repository: oozie-git
>
>
> Description
> -------
>
> New workflow xsd to allow changing launcher configuration (e.g. setting java system properties)
>
>
> Diffs
> -----
>
> client/src/main/resources/hive-action-0.7.xsd PRE-CREATION
> client/src/main/resources/hive2-action-0.3.xsd PRE-CREATION
> client/src/main/resources/oozie-common-0.1.xsd PRE-CREATION
> client/src/main/resources/oozie-workflow-0.6.xsd PRE-CREATION
> client/src/main/resources/shell-action-0.4.xsd PRE-CREATION
> client/src/main/resources/spark-action-0.3.xsd PRE-CREATION
> client/src/main/resources/sqoop-action-0.5.xsd PRE-CREATION
> core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java dc17950418ada9b4848ed5597702a29e7124ab38
> core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 8fdc50cce15271c13b20d1aaab2ffb95cb8fa711
> core/src/main/java/org/apache/oozie/service/LiteWorkflowStoreService.java ffc29af2f834da6d0890ea7215c5a62fd7cd693e
> core/src/main/java/org/apache/oozie/util/WritableUtils.java aa027e37ba23d4f481698e0bcd93c26c763a0b1f
> core/src/main/java/org/apache/oozie/workflow/lite/LiteWorkflowAppParser.java a74e5c759fe3e336e7e98c61b8f5ac52efa4100c
> core/src/main/java/org/apache/oozie/workflow/lite/LiteWorkflowLib.java 23df0867e5984c518e8424d79365997bec28f619
> core/src/main/resources/oozie-default.xml 0d174b5069aff36ca3a9d48722c5bba69623a046
> core/src/test/java/org/apache/oozie/action/hadoop/ActionExecutorTestCase.java d74160a09ee237f4067e00ded526e7ec94528a6b
> core/src/test/java/org/apache/oozie/action/hadoop/LauncherMainTester.java 6cee7a8948ca7f2c7aaf44d4f172d18a1308a482
> core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java c51c64a18c74c73aa9907027992033a9975bf4d7
> core/src/test/java/org/apache/oozie/workflow/lite/TestLiteWorkflowAppParser.java 21332404bea79fe0a3cea045d57b3e52a1f15cae
> core/src/test/resources/wf-schema-global-launcherconf-override.xml PRE-CREATION
> core/src/test/resources/wf-schema-global-launcherconf.xml PRE-CREATION
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java 6a98d6ef3a09a7a75272f3f0f9a9c2fc5472e76d
>
>
> Diff: https://reviews.apache.org/r/61529/diff/2/
>
>
> Testing
> -------
>
> - TestJavaActionExecutor, TestLiteWorkflowAppParser tests passed
> - java-main example on pseudo hadoop succeeded, modified workflow xml so that verbose class loading information is printed out of stdout of the launcher job
>
>
> Thanks,
>
> Attila Sasvari
>
>
Re: Review Request 61529: OOZIE-2687 Create XML schema for launcher
configurations
Posted by Peter Cseh <ge...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61529/#review182566
-----------------------------------------------------------
client/src/main/resources/oozie-common-0.1.xsd
Lines 27 (patched)
<https://reviews.apache.org/r/61529/#comment258482>
How do you imagine handling env? I could not see any examples for it.
core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
Line 1115 (original), 1244 (patched)
<https://reviews.apache.org/r/61529/#comment258483>
lancuer.env property is not parsed here.
core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
Line 1376 (original), 1510 (patched)
<https://reviews.apache.org/r/61529/#comment258477>
Have we decided to use a special <mr-jars> tag or createa a "map-reduce" sharelib that can be referenced?
core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java
Line 80 (original)
<https://reviews.apache.org/r/61529/#comment258478>
Why remove this check?
core/src/main/resources/oozie-default.xml
Line 3077 (original), 3077-3106 (patched)
<https://reviews.apache.org/r/61529/#comment258479>
Please file a follow-up jira to allow setting default core/memory/queue for launcher per action type
core/src/test/java/org/apache/oozie/action/hadoop/ActionExecutorTestCase.java
Lines 308-310 (patched)
<https://reviews.apache.org/r/61529/#comment258480>
Please add queue, priority, env and java-opts into this test as well
core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java
Line 66 (original), 66 (patched)
<https://reviews.apache.org/r/61529/#comment258481>
Don't use * imports
core/src/test/resources/wf-schema-global-launcherconf-override.xml
Lines 9-11 (patched)
<https://reviews.apache.org/r/61529/#comment258486>
same
core/src/test/resources/wf-schema-global-launcherconf.xml
Lines 9 (patched)
<https://reviews.apache.org/r/61529/#comment258484>
whitespaces!
core/src/test/resources/wf-schema-global-launcherconf.xml
Lines 11 (patched)
<https://reviews.apache.org/r/61529/#comment258485>
Rat plugin will fail on this.
sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java
Lines 59 (patched)
<https://reviews.apache.org/r/61529/#comment258487>
whitespace!
- Peter Cseh
On Aug. 10, 2017, 9:40 a.m., Attila Sasvari wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61529/
> -----------------------------------------------------------
>
> (Updated Aug. 10, 2017, 9:40 a.m.)
>
>
> Review request for oozie and Peter Cseh.
>
>
> Repository: oozie-git
>
>
> Description
> -------
>
> New workflow xsd to allow changing launcher configuration (e.g. setting java system properties)
>
>
> Diffs
> -----
>
> client/src/main/resources/hive-action-0.7.xsd PRE-CREATION
> client/src/main/resources/hive2-action-0.3.xsd PRE-CREATION
> client/src/main/resources/oozie-common-0.1.xsd PRE-CREATION
> client/src/main/resources/oozie-workflow-0.6.xsd PRE-CREATION
> client/src/main/resources/shell-action-0.4.xsd PRE-CREATION
> client/src/main/resources/spark-action-0.3.xsd PRE-CREATION
> client/src/main/resources/sqoop-action-0.5.xsd PRE-CREATION
> core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java dc17950418ada9b4848ed5597702a29e7124ab38
> core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 8fdc50cce15271c13b20d1aaab2ffb95cb8fa711
> core/src/main/java/org/apache/oozie/service/LiteWorkflowStoreService.java ffc29af2f834da6d0890ea7215c5a62fd7cd693e
> core/src/main/java/org/apache/oozie/util/WritableUtils.java aa027e37ba23d4f481698e0bcd93c26c763a0b1f
> core/src/main/java/org/apache/oozie/workflow/lite/LiteWorkflowAppParser.java a74e5c759fe3e336e7e98c61b8f5ac52efa4100c
> core/src/main/java/org/apache/oozie/workflow/lite/LiteWorkflowLib.java 23df0867e5984c518e8424d79365997bec28f619
> core/src/main/resources/oozie-default.xml 0d174b5069aff36ca3a9d48722c5bba69623a046
> core/src/test/java/org/apache/oozie/action/hadoop/ActionExecutorTestCase.java d74160a09ee237f4067e00ded526e7ec94528a6b
> core/src/test/java/org/apache/oozie/action/hadoop/LauncherMainTester.java 6cee7a8948ca7f2c7aaf44d4f172d18a1308a482
> core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java c51c64a18c74c73aa9907027992033a9975bf4d7
> core/src/test/java/org/apache/oozie/workflow/lite/TestLiteWorkflowAppParser.java 21332404bea79fe0a3cea045d57b3e52a1f15cae
> core/src/test/resources/wf-schema-global-launcherconf-override.xml PRE-CREATION
> core/src/test/resources/wf-schema-global-launcherconf.xml PRE-CREATION
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java 6a98d6ef3a09a7a75272f3f0f9a9c2fc5472e76d
>
>
> Diff: https://reviews.apache.org/r/61529/diff/1/
>
>
> Testing
> -------
>
> - TestJavaActionExecutor, TestLiteWorkflowAppParser tests passed
> - java-main example on pseudo hadoop succeeded, modified workflow xml so that verbose class loading information is printed out of stdout of the launcher job
>
>
> Thanks,
>
> Attila Sasvari
>
>
Re: Review Request 61529: OOZIE-2687 Create XML schema for launcher
configurations
Posted by Attila Sasvari <as...@cloudera.com>.
> On Aug. 10, 2017, 9:45 a.m., Peter Cseh wrote:
> > client/src/main/resources/oozie-workflow-0.6.xsd
> > Lines 143-147 (patched)
> > <https://reviews.apache.org/r/61529/diff/1/?file=1794148#file1794148line143>
> >
> > Can you extract these into separate xsd files? I think each action should get it's own sooner or later and this looks like a proper time for it.
It is a good idea, and would make the scope of this task much bigger and harder to review. I created a follow-up JIRA https://issues.apache.org/jira/browse/OOZIE-3033
> On Aug. 10, 2017, 9:45 a.m., Peter Cseh wrote:
> > core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
> > Lines 872-877 (patched)
> > <https://reviews.apache.org/r/61529/diff/1/?file=1794152#file1794152line873>
> >
> > What't the intention behind this?
Setting mapreduce job names. Should be handled by OOZIE-2916. Thanks for pointing this out. Removed.
- Attila
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61529/#review182565
-----------------------------------------------------------
On Aug. 10, 2017, 9:40 a.m., Attila Sasvari wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61529/
> -----------------------------------------------------------
>
> (Updated Aug. 10, 2017, 9:40 a.m.)
>
>
> Review request for oozie and Peter Cseh.
>
>
> Repository: oozie-git
>
>
> Description
> -------
>
> New workflow xsd to allow changing launcher configuration (e.g. setting java system properties)
>
>
> Diffs
> -----
>
> client/src/main/resources/hive-action-0.7.xsd PRE-CREATION
> client/src/main/resources/hive2-action-0.3.xsd PRE-CREATION
> client/src/main/resources/oozie-common-0.1.xsd PRE-CREATION
> client/src/main/resources/oozie-workflow-0.6.xsd PRE-CREATION
> client/src/main/resources/shell-action-0.4.xsd PRE-CREATION
> client/src/main/resources/spark-action-0.3.xsd PRE-CREATION
> client/src/main/resources/sqoop-action-0.5.xsd PRE-CREATION
> core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java dc17950418ada9b4848ed5597702a29e7124ab38
> core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 8fdc50cce15271c13b20d1aaab2ffb95cb8fa711
> core/src/main/java/org/apache/oozie/service/LiteWorkflowStoreService.java ffc29af2f834da6d0890ea7215c5a62fd7cd693e
> core/src/main/java/org/apache/oozie/util/WritableUtils.java aa027e37ba23d4f481698e0bcd93c26c763a0b1f
> core/src/main/java/org/apache/oozie/workflow/lite/LiteWorkflowAppParser.java a74e5c759fe3e336e7e98c61b8f5ac52efa4100c
> core/src/main/java/org/apache/oozie/workflow/lite/LiteWorkflowLib.java 23df0867e5984c518e8424d79365997bec28f619
> core/src/main/resources/oozie-default.xml 0d174b5069aff36ca3a9d48722c5bba69623a046
> core/src/test/java/org/apache/oozie/action/hadoop/ActionExecutorTestCase.java d74160a09ee237f4067e00ded526e7ec94528a6b
> core/src/test/java/org/apache/oozie/action/hadoop/LauncherMainTester.java 6cee7a8948ca7f2c7aaf44d4f172d18a1308a482
> core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java c51c64a18c74c73aa9907027992033a9975bf4d7
> core/src/test/java/org/apache/oozie/workflow/lite/TestLiteWorkflowAppParser.java 21332404bea79fe0a3cea045d57b3e52a1f15cae
> core/src/test/resources/wf-schema-global-launcherconf-override.xml PRE-CREATION
> core/src/test/resources/wf-schema-global-launcherconf.xml PRE-CREATION
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java 6a98d6ef3a09a7a75272f3f0f9a9c2fc5472e76d
>
>
> Diff: https://reviews.apache.org/r/61529/diff/2/
>
>
> Testing
> -------
>
> - TestJavaActionExecutor, TestLiteWorkflowAppParser tests passed
> - java-main example on pseudo hadoop succeeded, modified workflow xml so that verbose class loading information is printed out of stdout of the launcher job
>
>
> Thanks,
>
> Attila Sasvari
>
>
Re: Review Request 61529: OOZIE-2687 Create XML schema for launcher
configurations
Posted by Peter Cseh <ge...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61529/#review182565
-----------------------------------------------------------
client/src/main/resources/oozie-workflow-0.6.xsd
Lines 143-147 (patched)
<https://reviews.apache.org/r/61529/#comment258472>
Can you extract these into separate xsd files? I think each action should get it's own sooner or later and this looks like a proper time for it.
core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
Lines 290-295 (patched)
<https://reviews.apache.org/r/61529/#comment258473>
This was moved to MRActionExecutor, don't put it back here.
core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
Lines 335-339 (patched)
<https://reviews.apache.org/r/61529/#comment258474>
MRActionExecutor. Remove this.
core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
Lines 872-877 (patched)
<https://reviews.apache.org/r/61529/#comment258475>
What't the intention behind this?
core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
Lines 999-1005 (patched)
<https://reviews.apache.org/r/61529/#comment258476>
Same as in lin 872
- Peter Cseh
On Aug. 10, 2017, 9:40 a.m., Attila Sasvari wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61529/
> -----------------------------------------------------------
>
> (Updated Aug. 10, 2017, 9:40 a.m.)
>
>
> Review request for oozie and Peter Cseh.
>
>
> Repository: oozie-git
>
>
> Description
> -------
>
> New workflow xsd to allow changing launcher configuration (e.g. setting java system properties)
>
>
> Diffs
> -----
>
> client/src/main/resources/hive-action-0.7.xsd PRE-CREATION
> client/src/main/resources/hive2-action-0.3.xsd PRE-CREATION
> client/src/main/resources/oozie-common-0.1.xsd PRE-CREATION
> client/src/main/resources/oozie-workflow-0.6.xsd PRE-CREATION
> client/src/main/resources/shell-action-0.4.xsd PRE-CREATION
> client/src/main/resources/spark-action-0.3.xsd PRE-CREATION
> client/src/main/resources/sqoop-action-0.5.xsd PRE-CREATION
> core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java dc17950418ada9b4848ed5597702a29e7124ab38
> core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 8fdc50cce15271c13b20d1aaab2ffb95cb8fa711
> core/src/main/java/org/apache/oozie/service/LiteWorkflowStoreService.java ffc29af2f834da6d0890ea7215c5a62fd7cd693e
> core/src/main/java/org/apache/oozie/util/WritableUtils.java aa027e37ba23d4f481698e0bcd93c26c763a0b1f
> core/src/main/java/org/apache/oozie/workflow/lite/LiteWorkflowAppParser.java a74e5c759fe3e336e7e98c61b8f5ac52efa4100c
> core/src/main/java/org/apache/oozie/workflow/lite/LiteWorkflowLib.java 23df0867e5984c518e8424d79365997bec28f619
> core/src/main/resources/oozie-default.xml 0d174b5069aff36ca3a9d48722c5bba69623a046
> core/src/test/java/org/apache/oozie/action/hadoop/ActionExecutorTestCase.java d74160a09ee237f4067e00ded526e7ec94528a6b
> core/src/test/java/org/apache/oozie/action/hadoop/LauncherMainTester.java 6cee7a8948ca7f2c7aaf44d4f172d18a1308a482
> core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java c51c64a18c74c73aa9907027992033a9975bf4d7
> core/src/test/java/org/apache/oozie/workflow/lite/TestLiteWorkflowAppParser.java 21332404bea79fe0a3cea045d57b3e52a1f15cae
> core/src/test/resources/wf-schema-global-launcherconf-override.xml PRE-CREATION
> core/src/test/resources/wf-schema-global-launcherconf.xml PRE-CREATION
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java 6a98d6ef3a09a7a75272f3f0f9a9c2fc5472e76d
>
>
> Diff: https://reviews.apache.org/r/61529/diff/1/
>
>
> Testing
> -------
>
> - TestJavaActionExecutor, TestLiteWorkflowAppParser tests passed
> - java-main example on pseudo hadoop succeeded, modified workflow xml so that verbose class loading information is printed out of stdout of the launcher job
>
>
> Thanks,
>
> Attila Sasvari
>
>