You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@oozie.apache.org by Purshotam Shah <pu...@yahoo-inc.com> on 2014/04/14 23:26:15 UTC
Re: Review Request 20006: OOZIE-1769 An option to update coord
properties/definition
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20006/
-----------------------------------------------------------
(Updated April 14, 2014, 9:26 p.m.)
Review request for oozie, Ryota Egashira and Rohini Palaniswamy.
Changes
-------
Review comment.
Bugs: OOZIE-1769
https://issues.apache.org/jira/browse/OOZIE-1769
Repository: oozie-git
Description
-------
To change the coord definition user can update coord definition in hdfs and issue a update command. Existing coordinator definition will be replaced by new definition. The refreshed coordinator
would keep the same coordinator ID, state, and coordinator actions.
User can also provide -dryrun to validate changes.
All created coord action(including in waiting) will use old configuration.
User can rerun actions with -refresh option, -refresh option will use new configuration to rerun coord action.
Ex.
1. $ oozie job -update <coordID> -config examples/apps/aggregator/job.properties
Will update coord definition and action.
2. $ oozie job -update <coordID>
Will fetch coord definition path from bundle and update coord definition
Diffs (updated)
-----
client/src/main/java/org/apache/oozie/cli/OozieCLI.java e1f551d
client/src/main/java/org/apache/oozie/client/OozieClient.java 46c4288
client/src/main/java/org/apache/oozie/client/rest/JsonTags.java ae391c7
client/src/main/java/org/apache/oozie/client/rest/RestConstants.java a7fe06e
core/pom.xml f0547a3
core/src/main/java/org/apache/oozie/CoordinatorEngine.java 3f10024
core/src/main/java/org/apache/oozie/CoordinatorJobBean.java 5eb134b
core/src/main/java/org/apache/oozie/ErrorCode.java f69d7a2
core/src/main/java/org/apache/oozie/command/coord/CoordSubmitXCommand.java 9e6a3d5
core/src/main/java/org/apache/oozie/command/coord/CoordUpdateXCommand.java e69de29
core/src/main/java/org/apache/oozie/executor/jpa/CoordJobQueryExecutor.java 5f7744c
core/src/main/java/org/apache/oozie/servlet/BaseJobServlet.java aa43e68
core/src/main/java/org/apache/oozie/servlet/V0JobServlet.java 443ab6d
core/src/main/java/org/apache/oozie/servlet/V1JobServlet.java ac399e9
core/src/main/java/org/apache/oozie/servlet/V2JobServlet.java 2d26599
core/src/test/java/org/apache/oozie/command/coord/TestCoordUpdateXCommand.java e69de29
core/src/test/resources/coord-update-test.xml e69de29
docs/src/site/twiki/DG_CommandLineTool.twiki 351f0f2
docs/src/site/twiki/WebServicesAPI.twiki 37c0bc0
pom.xml cb10007
Diff: https://reviews.apache.org/r/20006/diff/
Testing
-------
Invalid xml.
purushah$ /Users/purushah/views/coor_swap/oozie/distro/target/oozie-4.1.0-SNAPSHOT-distro/oozie-4.1.0-SNAPSHOT/bin/oozie job -oozie http://localhost:11000/oozie -config temp/job.properties -update 0000005-140402104721140-oozie-puru-C -dryrun
Picked up JAVA_TOOL_OPTIONS: -Djava.awt.headless=true
Error: E1023 : E1023: Coord Job update Error: [E0701: XML schema error, cvc-complex-type.2.4.a: Invalid content was found starting with element 'datasets'. One of '{"uri:oozie:coordinator:0.2":execution, "uri:oozie:coordinator:0.2":throttle}' is expected.]
purushah$
valid XML.
$ oozie job -oozie http://localhost:11000/oozie -config temp/job.properties -update 0000005-140402104721140-oozie-puru-C -dryrun
**********Job definition changes**********
@@ -3,8 +3,8 @@
<concurrency>1</concurrency>
</controls>
<input-events>
- <data-in name="input" dataset="raw-logs">
- <dataset name="raw-logs" frequency="20" initial-instance="2010-01-01T00:00Z" timezone="UTC" freq_timeunit="MINUTE" end_of_duration="NONE">
+ <data-in name="input" dataset="raw-logs-rename">
+ <dataset name="raw-logs-rename" frequency="20" initial-instance="2010-01-01T00:00Z" timezone="UTC" freq_timeunit="MINUTE" end_of_duration="NONE">
<uri-template>hdfs://localhost:9000/user/purushah/examples/input-data/rawLogs/</uri-template>
<done-flag />
</dataset>
**********************************
**********Job conf changes**********
@@ -8,10 +8,6 @@
<value>hdfs://localhost:9000/user/purushah/examples/apps/aggregator/coordinator.xml</value>
</property>
<property>
- <name>aaaaaa</name>
- <value>b</value>
- </property>
- <property>
<name>user.name</name>
<value>purushah</value>
</property>
@@ -28,6 +24,10 @@
<value>hdfs://localhost:9000</value>
</property>
<property>
+ <name>adding</name>
+ <value>new</value>
+ </property>
+ <property>
<name>jobTracker</name>
<value>localhost:9001</value>
</property>
**********************************
$
$ /Users/purushah/views/coor_swap/oozie/distro/target/oozie-4.1.0-SNAPSHOT-distro/oozie-4.1.0-SNAPSHOT/bin/oozie job -oozie http://localhost:11000/oozie -config temp/job.properties -update 0000000-140408140947673-oozie-puru-C
Picked up JAVA_TOOL_OPTIONS: -Djava.awt.headless=true
Error: E0803 : E0803: IO error, E1023: Coord Job update Error: [Frequency can't be changes. Old frqenecy 60 new frequency 180]
$
Tested Doc.........
DG_CommandLineTool.html#Updating_coordinator_definition_and_properties
Updating coordinator definition and properties
Existing coordinator definition will be replaced by new definition. The refreshed coordinator would keep the same coordinator ID, state, and coordinator actions. All created coord action(including in WAITING) will use old configuration. One can rerun actions with -refresh option, -refresh option will use new configuration to rerun coord action
Update command also verifies coordinator definition like submit command, if there is any issue with definition, update will fail. Update command with -dryrun will show coordinator definition and properties differences. Config option is optional, if not specified existing coordinator property is used to find coordinator path.
Update command doesn't update frequency, end time, start time, timzone, name, execution order. Update command will fail if any of this element has changed in job definition.
$ oozie job -oozie http://localhost:11000/oozie -config job.properties -update 0000005-140402104721140-oozie-puru-C -dryrun**********Job definition changes**********
@@ -3,8 +3,8 @@
<concurrency>1</concurrency>
</controls>
<input-events>
- <data-in name="input" dataset="raw-logs">
- <dataset name="raw-logs" frequency="20" initial-instance="2010-01-01T00:00Z" timezone="UTC" freq_timeunit="MINUTE" end_of_duration="NONE">
+ <data-in name="input" dataset="raw-logs-rename">
+ <dataset name="raw-logs-rename" frequency="20" initial-instance="2010-01-01T00:00Z" timezone="UTC" freq_timeunit="MINUTE" end_of_duration="NONE">
<uri-template>hdfs://localhost:9000/user/purushah/examples/input-data/rawLogs/</uri-template>
<done-flag />
</dataset>
**********************************
**********Job conf changes**********
@@ -8,10 +8,6 @@
<value>hdfs://localhost:9000/user/purushah/examples/apps/aggregator/coordinator.xml</value>
</property>
<property>
- <name>old</name>
- <value>test</value>
- </property>
- <property>
<name>user.name</name>
<value>purushah</value>
</property>
@@ -28,6 +24,10 @@
<value>hdfs://localhost:9000</value>
</property>
<property>
+ <name>adding</name>
+ <value>new</value>
+ </property>
+ <property>
<name>jobTracker</name>
<value>localhost:9001</value>
</property>
**********************************
WebServicesAPI.html#Updating_coordinator_definition_and_properties
Existing coordinator definition and properties will be replaced by new definition and properties. Refer Updating coordinator definition and properties
PUT oozie/v2/job/0000000-140414102048137-oozie-puru-C?action=update
Response:
HTTP/1.1 200 OK
Content-Type: application/json;charset=UTF-8
{"update":
{"diff":"**********Job definition changes**********\n******************************************\n**********Job conf changes****************\n@@ -8,16 +8,12 @@\n
<value>hdfs:\/\/localhost:9000\/user\/purushah\/examples\/apps\/aggregator\/coordinator.xml<\/value>\r\n <\/property>\r\n <property>\r\n
- <name>user.name<\/name>\r\n
- <value>purushah<\/value>\r\n
- <\/property>\r\n
- <property>\r\n <name>start<\/name>\r\n
<value>2010-01-01T01:00Z<\/value>\r\n <\/property>\r\n <property>\r\n
- <name>newproperty<\/name>\r\n
- <value>new<\/value>\r\n
+ <name>user.name<\/name>\r\n
+ <value>purushah<\/value>\r\n <\/property>\r\n <property>\r\n
<name>queueName<\/name>\r\n******************************************\n"
}
}
Thanks,
Purshotam Shah
Re: Review Request 20006: OOZIE-1769 An option to update coord
properties/definition
Posted by Rohini Palaniswamy <ro...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20006/#review40421
-----------------------------------------------------------
Ship it!
Ship It!
- Rohini Palaniswamy
On April 14, 2014, 9:26 p.m., Purshotam Shah wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/20006/
> -----------------------------------------------------------
>
> (Updated April 14, 2014, 9:26 p.m.)
>
>
> Review request for oozie, Ryota Egashira and Rohini Palaniswamy.
>
>
> Bugs: OOZIE-1769
> https://issues.apache.org/jira/browse/OOZIE-1769
>
>
> Repository: oozie-git
>
>
> Description
> -------
>
> To change the coord definition user can update coord definition in hdfs and issue a update command. Existing coordinator definition will be replaced by new definition. The refreshed coordinator
> would keep the same coordinator ID, state, and coordinator actions.
> User can also provide -dryrun to validate changes.
> All created coord action(including in waiting) will use old configuration.
> User can rerun actions with -refresh option, -refresh option will use new configuration to rerun coord action.
> Ex.
> 1. $ oozie job -update <coordID> -config examples/apps/aggregator/job.properties
> Will update coord definition and action.
> 2. $ oozie job -update <coordID>
> Will fetch coord definition path from bundle and update coord definition
>
>
> Diffs
> -----
>
> client/src/main/java/org/apache/oozie/cli/OozieCLI.java e1f551d
> client/src/main/java/org/apache/oozie/client/OozieClient.java 46c4288
> client/src/main/java/org/apache/oozie/client/rest/JsonTags.java ae391c7
> client/src/main/java/org/apache/oozie/client/rest/RestConstants.java a7fe06e
> core/pom.xml f0547a3
> core/src/main/java/org/apache/oozie/CoordinatorEngine.java 3f10024
> core/src/main/java/org/apache/oozie/CoordinatorJobBean.java 5eb134b
> core/src/main/java/org/apache/oozie/ErrorCode.java f69d7a2
> core/src/main/java/org/apache/oozie/command/coord/CoordSubmitXCommand.java 9e6a3d5
> core/src/main/java/org/apache/oozie/command/coord/CoordUpdateXCommand.java e69de29
> core/src/main/java/org/apache/oozie/executor/jpa/CoordJobQueryExecutor.java 5f7744c
> core/src/main/java/org/apache/oozie/servlet/BaseJobServlet.java aa43e68
> core/src/main/java/org/apache/oozie/servlet/V0JobServlet.java 443ab6d
> core/src/main/java/org/apache/oozie/servlet/V1JobServlet.java ac399e9
> core/src/main/java/org/apache/oozie/servlet/V2JobServlet.java 2d26599
> core/src/test/java/org/apache/oozie/command/coord/TestCoordUpdateXCommand.java e69de29
> core/src/test/resources/coord-update-test.xml e69de29
> docs/src/site/twiki/DG_CommandLineTool.twiki 351f0f2
> docs/src/site/twiki/WebServicesAPI.twiki 37c0bc0
> pom.xml cb10007
>
> Diff: https://reviews.apache.org/r/20006/diff/
>
>
> Testing
> -------
>
> Invalid xml.
> purushah$ /Users/purushah/views/coor_swap/oozie/distro/target/oozie-4.1.0-SNAPSHOT-distro/oozie-4.1.0-SNAPSHOT/bin/oozie job -oozie http://localhost:11000/oozie -config temp/job.properties -update 0000005-140402104721140-oozie-puru-C -dryrun
> Picked up JAVA_TOOL_OPTIONS: -Djava.awt.headless=true
> Error: E1023 : E1023: Coord Job update Error: [E0701: XML schema error, cvc-complex-type.2.4.a: Invalid content was found starting with element 'datasets'. One of '{"uri:oozie:coordinator:0.2":execution, "uri:oozie:coordinator:0.2":throttle}' is expected.]
> purushah$
>
>
> valid XML.
> $ oozie job -oozie http://localhost:11000/oozie -config temp/job.properties -update 0000005-140402104721140-oozie-puru-C -dryrun
> **********Job definition changes**********
> @@ -3,8 +3,8 @@
> <concurrency>1</concurrency>
> </controls>
> <input-events>
> - <data-in name="input" dataset="raw-logs">
> - <dataset name="raw-logs" frequency="20" initial-instance="2010-01-01T00:00Z" timezone="UTC" freq_timeunit="MINUTE" end_of_duration="NONE">
> + <data-in name="input" dataset="raw-logs-rename">
> + <dataset name="raw-logs-rename" frequency="20" initial-instance="2010-01-01T00:00Z" timezone="UTC" freq_timeunit="MINUTE" end_of_duration="NONE">
> <uri-template>hdfs://localhost:9000/user/purushah/examples/input-data/rawLogs/</uri-template>
> <done-flag />
> </dataset>
> **********************************
> **********Job conf changes**********
> @@ -8,10 +8,6 @@
> <value>hdfs://localhost:9000/user/purushah/examples/apps/aggregator/coordinator.xml</value>
> </property>
> <property>
> - <name>aaaaaa</name>
> - <value>b</value>
> - </property>
> - <property>
> <name>user.name</name>
> <value>purushah</value>
> </property>
> @@ -28,6 +24,10 @@
> <value>hdfs://localhost:9000</value>
> </property>
> <property>
> + <name>adding</name>
> + <value>new</value>
> + </property>
> + <property>
> <name>jobTracker</name>
> <value>localhost:9001</value>
> </property>
> **********************************
> $
>
> $ /Users/purushah/views/coor_swap/oozie/distro/target/oozie-4.1.0-SNAPSHOT-distro/oozie-4.1.0-SNAPSHOT/bin/oozie job -oozie http://localhost:11000/oozie -config temp/job.properties -update 0000000-140408140947673-oozie-puru-C
> Picked up JAVA_TOOL_OPTIONS: -Djava.awt.headless=true
> Error: E0803 : E0803: IO error, E1023: Coord Job update Error: [Frequency can't be changes. Old frqenecy 60 new frequency 180]
> $
>
> Tested Doc.........
> DG_CommandLineTool.html#Updating_coordinator_definition_and_properties
>
> Updating coordinator definition and properties
>
> Existing coordinator definition will be replaced by new definition. The refreshed coordinator would keep the same coordinator ID, state, and coordinator actions. All created coord action(including in WAITING) will use old configuration. One can rerun actions with -refresh option, -refresh option will use new configuration to rerun coord action
>
> Update command also verifies coordinator definition like submit command, if there is any issue with definition, update will fail. Update command with -dryrun will show coordinator definition and properties differences. Config option is optional, if not specified existing coordinator property is used to find coordinator path.
>
> Update command doesn't update frequency, end time, start time, timzone, name, execution order. Update command will fail if any of this element has changed in job definition.
>
> $ oozie job -oozie http://localhost:11000/oozie -config job.properties -update 0000005-140402104721140-oozie-puru-C -dryrun**********Job definition changes**********
> @@ -3,8 +3,8 @@
> <concurrency>1</concurrency>
> </controls>
> <input-events>
> - <data-in name="input" dataset="raw-logs">
> - <dataset name="raw-logs" frequency="20" initial-instance="2010-01-01T00:00Z" timezone="UTC" freq_timeunit="MINUTE" end_of_duration="NONE">
> + <data-in name="input" dataset="raw-logs-rename">
> + <dataset name="raw-logs-rename" frequency="20" initial-instance="2010-01-01T00:00Z" timezone="UTC" freq_timeunit="MINUTE" end_of_duration="NONE">
> <uri-template>hdfs://localhost:9000/user/purushah/examples/input-data/rawLogs/</uri-template>
> <done-flag />
> </dataset>
> **********************************
> **********Job conf changes**********
> @@ -8,10 +8,6 @@
> <value>hdfs://localhost:9000/user/purushah/examples/apps/aggregator/coordinator.xml</value>
> </property>
> <property>
> - <name>old</name>
> - <value>test</value>
> - </property>
> - <property>
> <name>user.name</name>
> <value>purushah</value>
> </property>
> @@ -28,6 +24,10 @@
> <value>hdfs://localhost:9000</value>
> </property>
> <property>
> + <name>adding</name>
> + <value>new</value>
> + </property>
> + <property>
> <name>jobTracker</name>
> <value>localhost:9001</value>
> </property>
> **********************************
>
>
> WebServicesAPI.html#Updating_coordinator_definition_and_properties
>
> Existing coordinator definition and properties will be replaced by new definition and properties. Refer Updating coordinator definition and properties
>
> PUT oozie/v2/job/0000000-140414102048137-oozie-puru-C?action=update
> Response:
>
> HTTP/1.1 200 OK
> Content-Type: application/json;charset=UTF-8
> {"update":
> {"diff":"**********Job definition changes**********\n******************************************\n**********Job conf changes****************\n@@ -8,16 +8,12 @@\n
> <value>hdfs:\/\/localhost:9000\/user\/purushah\/examples\/apps\/aggregator\/coordinator.xml<\/value>\r\n <\/property>\r\n <property>\r\n
> - <name>user.name<\/name>\r\n
> - <value>purushah<\/value>\r\n
> - <\/property>\r\n
> - <property>\r\n <name>start<\/name>\r\n
> <value>2010-01-01T01:00Z<\/value>\r\n <\/property>\r\n <property>\r\n
> - <name>newproperty<\/name>\r\n
> - <value>new<\/value>\r\n
> + <name>user.name<\/name>\r\n
> + <value>purushah<\/value>\r\n <\/property>\r\n <property>\r\n
> <name>queueName<\/name>\r\n******************************************\n"
> }
> }
>
>
> Thanks,
>
> Purshotam Shah
>
>