You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@oozie.apache.org by Purshotam Shah <pu...@yahoo-inc.com> on 2013/09/24 00:57:21 UTC
Review Request 14291: OOZIE-1554 Support variables for coord
data-in/data-out dataset
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14291/
-----------------------------------------------------------
Review request for oozie.
Bugs: OOZIE-1554
https://issues.apache.org/jira/browse/OOZIE-1554
Repository: oozie
Description
-------
Problem
__________________________________________________________________________________________________
One would like to have a centralized list of datasets,
and use the <include> tag to make them available to every coordinator. One
would like to re-use our coordinator code, as most of his processing follows
the same steps, but with differing input and output feeds.
He need to be able to set the data-in and data-out dataset
values to variables;
My bundle coordinator entry looks like this:
<coordinator name="data1-2">
<app-path>/user/harveyc/oozie_test/src/test_coordA.xml</app-path>
<configuration>
<property><name>wf_name</name><value>1-2</value></property>
<property><name>dataset_A</name><value>dataA</value></property>
<property><name>dataset_B</name><value>dataB</value></property>
</configuration>
</coordinator>
Coord looks
<coordinator-app name="COORD_A_TEST" frequency="${coord:minutes(1)}"
start="${startTime}" end="${endTime}" timezone="${timezoneCode}"
xmlns:sla="uri:oozie:sla:0.1" xmlns="uri:oozie:coordinator:0.2">
<datasets>
<include>${nameNode}/user/harveyc/oozie_test/datasets/test_datasets.xml</include>
</datasets>
<input-events>
<data-in name="inputDataA" dataset="${dataset_A}">
<instance>${coord:current(0)}</instance>
</data-in>
</input-events>
<output-events>
<data-out name="outputDataB" dataset="${dataset_B}">
<instance>${coord:current(0)}</instance>
</data-out>
</output-events>
<action>
<workflow>
<app-path>/user/harveyc/oozie_test/src/wf_touchz.xml</app-path>
<configuration>
<property><name>name</name><value>${wf_name}</value></property>
<property><name>touchzpathb</name><value>${coord:dataOut('outputDataB')}</value></property>
</configuration>
</workflow>
</action>
</coordinator-app>
Test_datasets.xml looks like this:
<datasets>
<dataset name="dataA" frequency="${coord:minutes(1)}"
initial-instance="${ds_startTime}" timezone="${timezoneCode}">
<uri-template>${nameNode}/user/harveyc/oozie_test/data1/${YEAR}${MONTH}${DAY}${HOUR}${MINUTE}</uri-template>
</dataset>
<dataset name="dataB" frequency="${coord:minutes(1)}"
initial-instance="${ds_startTime}" timezone="${timezoneCode}">
<uri-template>${nameNode}/user/harveyc/oozie_test/data2/${YEAR}${MONTH}${DAY}${HOUR}${MINUTE}</uri-template>
</dataset>
</datasets>
The error is:
Error: Invalid workflow-app, org.xml.sax.SAXParseException; lineNumber: 8;
columnNumber: 57; cvc-pattern-valid: Value '${dataset_A}' is not facet-valid
with respect to pattern '([a-zA-Z]([\-_a-zA-Z0-9])*){1,39}' for type
'IDENTIFIER'.
__________________________________________________________________________________________________
Fix
__________________________________________________________________________________________________
Changed the schema to include all valid string for dataset attribute for input-events "data-in and date-out".
input-events datasets are el evaluated before looking for valid datasets.
__________________________________________________________________________________________________
Diffs
-----
http://svn.apache.org/repos/asf/oozie/trunk/client/src/main/resources/oozie-coordinator-0.1.xsd 1524864
http://svn.apache.org/repos/asf/oozie/trunk/client/src/main/resources/oozie-coordinator-0.2.xsd 1524864
http://svn.apache.org/repos/asf/oozie/trunk/client/src/main/resources/oozie-coordinator-0.3.xsd 1524864
http://svn.apache.org/repos/asf/oozie/trunk/client/src/main/resources/oozie-coordinator-0.4.xsd 1524864
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordSubmitXCommand.java 1525674
http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/java/org/apache/oozie/command/coord/TestCoordSubmitXCommand.java 1525674
http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/resources/coord-el-dataset-1.xml PRE-CREATION
http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/resources/coord-el-dataset-2.xml PRE-CREATION
http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/resources/coord-el-dataset-3.xml PRE-CREATION
http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/resources/coord-el-dataset-4.xml PRE-CREATION
Diff: https://reviews.apache.org/r/14291/diff/
Testing
-------
Added test cases to test it for all xsd's.
Thanks,
Purshotam Shah
Re: Review Request 14291: OOZIE-1554 Support variables for coord
data-in/data-out dataset
Posted by Rohini Palaniswamy <ro...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14291/#review28693
-----------------------------------------------------------
Ship it!
Ship It!
- Rohini Palaniswamy
On Oct. 18, 2013, 7:07 p.m., Purshotam Shah wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/14291/
> -----------------------------------------------------------
>
> (Updated Oct. 18, 2013, 7:07 p.m.)
>
>
> Review request for oozie.
>
>
> Bugs: OOZIE-1554
> https://issues.apache.org/jira/browse/OOZIE-1554
>
>
> Repository: oozie
>
>
> Description
> -------
>
> Problem
> __________________________________________________________________________________________________
> One would like to have a centralized list of datasets,
> and use the <include> tag to make them available to every coordinator. One
> would like to re-use our coordinator code, as most of his processing follows
> the same steps, but with differing input and output feeds.
>
> He need to be able to set the data-in and data-out dataset
> values to variables;
>
>
> My bundle coordinator entry looks like this:
> <coordinator name="data1-2">
> <app-path>/user/harveyc/oozie_test/src/test_coordA.xml</app-path>
> <configuration>
> <property><name>wf_name</name><value>1-2</value></property>
> <property><name>dataset_A</name><value>dataA</value></property>
> <property><name>dataset_B</name><value>dataB</value></property>
> </configuration>
> </coordinator>
>
>
> Coord looks
> <coordinator-app name="COORD_A_TEST" frequency="${coord:minutes(1)}"
> start="${startTime}" end="${endTime}" timezone="${timezoneCode}"
> xmlns:sla="uri:oozie:sla:0.1" xmlns="uri:oozie:coordinator:0.2">
> <datasets>
>
> <include>${nameNode}/user/harveyc/oozie_test/datasets/test_datasets.xml</include>
> </datasets>
> <input-events>
> <data-in name="inputDataA" dataset="${dataset_A}">
> <instance>${coord:current(0)}</instance>
> </data-in>
> </input-events>
> <output-events>
> <data-out name="outputDataB" dataset="${dataset_B}">
> <instance>${coord:current(0)}</instance>
> </data-out>
> </output-events>
> <action>
> <workflow>
> <app-path>/user/harveyc/oozie_test/src/wf_touchz.xml</app-path>
> <configuration>
> <property><name>name</name><value>${wf_name}</value></property>
>
> <property><name>touchzpathb</name><value>${coord:dataOut('outputDataB')}</value></property>
> </configuration>
> </workflow>
> </action>
> </coordinator-app>
>
> Test_datasets.xml looks like this:
> <datasets>
> <dataset name="dataA" frequency="${coord:minutes(1)}"
> initial-instance="${ds_startTime}" timezone="${timezoneCode}">
>
> <uri-template>${nameNode}/user/harveyc/oozie_test/data1/${YEAR}${MONTH}${DAY}${HOUR}${MINUTE}</uri-template>
> </dataset>
> <dataset name="dataB" frequency="${coord:minutes(1)}"
> initial-instance="${ds_startTime}" timezone="${timezoneCode}">
>
> <uri-template>${nameNode}/user/harveyc/oozie_test/data2/${YEAR}${MONTH}${DAY}${HOUR}${MINUTE}</uri-template>
> </dataset>
> </datasets>
>
> The error is:
> Error: Invalid workflow-app, org.xml.sax.SAXParseException; lineNumber: 8;
> columnNumber: 57; cvc-pattern-valid: Value '${dataset_A}' is not facet-valid
> with respect to pattern '([a-zA-Z]([\-_a-zA-Z0-9])*){1,39}' for type
> 'IDENTIFIER'.
> __________________________________________________________________________________________________
>
> Fix
> __________________________________________________________________________________________________
>
> Changed the schema to include all valid string for dataset attribute for input-events "data-in and date-out".
>
> input-events datasets are el evaluated before looking for valid datasets.
> __________________________________________________________________________________________________
>
>
> Diffs
> -----
>
> http://svn.apache.org/repos/asf/oozie/trunk/client/src/main/resources/oozie-coordinator-0.1.xsd 1533583
> http://svn.apache.org/repos/asf/oozie/trunk/client/src/main/resources/oozie-coordinator-0.2.xsd 1533583
> http://svn.apache.org/repos/asf/oozie/trunk/client/src/main/resources/oozie-coordinator-0.3.xsd 1533583
> http://svn.apache.org/repos/asf/oozie/trunk/client/src/main/resources/oozie-coordinator-0.4.xsd 1533583
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordSubmitXCommand.java 1533583
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/java/org/apache/oozie/command/coord/TestCoordSubmitXCommand.java 1533583
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/resources/coord-el-dataset-4.xml PRE-CREATION
>
> Diff: https://reviews.apache.org/r/14291/diff/
>
>
> Testing
> -------
>
> Added test cases to test it for all xsd's.
>
>
> Thanks,
>
> Purshotam Shah
>
>
Re: Review Request 14291: OOZIE-1554 Support variables for coord
data-in/data-out dataset
Posted by Purshotam Shah <pu...@yahoo-inc.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14291/
-----------------------------------------------------------
(Updated Oct. 18, 2013, 7:07 p.m.)
Review request for oozie.
Changes
-------
Incorporating review comment.
Bugs: OOZIE-1554
https://issues.apache.org/jira/browse/OOZIE-1554
Repository: oozie
Description
-------
Problem
__________________________________________________________________________________________________
One would like to have a centralized list of datasets,
and use the <include> tag to make them available to every coordinator. One
would like to re-use our coordinator code, as most of his processing follows
the same steps, but with differing input and output feeds.
He need to be able to set the data-in and data-out dataset
values to variables;
My bundle coordinator entry looks like this:
<coordinator name="data1-2">
<app-path>/user/harveyc/oozie_test/src/test_coordA.xml</app-path>
<configuration>
<property><name>wf_name</name><value>1-2</value></property>
<property><name>dataset_A</name><value>dataA</value></property>
<property><name>dataset_B</name><value>dataB</value></property>
</configuration>
</coordinator>
Coord looks
<coordinator-app name="COORD_A_TEST" frequency="${coord:minutes(1)}"
start="${startTime}" end="${endTime}" timezone="${timezoneCode}"
xmlns:sla="uri:oozie:sla:0.1" xmlns="uri:oozie:coordinator:0.2">
<datasets>
<include>${nameNode}/user/harveyc/oozie_test/datasets/test_datasets.xml</include>
</datasets>
<input-events>
<data-in name="inputDataA" dataset="${dataset_A}">
<instance>${coord:current(0)}</instance>
</data-in>
</input-events>
<output-events>
<data-out name="outputDataB" dataset="${dataset_B}">
<instance>${coord:current(0)}</instance>
</data-out>
</output-events>
<action>
<workflow>
<app-path>/user/harveyc/oozie_test/src/wf_touchz.xml</app-path>
<configuration>
<property><name>name</name><value>${wf_name}</value></property>
<property><name>touchzpathb</name><value>${coord:dataOut('outputDataB')}</value></property>
</configuration>
</workflow>
</action>
</coordinator-app>
Test_datasets.xml looks like this:
<datasets>
<dataset name="dataA" frequency="${coord:minutes(1)}"
initial-instance="${ds_startTime}" timezone="${timezoneCode}">
<uri-template>${nameNode}/user/harveyc/oozie_test/data1/${YEAR}${MONTH}${DAY}${HOUR}${MINUTE}</uri-template>
</dataset>
<dataset name="dataB" frequency="${coord:minutes(1)}"
initial-instance="${ds_startTime}" timezone="${timezoneCode}">
<uri-template>${nameNode}/user/harveyc/oozie_test/data2/${YEAR}${MONTH}${DAY}${HOUR}${MINUTE}</uri-template>
</dataset>
</datasets>
The error is:
Error: Invalid workflow-app, org.xml.sax.SAXParseException; lineNumber: 8;
columnNumber: 57; cvc-pattern-valid: Value '${dataset_A}' is not facet-valid
with respect to pattern '([a-zA-Z]([\-_a-zA-Z0-9])*){1,39}' for type
'IDENTIFIER'.
__________________________________________________________________________________________________
Fix
__________________________________________________________________________________________________
Changed the schema to include all valid string for dataset attribute for input-events "data-in and date-out".
input-events datasets are el evaluated before looking for valid datasets.
__________________________________________________________________________________________________
Diffs (updated)
-----
http://svn.apache.org/repos/asf/oozie/trunk/client/src/main/resources/oozie-coordinator-0.1.xsd 1533583
http://svn.apache.org/repos/asf/oozie/trunk/client/src/main/resources/oozie-coordinator-0.2.xsd 1533583
http://svn.apache.org/repos/asf/oozie/trunk/client/src/main/resources/oozie-coordinator-0.3.xsd 1533583
http://svn.apache.org/repos/asf/oozie/trunk/client/src/main/resources/oozie-coordinator-0.4.xsd 1533583
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordSubmitXCommand.java 1533583
http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/java/org/apache/oozie/command/coord/TestCoordSubmitXCommand.java 1533583
http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/resources/coord-el-dataset-4.xml PRE-CREATION
Diff: https://reviews.apache.org/r/14291/diff/
Testing
-------
Added test cases to test it for all xsd's.
Thanks,
Purshotam Shah
Re: Review Request 14291: OOZIE-1554 Support variables for coord
data-in/data-out dataset
Posted by Purshotam Shah <pu...@yahoo-inc.com>.
> On Oct. 16, 2013, 8:56 p.m., Rohini Palaniswamy wrote:
> > http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordSubmitXCommand.java, line 824
> > <https://reviews.apache.org/r/14291/diff/2/?file=356304#file356304line824>
> >
> > Why clone? where is the clone being done?
May be copy paste issue, remove it.
> On Oct. 16, 2013, 8:56 p.m., Rohini Palaniswamy wrote:
> > http://svn.apache.org/repos/asf/oozie/trunk/client/src/main/resources/oozie-coordinator-0.1.xsd, line 91
> > <https://reviews.apache.org/r/14291/diff/2/?file=356300#file356300line91>
> >
> > Usually we create a new xsd and not update old xsds. But this is minor and if Virag says they have done this before I am ok.
Keeping as it is.
- Purshotam
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14291/#review27083
-----------------------------------------------------------
On Oct. 18, 2013, 7:07 p.m., Purshotam Shah wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/14291/
> -----------------------------------------------------------
>
> (Updated Oct. 18, 2013, 7:07 p.m.)
>
>
> Review request for oozie.
>
>
> Bugs: OOZIE-1554
> https://issues.apache.org/jira/browse/OOZIE-1554
>
>
> Repository: oozie
>
>
> Description
> -------
>
> Problem
> __________________________________________________________________________________________________
> One would like to have a centralized list of datasets,
> and use the <include> tag to make them available to every coordinator. One
> would like to re-use our coordinator code, as most of his processing follows
> the same steps, but with differing input and output feeds.
>
> He need to be able to set the data-in and data-out dataset
> values to variables;
>
>
> My bundle coordinator entry looks like this:
> <coordinator name="data1-2">
> <app-path>/user/harveyc/oozie_test/src/test_coordA.xml</app-path>
> <configuration>
> <property><name>wf_name</name><value>1-2</value></property>
> <property><name>dataset_A</name><value>dataA</value></property>
> <property><name>dataset_B</name><value>dataB</value></property>
> </configuration>
> </coordinator>
>
>
> Coord looks
> <coordinator-app name="COORD_A_TEST" frequency="${coord:minutes(1)}"
> start="${startTime}" end="${endTime}" timezone="${timezoneCode}"
> xmlns:sla="uri:oozie:sla:0.1" xmlns="uri:oozie:coordinator:0.2">
> <datasets>
>
> <include>${nameNode}/user/harveyc/oozie_test/datasets/test_datasets.xml</include>
> </datasets>
> <input-events>
> <data-in name="inputDataA" dataset="${dataset_A}">
> <instance>${coord:current(0)}</instance>
> </data-in>
> </input-events>
> <output-events>
> <data-out name="outputDataB" dataset="${dataset_B}">
> <instance>${coord:current(0)}</instance>
> </data-out>
> </output-events>
> <action>
> <workflow>
> <app-path>/user/harveyc/oozie_test/src/wf_touchz.xml</app-path>
> <configuration>
> <property><name>name</name><value>${wf_name}</value></property>
>
> <property><name>touchzpathb</name><value>${coord:dataOut('outputDataB')}</value></property>
> </configuration>
> </workflow>
> </action>
> </coordinator-app>
>
> Test_datasets.xml looks like this:
> <datasets>
> <dataset name="dataA" frequency="${coord:minutes(1)}"
> initial-instance="${ds_startTime}" timezone="${timezoneCode}">
>
> <uri-template>${nameNode}/user/harveyc/oozie_test/data1/${YEAR}${MONTH}${DAY}${HOUR}${MINUTE}</uri-template>
> </dataset>
> <dataset name="dataB" frequency="${coord:minutes(1)}"
> initial-instance="${ds_startTime}" timezone="${timezoneCode}">
>
> <uri-template>${nameNode}/user/harveyc/oozie_test/data2/${YEAR}${MONTH}${DAY}${HOUR}${MINUTE}</uri-template>
> </dataset>
> </datasets>
>
> The error is:
> Error: Invalid workflow-app, org.xml.sax.SAXParseException; lineNumber: 8;
> columnNumber: 57; cvc-pattern-valid: Value '${dataset_A}' is not facet-valid
> with respect to pattern '([a-zA-Z]([\-_a-zA-Z0-9])*){1,39}' for type
> 'IDENTIFIER'.
> __________________________________________________________________________________________________
>
> Fix
> __________________________________________________________________________________________________
>
> Changed the schema to include all valid string for dataset attribute for input-events "data-in and date-out".
>
> input-events datasets are el evaluated before looking for valid datasets.
> __________________________________________________________________________________________________
>
>
> Diffs
> -----
>
> http://svn.apache.org/repos/asf/oozie/trunk/client/src/main/resources/oozie-coordinator-0.1.xsd 1533583
> http://svn.apache.org/repos/asf/oozie/trunk/client/src/main/resources/oozie-coordinator-0.2.xsd 1533583
> http://svn.apache.org/repos/asf/oozie/trunk/client/src/main/resources/oozie-coordinator-0.3.xsd 1533583
> http://svn.apache.org/repos/asf/oozie/trunk/client/src/main/resources/oozie-coordinator-0.4.xsd 1533583
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordSubmitXCommand.java 1533583
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/java/org/apache/oozie/command/coord/TestCoordSubmitXCommand.java 1533583
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/resources/coord-el-dataset-4.xml PRE-CREATION
>
> Diff: https://reviews.apache.org/r/14291/diff/
>
>
> Testing
> -------
>
> Added test cases to test it for all xsd's.
>
>
> Thanks,
>
> Purshotam Shah
>
>
Re: Review Request 14291: OOZIE-1554 Support variables for coord
data-in/data-out dataset
Posted by Rohini Palaniswamy <ro...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14291/#review27083
-----------------------------------------------------------
http://svn.apache.org/repos/asf/oozie/trunk/client/src/main/resources/oozie-coordinator-0.1.xsd
<https://reviews.apache.org/r/14291/#comment52731>
Usually we create a new xsd and not update old xsds. But this is minor and if Virag says they have done this before I am ok.
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordSubmitXCommand.java
<https://reviews.apache.org/r/14291/#comment52723>
eAppXml. Also pass evalInst instead of using the global variable
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordSubmitXCommand.java
<https://reviews.apache.org/r/14291/#comment52727>
Why clone? where is the clone being done?
http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/java/org/apache/oozie/command/coord/TestCoordSubmitXCommand.java
<https://reviews.apache.org/r/14291/#comment52729>
Do only 1 testcase for latest schema - 0.4 xsd. Don't have to repeat it for each xsd.
http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/resources/coord-el-dataset-4.xml
<https://reviews.apache.org/r/14291/#comment52728>
Make this also a variable
- Rohini Palaniswamy
On Sept. 23, 2013, 10:57 p.m., Purshotam Shah wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/14291/
> -----------------------------------------------------------
>
> (Updated Sept. 23, 2013, 10:57 p.m.)
>
>
> Review request for oozie.
>
>
> Bugs: OOZIE-1554
> https://issues.apache.org/jira/browse/OOZIE-1554
>
>
> Repository: oozie
>
>
> Description
> -------
>
> Problem
> __________________________________________________________________________________________________
> One would like to have a centralized list of datasets,
> and use the <include> tag to make them available to every coordinator. One
> would like to re-use our coordinator code, as most of his processing follows
> the same steps, but with differing input and output feeds.
>
> He need to be able to set the data-in and data-out dataset
> values to variables;
>
>
> My bundle coordinator entry looks like this:
> <coordinator name="data1-2">
> <app-path>/user/harveyc/oozie_test/src/test_coordA.xml</app-path>
> <configuration>
> <property><name>wf_name</name><value>1-2</value></property>
> <property><name>dataset_A</name><value>dataA</value></property>
> <property><name>dataset_B</name><value>dataB</value></property>
> </configuration>
> </coordinator>
>
>
> Coord looks
> <coordinator-app name="COORD_A_TEST" frequency="${coord:minutes(1)}"
> start="${startTime}" end="${endTime}" timezone="${timezoneCode}"
> xmlns:sla="uri:oozie:sla:0.1" xmlns="uri:oozie:coordinator:0.2">
> <datasets>
>
> <include>${nameNode}/user/harveyc/oozie_test/datasets/test_datasets.xml</include>
> </datasets>
> <input-events>
> <data-in name="inputDataA" dataset="${dataset_A}">
> <instance>${coord:current(0)}</instance>
> </data-in>
> </input-events>
> <output-events>
> <data-out name="outputDataB" dataset="${dataset_B}">
> <instance>${coord:current(0)}</instance>
> </data-out>
> </output-events>
> <action>
> <workflow>
> <app-path>/user/harveyc/oozie_test/src/wf_touchz.xml</app-path>
> <configuration>
> <property><name>name</name><value>${wf_name}</value></property>
>
> <property><name>touchzpathb</name><value>${coord:dataOut('outputDataB')}</value></property>
> </configuration>
> </workflow>
> </action>
> </coordinator-app>
>
> Test_datasets.xml looks like this:
> <datasets>
> <dataset name="dataA" frequency="${coord:minutes(1)}"
> initial-instance="${ds_startTime}" timezone="${timezoneCode}">
>
> <uri-template>${nameNode}/user/harveyc/oozie_test/data1/${YEAR}${MONTH}${DAY}${HOUR}${MINUTE}</uri-template>
> </dataset>
> <dataset name="dataB" frequency="${coord:minutes(1)}"
> initial-instance="${ds_startTime}" timezone="${timezoneCode}">
>
> <uri-template>${nameNode}/user/harveyc/oozie_test/data2/${YEAR}${MONTH}${DAY}${HOUR}${MINUTE}</uri-template>
> </dataset>
> </datasets>
>
> The error is:
> Error: Invalid workflow-app, org.xml.sax.SAXParseException; lineNumber: 8;
> columnNumber: 57; cvc-pattern-valid: Value '${dataset_A}' is not facet-valid
> with respect to pattern '([a-zA-Z]([\-_a-zA-Z0-9])*){1,39}' for type
> 'IDENTIFIER'.
> __________________________________________________________________________________________________
>
> Fix
> __________________________________________________________________________________________________
>
> Changed the schema to include all valid string for dataset attribute for input-events "data-in and date-out".
>
> input-events datasets are el evaluated before looking for valid datasets.
> __________________________________________________________________________________________________
>
>
> Diffs
> -----
>
> http://svn.apache.org/repos/asf/oozie/trunk/client/src/main/resources/oozie-coordinator-0.1.xsd 1524864
> http://svn.apache.org/repos/asf/oozie/trunk/client/src/main/resources/oozie-coordinator-0.2.xsd 1524864
> http://svn.apache.org/repos/asf/oozie/trunk/client/src/main/resources/oozie-coordinator-0.3.xsd 1524864
> http://svn.apache.org/repos/asf/oozie/trunk/client/src/main/resources/oozie-coordinator-0.4.xsd 1524864
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordSubmitXCommand.java 1525674
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/java/org/apache/oozie/command/coord/TestCoordSubmitXCommand.java 1525674
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/resources/coord-el-dataset-1.xml PRE-CREATION
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/resources/coord-el-dataset-2.xml PRE-CREATION
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/resources/coord-el-dataset-3.xml PRE-CREATION
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/resources/coord-el-dataset-4.xml PRE-CREATION
>
> Diff: https://reviews.apache.org/r/14291/diff/
>
>
> Testing
> -------
>
> Added test cases to test it for all xsd's.
>
>
> Thanks,
>
> Purshotam Shah
>
>
Re: Review Request 14291: OOZIE-1554 Support variables for coord
data-in/data-out dataset
Posted by Purshotam Shah <pu...@yahoo-inc.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14291/
-----------------------------------------------------------
(Updated Sept. 23, 2013, 10:57 p.m.)
Review request for oozie.
Bugs: OOZIE-1554
https://issues.apache.org/jira/browse/OOZIE-1554
Repository: oozie
Description
-------
Problem
__________________________________________________________________________________________________
One would like to have a centralized list of datasets,
and use the <include> tag to make them available to every coordinator. One
would like to re-use our coordinator code, as most of his processing follows
the same steps, but with differing input and output feeds.
He need to be able to set the data-in and data-out dataset
values to variables;
My bundle coordinator entry looks like this:
<coordinator name="data1-2">
<app-path>/user/harveyc/oozie_test/src/test_coordA.xml</app-path>
<configuration>
<property><name>wf_name</name><value>1-2</value></property>
<property><name>dataset_A</name><value>dataA</value></property>
<property><name>dataset_B</name><value>dataB</value></property>
</configuration>
</coordinator>
Coord looks
<coordinator-app name="COORD_A_TEST" frequency="${coord:minutes(1)}"
start="${startTime}" end="${endTime}" timezone="${timezoneCode}"
xmlns:sla="uri:oozie:sla:0.1" xmlns="uri:oozie:coordinator:0.2">
<datasets>
<include>${nameNode}/user/harveyc/oozie_test/datasets/test_datasets.xml</include>
</datasets>
<input-events>
<data-in name="inputDataA" dataset="${dataset_A}">
<instance>${coord:current(0)}</instance>
</data-in>
</input-events>
<output-events>
<data-out name="outputDataB" dataset="${dataset_B}">
<instance>${coord:current(0)}</instance>
</data-out>
</output-events>
<action>
<workflow>
<app-path>/user/harveyc/oozie_test/src/wf_touchz.xml</app-path>
<configuration>
<property><name>name</name><value>${wf_name}</value></property>
<property><name>touchzpathb</name><value>${coord:dataOut('outputDataB')}</value></property>
</configuration>
</workflow>
</action>
</coordinator-app>
Test_datasets.xml looks like this:
<datasets>
<dataset name="dataA" frequency="${coord:minutes(1)}"
initial-instance="${ds_startTime}" timezone="${timezoneCode}">
<uri-template>${nameNode}/user/harveyc/oozie_test/data1/${YEAR}${MONTH}${DAY}${HOUR}${MINUTE}</uri-template>
</dataset>
<dataset name="dataB" frequency="${coord:minutes(1)}"
initial-instance="${ds_startTime}" timezone="${timezoneCode}">
<uri-template>${nameNode}/user/harveyc/oozie_test/data2/${YEAR}${MONTH}${DAY}${HOUR}${MINUTE}</uri-template>
</dataset>
</datasets>
The error is:
Error: Invalid workflow-app, org.xml.sax.SAXParseException; lineNumber: 8;
columnNumber: 57; cvc-pattern-valid: Value '${dataset_A}' is not facet-valid
with respect to pattern '([a-zA-Z]([\-_a-zA-Z0-9])*){1,39}' for type
'IDENTIFIER'.
__________________________________________________________________________________________________
Fix
__________________________________________________________________________________________________
Changed the schema to include all valid string for dataset attribute for input-events "data-in and date-out".
input-events datasets are el evaluated before looking for valid datasets.
__________________________________________________________________________________________________
Diffs
-----
http://svn.apache.org/repos/asf/oozie/trunk/client/src/main/resources/oozie-coordinator-0.1.xsd 1524864
http://svn.apache.org/repos/asf/oozie/trunk/client/src/main/resources/oozie-coordinator-0.2.xsd 1524864
http://svn.apache.org/repos/asf/oozie/trunk/client/src/main/resources/oozie-coordinator-0.3.xsd 1524864
http://svn.apache.org/repos/asf/oozie/trunk/client/src/main/resources/oozie-coordinator-0.4.xsd 1524864
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordSubmitXCommand.java 1525674
http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/java/org/apache/oozie/command/coord/TestCoordSubmitXCommand.java 1525674
http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/resources/coord-el-dataset-1.xml PRE-CREATION
http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/resources/coord-el-dataset-2.xml PRE-CREATION
http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/resources/coord-el-dataset-3.xml PRE-CREATION
http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/resources/coord-el-dataset-4.xml PRE-CREATION
Diff: https://reviews.apache.org/r/14291/diff/
Testing
-------
Added test cases to test it for all xsd's.
Thanks,
Purshotam Shah