You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@oozie.apache.org by Purshotam Shah <pu...@yahoo-inc.com> on 2013/09/20 00:20:14 UTC
Review Request 14239: OOZIE-1504 parameterize coord EL functions.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14239/
-----------------------------------------------------------
Review request for oozie.
Bugs: OOZIE-1504
https://issues.apache.org/jira/browse/OOZIE-1504
Repository: oozie
Description
-------
Use-case
-----------------------------
A workflow that processes input datasets for the past 30 days:
<input-events>
<data-in name="event_input_path_format1" dataset="EVENT_INPUT_FORMAT1">
<start-instance>$
{coord:current(-30)}
</start-instance>
<end-instance>$
{coord:current(-1)}
</end-instance>
</data-in>
<data-in name="event_input_path_format2" dataset="EVENT_INPUT_FORMAT2">
<start-instance>$
{coord:current(-30)}
</start-instance>
<end-instance>$
{coord:current(-1)}
</end-instance>
</data-in>
</input-events>
Instead, one wants it to start on the same specific date so that each day, it
processes one more data set than the last day (the datasets are daily). Something like the following is not supported
<input-events>
<data-in name="event_input_path_format1" dataset="EVENT_INPUT_FORMAT1">
<start-instance>2013-03-15T00:00Z</start-instance>
<end-instance>$
{coord:current(-1)}
</end-instance>
</data-in>
<data-in name="event_input_path_format2" dataset="EVENT_INPUT_FORMAT2">
<start-instance>2013-03-15T00:00Z</start-instance>
<end-instance>$
{coord:current(-1)}
</end-instance>
</data-in>
</input-events>
---------------------------------
Fix support providing absolute value for instance range. One can have a combination of absolute date with coord:current().
Other EL function are not supported with absolute date.
coord:Current() can be parameterize.
like
<end-instance>${coord:latest(end_date)}</end-instance> // where end_date is one of the specified property.
Diffs
-----
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordCommandUtils.java 1523758
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/coord/CoordELFunctions.java 1523758
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/resources/oozie-default.xml 1523758
http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/java/org/apache/oozie/command/coord/TestCoordCommandUtils.java 1523758
http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/java/org/apache/oozie/command/coord/TestCoordMaterializeTransitionXCommand.java 1523758
http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/java/org/apache/oozie/coord/TestCoordELEvaluator.java 1523758
http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/java/org/apache/oozie/coord/TestCoordELFunctions.java 1523758
http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/resources/cord-action-for-parameterization-1.xml PRE-CREATION
http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/resources/cord-action-for-parameterization-2.xml PRE-CREATION
http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/resources/cord-action-for-parameterization-3.xml PRE-CREATION
http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/resources/cord-action-for-parameterization-4.xml PRE-CREATION
http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/resources/cord-action-for-parameterization.xml PRE-CREATION
Diff: https://reviews.apache.org/r/14239/diff/
Testing
-------
Added multiple test cases to verify combination of scenarios.
Thanks,
Purshotam Shah
Re: Review Request 14239: OOZIE-1504 parameterize coord EL functions.
Posted by Mona Chitnis <mo...@yahoo.in>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14239/#review27730
-----------------------------------------------------------
In addition to phase 2 (ph2_*) functions which do the actual resolution, add phase 1 (ph1_*) counterparts to echo the function name. This is required during initial submission and materialization phase.
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordCommandUtils.java
<https://reviews.apache.org/r/14239/#comment53852>
slightly confusing condition. Throw exception if the date is not absolute?
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/coord/CoordELFunctions.java
<https://reviews.apache.org/r/14239/#comment53854>
Also print the startTime and endTime (indexes) here so we know what exactly threw the exception
http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/resources/cord-action-for-parameterization-1.xml
<https://reviews.apache.org/r/14239/#comment53856>
same comment as above
http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/resources/cord-action-for-parameterization-1.xml
<https://reviews.apache.org/r/14239/#comment53858>
dataset name='logs'
http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/resources/cord-action-for-parameterization-1.xml
<https://reviews.apache.org/r/14239/#comment53857>
should be dataIn('input') i.e. name given to the data-in
http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/resources/cord-action-for-parameterization-2.xml
<https://reviews.apache.org/r/14239/#comment53861>
same as above
http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/resources/cord-action-for-parameterization-2.xml
<https://reviews.apache.org/r/14239/#comment53860>
same comment as above
http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/resources/cord-action-for-parameterization-3.xml
<https://reviews.apache.org/r/14239/#comment53862>
same as above
http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/resources/cord-action-for-parameterization-3.xml
<https://reviews.apache.org/r/14239/#comment53867>
do you need two dataset definitions here? I see both are the same
http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/resources/cord-action-for-parameterization-3.xml
<https://reviews.apache.org/r/14239/#comment53863>
same comment as above
http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/resources/cord-action-for-parameterization-4.xml
<https://reviews.apache.org/r/14239/#comment53864>
same as above
http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/resources/cord-action-for-parameterization-4.xml
<https://reviews.apache.org/r/14239/#comment53865>
same as above
http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/resources/cord-action-for-parameterization.xml
<https://reviews.apache.org/r/14239/#comment53855>
maintain the original structure in this comment block, by reverting the auto format here
http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/resources/cord-action-for-parameterization.xml
<https://reviews.apache.org/r/14239/#comment53866>
do you need two dataset definitions here? I see both are the same
http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/resources/cord-action-for-parameterization.xml
<https://reviews.apache.org/r/14239/#comment53859>
should be dataIn('input') i.e. name given to the data-in
- Mona Chitnis
On Sept. 19, 2013, 10:20 p.m., Purshotam Shah wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/14239/
> -----------------------------------------------------------
>
> (Updated Sept. 19, 2013, 10:20 p.m.)
>
>
> Review request for oozie.
>
>
> Bugs: OOZIE-1504
> https://issues.apache.org/jira/browse/OOZIE-1504
>
>
> Repository: oozie
>
>
> Description
> -------
>
> Use-case
> -----------------------------
> A workflow that processes input datasets for the past 30 days:
> <input-events>
> <data-in name="event_input_path_format1" dataset="EVENT_INPUT_FORMAT1">
> <start-instance>$
> {coord:current(-30)}
> </start-instance>
> <end-instance>$
> {coord:current(-1)}
> </end-instance>
> </data-in>
> <data-in name="event_input_path_format2" dataset="EVENT_INPUT_FORMAT2">
> <start-instance>$
> {coord:current(-30)}
> </start-instance>
> <end-instance>$
> {coord:current(-1)}
> </end-instance>
> </data-in>
> </input-events>
> Instead, one wants it to start on the same specific date so that each day, it
> processes one more data set than the last day (the datasets are daily). Something like the following is not supported
> <input-events>
> <data-in name="event_input_path_format1" dataset="EVENT_INPUT_FORMAT1">
> <start-instance>2013-03-15T00:00Z</start-instance>
> <end-instance>$
> {coord:current(-1)}
> </end-instance>
> </data-in>
> <data-in name="event_input_path_format2" dataset="EVENT_INPUT_FORMAT2">
> <start-instance>2013-03-15T00:00Z</start-instance>
> <end-instance>$
> {coord:current(-1)}
> </end-instance>
> </data-in>
> </input-events>
> ---------------------------------
>
>
> Fix support providing absolute value for instance range. One can have a combination of absolute date with coord:current().
> Other EL function are not supported with absolute date.
>
> coord:Current() can be parameterize.
>
> like
> <end-instance>${coord:latest(end_date)}</end-instance> // where end_date is one of the specified property.
>
>
> Diffs
> -----
>
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordCommandUtils.java 1523758
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/coord/CoordELFunctions.java 1523758
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/resources/oozie-default.xml 1523758
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/java/org/apache/oozie/command/coord/TestCoordCommandUtils.java 1523758
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/java/org/apache/oozie/command/coord/TestCoordMaterializeTransitionXCommand.java 1523758
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/java/org/apache/oozie/coord/TestCoordELEvaluator.java 1523758
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/java/org/apache/oozie/coord/TestCoordELFunctions.java 1523758
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/resources/cord-action-for-parameterization-1.xml PRE-CREATION
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/resources/cord-action-for-parameterization-2.xml PRE-CREATION
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/resources/cord-action-for-parameterization-3.xml PRE-CREATION
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/resources/cord-action-for-parameterization-4.xml PRE-CREATION
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/resources/cord-action-for-parameterization.xml PRE-CREATION
>
> Diff: https://reviews.apache.org/r/14239/diff/
>
>
> Testing
> -------
>
> Added multiple test cases to verify combination of scenarios.
>
>
> Thanks,
>
> Purshotam Shah
>
>
Re: Review Request 14239: OOZIE-1504 parameterize coord EL functions.
Posted by Purshotam Shah <pu...@yahoo-inc.com>.
> On Nov. 27, 2013, 10:12 p.m., Rohini Palaniswamy wrote:
> > http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordCommandUtils.java, lines 237-240
> > <https://reviews.apache.org/r/14239/diff/3/?file=391354#file391354line237>
> >
> > Not required as checkIfBothSameType takes care of this
We want to throw different error message if start is absolute and end is not current.
- Purshotam
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14239/#review29516
-----------------------------------------------------------
On Nov. 27, 2013, 1:42 a.m., Purshotam Shah wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/14239/
> -----------------------------------------------------------
>
> (Updated Nov. 27, 2013, 1:42 a.m.)
>
>
> Review request for oozie.
>
>
> Bugs: OOZIE-1504
> https://issues.apache.org/jira/browse/OOZIE-1504
>
>
> Repository: oozie
>
>
> Description
> -------
>
> Use-case
> -----------------------------
> A workflow that processes input datasets for the past 30 days:
> <input-events>
> <data-in name="event_input_path_format1" dataset="EVENT_INPUT_FORMAT1">
> <start-instance>$
> {coord:current(-30)}
> </start-instance>
> <end-instance>$
> {coord:current(-1)}
> </end-instance>
> </data-in>
> <data-in name="event_input_path_format2" dataset="EVENT_INPUT_FORMAT2">
> <start-instance>$
> {coord:current(-30)}
> </start-instance>
> <end-instance>$
> {coord:current(-1)}
> </end-instance>
> </data-in>
> </input-events>
> Instead, one wants it to start on the same specific date so that each day, it
> processes one more data set than the last day (the datasets are daily). Something like the following is not supported
> <input-events>
> <data-in name="event_input_path_format1" dataset="EVENT_INPUT_FORMAT1">
> <start-instance>${coord:absolute(2013-03-15T00:00Z)}</start-instance>
> <end-instance>$
> {coord:current(-1)}
> </end-instance>
> </data-in>
> <data-in name="event_input_path_format2" dataset="EVENT_INPUT_FORMAT2">
> <start-instance>${coord:absolute(2013-03-15T00:00Z)}</start-instance>
> <end-instance>$
> {coord:current(-1)}
> </end-instance>
> </data-in>
> </input-events>
> ---------------------------------
>
>
> Fix support providing absolute value for instance range. One can have a combination of absolute date with coord:current().
> Other EL function are not supported with absolute date.
>
> coord:Current() can be parameterize.
>
> like
> <end-instance>${coord:latest(end_date)}</end-instance> // where end_date is one of the specified property.
>
>
> Diffs
> -----
>
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordCommandUtils.java 1544636
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/coord/CoordELFunctions.java 1544636
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/resources/oozie-default.xml 1544636
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/java/org/apache/oozie/command/coord/TestCoordCommandUtils.java 1544636
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/java/org/apache/oozie/coord/TestCoordELFunctions.java 1544636
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/resources/coord-dataset-absolute.xml PRE-CREATION
> http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/CoordinatorFunctionalSpec.twiki 1544636
> http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/WebServicesAPI.twiki 1544636
>
> Diff: https://reviews.apache.org/r/14239/diff/
>
>
> Testing
> -------
>
> Added multiple test cases to verify combination of scenarios.
>
>
> Thanks,
>
> Purshotam Shah
>
>
Re: Review Request 14239: OOZIE-1504 parameterize coord EL functions.
Posted by Rohini Palaniswamy <ro...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14239/#review29516
-----------------------------------------------------------
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordCommandUtils.java
<https://reviews.apache.org/r/14239/#comment56829>
else block in {}
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordCommandUtils.java
<https://reviews.apache.org/r/14239/#comment56828>
Not required as checkIfBothSameType takes care of this
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/coord/CoordELFunctions.java
<https://reviews.apache.org/r/14239/#comment56839>
startInstance
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/coord/CoordELFunctions.java
<https://reviews.apache.org/r/14239/#comment56838>
startInstance = " + startInstance is equivalent to current (" + instanceCount[0] + ") but end is specified as current (" + end + ")");
http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/java/org/apache/oozie/command/coord/TestCoordCommandUtils.java
<https://reviews.apache.org/r/14239/#comment56835>
Avoid having a lot of newlines
http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/java/org/apache/oozie/command/coord/TestCoordCommandUtils.java
<https://reviews.apache.org/r/14239/#comment56836>
This is wrong. It should have 2009-08-20 in all the asserts. Please verify code logic
http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/java/org/apache/oozie/command/coord/TestCoordCommandUtils.java
<https://reviews.apache.org/r/14239/#comment56834>
Can you make the fail() message more meaningful
http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/java/org/apache/oozie/coord/TestCoordELFunctions.java
<https://reviews.apache.org/r/14239/#comment56833>
Can you change the fail() message to "start-instance is greater than the end-instance and there was no exception"
http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/CoordinatorFunctionalSpec.twiki
<https://reviews.apache.org/r/14239/#comment56831>
your processing
- Rohini Palaniswamy
On Nov. 27, 2013, 1:42 a.m., Purshotam Shah wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/14239/
> -----------------------------------------------------------
>
> (Updated Nov. 27, 2013, 1:42 a.m.)
>
>
> Review request for oozie.
>
>
> Bugs: OOZIE-1504
> https://issues.apache.org/jira/browse/OOZIE-1504
>
>
> Repository: oozie
>
>
> Description
> -------
>
> Use-case
> -----------------------------
> A workflow that processes input datasets for the past 30 days:
> <input-events>
> <data-in name="event_input_path_format1" dataset="EVENT_INPUT_FORMAT1">
> <start-instance>$
> {coord:current(-30)}
> </start-instance>
> <end-instance>$
> {coord:current(-1)}
> </end-instance>
> </data-in>
> <data-in name="event_input_path_format2" dataset="EVENT_INPUT_FORMAT2">
> <start-instance>$
> {coord:current(-30)}
> </start-instance>
> <end-instance>$
> {coord:current(-1)}
> </end-instance>
> </data-in>
> </input-events>
> Instead, one wants it to start on the same specific date so that each day, it
> processes one more data set than the last day (the datasets are daily). Something like the following is not supported
> <input-events>
> <data-in name="event_input_path_format1" dataset="EVENT_INPUT_FORMAT1">
> <start-instance>${coord:absolute(2013-03-15T00:00Z)}</start-instance>
> <end-instance>$
> {coord:current(-1)}
> </end-instance>
> </data-in>
> <data-in name="event_input_path_format2" dataset="EVENT_INPUT_FORMAT2">
> <start-instance>${coord:absolute(2013-03-15T00:00Z)}</start-instance>
> <end-instance>$
> {coord:current(-1)}
> </end-instance>
> </data-in>
> </input-events>
> ---------------------------------
>
>
> Fix support providing absolute value for instance range. One can have a combination of absolute date with coord:current().
> Other EL function are not supported with absolute date.
>
> coord:Current() can be parameterize.
>
> like
> <end-instance>${coord:latest(end_date)}</end-instance> // where end_date is one of the specified property.
>
>
> Diffs
> -----
>
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordCommandUtils.java 1544636
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/coord/CoordELFunctions.java 1544636
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/resources/oozie-default.xml 1544636
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/java/org/apache/oozie/command/coord/TestCoordCommandUtils.java 1544636
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/java/org/apache/oozie/coord/TestCoordELFunctions.java 1544636
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/resources/coord-dataset-absolute.xml PRE-CREATION
> http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/CoordinatorFunctionalSpec.twiki 1544636
> http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/WebServicesAPI.twiki 1544636
>
> Diff: https://reviews.apache.org/r/14239/diff/
>
>
> Testing
> -------
>
> Added multiple test cases to verify combination of scenarios.
>
>
> Thanks,
>
> Purshotam Shah
>
>
Re: Review Request 14239: OOZIE-1504 parameterize coord EL functions.
Posted by Rohini Palaniswamy <ro...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14239/#review29751
-----------------------------------------------------------
http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/java/org/apache/oozie/command/coord/TestCoordCommandUtils.java
<https://reviews.apache.org/r/14239/#comment57221>
Only the test case has been changed. You need to fix the logic. The expected instances is still not right even with this. There should have been 3 instances for this case.
Please have tests for 3 cases of nominal time - same as absolute instance, > absolute instance, < absolute instance . Keep the coord job start time earlier than the dataset initial instance, so that you can do the < start instance case. We also need to throw an exception if the absolute time is < dataset initial instance.
- Rohini Palaniswamy
On Nov. 28, 2013, 12:28 a.m., Purshotam Shah wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/14239/
> -----------------------------------------------------------
>
> (Updated Nov. 28, 2013, 12:28 a.m.)
>
>
> Review request for oozie.
>
>
> Bugs: OOZIE-1504
> https://issues.apache.org/jira/browse/OOZIE-1504
>
>
> Repository: oozie
>
>
> Description
> -------
>
> Use-case
> -----------------------------
> A workflow that processes input datasets for the past 30 days:
> <input-events>
> <data-in name="event_input_path_format1" dataset="EVENT_INPUT_FORMAT1">
> <start-instance>$
> {coord:current(-30)}
> </start-instance>
> <end-instance>$
> {coord:current(-1)}
> </end-instance>
> </data-in>
> <data-in name="event_input_path_format2" dataset="EVENT_INPUT_FORMAT2">
> <start-instance>$
> {coord:current(-30)}
> </start-instance>
> <end-instance>$
> {coord:current(-1)}
> </end-instance>
> </data-in>
> </input-events>
> Instead, one wants it to start on the same specific date so that each day, it
> processes one more data set than the last day (the datasets are daily). Something like the following is not supported
> <input-events>
> <data-in name="event_input_path_format1" dataset="EVENT_INPUT_FORMAT1">
> <start-instance>${coord:absolute(2013-03-15T00:00Z)}</start-instance>
> <end-instance>$
> {coord:current(-1)}
> </end-instance>
> </data-in>
> <data-in name="event_input_path_format2" dataset="EVENT_INPUT_FORMAT2">
> <start-instance>${coord:absolute(2013-03-15T00:00Z)}</start-instance>
> <end-instance>$
> {coord:current(-1)}
> </end-instance>
> </data-in>
> </input-events>
> ---------------------------------
>
>
> Fix support providing absolute value for instance range. One can have a combination of absolute date with coord:current().
> Other EL function are not supported with absolute date.
>
> coord:Current() can be parameterize.
>
> like
> <end-instance>${coord:latest(end_date)}</end-instance> // where end_date is one of the specified property.
>
>
> Diffs
> -----
>
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordCommandUtils.java 1544636
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/coord/CoordELFunctions.java 1544636
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/resources/oozie-default.xml 1544636
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/java/org/apache/oozie/command/coord/TestCoordCommandUtils.java 1544636
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/java/org/apache/oozie/coord/TestCoordELFunctions.java 1544636
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/resources/coord-dataset-absolute.xml PRE-CREATION
> http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/CoordinatorFunctionalSpec.twiki 1544636
> http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/WebServicesAPI.twiki 1544636
>
> Diff: https://reviews.apache.org/r/14239/diff/
>
>
> Testing
> -------
>
> Added multiple test cases to verify combination of scenarios.
>
>
> Thanks,
>
> Purshotam Shah
>
>
Re: Review Request 14239: OOZIE-1504 parameterize coord EL functions.
Posted by Rohini Palaniswamy <ro...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14239/#review30501
-----------------------------------------------------------
Ship it!
Ship It!
- Rohini Palaniswamy
On Dec. 5, 2013, 12:48 a.m., Purshotam Shah wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/14239/
> -----------------------------------------------------------
>
> (Updated Dec. 5, 2013, 12:48 a.m.)
>
>
> Review request for oozie.
>
>
> Bugs: OOZIE-1504
> https://issues.apache.org/jira/browse/OOZIE-1504
>
>
> Repository: oozie
>
>
> Description
> -------
>
> Use-case
> -----------------------------
> A workflow that processes input datasets for the past 30 days:
> <input-events>
> <data-in name="event_input_path_format1" dataset="EVENT_INPUT_FORMAT1">
> <start-instance>$
> {coord:current(-30)}
> </start-instance>
> <end-instance>$
> {coord:current(-1)}
> </end-instance>
> </data-in>
> <data-in name="event_input_path_format2" dataset="EVENT_INPUT_FORMAT2">
> <start-instance>$
> {coord:current(-30)}
> </start-instance>
> <end-instance>$
> {coord:current(-1)}
> </end-instance>
> </data-in>
> </input-events>
> Instead, one wants it to start on the same specific date so that each day, it
> processes one more data set than the last day (the datasets are daily). Something like the following is not supported
> <input-events>
> <data-in name="event_input_path_format1" dataset="EVENT_INPUT_FORMAT1">
> <start-instance>${coord:absolute(2013-03-15T00:00Z)}</start-instance>
> <end-instance>$
> {coord:current(-1)}
> </end-instance>
> </data-in>
> <data-in name="event_input_path_format2" dataset="EVENT_INPUT_FORMAT2">
> <start-instance>${coord:absolute(2013-03-15T00:00Z)}</start-instance>
> <end-instance>$
> {coord:current(-1)}
> </end-instance>
> </data-in>
> </input-events>
> ---------------------------------
>
>
> Fix support providing absolute value for instance range. One can have a combination of absolute date with coord:current().
> Other EL function are not supported with absolute date.
>
> coord:Current() can be parameterize.
>
> like
> <end-instance>${coord:latest(end_date)}</end-instance> // where end_date is one of the specified property.
>
>
> Diffs
> -----
>
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordCommandUtils.java 1544636
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/coord/CoordELFunctions.java 1544636
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/resources/oozie-default.xml 1544636
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/java/org/apache/oozie/command/coord/TestCoordCommandUtils.java 1544636
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/java/org/apache/oozie/coord/TestCoordELFunctions.java 1544636
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/resources/coord-dataset-absolute.xml PRE-CREATION
> http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/CoordinatorFunctionalSpec.twiki 1544636
> http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/WebServicesAPI.twiki 1544636
>
> Diff: https://reviews.apache.org/r/14239/diff/
>
>
> Testing
> -------
>
> Added multiple test cases to verify combination of scenarios.
>
>
> Thanks,
>
> Purshotam Shah
>
>
Re: Review Request 14239: OOZIE-1504 parameterize coord EL functions.
Posted by Purshotam Shah <pu...@yahoo-inc.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14239/
-----------------------------------------------------------
(Updated Dec. 5, 2013, 12:48 a.m.)
Review request for oozie.
Changes
-------
Fixing offset logic issue.
Bugs: OOZIE-1504
https://issues.apache.org/jira/browse/OOZIE-1504
Repository: oozie
Description
-------
Use-case
-----------------------------
A workflow that processes input datasets for the past 30 days:
<input-events>
<data-in name="event_input_path_format1" dataset="EVENT_INPUT_FORMAT1">
<start-instance>$
{coord:current(-30)}
</start-instance>
<end-instance>$
{coord:current(-1)}
</end-instance>
</data-in>
<data-in name="event_input_path_format2" dataset="EVENT_INPUT_FORMAT2">
<start-instance>$
{coord:current(-30)}
</start-instance>
<end-instance>$
{coord:current(-1)}
</end-instance>
</data-in>
</input-events>
Instead, one wants it to start on the same specific date so that each day, it
processes one more data set than the last day (the datasets are daily). Something like the following is not supported
<input-events>
<data-in name="event_input_path_format1" dataset="EVENT_INPUT_FORMAT1">
<start-instance>${coord:absolute(2013-03-15T00:00Z)}</start-instance>
<end-instance>$
{coord:current(-1)}
</end-instance>
</data-in>
<data-in name="event_input_path_format2" dataset="EVENT_INPUT_FORMAT2">
<start-instance>${coord:absolute(2013-03-15T00:00Z)}</start-instance>
<end-instance>$
{coord:current(-1)}
</end-instance>
</data-in>
</input-events>
---------------------------------
Fix support providing absolute value for instance range. One can have a combination of absolute date with coord:current().
Other EL function are not supported with absolute date.
coord:Current() can be parameterize.
like
<end-instance>${coord:latest(end_date)}</end-instance> // where end_date is one of the specified property.
Diffs (updated)
-----
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordCommandUtils.java 1544636
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/coord/CoordELFunctions.java 1544636
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/resources/oozie-default.xml 1544636
http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/java/org/apache/oozie/command/coord/TestCoordCommandUtils.java 1544636
http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/java/org/apache/oozie/coord/TestCoordELFunctions.java 1544636
http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/resources/coord-dataset-absolute.xml PRE-CREATION
http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/CoordinatorFunctionalSpec.twiki 1544636
http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/WebServicesAPI.twiki 1544636
Diff: https://reviews.apache.org/r/14239/diff/
Testing
-------
Added multiple test cases to verify combination of scenarios.
Thanks,
Purshotam Shah
Re: Review Request 14239: OOZIE-1504 parameterize coord EL functions.
Posted by Purshotam Shah <pu...@yahoo-inc.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14239/
-----------------------------------------------------------
(Updated Nov. 28, 2013, 12:28 a.m.)
Review request for oozie.
Changes
-------
review comment.
Bugs: OOZIE-1504
https://issues.apache.org/jira/browse/OOZIE-1504
Repository: oozie
Description
-------
Use-case
-----------------------------
A workflow that processes input datasets for the past 30 days:
<input-events>
<data-in name="event_input_path_format1" dataset="EVENT_INPUT_FORMAT1">
<start-instance>$
{coord:current(-30)}
</start-instance>
<end-instance>$
{coord:current(-1)}
</end-instance>
</data-in>
<data-in name="event_input_path_format2" dataset="EVENT_INPUT_FORMAT2">
<start-instance>$
{coord:current(-30)}
</start-instance>
<end-instance>$
{coord:current(-1)}
</end-instance>
</data-in>
</input-events>
Instead, one wants it to start on the same specific date so that each day, it
processes one more data set than the last day (the datasets are daily). Something like the following is not supported
<input-events>
<data-in name="event_input_path_format1" dataset="EVENT_INPUT_FORMAT1">
<start-instance>${coord:absolute(2013-03-15T00:00Z)}</start-instance>
<end-instance>$
{coord:current(-1)}
</end-instance>
</data-in>
<data-in name="event_input_path_format2" dataset="EVENT_INPUT_FORMAT2">
<start-instance>${coord:absolute(2013-03-15T00:00Z)}</start-instance>
<end-instance>$
{coord:current(-1)}
</end-instance>
</data-in>
</input-events>
---------------------------------
Fix support providing absolute value for instance range. One can have a combination of absolute date with coord:current().
Other EL function are not supported with absolute date.
coord:Current() can be parameterize.
like
<end-instance>${coord:latest(end_date)}</end-instance> // where end_date is one of the specified property.
Diffs (updated)
-----
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordCommandUtils.java 1544636
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/coord/CoordELFunctions.java 1544636
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/resources/oozie-default.xml 1544636
http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/java/org/apache/oozie/command/coord/TestCoordCommandUtils.java 1544636
http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/java/org/apache/oozie/coord/TestCoordELFunctions.java 1544636
http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/resources/coord-dataset-absolute.xml PRE-CREATION
http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/CoordinatorFunctionalSpec.twiki 1544636
http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/WebServicesAPI.twiki 1544636
Diff: https://reviews.apache.org/r/14239/diff/
Testing
-------
Added multiple test cases to verify combination of scenarios.
Thanks,
Purshotam Shah
Re: Review Request 14239: OOZIE-1504 parameterize coord EL functions.
Posted by Purshotam Shah <pu...@yahoo-inc.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14239/
-----------------------------------------------------------
(Updated Nov. 27, 2013, 1:42 a.m.)
Review request for oozie.
Changes
-------
Review comment..
Bugs: OOZIE-1504
https://issues.apache.org/jira/browse/OOZIE-1504
Repository: oozie
Description
-------
Use-case
-----------------------------
A workflow that processes input datasets for the past 30 days:
<input-events>
<data-in name="event_input_path_format1" dataset="EVENT_INPUT_FORMAT1">
<start-instance>$
{coord:current(-30)}
</start-instance>
<end-instance>$
{coord:current(-1)}
</end-instance>
</data-in>
<data-in name="event_input_path_format2" dataset="EVENT_INPUT_FORMAT2">
<start-instance>$
{coord:current(-30)}
</start-instance>
<end-instance>$
{coord:current(-1)}
</end-instance>
</data-in>
</input-events>
Instead, one wants it to start on the same specific date so that each day, it
processes one more data set than the last day (the datasets are daily). Something like the following is not supported
<input-events>
<data-in name="event_input_path_format1" dataset="EVENT_INPUT_FORMAT1">
<start-instance>${coord:absolute(2013-03-15T00:00Z)}</start-instance>
<end-instance>$
{coord:current(-1)}
</end-instance>
</data-in>
<data-in name="event_input_path_format2" dataset="EVENT_INPUT_FORMAT2">
<start-instance>${coord:absolute(2013-03-15T00:00Z)}</start-instance>
<end-instance>$
{coord:current(-1)}
</end-instance>
</data-in>
</input-events>
---------------------------------
Fix support providing absolute value for instance range. One can have a combination of absolute date with coord:current().
Other EL function are not supported with absolute date.
coord:Current() can be parameterize.
like
<end-instance>${coord:latest(end_date)}</end-instance> // where end_date is one of the specified property.
Diffs (updated)
-----
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordCommandUtils.java 1544636
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/coord/CoordELFunctions.java 1544636
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/resources/oozie-default.xml 1544636
http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/java/org/apache/oozie/command/coord/TestCoordCommandUtils.java 1544636
http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/java/org/apache/oozie/coord/TestCoordELFunctions.java 1544636
http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/resources/coord-dataset-absolute.xml PRE-CREATION
http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/CoordinatorFunctionalSpec.twiki 1544636
http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/WebServicesAPI.twiki 1544636
Diff: https://reviews.apache.org/r/14239/diff/
Testing
-------
Added multiple test cases to verify combination of scenarios.
Thanks,
Purshotam Shah
Re: Review Request 14239: OOZIE-1504 parameterize coord EL functions.
Posted by Purshotam Shah <pu...@yahoo-inc.com>.
> On Nov. 25, 2013, 10:11 p.m., Rohini Palaniswamy wrote:
> > http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/coord/CoordELFunctions.java, line 731
> > <https://reviews.apache.org/r/14239/diff/2/?file=390294#file390294line731>
> >
> > You need to handle timezone
getCurrentInstance take care of setting timezone .
private static Calendar getCurrentInstance(Date effectiveTime, int instanceCount[], ELEvaluator eval) {
Date datasetInitialInstance = getInitialInstance(eval);
TimeUnit dsTimeUnit = getDSTimeUnit(eval);
TimeZone dsTZ = getDatasetTZ(eval);
int dsFreq = getDSFrequency(eval);
// Convert Date to Calendar for corresponding TZ
Calendar current = Calendar.getInstance();
current.setTime(datasetInitialInstance);
current.setTimeZone(dsTZ);
Calendar calEffectiveTime = Calendar.getInstance();
calEffectiveTime.setTime(effectiveTime);
calEffectiveTime.setTimeZone(dsTZ);
> On Nov. 25, 2013, 10:11 p.m., Rohini Palaniswamy wrote:
> > http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/java/org/apache/oozie/coord/TestCoordELFunctions.java, line 999
> > <https://reviews.apache.org/r/14239/diff/2/?file=390297#file390297line999>
> >
> > formatting.
> >
> > This should throw an exception and not return ""
- Purshotam
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14239/#review29376
-----------------------------------------------------------
On Nov. 27, 2013, 1:42 a.m., Purshotam Shah wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/14239/
> -----------------------------------------------------------
>
> (Updated Nov. 27, 2013, 1:42 a.m.)
>
>
> Review request for oozie.
>
>
> Bugs: OOZIE-1504
> https://issues.apache.org/jira/browse/OOZIE-1504
>
>
> Repository: oozie
>
>
> Description
> -------
>
> Use-case
> -----------------------------
> A workflow that processes input datasets for the past 30 days:
> <input-events>
> <data-in name="event_input_path_format1" dataset="EVENT_INPUT_FORMAT1">
> <start-instance>$
> {coord:current(-30)}
> </start-instance>
> <end-instance>$
> {coord:current(-1)}
> </end-instance>
> </data-in>
> <data-in name="event_input_path_format2" dataset="EVENT_INPUT_FORMAT2">
> <start-instance>$
> {coord:current(-30)}
> </start-instance>
> <end-instance>$
> {coord:current(-1)}
> </end-instance>
> </data-in>
> </input-events>
> Instead, one wants it to start on the same specific date so that each day, it
> processes one more data set than the last day (the datasets are daily). Something like the following is not supported
> <input-events>
> <data-in name="event_input_path_format1" dataset="EVENT_INPUT_FORMAT1">
> <start-instance>${coord:absolute(2013-03-15T00:00Z)}</start-instance>
> <end-instance>$
> {coord:current(-1)}
> </end-instance>
> </data-in>
> <data-in name="event_input_path_format2" dataset="EVENT_INPUT_FORMAT2">
> <start-instance>${coord:absolute(2013-03-15T00:00Z)}</start-instance>
> <end-instance>$
> {coord:current(-1)}
> </end-instance>
> </data-in>
> </input-events>
> ---------------------------------
>
>
> Fix support providing absolute value for instance range. One can have a combination of absolute date with coord:current().
> Other EL function are not supported with absolute date.
>
> coord:Current() can be parameterize.
>
> like
> <end-instance>${coord:latest(end_date)}</end-instance> // where end_date is one of the specified property.
>
>
> Diffs
> -----
>
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordCommandUtils.java 1544636
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/coord/CoordELFunctions.java 1544636
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/resources/oozie-default.xml 1544636
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/java/org/apache/oozie/command/coord/TestCoordCommandUtils.java 1544636
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/java/org/apache/oozie/coord/TestCoordELFunctions.java 1544636
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/resources/coord-dataset-absolute.xml PRE-CREATION
> http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/CoordinatorFunctionalSpec.twiki 1544636
> http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/WebServicesAPI.twiki 1544636
>
> Diff: https://reviews.apache.org/r/14239/diff/
>
>
> Testing
> -------
>
> Added multiple test cases to verify combination of scenarios.
>
>
> Thanks,
>
> Purshotam Shah
>
>
Re: Review Request 14239: OOZIE-1504 parameterize coord EL functions.
Posted by Rohini Palaniswamy <ro...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14239/#review29376
-----------------------------------------------------------
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordCommandUtils.java
<https://reviews.apache.org/r/14239/#comment56621>
formatting
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordCommandUtils.java
<https://reviews.apache.org/r/14239/#comment56595>
Can you rename this as parseOneStringArg
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordCommandUtils.java
<https://reviews.apache.org/r/14239/#comment56620>
formatting
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordCommandUtils.java
<https://reviews.apache.org/r/14239/#comment56573>
This condition and else block needs to be fixed. Check is wrong.
formatting.
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordCommandUtils.java
<https://reviews.apache.org/r/14239/#comment56574>
You are going to throw exception even if they are same
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordCommandUtils.java
<https://reviews.apache.org/r/14239/#comment56575>
Move function to DateUtils and rename the function to validateDateOozieTZ
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordCommandUtils.java
<https://reviews.apache.org/r/14239/#comment56619>
formatting
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/coord/CoordELFunctions.java
<https://reviews.apache.org/r/14239/#comment56601>
You need to handle timezone
http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/java/org/apache/oozie/command/coord/TestCoordCommandUtils.java
<https://reviews.apache.org/r/14239/#comment56605>
Have one positive test in TestCoordActionInputCheckxcommand to test more e2e flow.
Rename test methods and test files. These are tests for EL functions and not parameterizing.
Please add some comments to the tests. It is difficult to understand what the test is doing without thorough reading of code.
http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/java/org/apache/oozie/command/coord/TestCoordCommandUtils.java
<https://reviews.apache.org/r/14239/#comment56622>
Extract common test code to one method
http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/java/org/apache/oozie/command/coord/TestCoordCommandUtils.java
<https://reviews.apache.org/r/14239/#comment56602>
Remove the try catch block in all test methods. Redundant.
http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/java/org/apache/oozie/command/coord/TestCoordCommandUtils.java
<https://reviews.apache.org/r/14239/#comment56618>
Do in one assertEquals. Same for other methods
http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/java/org/apache/oozie/command/coord/TestCoordCommandUtils.java
<https://reviews.apache.org/r/14239/#comment56604>
why is this changed?
http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/java/org/apache/oozie/coord/TestCoordELFunctions.java
<https://reviews.apache.org/r/14239/#comment56608>
formatting.
This should throw an exception and not return ""
http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/java/org/apache/oozie/coord/TestCoordELFunctions.java
<https://reviews.apache.org/r/14239/#comment56609>
Too many new lines. Group expr and corresponding assertEquals without new lines in between them
http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/java/org/apache/oozie/coord/TestCoordELFunctions.java
<https://reviews.apache.org/r/14239/#comment56607>
Remove try catch block
http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/resources/cord-action-for-parameterization-2.xml
<https://reviews.apache.org/r/14239/#comment56623>
Combine this all into previous test case with multiple data-in
http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/resources/cord-action-for-parameterization-3.xml
<https://reviews.apache.org/r/14239/#comment56611>
Combine this all into one test case with multiple data-in
http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/resources/cord-action-for-parameterization.xml
<https://reviews.apache.org/r/14239/#comment56610>
Why is this required as a test?
http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/CoordinatorFunctionalSpec.twiki
<https://reviews.apache.org/r/14239/#comment56613>
Not materialization time. It is a dataset instance.
http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/CoordinatorFunctionalSpec.twiki
<https://reviews.apache.org/r/14239/#comment56614>
Explain the use case as to where/why coord:absolute can be used.
http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/CoordinatorFunctionalSpec.twiki
<https://reviews.apache.org/r/14239/#comment56616>
Make this daily as well for easy understanding
http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/CoordinatorFunctionalSpec.twiki
<https://reviews.apache.org/r/14239/#comment56617>
Make this current(0) or current(-1)
http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/CoordinatorFunctionalSpec.twiki
<https://reviews.apache.org/r/14239/#comment56615>
This is wrong. Should include the missing as well. Also need to tell what the values will be for a given nominal time.
- Rohini Palaniswamy
On Nov. 23, 2013, 12:24 a.m., Purshotam Shah wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/14239/
> -----------------------------------------------------------
>
> (Updated Nov. 23, 2013, 12:24 a.m.)
>
>
> Review request for oozie.
>
>
> Bugs: OOZIE-1504
> https://issues.apache.org/jira/browse/OOZIE-1504
>
>
> Repository: oozie
>
>
> Description
> -------
>
> Use-case
> -----------------------------
> A workflow that processes input datasets for the past 30 days:
> <input-events>
> <data-in name="event_input_path_format1" dataset="EVENT_INPUT_FORMAT1">
> <start-instance>$
> {coord:current(-30)}
> </start-instance>
> <end-instance>$
> {coord:current(-1)}
> </end-instance>
> </data-in>
> <data-in name="event_input_path_format2" dataset="EVENT_INPUT_FORMAT2">
> <start-instance>$
> {coord:current(-30)}
> </start-instance>
> <end-instance>$
> {coord:current(-1)}
> </end-instance>
> </data-in>
> </input-events>
> Instead, one wants it to start on the same specific date so that each day, it
> processes one more data set than the last day (the datasets are daily). Something like the following is not supported
> <input-events>
> <data-in name="event_input_path_format1" dataset="EVENT_INPUT_FORMAT1">
> <start-instance>${coord:absolute(2013-03-15T00:00Z)}</start-instance>
> <end-instance>$
> {coord:current(-1)}
> </end-instance>
> </data-in>
> <data-in name="event_input_path_format2" dataset="EVENT_INPUT_FORMAT2">
> <start-instance>${coord:absolute(2013-03-15T00:00Z)}</start-instance>
> <end-instance>$
> {coord:current(-1)}
> </end-instance>
> </data-in>
> </input-events>
> ---------------------------------
>
>
> Fix support providing absolute value for instance range. One can have a combination of absolute date with coord:current().
> Other EL function are not supported with absolute date.
>
> coord:Current() can be parameterize.
>
> like
> <end-instance>${coord:latest(end_date)}</end-instance> // where end_date is one of the specified property.
>
>
> Diffs
> -----
>
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordCommandUtils.java 1544636
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/coord/CoordELFunctions.java 1544636
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/resources/oozie-default.xml 1544636
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/java/org/apache/oozie/command/coord/TestCoordCommandUtils.java 1544636
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/java/org/apache/oozie/coord/TestCoordELFunctions.java 1544636
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/resources/cord-action-for-parameterization-1.xml PRE-CREATION
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/resources/cord-action-for-parameterization-2.xml PRE-CREATION
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/resources/cord-action-for-parameterization-3.xml PRE-CREATION
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/resources/cord-action-for-parameterization.xml PRE-CREATION
> http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/CoordinatorFunctionalSpec.twiki 1544636
> http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/WebServicesAPI.twiki 1544636
>
> Diff: https://reviews.apache.org/r/14239/diff/
>
>
> Testing
> -------
>
> Added multiple test cases to verify combination of scenarios.
>
>
> Thanks,
>
> Purshotam Shah
>
>
Re: Review Request 14239: OOZIE-1504 parameterize coord EL functions.
Posted by Purshotam Shah <pu...@yahoo-inc.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14239/
-----------------------------------------------------------
(Updated Nov. 23, 2013, 12:24 a.m.)
Review request for oozie.
Changes
-------
Review comments.
Bugs: OOZIE-1504
https://issues.apache.org/jira/browse/OOZIE-1504
Repository: oozie
Description (updated)
-------
Use-case
-----------------------------
A workflow that processes input datasets for the past 30 days:
<input-events>
<data-in name="event_input_path_format1" dataset="EVENT_INPUT_FORMAT1">
<start-instance>$
{coord:current(-30)}
</start-instance>
<end-instance>$
{coord:current(-1)}
</end-instance>
</data-in>
<data-in name="event_input_path_format2" dataset="EVENT_INPUT_FORMAT2">
<start-instance>$
{coord:current(-30)}
</start-instance>
<end-instance>$
{coord:current(-1)}
</end-instance>
</data-in>
</input-events>
Instead, one wants it to start on the same specific date so that each day, it
processes one more data set than the last day (the datasets are daily). Something like the following is not supported
<input-events>
<data-in name="event_input_path_format1" dataset="EVENT_INPUT_FORMAT1">
<start-instance>${coord:absolute(2013-03-15T00:00Z)}</start-instance>
<end-instance>$
{coord:current(-1)}
</end-instance>
</data-in>
<data-in name="event_input_path_format2" dataset="EVENT_INPUT_FORMAT2">
<start-instance>${coord:absolute(2013-03-15T00:00Z)}</start-instance>
<end-instance>$
{coord:current(-1)}
</end-instance>
</data-in>
</input-events>
---------------------------------
Fix support providing absolute value for instance range. One can have a combination of absolute date with coord:current().
Other EL function are not supported with absolute date.
coord:Current() can be parameterize.
like
<end-instance>${coord:latest(end_date)}</end-instance> // where end_date is one of the specified property.
Diffs (updated)
-----
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/command/coord/CoordCommandUtils.java 1544636
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/coord/CoordELFunctions.java 1544636
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/resources/oozie-default.xml 1544636
http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/java/org/apache/oozie/command/coord/TestCoordCommandUtils.java 1544636
http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/java/org/apache/oozie/coord/TestCoordELFunctions.java 1544636
http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/resources/cord-action-for-parameterization-1.xml PRE-CREATION
http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/resources/cord-action-for-parameterization-2.xml PRE-CREATION
http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/resources/cord-action-for-parameterization-3.xml PRE-CREATION
http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/resources/cord-action-for-parameterization.xml PRE-CREATION
http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/CoordinatorFunctionalSpec.twiki 1544636
http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/WebServicesAPI.twiki 1544636
Diff: https://reviews.apache.org/r/14239/diff/
Testing
-------
Added multiple test cases to verify combination of scenarios.
Thanks,
Purshotam Shah