You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@oozie.apache.org by Satish Mittal <sa...@apache.org> on 2014/06/13 19:40:34 UTC
Re: Review Request 19449: OOZIE-1741: Add new coord EL function to get input
partitions value string
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/19449/
-----------------------------------------------------------
(Updated June 13, 2014, 5:40 p.m.)
Review request for oozie, Ryota Egashira and Rohini Palaniswamy.
Changes
-------
Attaching revised patch with dataInPartitions() and type as hive-export.
Repository: oozie-git
Description
-------
Currently oozie provides coord EL functions to get partition filter string (where key-values are separated by AND) for various action types - java/pig/hive. However this doesn't work in other cases, e.g. hive action that performs export/import of hive partition. In that case, the partition value is expected to be in a different format: (col1="value1",col2="value2"...).
If we use any existing EL function like coord:dataInPartitionFilter, the value of partition looks like:
(minute='00' AND month='03' AND year='2014' AND hour='09' AND day='11')
In this case, Hive export fails with error:
FAILED: ParseException line 1:51 mismatched input 'AND' expecting ) near ''00'' in export statement
We need to add a new EL function that returns partition value in the above format.
Diffs (updated)
-----
core/src/main/java/org/apache/oozie/coord/HCatELFunctions.java e5f0146
core/src/main/resources/oozie-default.xml 63a91fa
core/src/test/java/org/apache/oozie/coord/TestHCatELFunctions.java f63b682
docs/src/site/twiki/CoordinatorFunctionalSpec.twiki 11018b0
sharelib/hcatalog/src/main/java/org/apache/oozie/util/HCatURI.java d797f9b
Diff: https://reviews.apache.org/r/19449/diff/
Testing
-------
- Manually verified the new EL function on my local setup.
- Added new test cases for the same.
Thanks,
Satish Mittal
Re: Review Request 19449: OOZIE-1741: Add new coord EL function to get input
partitions value string
Posted by Satish Mittal <sa...@apache.org>.
> On June 17, 2014, 6:37 a.m., shwethags wrote:
> > core/src/main/java/org/apache/oozie/coord/HCatELFunctions.java, line 303
> > <https://reviews.apache.org/r/19449/diff/5/?file=610674#file610674line303>
> >
> > There should be validation on type in the beginning of the function
Added validation for any other type than hive-export.
> On June 17, 2014, 6:37 a.m., shwethags wrote:
> > core/src/test/java/org/apache/oozie/coord/TestHCatELFunctions.java, line 292
> > <https://reviews.apache.org/r/19449/diff/5/?file=610676#file610676line292>
> >
> > why do you need assertEquals here?
-ve test case; was written similar to other tests in this class. Have removed assertEquals now.
> On June 17, 2014, 6:37 a.m., shwethags wrote:
> > core/src/test/java/org/apache/oozie/coord/TestHCatELFunctions.java, line 301
> > <https://reviews.apache.org/r/19449/diff/5/?file=610676#file610676line301>
> >
> > why do you need assertEquals here?
-ve test case; was written similar to other tests in this class. Have removed assertEquals now.
> On June 17, 2014, 6:37 a.m., shwethags wrote:
> > core/src/test/java/org/apache/oozie/coord/TestHCatELFunctions.java, line 305
> > <https://reviews.apache.org/r/19449/diff/5/?file=610676#file610676line305>
> >
> > Can you add another test with different type (not hive-export)
Added a -ve test scenario in testDataInPartitions() to execute EL with a different type than hive-export.
> On June 17, 2014, 6:37 a.m., shwethags wrote:
> > docs/src/site/twiki/CoordinatorFunctionalSpec.twiki, line 2918
> > <https://reviews.apache.org/r/19449/diff/5/?file=610677#file610677line2918>
> >
> > Uses dataInPartitions for output
Have renamed various property names in example from IN/OUT prefixes to EXPORT/IMPORT prefixes for better clarity.
- Satish
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/19449/#review45914
-----------------------------------------------------------
On June 17, 2014, 11:01 a.m., Satish Mittal wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/19449/
> -----------------------------------------------------------
>
> (Updated June 17, 2014, 11:01 a.m.)
>
>
> Review request for oozie, Ryota Egashira and Rohini Palaniswamy.
>
>
> Repository: oozie-git
>
>
> Description
> -------
>
> Currently oozie provides coord EL functions to get partition filter string (where key-values are separated by AND) for various action types - java/pig/hive. However this doesn't work in other cases, e.g. hive action that performs export/import of hive partition. In that case, the partition value is expected to be in a different format: (col1="value1",col2="value2"...).
>
> If we use any existing EL function like coord:dataInPartitionFilter, the value of partition looks like:
> (minute='00' AND month='03' AND year='2014' AND hour='09' AND day='11')
> In this case, Hive export fails with error:
> FAILED: ParseException line 1:51 mismatched input 'AND' expecting ) near ''00'' in export statement
>
> We need to add a new EL function that returns partition value in the above format.
>
>
> Diffs
> -----
>
> core/src/main/java/org/apache/oozie/coord/HCatELFunctions.java e5f0146
> core/src/main/resources/oozie-default.xml b944d3d
> core/src/test/java/org/apache/oozie/coord/TestHCatELFunctions.java f63b682
> docs/src/site/twiki/CoordinatorFunctionalSpec.twiki 961377e
> sharelib/hcatalog/src/main/java/org/apache/oozie/util/HCatURI.java d797f9b
>
> Diff: https://reviews.apache.org/r/19449/diff/
>
>
> Testing
> -------
>
> - Manually verified the new EL function on my local setup.
> - Added new test cases for the same.
>
>
> Thanks,
>
> Satish Mittal
>
>
Re: Review Request 19449: OOZIE-1741: Add new coord EL function to get input
partitions value string
Posted by sh...@inmobi.com.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/19449/#review45914
-----------------------------------------------------------
core/src/main/java/org/apache/oozie/coord/HCatELFunctions.java
<https://reviews.apache.org/r/19449/#comment81040>
Can you specify dataInName in the exceptions. When there are multiple data ins for a coord, its difficult to figure out for which data in the error is for
core/src/main/java/org/apache/oozie/coord/HCatELFunctions.java
<https://reviews.apache.org/r/19449/#comment81039>
There should be validation on type in the beginning of the function
core/src/test/java/org/apache/oozie/coord/TestHCatELFunctions.java
<https://reviews.apache.org/r/19449/#comment81041>
why do you need assertEquals here?
core/src/test/java/org/apache/oozie/coord/TestHCatELFunctions.java
<https://reviews.apache.org/r/19449/#comment81042>
why do you need assertEquals here?
core/src/test/java/org/apache/oozie/coord/TestHCatELFunctions.java
<https://reviews.apache.org/r/19449/#comment81043>
Can you add another test with different type (not hive-export)
docs/src/site/twiki/CoordinatorFunctionalSpec.twiki
<https://reviews.apache.org/r/19449/#comment81044>
Uses dataInPartitions for output
docs/src/site/twiki/CoordinatorFunctionalSpec.twiki
<https://reviews.apache.org/r/19449/#comment81046>
extra)
sharelib/hcatalog/src/main/java/org/apache/oozie/util/HCatURI.java
<https://reviews.apache.org/r/19449/#comment81045>
throw unhandled type
- shwethags
On June 16, 2014, 1:09 p.m., Satish Mittal wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/19449/
> -----------------------------------------------------------
>
> (Updated June 16, 2014, 1:09 p.m.)
>
>
> Review request for oozie, Ryota Egashira and Rohini Palaniswamy.
>
>
> Repository: oozie-git
>
>
> Description
> -------
>
> Currently oozie provides coord EL functions to get partition filter string (where key-values are separated by AND) for various action types - java/pig/hive. However this doesn't work in other cases, e.g. hive action that performs export/import of hive partition. In that case, the partition value is expected to be in a different format: (col1="value1",col2="value2"...).
>
> If we use any existing EL function like coord:dataInPartitionFilter, the value of partition looks like:
> (minute='00' AND month='03' AND year='2014' AND hour='09' AND day='11')
> In this case, Hive export fails with error:
> FAILED: ParseException line 1:51 mismatched input 'AND' expecting ) near ''00'' in export statement
>
> We need to add a new EL function that returns partition value in the above format.
>
>
> Diffs
> -----
>
> core/src/main/java/org/apache/oozie/coord/HCatELFunctions.java e5f0146
> core/src/main/resources/oozie-default.xml c9d9591
> core/src/test/java/org/apache/oozie/coord/TestHCatELFunctions.java f63b682
> docs/src/site/twiki/CoordinatorFunctionalSpec.twiki 11018b0
> sharelib/hcatalog/src/main/java/org/apache/oozie/util/HCatURI.java d797f9b
>
> Diff: https://reviews.apache.org/r/19449/diff/
>
>
> Testing
> -------
>
> - Manually verified the new EL function on my local setup.
> - Added new test cases for the same.
>
>
> Thanks,
>
> Satish Mittal
>
>
Re: Review Request 19449: OOZIE-1741: Add new coord EL function to get input
partitions value string
Posted by sh...@inmobi.com.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/19449/#review46036
-----------------------------------------------------------
Ship it!
Ship It!
- shwethags
On June 17, 2014, 11:01 a.m., Satish Mittal wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/19449/
> -----------------------------------------------------------
>
> (Updated June 17, 2014, 11:01 a.m.)
>
>
> Review request for oozie, Ryota Egashira and Rohini Palaniswamy.
>
>
> Repository: oozie-git
>
>
> Description
> -------
>
> Currently oozie provides coord EL functions to get partition filter string (where key-values are separated by AND) for various action types - java/pig/hive. However this doesn't work in other cases, e.g. hive action that performs export/import of hive partition. In that case, the partition value is expected to be in a different format: (col1="value1",col2="value2"...).
>
> If we use any existing EL function like coord:dataInPartitionFilter, the value of partition looks like:
> (minute='00' AND month='03' AND year='2014' AND hour='09' AND day='11')
> In this case, Hive export fails with error:
> FAILED: ParseException line 1:51 mismatched input 'AND' expecting ) near ''00'' in export statement
>
> We need to add a new EL function that returns partition value in the above format.
>
>
> Diffs
> -----
>
> core/src/main/java/org/apache/oozie/coord/HCatELFunctions.java e5f0146
> core/src/main/resources/oozie-default.xml b944d3d
> core/src/test/java/org/apache/oozie/coord/TestHCatELFunctions.java f63b682
> docs/src/site/twiki/CoordinatorFunctionalSpec.twiki 961377e
> sharelib/hcatalog/src/main/java/org/apache/oozie/util/HCatURI.java d797f9b
>
> Diff: https://reviews.apache.org/r/19449/diff/
>
>
> Testing
> -------
>
> - Manually verified the new EL function on my local setup.
> - Added new test cases for the same.
>
>
> Thanks,
>
> Satish Mittal
>
>
Re: Review Request 19449: OOZIE-1741: Add new coord EL function to get input
partitions value string
Posted by Satish Mittal <sa...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/19449/
-----------------------------------------------------------
(Updated June 17, 2014, 11:01 a.m.)
Review request for oozie, Ryota Egashira and Rohini Palaniswamy.
Repository: oozie-git
Description
-------
Currently oozie provides coord EL functions to get partition filter string (where key-values are separated by AND) for various action types - java/pig/hive. However this doesn't work in other cases, e.g. hive action that performs export/import of hive partition. In that case, the partition value is expected to be in a different format: (col1="value1",col2="value2"...).
If we use any existing EL function like coord:dataInPartitionFilter, the value of partition looks like:
(minute='00' AND month='03' AND year='2014' AND hour='09' AND day='11')
In this case, Hive export fails with error:
FAILED: ParseException line 1:51 mismatched input 'AND' expecting ) near ''00'' in export statement
We need to add a new EL function that returns partition value in the above format.
Diffs (updated)
-----
core/src/main/java/org/apache/oozie/coord/HCatELFunctions.java e5f0146
core/src/main/resources/oozie-default.xml b944d3d
core/src/test/java/org/apache/oozie/coord/TestHCatELFunctions.java f63b682
docs/src/site/twiki/CoordinatorFunctionalSpec.twiki 961377e
sharelib/hcatalog/src/main/java/org/apache/oozie/util/HCatURI.java d797f9b
Diff: https://reviews.apache.org/r/19449/diff/
Testing
-------
- Manually verified the new EL function on my local setup.
- Added new test cases for the same.
Thanks,
Satish Mittal
Re: Review Request 19449: OOZIE-1741: Add new coord EL function to get input
partitions value string
Posted by Satish Mittal <sa...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/19449/
-----------------------------------------------------------
(Updated June 16, 2014, 1:09 p.m.)
Review request for oozie, Ryota Egashira and Rohini Palaniswamy.
Changes
-------
Updated patch after review comments.
Repository: oozie-git
Description
-------
Currently oozie provides coord EL functions to get partition filter string (where key-values are separated by AND) for various action types - java/pig/hive. However this doesn't work in other cases, e.g. hive action that performs export/import of hive partition. In that case, the partition value is expected to be in a different format: (col1="value1",col2="value2"...).
If we use any existing EL function like coord:dataInPartitionFilter, the value of partition looks like:
(minute='00' AND month='03' AND year='2014' AND hour='09' AND day='11')
In this case, Hive export fails with error:
FAILED: ParseException line 1:51 mismatched input 'AND' expecting ) near ''00'' in export statement
We need to add a new EL function that returns partition value in the above format.
Diffs (updated)
-----
core/src/main/java/org/apache/oozie/coord/HCatELFunctions.java e5f0146
core/src/main/resources/oozie-default.xml c9d9591
core/src/test/java/org/apache/oozie/coord/TestHCatELFunctions.java f63b682
docs/src/site/twiki/CoordinatorFunctionalSpec.twiki 11018b0
sharelib/hcatalog/src/main/java/org/apache/oozie/util/HCatURI.java d797f9b
Diff: https://reviews.apache.org/r/19449/diff/
Testing
-------
- Manually verified the new EL function on my local setup.
- Added new test cases for the same.
Thanks,
Satish Mittal
Re: Review Request 19449: OOZIE-1741: Add new coord EL function to get input
partitions value string
Posted by sh...@inmobi.com.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/19449/#review45732
-----------------------------------------------------------
core/src/main/java/org/apache/oozie/coord/HCatELFunctions.java
<https://reviews.apache.org/r/19449/#comment80701>
pass e as argument. Not in message (so that stactrace is available)
- shwethags
On June 13, 2014, 5:40 p.m., Satish Mittal wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/19449/
> -----------------------------------------------------------
>
> (Updated June 13, 2014, 5:40 p.m.)
>
>
> Review request for oozie, Ryota Egashira and Rohini Palaniswamy.
>
>
> Repository: oozie-git
>
>
> Description
> -------
>
> Currently oozie provides coord EL functions to get partition filter string (where key-values are separated by AND) for various action types - java/pig/hive. However this doesn't work in other cases, e.g. hive action that performs export/import of hive partition. In that case, the partition value is expected to be in a different format: (col1="value1",col2="value2"...).
>
> If we use any existing EL function like coord:dataInPartitionFilter, the value of partition looks like:
> (minute='00' AND month='03' AND year='2014' AND hour='09' AND day='11')
> In this case, Hive export fails with error:
> FAILED: ParseException line 1:51 mismatched input 'AND' expecting ) near ''00'' in export statement
>
> We need to add a new EL function that returns partition value in the above format.
>
>
> Diffs
> -----
>
> core/src/main/java/org/apache/oozie/coord/HCatELFunctions.java e5f0146
> core/src/main/resources/oozie-default.xml 63a91fa
> core/src/test/java/org/apache/oozie/coord/TestHCatELFunctions.java f63b682
> docs/src/site/twiki/CoordinatorFunctionalSpec.twiki 11018b0
> sharelib/hcatalog/src/main/java/org/apache/oozie/util/HCatURI.java d797f9b
>
> Diff: https://reviews.apache.org/r/19449/diff/
>
>
> Testing
> -------
>
> - Manually verified the new EL function on my local setup.
> - Added new test cases for the same.
>
>
> Thanks,
>
> Satish Mittal
>
>
Re: Review Request 19449: OOZIE-1741: Add new coord EL function to get input
partitions value string
Posted by Satish Mittal <sa...@apache.org>.
> On June 16, 2014, 2:02 a.m., Rohini Palaniswamy wrote:
> > core/src/main/java/org/apache/oozie/coord/HCatELFunctions.java, lines 293-298
> > <https://reviews.apache.org/r/19449/diff/4/?file=609416#file609416line293>
> >
> > Can you put this code in if (type equals hive-export) block?
Done.
> On June 16, 2014, 2:02 a.m., Rohini Palaniswamy wrote:
> > core/src/test/java/org/apache/oozie/coord/TestHCatELFunctions.java, line 301
> > <https://reviews.apache.org/r/19449/diff/4/?file=609418#file609418line301>
> >
> > Aren't there 2 parameters here? Why would it fail?
The expected string has 2 parameters, but the actual one (expr = "${coord:dataInPartitions('ABCD')}") has only one param. You have probably missed that.
> On June 16, 2014, 2:02 a.m., Rohini Palaniswamy wrote:
> > docs/src/site/twiki/CoordinatorFunctionalSpec.twiki, line 2845
> > <https://reviews.apache.org/r/19449/diff/4/?file=609419#file609419line2845>
> >
> > This needs to be changed to type
Done
> On June 16, 2014, 2:02 a.m., Rohini Palaniswamy wrote:
> > docs/src/site/twiki/CoordinatorFunctionalSpec.twiki, line 2848
> > <https://reviews.apache.org/r/19449/diff/4/?file=609419#file609419line2848>
> >
> > Can you say that currently the only type supported is hive-export and type hive-export can be used in Hive action to create ....
> >
> > Also mention that there can be only one partition instance
Done
> On June 16, 2014, 2:02 a.m., Rohini Palaniswamy wrote:
> > docs/src/site/twiki/CoordinatorFunctionalSpec.twiki, line 3007
> > <https://reviews.apache.org/r/19449/diff/4/?file=609419#file609419line3007>
> >
> > We should make the EL function return the value without the enclosing braces( ) as most of the others do not. i.e
> >
> > export table ${sourceDatabase}.${sourceTable} partition (${sourcePartition}) to '${sourceStagingDir}';
Done
- Satish
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/19449/#review45715
-----------------------------------------------------------
On June 16, 2014, 1:09 p.m., Satish Mittal wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/19449/
> -----------------------------------------------------------
>
> (Updated June 16, 2014, 1:09 p.m.)
>
>
> Review request for oozie, Ryota Egashira and Rohini Palaniswamy.
>
>
> Repository: oozie-git
>
>
> Description
> -------
>
> Currently oozie provides coord EL functions to get partition filter string (where key-values are separated by AND) for various action types - java/pig/hive. However this doesn't work in other cases, e.g. hive action that performs export/import of hive partition. In that case, the partition value is expected to be in a different format: (col1="value1",col2="value2"...).
>
> If we use any existing EL function like coord:dataInPartitionFilter, the value of partition looks like:
> (minute='00' AND month='03' AND year='2014' AND hour='09' AND day='11')
> In this case, Hive export fails with error:
> FAILED: ParseException line 1:51 mismatched input 'AND' expecting ) near ''00'' in export statement
>
> We need to add a new EL function that returns partition value in the above format.
>
>
> Diffs
> -----
>
> core/src/main/java/org/apache/oozie/coord/HCatELFunctions.java e5f0146
> core/src/main/resources/oozie-default.xml c9d9591
> core/src/test/java/org/apache/oozie/coord/TestHCatELFunctions.java f63b682
> docs/src/site/twiki/CoordinatorFunctionalSpec.twiki 11018b0
> sharelib/hcatalog/src/main/java/org/apache/oozie/util/HCatURI.java d797f9b
>
> Diff: https://reviews.apache.org/r/19449/diff/
>
>
> Testing
> -------
>
> - Manually verified the new EL function on my local setup.
> - Added new test cases for the same.
>
>
> Thanks,
>
> Satish Mittal
>
>
Re: Review Request 19449: OOZIE-1741: Add new coord EL function to get input
partitions value string
Posted by Rohini Palaniswamy <ro...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/19449/#review45715
-----------------------------------------------------------
core/src/main/java/org/apache/oozie/coord/HCatELFunctions.java
<https://reviews.apache.org/r/19449/#comment80686>
Can you put this code in if (type equals hive-export) block?
core/src/test/java/org/apache/oozie/coord/TestHCatELFunctions.java
<https://reviews.apache.org/r/19449/#comment80687>
Aren't there 2 parameters here? Why would it fail?
docs/src/site/twiki/CoordinatorFunctionalSpec.twiki
<https://reviews.apache.org/r/19449/#comment80688>
This needs to be changed to type
docs/src/site/twiki/CoordinatorFunctionalSpec.twiki
<https://reviews.apache.org/r/19449/#comment80689>
Can you say that currently the only type supported is hive-export and type hive-export can be used in Hive action to create ....
Also mention that there can be only one partition instance
docs/src/site/twiki/CoordinatorFunctionalSpec.twiki
<https://reviews.apache.org/r/19449/#comment80691>
We should make the EL function return the value without the enclosing braces( ) as most of the others do not. i.e
export table ${sourceDatabase}.${sourceTable} partition (${sourcePartition}) to '${sourceStagingDir}';
- Rohini Palaniswamy
On June 13, 2014, 5:40 p.m., Satish Mittal wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/19449/
> -----------------------------------------------------------
>
> (Updated June 13, 2014, 5:40 p.m.)
>
>
> Review request for oozie, Ryota Egashira and Rohini Palaniswamy.
>
>
> Repository: oozie-git
>
>
> Description
> -------
>
> Currently oozie provides coord EL functions to get partition filter string (where key-values are separated by AND) for various action types - java/pig/hive. However this doesn't work in other cases, e.g. hive action that performs export/import of hive partition. In that case, the partition value is expected to be in a different format: (col1="value1",col2="value2"...).
>
> If we use any existing EL function like coord:dataInPartitionFilter, the value of partition looks like:
> (minute='00' AND month='03' AND year='2014' AND hour='09' AND day='11')
> In this case, Hive export fails with error:
> FAILED: ParseException line 1:51 mismatched input 'AND' expecting ) near ''00'' in export statement
>
> We need to add a new EL function that returns partition value in the above format.
>
>
> Diffs
> -----
>
> core/src/main/java/org/apache/oozie/coord/HCatELFunctions.java e5f0146
> core/src/main/resources/oozie-default.xml 63a91fa
> core/src/test/java/org/apache/oozie/coord/TestHCatELFunctions.java f63b682
> docs/src/site/twiki/CoordinatorFunctionalSpec.twiki 11018b0
> sharelib/hcatalog/src/main/java/org/apache/oozie/util/HCatURI.java d797f9b
>
> Diff: https://reviews.apache.org/r/19449/diff/
>
>
> Testing
> -------
>
> - Manually verified the new EL function on my local setup.
> - Added new test cases for the same.
>
>
> Thanks,
>
> Satish Mittal
>
>