You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@oozie.apache.org by Mona Chitnis <mo...@yahoo.in> on 2012/12/18 21:59:55 UTC
Review Request: OOZIE-1123 EL Functions for hcatalog
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8673/
-----------------------------------------------------------
Review request for oozie.
Description
-------
See https://issues.apache.org/jira/browse/OOZIE-1123
All HCatalog related EL functions separated out into another class. Changed nomenclature instead of having 'meta'
WIP - URIHandler to determine applicable EL Functions
WIP - partition-Min and partition-Max. Also need to account for input-event having range of instances e.g. current(0) to current(-n)
This addresses bug OOZIE-1123.
https://issues.apache.org/jira/browse/OOZIE-1123
Diffs
-----
branches/hcat-intre/core/src/main/java/org/apache/oozie/coord/CoordELFunctions.java 1423208
branches/hcat-intre/core/src/main/java/org/apache/oozie/coord/HCatELFunctions.java PRE-CREATION
branches/hcat-intre/core/src/main/resources/oozie-default.xml 1423208
branches/hcat-intre/core/src/test/java/org/apache/oozie/coord/TestCoordELFunctions.java 1423208
branches/hcat-intre/core/src/test/java/org/apache/oozie/coord/TestHCatELFunctions.java PRE-CREATION
branches/hcat-intre/pom.xml 1423208
branches/hcat-intre/sharelib/hive/pom.xml 1423208
Diff: https://reviews.apache.org/r/8673/diff/
Testing
-------
unit tests added. end-to-end test will be done using Pig job
Thanks,
Mona Chitnis
Re: Review Request: OOZIE-1123 EL Functions for hcatalog
Posted by Mona Chitnis <mo...@yahoo.in>.
> On Dec. 20, 2012, 8:07 p.m., Virag Kothari wrote:
> > branches/hcat-intre/core/src/main/java/org/apache/oozie/util/HCatURI.java, line 249
> > <https://reviews.apache.org/r/8673/diff/4/?file=241485#file241485line249>
> >
> > shouldn't the partition quotes be appended by the pig script (application)
The HCatStorer code was not taking the entire string e.g. 'partitionkey=value' unless it is enclosed in quotes, and was ending up delimiting at =. So this was added after end-to-end testing
> On Dec. 20, 2012, 8:07 p.m., Virag Kothari wrote:
> > branches/hcat-intre/core/src/main/java/org/apache/oozie/coord/HCatELFunctions.java, line 299
> > <https://reviews.apache.org/r/8673/diff/4/?file=241484#file241484line299>
> >
> > I dont think arrray length can ever be 0. Can you check it?
you are right. removing that block
> On Dec. 20, 2012, 8:07 p.m., Virag Kothari wrote:
> > branches/hcat-intre/core/src/main/java/org/apache/oozie/coord/HCatELFunctions.java, line 204
> > <https://reviews.apache.org/r/8673/diff/4/?file=241484#file241484line204>
> >
> > instead of doing substring and split to get rid of the first element of uriList, its better to have a for loop which doesn't read the first element.
> > Same for max partition el function
done
> On Dec. 20, 2012, 8:07 p.m., Virag Kothari wrote:
> > branches/hcat-intre/core/src/main/java/org/apache/oozie/coord/HCatELFunctions.java, line 202
> > <https://reviews.apache.org/r/8673/diff/4/?file=241484#file241484line202>
> >
> > Can we have error handling when minPartition is null?..same comment for maxPartition el function
if it is found to be null, all further comparisons are also invalid and this qualifies for an exception. adding it.
> On Dec. 20, 2012, 8:07 p.m., Virag Kothari wrote:
> > branches/hcat-intre/core/src/main/java/org/apache/oozie/coord/HCatELFunctions.java, line 177
> > <https://reviews.apache.org/r/8673/diff/4/?file=241484#file241484line177>
> >
> > As dataOut uri is a single uri and not an array of URI, why not call the tofilter() of HcatURI directly.
> > The createPartitionFilter does string operations which doesn't seem required for dataOut.
done. thanks
- Mona
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8673/#review14769
-----------------------------------------------------------
On Dec. 20, 2012, 7 p.m., Mona Chitnis wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/8673/
> -----------------------------------------------------------
>
> (Updated Dec. 20, 2012, 7 p.m.)
>
>
> Review request for oozie.
>
>
> Description
> -------
>
> See https://issues.apache.org/jira/browse/OOZIE-1123
> All HCatalog related EL functions separated out into another class. Changed nomenclature instead of having 'meta'
> WIP - URIHandler to determine EL Functions to FS and HCat URIs and throw error otherwise
>
>
> This addresses bug OOZIE-1123.
> https://issues.apache.org/jira/browse/OOZIE-1123
>
>
> Diffs
> -----
>
> branches/hcat-intre/client/src/main/java/org/apache/oozie/cli/OozieCLI.java 1424233
> branches/hcat-intre/core/src/main/java/org/apache/oozie/command/coord/CoordActionUpdatePushMissingDependency.java 1424233
> branches/hcat-intre/core/src/main/java/org/apache/oozie/command/coord/CoordSubmitXCommand.java 1424233
> branches/hcat-intre/core/src/main/java/org/apache/oozie/coord/CoordELFunctions.java 1424233
> branches/hcat-intre/core/src/main/java/org/apache/oozie/coord/HCatELFunctions.java PRE-CREATION
> branches/hcat-intre/core/src/main/java/org/apache/oozie/util/HCatURI.java 1424233
> branches/hcat-intre/core/src/main/java/org/apache/oozie/util/PartitionWrapper.java 1424233
> branches/hcat-intre/core/src/main/java/org/apache/oozie/util/XmlUtils.java 1424233
> branches/hcat-intre/core/src/main/resources/oozie-default.xml 1424233
> branches/hcat-intre/core/src/test/java/org/apache/oozie/coord/TestCoordELFunctions.java 1424233
> branches/hcat-intre/core/src/test/java/org/apache/oozie/coord/TestHCatELFunctions.java PRE-CREATION
> branches/hcat-intre/core/src/test/java/org/apache/oozie/util/TestHCatURI.java 1424233
> branches/hcat-intre/core/src/test/java/org/apache/oozie/util/TestXmlUtils.java 1424233
> branches/hcat-intre/examples/src/main/apps/hcatalog/README PRE-CREATION
> branches/hcat-intre/examples/src/main/apps/hcatalog/coordinator.xml PRE-CREATION
> branches/hcat-intre/examples/src/main/apps/hcatalog/id.pig PRE-CREATION
> branches/hcat-intre/examples/src/main/apps/hcatalog/job.properties PRE-CREATION
> branches/hcat-intre/examples/src/main/apps/hcatalog/workflow.xml PRE-CREATION
> branches/hcat-intre/pom.xml 1424233
> branches/hcat-intre/sharelib/hive/pom.xml 1424233
>
> Diff: https://reviews.apache.org/r/8673/diff/
>
>
> Testing
> -------
>
> unit tests added. end-to-end test will be done using Pig job
>
>
> Thanks,
>
> Mona Chitnis
>
>
Re: Review Request: OOZIE-1123 EL Functions for hcatalog
Posted by Virag Kothari <vi...@yahoo-inc.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8673/#review14769
-----------------------------------------------------------
looks good to me..few comments..
branches/hcat-intre/core/src/main/java/org/apache/oozie/coord/HCatELFunctions.java
<https://reviews.apache.org/r/8673/#comment31292>
As dataOut uri is a single uri and not an array of URI, why not call the tofilter() of HcatURI directly.
The createPartitionFilter does string operations which doesn't seem required for dataOut.
branches/hcat-intre/core/src/main/java/org/apache/oozie/coord/HCatELFunctions.java
<https://reviews.apache.org/r/8673/#comment31294>
Can we have error handling when minPartition is null?..same comment for maxPartition el function
branches/hcat-intre/core/src/main/java/org/apache/oozie/coord/HCatELFunctions.java
<https://reviews.apache.org/r/8673/#comment31312>
instead of doing substring and split to get rid of the first element of uriList, its better to have a for loop which doesn't read the first element.
Same for max partition el function
branches/hcat-intre/core/src/main/java/org/apache/oozie/coord/HCatELFunctions.java
<https://reviews.apache.org/r/8673/#comment31291>
I dont think arrray length can ever be 0. Can you check it?
branches/hcat-intre/core/src/main/java/org/apache/oozie/util/HCatURI.java
<https://reviews.apache.org/r/8673/#comment31313>
shouldn't the partition quotes be appended by the pig script (application)
branches/hcat-intre/core/src/main/java/org/apache/oozie/util/PartitionWrapper.java
<https://reviews.apache.org/r/8673/#comment31314>
1138 replaces all getServerEndpoint with getServer. Its better to keep this change in that patch
branches/hcat-intre/core/src/test/java/org/apache/oozie/coord/TestHCatELFunctions.java
<https://reviews.apache.org/r/8673/#comment31316>
add @Test annotations for all test cases
- Virag Kothari
On Dec. 20, 2012, 7 p.m., Mona Chitnis wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/8673/
> -----------------------------------------------------------
>
> (Updated Dec. 20, 2012, 7 p.m.)
>
>
> Review request for oozie.
>
>
> Description
> -------
>
> See https://issues.apache.org/jira/browse/OOZIE-1123
> All HCatalog related EL functions separated out into another class. Changed nomenclature instead of having 'meta'
> WIP - URIHandler to determine EL Functions to FS and HCat URIs and throw error otherwise
>
>
> This addresses bug OOZIE-1123.
> https://issues.apache.org/jira/browse/OOZIE-1123
>
>
> Diffs
> -----
>
> branches/hcat-intre/client/src/main/java/org/apache/oozie/cli/OozieCLI.java 1424233
> branches/hcat-intre/core/src/main/java/org/apache/oozie/command/coord/CoordActionUpdatePushMissingDependency.java 1424233
> branches/hcat-intre/core/src/main/java/org/apache/oozie/command/coord/CoordSubmitXCommand.java 1424233
> branches/hcat-intre/core/src/main/java/org/apache/oozie/coord/CoordELFunctions.java 1424233
> branches/hcat-intre/core/src/main/java/org/apache/oozie/coord/HCatELFunctions.java PRE-CREATION
> branches/hcat-intre/core/src/main/java/org/apache/oozie/util/HCatURI.java 1424233
> branches/hcat-intre/core/src/main/java/org/apache/oozie/util/PartitionWrapper.java 1424233
> branches/hcat-intre/core/src/main/java/org/apache/oozie/util/XmlUtils.java 1424233
> branches/hcat-intre/core/src/main/resources/oozie-default.xml 1424233
> branches/hcat-intre/core/src/test/java/org/apache/oozie/coord/TestCoordELFunctions.java 1424233
> branches/hcat-intre/core/src/test/java/org/apache/oozie/coord/TestHCatELFunctions.java PRE-CREATION
> branches/hcat-intre/core/src/test/java/org/apache/oozie/util/TestHCatURI.java 1424233
> branches/hcat-intre/core/src/test/java/org/apache/oozie/util/TestXmlUtils.java 1424233
> branches/hcat-intre/examples/src/main/apps/hcatalog/README PRE-CREATION
> branches/hcat-intre/examples/src/main/apps/hcatalog/coordinator.xml PRE-CREATION
> branches/hcat-intre/examples/src/main/apps/hcatalog/id.pig PRE-CREATION
> branches/hcat-intre/examples/src/main/apps/hcatalog/job.properties PRE-CREATION
> branches/hcat-intre/examples/src/main/apps/hcatalog/workflow.xml PRE-CREATION
> branches/hcat-intre/pom.xml 1424233
> branches/hcat-intre/sharelib/hive/pom.xml 1424233
>
> Diff: https://reviews.apache.org/r/8673/diff/
>
>
> Testing
> -------
>
> unit tests added. end-to-end test will be done using Pig job
>
>
> Thanks,
>
> Mona Chitnis
>
>
Re: Review Request: OOZIE-1123 EL Functions for hcatalog
Posted by Rohini Palaniswamy <ro...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8673/#review14799
-----------------------------------------------------------
branches/hcat-intre/client/src/main/java/org/apache/oozie/cli/OozieCLI.java
<https://reviews.apache.org/r/8673/#comment31418>
If this is the first error, even if there are other errors you will tell that the schema is valid. Need to give the escaped xml to the validator
branches/hcat-intre/core/src/main/java/org/apache/oozie/command/coord/CoordSubmitXCommand.java
<https://reviews.apache.org/r/8673/#comment31421>
Same as before. Pass the escaped xml for validation.
branches/hcat-intre/core/src/main/java/org/apache/oozie/coord/HCatELFunctions.java
<https://reviews.apache.org/r/8673/#comment31441>
dataOutPartitions ? Plural would make it clearer to user.
branches/hcat-intre/core/src/main/java/org/apache/oozie/coord/HCatELFunctions.java
<https://reviews.apache.org/r/8673/#comment31430>
Can we rename this to dataInPartitionPigFilter? Just realized we will need to add one for Hive later. One has == and other =
branches/hcat-intre/core/src/main/java/org/apache/oozie/coord/HCatELFunctions.java
<https://reviews.apache.org/r/8673/#comment31424>
error or warn
branches/hcat-intre/core/src/main/java/org/apache/oozie/coord/HCatELFunctions.java
<https://reviews.apache.org/r/8673/#comment31432>
Shouldn't this be a comma separated list? Use a different method in HCatURI for it as it is not a filter.
branches/hcat-intre/core/src/main/java/org/apache/oozie/coord/HCatELFunctions.java
<https://reviews.apache.org/r/8673/#comment31435>
We need something that does string compare if string and num compare if number (natural sorting). Else 11 will come before 2. Cases that could happen are if they are just doing date as the column without the timestamp or a monotonically increasing sequence number as the column. We could first try sorting by number. If there is an exception we could switch to sorting by string. That seems to be the simplest thing we can do. natural sorting seems to be tricky.
branches/hcat-intre/core/src/main/java/org/apache/oozie/coord/HCatELFunctions.java
<https://reviews.apache.org/r/8673/#comment31437>
same comments as min
branches/hcat-intre/core/src/main/java/org/apache/oozie/coord/HCatELFunctions.java
<https://reviews.apache.org/r/8673/#comment31439>
Either use enum for type (no string compare) or have two separate methods .
branches/hcat-intre/core/src/main/java/org/apache/oozie/util/HCatURI.java
<https://reviews.apache.org/r/8673/#comment31428>
Use two different methods specifying the intent (toPigPartitionFilter() and toPartitionStringHCatStorer()). This is hacky.
- Rohini Palaniswamy
On Dec. 21, 2012, 12:04 a.m., Mona Chitnis wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/8673/
> -----------------------------------------------------------
>
> (Updated Dec. 21, 2012, 12:04 a.m.)
>
>
> Review request for oozie.
>
>
> Description
> -------
>
> See https://issues.apache.org/jira/browse/OOZIE-1123
> All HCatalog related EL functions separated out into another class. Changed nomenclature instead of having 'meta'
> WIP - URIHandler to determine EL Functions to FS and HCat URIs and throw error otherwise
>
>
> This addresses bug OOZIE-1123.
> https://issues.apache.org/jira/browse/OOZIE-1123
>
>
> Diffs
> -----
>
> branches/hcat-intre/client/src/main/java/org/apache/oozie/cli/OozieCLI.java 1424233
> branches/hcat-intre/core/src/main/java/org/apache/oozie/command/coord/CoordActionUpdatePushMissingDependency.java 1424233
> branches/hcat-intre/core/src/main/java/org/apache/oozie/command/coord/CoordSubmitXCommand.java 1424233
> branches/hcat-intre/core/src/main/java/org/apache/oozie/coord/CoordELFunctions.java 1424233
> branches/hcat-intre/core/src/main/java/org/apache/oozie/coord/HCatELFunctions.java PRE-CREATION
> branches/hcat-intre/core/src/main/java/org/apache/oozie/util/HCatURI.java 1424233
> branches/hcat-intre/core/src/main/java/org/apache/oozie/util/XmlUtils.java 1424233
> branches/hcat-intre/core/src/main/resources/oozie-default.xml 1424233
> branches/hcat-intre/core/src/test/java/org/apache/oozie/coord/TestCoordELFunctions.java 1424233
> branches/hcat-intre/core/src/test/java/org/apache/oozie/coord/TestHCatELFunctions.java PRE-CREATION
> branches/hcat-intre/core/src/test/java/org/apache/oozie/util/TestHCatURI.java 1424233
> branches/hcat-intre/core/src/test/java/org/apache/oozie/util/TestXmlUtils.java 1424233
> branches/hcat-intre/examples/src/main/apps/hcatalog/README PRE-CREATION
> branches/hcat-intre/examples/src/main/apps/hcatalog/coordinator.xml PRE-CREATION
> branches/hcat-intre/examples/src/main/apps/hcatalog/id.pig PRE-CREATION
> branches/hcat-intre/examples/src/main/apps/hcatalog/job.properties PRE-CREATION
> branches/hcat-intre/examples/src/main/apps/hcatalog/workflow.xml PRE-CREATION
> branches/hcat-intre/pom.xml 1424233
> branches/hcat-intre/sharelib/hive/pom.xml 1424233
>
> Diff: https://reviews.apache.org/r/8673/diff/
>
>
> Testing
> -------
>
> unit tests added. end-to-end test will be done using Pig job
>
>
> Thanks,
>
> Mona Chitnis
>
>
Re: Review Request: OOZIE-1123 EL Functions for hcatalog
Posted by Virag Kothari <vi...@yahoo-inc.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8673/#review14849
-----------------------------------------------------------
Ship it!
Ship It!
- Virag Kothari
On Dec. 21, 2012, 11:25 p.m., Mona Chitnis wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/8673/
> -----------------------------------------------------------
>
> (Updated Dec. 21, 2012, 11:25 p.m.)
>
>
> Review request for oozie.
>
>
> Description
> -------
>
> See https://issues.apache.org/jira/browse/OOZIE-1123
> All HCatalog related EL functions separated out into another class. Changed nomenclature instead of having 'meta'
> WIP - URIHandler to determine EL Functions to FS and HCat URIs and throw error otherwise
>
>
> This addresses bug OOZIE-1123.
> https://issues.apache.org/jira/browse/OOZIE-1123
>
>
> Diffs
> -----
>
> branches/hcat-intre/core/src/main/java/org/apache/oozie/command/coord/CoordActionUpdatePushMissingDependency.java 1425179
> branches/hcat-intre/core/src/main/java/org/apache/oozie/coord/CoordELFunctions.java 1425179
> branches/hcat-intre/core/src/main/java/org/apache/oozie/coord/HCatELFunctions.java PRE-CREATION
> branches/hcat-intre/core/src/main/java/org/apache/oozie/util/HCatURI.java 1425179
> branches/hcat-intre/core/src/main/resources/oozie-default.xml 1425179
> branches/hcat-intre/core/src/test/java/org/apache/oozie/coord/TestCoordELFunctions.java 1425179
> branches/hcat-intre/core/src/test/java/org/apache/oozie/coord/TestHCatELFunctions.java PRE-CREATION
> branches/hcat-intre/core/src/test/java/org/apache/oozie/util/TestHCatURI.java 1425179
> branches/hcat-intre/examples/src/main/apps/hcatalog/README PRE-CREATION
> branches/hcat-intre/examples/src/main/apps/hcatalog/coordinator.xml PRE-CREATION
> branches/hcat-intre/examples/src/main/apps/hcatalog/id.pig PRE-CREATION
> branches/hcat-intre/examples/src/main/apps/hcatalog/job.properties PRE-CREATION
> branches/hcat-intre/examples/src/main/apps/hcatalog/workflow.xml PRE-CREATION
> branches/hcat-intre/pom.xml 1425179
> branches/hcat-intre/sharelib/hive/pom.xml 1425179
>
> Diff: https://reviews.apache.org/r/8673/diff/
>
>
> Testing
> -------
>
> unit tests added. end-to-end test will be done using Pig job
>
>
> Thanks,
>
> Mona Chitnis
>
>
Re: Review Request: OOZIE-1123 EL Functions for hcatalog
Posted by Rohini Palaniswamy <ro...@gmail.com>.
> On Dec. 21, 2012, 11:35 p.m., Rohini Palaniswamy wrote:
> > Xmlutils and TestXmlutils need to be removed from the patch before check in.
Sorry. I was seeing the diff with the previous version. They are actually not part of the patch.
- Rohini
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8673/#review14848
-----------------------------------------------------------
On Dec. 21, 2012, 11:25 p.m., Mona Chitnis wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/8673/
> -----------------------------------------------------------
>
> (Updated Dec. 21, 2012, 11:25 p.m.)
>
>
> Review request for oozie.
>
>
> Description
> -------
>
> See https://issues.apache.org/jira/browse/OOZIE-1123
> All HCatalog related EL functions separated out into another class. Changed nomenclature instead of having 'meta'
> WIP - URIHandler to determine EL Functions to FS and HCat URIs and throw error otherwise
>
>
> This addresses bug OOZIE-1123.
> https://issues.apache.org/jira/browse/OOZIE-1123
>
>
> Diffs
> -----
>
> branches/hcat-intre/core/src/main/java/org/apache/oozie/command/coord/CoordActionUpdatePushMissingDependency.java 1425179
> branches/hcat-intre/core/src/main/java/org/apache/oozie/coord/CoordELFunctions.java 1425179
> branches/hcat-intre/core/src/main/java/org/apache/oozie/coord/HCatELFunctions.java PRE-CREATION
> branches/hcat-intre/core/src/main/java/org/apache/oozie/util/HCatURI.java 1425179
> branches/hcat-intre/core/src/main/resources/oozie-default.xml 1425179
> branches/hcat-intre/core/src/test/java/org/apache/oozie/coord/TestCoordELFunctions.java 1425179
> branches/hcat-intre/core/src/test/java/org/apache/oozie/coord/TestHCatELFunctions.java PRE-CREATION
> branches/hcat-intre/core/src/test/java/org/apache/oozie/util/TestHCatURI.java 1425179
> branches/hcat-intre/examples/src/main/apps/hcatalog/README PRE-CREATION
> branches/hcat-intre/examples/src/main/apps/hcatalog/coordinator.xml PRE-CREATION
> branches/hcat-intre/examples/src/main/apps/hcatalog/id.pig PRE-CREATION
> branches/hcat-intre/examples/src/main/apps/hcatalog/job.properties PRE-CREATION
> branches/hcat-intre/examples/src/main/apps/hcatalog/workflow.xml PRE-CREATION
> branches/hcat-intre/pom.xml 1425179
> branches/hcat-intre/sharelib/hive/pom.xml 1425179
>
> Diff: https://reviews.apache.org/r/8673/diff/
>
>
> Testing
> -------
>
> unit tests added. end-to-end test will be done using Pig job
>
>
> Thanks,
>
> Mona Chitnis
>
>
Re: Review Request: OOZIE-1123 EL Functions for hcatalog
Posted by Rohini Palaniswamy <ro...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8673/#review14848
-----------------------------------------------------------
Ship it!
Xmlutils and TestXmlutils need to be removed from the patch before check in.
- Rohini Palaniswamy
On Dec. 21, 2012, 11:25 p.m., Mona Chitnis wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/8673/
> -----------------------------------------------------------
>
> (Updated Dec. 21, 2012, 11:25 p.m.)
>
>
> Review request for oozie.
>
>
> Description
> -------
>
> See https://issues.apache.org/jira/browse/OOZIE-1123
> All HCatalog related EL functions separated out into another class. Changed nomenclature instead of having 'meta'
> WIP - URIHandler to determine EL Functions to FS and HCat URIs and throw error otherwise
>
>
> This addresses bug OOZIE-1123.
> https://issues.apache.org/jira/browse/OOZIE-1123
>
>
> Diffs
> -----
>
> branches/hcat-intre/core/src/main/java/org/apache/oozie/command/coord/CoordActionUpdatePushMissingDependency.java 1425179
> branches/hcat-intre/core/src/main/java/org/apache/oozie/coord/CoordELFunctions.java 1425179
> branches/hcat-intre/core/src/main/java/org/apache/oozie/coord/HCatELFunctions.java PRE-CREATION
> branches/hcat-intre/core/src/main/java/org/apache/oozie/util/HCatURI.java 1425179
> branches/hcat-intre/core/src/main/resources/oozie-default.xml 1425179
> branches/hcat-intre/core/src/test/java/org/apache/oozie/coord/TestCoordELFunctions.java 1425179
> branches/hcat-intre/core/src/test/java/org/apache/oozie/coord/TestHCatELFunctions.java PRE-CREATION
> branches/hcat-intre/core/src/test/java/org/apache/oozie/util/TestHCatURI.java 1425179
> branches/hcat-intre/examples/src/main/apps/hcatalog/README PRE-CREATION
> branches/hcat-intre/examples/src/main/apps/hcatalog/coordinator.xml PRE-CREATION
> branches/hcat-intre/examples/src/main/apps/hcatalog/id.pig PRE-CREATION
> branches/hcat-intre/examples/src/main/apps/hcatalog/job.properties PRE-CREATION
> branches/hcat-intre/examples/src/main/apps/hcatalog/workflow.xml PRE-CREATION
> branches/hcat-intre/pom.xml 1425179
> branches/hcat-intre/sharelib/hive/pom.xml 1425179
>
> Diff: https://reviews.apache.org/r/8673/diff/
>
>
> Testing
> -------
>
> unit tests added. end-to-end test will be done using Pig job
>
>
> Thanks,
>
> Mona Chitnis
>
>
Re: Review Request: OOZIE-1123 EL Functions for hcatalog
Posted by Mona Chitnis <mo...@yahoo.in>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8673/
-----------------------------------------------------------
(Updated Dec. 21, 2012, 11:25 p.m.)
Review request for oozie.
Changes
-------
Addressed Rohini's comments
Description
-------
See https://issues.apache.org/jira/browse/OOZIE-1123
All HCatalog related EL functions separated out into another class. Changed nomenclature instead of having 'meta'
WIP - URIHandler to determine EL Functions to FS and HCat URIs and throw error otherwise
This addresses bug OOZIE-1123.
https://issues.apache.org/jira/browse/OOZIE-1123
Diffs (updated)
-----
branches/hcat-intre/core/src/main/java/org/apache/oozie/command/coord/CoordActionUpdatePushMissingDependency.java 1425179
branches/hcat-intre/core/src/main/java/org/apache/oozie/coord/CoordELFunctions.java 1425179
branches/hcat-intre/core/src/main/java/org/apache/oozie/coord/HCatELFunctions.java PRE-CREATION
branches/hcat-intre/core/src/main/java/org/apache/oozie/util/HCatURI.java 1425179
branches/hcat-intre/core/src/main/resources/oozie-default.xml 1425179
branches/hcat-intre/core/src/test/java/org/apache/oozie/coord/TestCoordELFunctions.java 1425179
branches/hcat-intre/core/src/test/java/org/apache/oozie/coord/TestHCatELFunctions.java PRE-CREATION
branches/hcat-intre/core/src/test/java/org/apache/oozie/util/TestHCatURI.java 1425179
branches/hcat-intre/examples/src/main/apps/hcatalog/README PRE-CREATION
branches/hcat-intre/examples/src/main/apps/hcatalog/coordinator.xml PRE-CREATION
branches/hcat-intre/examples/src/main/apps/hcatalog/id.pig PRE-CREATION
branches/hcat-intre/examples/src/main/apps/hcatalog/job.properties PRE-CREATION
branches/hcat-intre/examples/src/main/apps/hcatalog/workflow.xml PRE-CREATION
branches/hcat-intre/pom.xml 1425179
branches/hcat-intre/sharelib/hive/pom.xml 1425179
Diff: https://reviews.apache.org/r/8673/diff/
Testing
-------
unit tests added. end-to-end test will be done using Pig job
Thanks,
Mona Chitnis
Re: Review Request: OOZIE-1123 EL Functions for hcatalog
Posted by Mona Chitnis <mo...@yahoo.in>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8673/
-----------------------------------------------------------
(Updated Dec. 21, 2012, 12:04 a.m.)
Review request for oozie.
Changes
-------
Forgot to remove older HCat testcases from TestCoordELFunctions
Description
-------
See https://issues.apache.org/jira/browse/OOZIE-1123
All HCatalog related EL functions separated out into another class. Changed nomenclature instead of having 'meta'
WIP - URIHandler to determine EL Functions to FS and HCat URIs and throw error otherwise
This addresses bug OOZIE-1123.
https://issues.apache.org/jira/browse/OOZIE-1123
Diffs (updated)
-----
branches/hcat-intre/client/src/main/java/org/apache/oozie/cli/OozieCLI.java 1424233
branches/hcat-intre/core/src/main/java/org/apache/oozie/command/coord/CoordActionUpdatePushMissingDependency.java 1424233
branches/hcat-intre/core/src/main/java/org/apache/oozie/command/coord/CoordSubmitXCommand.java 1424233
branches/hcat-intre/core/src/main/java/org/apache/oozie/coord/CoordELFunctions.java 1424233
branches/hcat-intre/core/src/main/java/org/apache/oozie/coord/HCatELFunctions.java PRE-CREATION
branches/hcat-intre/core/src/main/java/org/apache/oozie/util/HCatURI.java 1424233
branches/hcat-intre/core/src/main/java/org/apache/oozie/util/XmlUtils.java 1424233
branches/hcat-intre/core/src/main/resources/oozie-default.xml 1424233
branches/hcat-intre/core/src/test/java/org/apache/oozie/coord/TestCoordELFunctions.java 1424233
branches/hcat-intre/core/src/test/java/org/apache/oozie/coord/TestHCatELFunctions.java PRE-CREATION
branches/hcat-intre/core/src/test/java/org/apache/oozie/util/TestHCatURI.java 1424233
branches/hcat-intre/core/src/test/java/org/apache/oozie/util/TestXmlUtils.java 1424233
branches/hcat-intre/examples/src/main/apps/hcatalog/README PRE-CREATION
branches/hcat-intre/examples/src/main/apps/hcatalog/coordinator.xml PRE-CREATION
branches/hcat-intre/examples/src/main/apps/hcatalog/id.pig PRE-CREATION
branches/hcat-intre/examples/src/main/apps/hcatalog/job.properties PRE-CREATION
branches/hcat-intre/examples/src/main/apps/hcatalog/workflow.xml PRE-CREATION
branches/hcat-intre/pom.xml 1424233
branches/hcat-intre/sharelib/hive/pom.xml 1424233
Diff: https://reviews.apache.org/r/8673/diff/
Testing
-------
unit tests added. end-to-end test will be done using Pig job
Thanks,
Mona Chitnis
Re: Review Request: OOZIE-1123 EL Functions for hcatalog
Posted by Mona Chitnis <mo...@yahoo.in>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8673/
-----------------------------------------------------------
(Updated Dec. 20, 2012, 11:09 p.m.)
Review request for oozie.
Changes
-------
Addressed Virag's review comments.
Simple utility in XmlUtils to escape '&' in HCat uri-template and used as stop-gap (to be enhanced via OOZIE-1141)
e-2-e Test can be run with example provided
Description
-------
See https://issues.apache.org/jira/browse/OOZIE-1123
All HCatalog related EL functions separated out into another class. Changed nomenclature instead of having 'meta'
WIP - URIHandler to determine EL Functions to FS and HCat URIs and throw error otherwise
This addresses bug OOZIE-1123.
https://issues.apache.org/jira/browse/OOZIE-1123
Diffs (updated)
-----
branches/hcat-intre/client/src/main/java/org/apache/oozie/cli/OozieCLI.java 1424233
branches/hcat-intre/core/src/main/java/org/apache/oozie/command/coord/CoordActionUpdatePushMissingDependency.java 1424233
branches/hcat-intre/core/src/main/java/org/apache/oozie/command/coord/CoordSubmitXCommand.java 1424233
branches/hcat-intre/core/src/main/java/org/apache/oozie/coord/CoordELFunctions.java 1424233
branches/hcat-intre/core/src/main/java/org/apache/oozie/coord/HCatELFunctions.java PRE-CREATION
branches/hcat-intre/core/src/main/java/org/apache/oozie/util/HCatURI.java 1424233
branches/hcat-intre/core/src/main/java/org/apache/oozie/util/XmlUtils.java 1424233
branches/hcat-intre/core/src/main/resources/oozie-default.xml 1424233
branches/hcat-intre/core/src/test/java/org/apache/oozie/coord/TestCoordELFunctions.java 1424233
branches/hcat-intre/core/src/test/java/org/apache/oozie/coord/TestHCatELFunctions.java PRE-CREATION
branches/hcat-intre/core/src/test/java/org/apache/oozie/util/TestHCatURI.java 1424233
branches/hcat-intre/core/src/test/java/org/apache/oozie/util/TestXmlUtils.java 1424233
branches/hcat-intre/examples/src/main/apps/hcatalog/README PRE-CREATION
branches/hcat-intre/examples/src/main/apps/hcatalog/coordinator.xml PRE-CREATION
branches/hcat-intre/examples/src/main/apps/hcatalog/id.pig PRE-CREATION
branches/hcat-intre/examples/src/main/apps/hcatalog/job.properties PRE-CREATION
branches/hcat-intre/examples/src/main/apps/hcatalog/workflow.xml PRE-CREATION
branches/hcat-intre/pom.xml 1424233
branches/hcat-intre/sharelib/hive/pom.xml 1424233
Diff: https://reviews.apache.org/r/8673/diff/
Testing
-------
unit tests added. end-to-end test will be done using Pig job
Thanks,
Mona Chitnis
Re: Review Request: OOZIE-1123 EL Functions for hcatalog
Posted by Mona Chitnis <mo...@yahoo.in>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8673/
-----------------------------------------------------------
(Updated Dec. 20, 2012, 7 p.m.)
Review request for oozie.
Changes
-------
1. Moved '&' encoding logic to common place XmlUtils, added small test-case
2. Oozie CLI -validate command to account for it
Description
-------
See https://issues.apache.org/jira/browse/OOZIE-1123
All HCatalog related EL functions separated out into another class. Changed nomenclature instead of having 'meta'
WIP - URIHandler to determine EL Functions to FS and HCat URIs and throw error otherwise
This addresses bug OOZIE-1123.
https://issues.apache.org/jira/browse/OOZIE-1123
Diffs (updated)
-----
branches/hcat-intre/client/src/main/java/org/apache/oozie/cli/OozieCLI.java 1424233
branches/hcat-intre/core/src/main/java/org/apache/oozie/command/coord/CoordActionUpdatePushMissingDependency.java 1424233
branches/hcat-intre/core/src/main/java/org/apache/oozie/command/coord/CoordSubmitXCommand.java 1424233
branches/hcat-intre/core/src/main/java/org/apache/oozie/coord/CoordELFunctions.java 1424233
branches/hcat-intre/core/src/main/java/org/apache/oozie/coord/HCatELFunctions.java PRE-CREATION
branches/hcat-intre/core/src/main/java/org/apache/oozie/util/HCatURI.java 1424233
branches/hcat-intre/core/src/main/java/org/apache/oozie/util/PartitionWrapper.java 1424233
branches/hcat-intre/core/src/main/java/org/apache/oozie/util/XmlUtils.java 1424233
branches/hcat-intre/core/src/main/resources/oozie-default.xml 1424233
branches/hcat-intre/core/src/test/java/org/apache/oozie/coord/TestCoordELFunctions.java 1424233
branches/hcat-intre/core/src/test/java/org/apache/oozie/coord/TestHCatELFunctions.java PRE-CREATION
branches/hcat-intre/core/src/test/java/org/apache/oozie/util/TestHCatURI.java 1424233
branches/hcat-intre/core/src/test/java/org/apache/oozie/util/TestXmlUtils.java 1424233
branches/hcat-intre/examples/src/main/apps/hcatalog/README PRE-CREATION
branches/hcat-intre/examples/src/main/apps/hcatalog/coordinator.xml PRE-CREATION
branches/hcat-intre/examples/src/main/apps/hcatalog/id.pig PRE-CREATION
branches/hcat-intre/examples/src/main/apps/hcatalog/job.properties PRE-CREATION
branches/hcat-intre/examples/src/main/apps/hcatalog/workflow.xml PRE-CREATION
branches/hcat-intre/pom.xml 1424233
branches/hcat-intre/sharelib/hive/pom.xml 1424233
Diff: https://reviews.apache.org/r/8673/diff/
Testing
-------
unit tests added. end-to-end test will be done using Pig job
Thanks,
Mona Chitnis
Re: Review Request: OOZIE-1123 EL Functions for hcatalog
Posted by Mona Chitnis <mo...@yahoo.in>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8673/
-----------------------------------------------------------
(Updated Dec. 20, 2012, 1:18 a.m.)
Review request for oozie.
Changes
-------
EL functions complete
Fix for encoding "ampersand" in the uri-template for HCat URI which fails coordinator XSD parsing otherwise
Added the examples/app/hcatalog for referencing my test example
Addressed minor comments by Rohini
Will work on incorporating a unit test for firing up this test in minicluster itself in another JIRA
Checking for applicable EL functions in URIHandler also scoped out
Description
-------
See https://issues.apache.org/jira/browse/OOZIE-1123
All HCatalog related EL functions separated out into another class. Changed nomenclature instead of having 'meta'
WIP - URIHandler to determine EL Functions to FS and HCat URIs and throw error otherwise
This addresses bug OOZIE-1123.
https://issues.apache.org/jira/browse/OOZIE-1123
Diffs (updated)
-----
branches/hcat-intre/core/src/main/java/org/apache/oozie/command/coord/CoordActionUpdatePushMissingDependency.java 1424233
branches/hcat-intre/core/src/main/java/org/apache/oozie/command/coord/CoordSubmitXCommand.java 1424233
branches/hcat-intre/core/src/main/java/org/apache/oozie/coord/CoordELFunctions.java 1424233
branches/hcat-intre/core/src/main/java/org/apache/oozie/coord/HCatELFunctions.java PRE-CREATION
branches/hcat-intre/core/src/main/java/org/apache/oozie/util/HCatURI.java 1424233
branches/hcat-intre/core/src/main/java/org/apache/oozie/util/PartitionWrapper.java 1424233
branches/hcat-intre/core/src/main/resources/oozie-default.xml 1424233
branches/hcat-intre/core/src/test/java/org/apache/oozie/coord/TestCoordELFunctions.java 1424233
branches/hcat-intre/core/src/test/java/org/apache/oozie/coord/TestHCatELFunctions.java PRE-CREATION
branches/hcat-intre/core/src/test/java/org/apache/oozie/util/TestHCatURI.java 1424233
branches/hcat-intre/examples/src/main/apps/hcatalog/README PRE-CREATION
branches/hcat-intre/examples/src/main/apps/hcatalog/coordinator.xml PRE-CREATION
branches/hcat-intre/examples/src/main/apps/hcatalog/id.pig PRE-CREATION
branches/hcat-intre/examples/src/main/apps/hcatalog/job.properties PRE-CREATION
branches/hcat-intre/examples/src/main/apps/hcatalog/workflow.xml PRE-CREATION
branches/hcat-intre/pom.xml 1424233
branches/hcat-intre/sharelib/hive/pom.xml 1424233
Diff: https://reviews.apache.org/r/8673/diff/
Testing
-------
unit tests added. end-to-end test will be done using Pig job
Thanks,
Mona Chitnis
Re: Review Request: OOZIE-1123 EL Functions for hcatalog
Posted by Mona Chitnis <mo...@yahoo.in>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8673/
-----------------------------------------------------------
(Updated Dec. 19, 2012, 9:59 a.m.)
Review request for oozie.
Changes
-------
made changes to EL functions according to how the pig script used with HCatalog actually accepts arguments
Description
-------
See https://issues.apache.org/jira/browse/OOZIE-1123
All HCatalog related EL functions separated out into another class. Changed nomenclature instead of having 'meta'
WIP - URIHandler to determine EL Functions to FS and HCat URIs and throw error otherwise
This addresses bug OOZIE-1123.
https://issues.apache.org/jira/browse/OOZIE-1123
Diffs (updated)
-----
branches/hcat-intre/core/src/main/java/org/apache/oozie/command/coord/CoordActionUpdatePushMissingDependency.java 1423676
branches/hcat-intre/core/src/main/java/org/apache/oozie/coord/CoordELFunctions.java 1423676
branches/hcat-intre/core/src/main/java/org/apache/oozie/coord/HCatELFunctions.java PRE-CREATION
branches/hcat-intre/core/src/main/java/org/apache/oozie/util/HCatURI.java 1423676
branches/hcat-intre/core/src/main/java/org/apache/oozie/util/PartitionWrapper.java 1423676
branches/hcat-intre/core/src/main/resources/oozie-default.xml 1423676
branches/hcat-intre/core/src/test/java/org/apache/oozie/coord/TestCoordELFunctions.java 1423676
branches/hcat-intre/core/src/test/java/org/apache/oozie/coord/TestHCatELFunctions.java PRE-CREATION
branches/hcat-intre/pom.xml 1423676
branches/hcat-intre/sharelib/hive/pom.xml 1423676
Diff: https://reviews.apache.org/r/8673/diff/
Testing
-------
unit tests added. end-to-end test will be done using Pig job
Thanks,
Mona Chitnis
Re: Review Request: OOZIE-1123 EL Functions for hcatalog
Posted by Mona Chitnis <mo...@yahoo.in>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8673/
-----------------------------------------------------------
(Updated Dec. 18, 2012, 11:17 p.m.)
Review request for oozie.
Changes
-------
dataInPartitionMin and dataInPartitionMax implemented
Description (updated)
-------
See https://issues.apache.org/jira/browse/OOZIE-1123
All HCatalog related EL functions separated out into another class. Changed nomenclature instead of having 'meta'
WIP - URIHandler to determine EL Functions to FS and HCat URIs and throw error otherwise
This addresses bug OOZIE-1123.
https://issues.apache.org/jira/browse/OOZIE-1123
Diffs (updated)
-----
branches/hcat-intre/core/src/main/java/org/apache/oozie/coord/CoordELFunctions.java 1423676
branches/hcat-intre/core/src/main/java/org/apache/oozie/coord/HCatELFunctions.java PRE-CREATION
branches/hcat-intre/core/src/main/resources/oozie-default.xml 1423676
branches/hcat-intre/core/src/test/java/org/apache/oozie/coord/TestCoordELFunctions.java 1423676
branches/hcat-intre/core/src/test/java/org/apache/oozie/coord/TestHCatELFunctions.java PRE-CREATION
branches/hcat-intre/pom.xml 1423676
branches/hcat-intre/sharelib/hive/pom.xml 1423676
Diff: https://reviews.apache.org/r/8673/diff/
Testing
-------
unit tests added. end-to-end test will be done using Pig job
Thanks,
Mona Chitnis