You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@oozie.apache.org by Mona Chitnis <mo...@yahoo.in> on 2012/12/18 21:59:55 UTC

Review Request: OOZIE-1123 EL Functions for hcatalog

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8673/
-----------------------------------------------------------

Review request for oozie.


Description
-------

See https://issues.apache.org/jira/browse/OOZIE-1123
All HCatalog related EL functions separated out into another class. Changed nomenclature instead of having 'meta'
WIP - URIHandler to determine applicable EL Functions
WIP - partition-Min and partition-Max. Also need to account for input-event having range of instances e.g. current(0) to current(-n)


This addresses bug OOZIE-1123.
    https://issues.apache.org/jira/browse/OOZIE-1123


Diffs
-----

  branches/hcat-intre/core/src/main/java/org/apache/oozie/coord/CoordELFunctions.java 1423208 
  branches/hcat-intre/core/src/main/java/org/apache/oozie/coord/HCatELFunctions.java PRE-CREATION 
  branches/hcat-intre/core/src/main/resources/oozie-default.xml 1423208 
  branches/hcat-intre/core/src/test/java/org/apache/oozie/coord/TestCoordELFunctions.java 1423208 
  branches/hcat-intre/core/src/test/java/org/apache/oozie/coord/TestHCatELFunctions.java PRE-CREATION 
  branches/hcat-intre/pom.xml 1423208 
  branches/hcat-intre/sharelib/hive/pom.xml 1423208 

Diff: https://reviews.apache.org/r/8673/diff/


Testing
-------

unit tests added. end-to-end test will be done using Pig job


Thanks,

Mona Chitnis


Re: Review Request: OOZIE-1123 EL Functions for hcatalog

Posted by Mona Chitnis <mo...@yahoo.in>.

> On Dec. 20, 2012, 8:07 p.m., Virag Kothari wrote:
> > branches/hcat-intre/core/src/main/java/org/apache/oozie/util/HCatURI.java, line 249
> > <https://reviews.apache.org/r/8673/diff/4/?file=241485#file241485line249>
> >
> >     shouldn't the partition quotes be appended by the pig script (application)

The HCatStorer code was not taking the entire string e.g. 'partitionkey=value' unless it is enclosed in quotes, and was ending up delimiting at =. So this was added after end-to-end testing


> On Dec. 20, 2012, 8:07 p.m., Virag Kothari wrote:
> > branches/hcat-intre/core/src/main/java/org/apache/oozie/coord/HCatELFunctions.java, line 299
> > <https://reviews.apache.org/r/8673/diff/4/?file=241484#file241484line299>
> >
> >     I dont think  arrray length can ever be 0. Can you check it?

you are right. removing that block


> On Dec. 20, 2012, 8:07 p.m., Virag Kothari wrote:
> > branches/hcat-intre/core/src/main/java/org/apache/oozie/coord/HCatELFunctions.java, line 204
> > <https://reviews.apache.org/r/8673/diff/4/?file=241484#file241484line204>
> >
> >     instead of doing substring and split to get rid of the first element of uriList, its better to have a for loop which doesn't read the first element.
> >     Same for max partition el function

done


> On Dec. 20, 2012, 8:07 p.m., Virag Kothari wrote:
> > branches/hcat-intre/core/src/main/java/org/apache/oozie/coord/HCatELFunctions.java, line 202
> > <https://reviews.apache.org/r/8673/diff/4/?file=241484#file241484line202>
> >
> >     Can we have error handling when minPartition is null?..same comment for maxPartition el function

if it is found to be null, all further comparisons are also invalid and this qualifies for an exception. adding it.


> On Dec. 20, 2012, 8:07 p.m., Virag Kothari wrote:
> > branches/hcat-intre/core/src/main/java/org/apache/oozie/coord/HCatELFunctions.java, line 177
> > <https://reviews.apache.org/r/8673/diff/4/?file=241484#file241484line177>
> >
> >     As dataOut uri is a single uri and not an array of URI, why not call the tofilter() of HcatURI directly.
> >     The createPartitionFilter does string operations which doesn't seem required for dataOut.

done. thanks


- Mona


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8673/#review14769
-----------------------------------------------------------


On Dec. 20, 2012, 7 p.m., Mona Chitnis wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/8673/
> -----------------------------------------------------------
> 
> (Updated Dec. 20, 2012, 7 p.m.)
> 
> 
> Review request for oozie.
> 
> 
> Description
> -------
> 
> See https://issues.apache.org/jira/browse/OOZIE-1123
> All HCatalog related EL functions separated out into another class. Changed nomenclature instead of having 'meta'
> WIP - URIHandler to determine EL Functions to FS and HCat URIs and throw error otherwise
> 
> 
> This addresses bug OOZIE-1123.
>     https://issues.apache.org/jira/browse/OOZIE-1123
> 
> 
> Diffs
> -----
> 
>   branches/hcat-intre/client/src/main/java/org/apache/oozie/cli/OozieCLI.java 1424233 
>   branches/hcat-intre/core/src/main/java/org/apache/oozie/command/coord/CoordActionUpdatePushMissingDependency.java 1424233 
>   branches/hcat-intre/core/src/main/java/org/apache/oozie/command/coord/CoordSubmitXCommand.java 1424233 
>   branches/hcat-intre/core/src/main/java/org/apache/oozie/coord/CoordELFunctions.java 1424233 
>   branches/hcat-intre/core/src/main/java/org/apache/oozie/coord/HCatELFunctions.java PRE-CREATION 
>   branches/hcat-intre/core/src/main/java/org/apache/oozie/util/HCatURI.java 1424233 
>   branches/hcat-intre/core/src/main/java/org/apache/oozie/util/PartitionWrapper.java 1424233 
>   branches/hcat-intre/core/src/main/java/org/apache/oozie/util/XmlUtils.java 1424233 
>   branches/hcat-intre/core/src/main/resources/oozie-default.xml 1424233 
>   branches/hcat-intre/core/src/test/java/org/apache/oozie/coord/TestCoordELFunctions.java 1424233 
>   branches/hcat-intre/core/src/test/java/org/apache/oozie/coord/TestHCatELFunctions.java PRE-CREATION 
>   branches/hcat-intre/core/src/test/java/org/apache/oozie/util/TestHCatURI.java 1424233 
>   branches/hcat-intre/core/src/test/java/org/apache/oozie/util/TestXmlUtils.java 1424233 
>   branches/hcat-intre/examples/src/main/apps/hcatalog/README PRE-CREATION 
>   branches/hcat-intre/examples/src/main/apps/hcatalog/coordinator.xml PRE-CREATION 
>   branches/hcat-intre/examples/src/main/apps/hcatalog/id.pig PRE-CREATION 
>   branches/hcat-intre/examples/src/main/apps/hcatalog/job.properties PRE-CREATION 
>   branches/hcat-intre/examples/src/main/apps/hcatalog/workflow.xml PRE-CREATION 
>   branches/hcat-intre/pom.xml 1424233 
>   branches/hcat-intre/sharelib/hive/pom.xml 1424233 
> 
> Diff: https://reviews.apache.org/r/8673/diff/
> 
> 
> Testing
> -------
> 
> unit tests added. end-to-end test will be done using Pig job
> 
> 
> Thanks,
> 
> Mona Chitnis
> 
>


Re: Review Request: OOZIE-1123 EL Functions for hcatalog

Posted by Virag Kothari <vi...@yahoo-inc.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8673/#review14769
-----------------------------------------------------------


looks good to me..few comments..


branches/hcat-intre/core/src/main/java/org/apache/oozie/coord/HCatELFunctions.java
<https://reviews.apache.org/r/8673/#comment31292>

    As dataOut uri is a single uri and not an array of URI, why not call the tofilter() of HcatURI directly.
    The createPartitionFilter does string operations which doesn't seem required for dataOut.



branches/hcat-intre/core/src/main/java/org/apache/oozie/coord/HCatELFunctions.java
<https://reviews.apache.org/r/8673/#comment31294>

    Can we have error handling when minPartition is null?..same comment for maxPartition el function



branches/hcat-intre/core/src/main/java/org/apache/oozie/coord/HCatELFunctions.java
<https://reviews.apache.org/r/8673/#comment31312>

    instead of doing substring and split to get rid of the first element of uriList, its better to have a for loop which doesn't read the first element.
    Same for max partition el function



branches/hcat-intre/core/src/main/java/org/apache/oozie/coord/HCatELFunctions.java
<https://reviews.apache.org/r/8673/#comment31291>

    I dont think  arrray length can ever be 0. Can you check it?



branches/hcat-intre/core/src/main/java/org/apache/oozie/util/HCatURI.java
<https://reviews.apache.org/r/8673/#comment31313>

    shouldn't the partition quotes be appended by the pig script (application)



branches/hcat-intre/core/src/main/java/org/apache/oozie/util/PartitionWrapper.java
<https://reviews.apache.org/r/8673/#comment31314>

    1138 replaces all getServerEndpoint with getServer. Its better to keep this change in that patch



branches/hcat-intre/core/src/test/java/org/apache/oozie/coord/TestHCatELFunctions.java
<https://reviews.apache.org/r/8673/#comment31316>

    add @Test annotations for all test cases


- Virag Kothari


On Dec. 20, 2012, 7 p.m., Mona Chitnis wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/8673/
> -----------------------------------------------------------
> 
> (Updated Dec. 20, 2012, 7 p.m.)
> 
> 
> Review request for oozie.
> 
> 
> Description
> -------
> 
> See https://issues.apache.org/jira/browse/OOZIE-1123
> All HCatalog related EL functions separated out into another class. Changed nomenclature instead of having 'meta'
> WIP - URIHandler to determine EL Functions to FS and HCat URIs and throw error otherwise
> 
> 
> This addresses bug OOZIE-1123.
>     https://issues.apache.org/jira/browse/OOZIE-1123
> 
> 
> Diffs
> -----
> 
>   branches/hcat-intre/client/src/main/java/org/apache/oozie/cli/OozieCLI.java 1424233 
>   branches/hcat-intre/core/src/main/java/org/apache/oozie/command/coord/CoordActionUpdatePushMissingDependency.java 1424233 
>   branches/hcat-intre/core/src/main/java/org/apache/oozie/command/coord/CoordSubmitXCommand.java 1424233 
>   branches/hcat-intre/core/src/main/java/org/apache/oozie/coord/CoordELFunctions.java 1424233 
>   branches/hcat-intre/core/src/main/java/org/apache/oozie/coord/HCatELFunctions.java PRE-CREATION 
>   branches/hcat-intre/core/src/main/java/org/apache/oozie/util/HCatURI.java 1424233 
>   branches/hcat-intre/core/src/main/java/org/apache/oozie/util/PartitionWrapper.java 1424233 
>   branches/hcat-intre/core/src/main/java/org/apache/oozie/util/XmlUtils.java 1424233 
>   branches/hcat-intre/core/src/main/resources/oozie-default.xml 1424233 
>   branches/hcat-intre/core/src/test/java/org/apache/oozie/coord/TestCoordELFunctions.java 1424233 
>   branches/hcat-intre/core/src/test/java/org/apache/oozie/coord/TestHCatELFunctions.java PRE-CREATION 
>   branches/hcat-intre/core/src/test/java/org/apache/oozie/util/TestHCatURI.java 1424233 
>   branches/hcat-intre/core/src/test/java/org/apache/oozie/util/TestXmlUtils.java 1424233 
>   branches/hcat-intre/examples/src/main/apps/hcatalog/README PRE-CREATION 
>   branches/hcat-intre/examples/src/main/apps/hcatalog/coordinator.xml PRE-CREATION 
>   branches/hcat-intre/examples/src/main/apps/hcatalog/id.pig PRE-CREATION 
>   branches/hcat-intre/examples/src/main/apps/hcatalog/job.properties PRE-CREATION 
>   branches/hcat-intre/examples/src/main/apps/hcatalog/workflow.xml PRE-CREATION 
>   branches/hcat-intre/pom.xml 1424233 
>   branches/hcat-intre/sharelib/hive/pom.xml 1424233 
> 
> Diff: https://reviews.apache.org/r/8673/diff/
> 
> 
> Testing
> -------
> 
> unit tests added. end-to-end test will be done using Pig job
> 
> 
> Thanks,
> 
> Mona Chitnis
> 
>


Re: Review Request: OOZIE-1123 EL Functions for hcatalog

Posted by Rohini Palaniswamy <ro...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8673/#review14799
-----------------------------------------------------------



branches/hcat-intre/client/src/main/java/org/apache/oozie/cli/OozieCLI.java
<https://reviews.apache.org/r/8673/#comment31418>

    If this is the first error, even if there are other errors you will tell that the schema is valid. Need to give the escaped xml to the validator



branches/hcat-intre/core/src/main/java/org/apache/oozie/command/coord/CoordSubmitXCommand.java
<https://reviews.apache.org/r/8673/#comment31421>

    Same as before. Pass the escaped xml for validation.



branches/hcat-intre/core/src/main/java/org/apache/oozie/coord/HCatELFunctions.java
<https://reviews.apache.org/r/8673/#comment31441>

    dataOutPartitions ? Plural would make it clearer to user.



branches/hcat-intre/core/src/main/java/org/apache/oozie/coord/HCatELFunctions.java
<https://reviews.apache.org/r/8673/#comment31430>

    Can we rename this to dataInPartitionPigFilter? Just realized we will need to add one for Hive later. One has == and other =



branches/hcat-intre/core/src/main/java/org/apache/oozie/coord/HCatELFunctions.java
<https://reviews.apache.org/r/8673/#comment31424>

    error or warn



branches/hcat-intre/core/src/main/java/org/apache/oozie/coord/HCatELFunctions.java
<https://reviews.apache.org/r/8673/#comment31432>

    Shouldn't this be a comma separated list? Use a different method in HCatURI for it as it is not a filter. 



branches/hcat-intre/core/src/main/java/org/apache/oozie/coord/HCatELFunctions.java
<https://reviews.apache.org/r/8673/#comment31435>

    We need something that does string compare if string and num compare if number (natural sorting). Else 11 will come before 2. Cases that could happen are if they are just doing date as the column without the timestamp or a monotonically increasing sequence number as the column.  We could first try sorting by number. If there is an exception we could switch to sorting by string. That seems to be the simplest thing we can do. natural sorting seems to be tricky. 
    
    



branches/hcat-intre/core/src/main/java/org/apache/oozie/coord/HCatELFunctions.java
<https://reviews.apache.org/r/8673/#comment31437>

    same comments as min



branches/hcat-intre/core/src/main/java/org/apache/oozie/coord/HCatELFunctions.java
<https://reviews.apache.org/r/8673/#comment31439>

    Either use enum for type (no string compare) or have two separate methods .



branches/hcat-intre/core/src/main/java/org/apache/oozie/util/HCatURI.java
<https://reviews.apache.org/r/8673/#comment31428>

    Use two different methods specifying the intent (toPigPartitionFilter() and toPartitionStringHCatStorer()). This is hacky. 


- Rohini Palaniswamy


On Dec. 21, 2012, 12:04 a.m., Mona Chitnis wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/8673/
> -----------------------------------------------------------
> 
> (Updated Dec. 21, 2012, 12:04 a.m.)
> 
> 
> Review request for oozie.
> 
> 
> Description
> -------
> 
> See https://issues.apache.org/jira/browse/OOZIE-1123
> All HCatalog related EL functions separated out into another class. Changed nomenclature instead of having 'meta'
> WIP - URIHandler to determine EL Functions to FS and HCat URIs and throw error otherwise
> 
> 
> This addresses bug OOZIE-1123.
>     https://issues.apache.org/jira/browse/OOZIE-1123
> 
> 
> Diffs
> -----
> 
>   branches/hcat-intre/client/src/main/java/org/apache/oozie/cli/OozieCLI.java 1424233 
>   branches/hcat-intre/core/src/main/java/org/apache/oozie/command/coord/CoordActionUpdatePushMissingDependency.java 1424233 
>   branches/hcat-intre/core/src/main/java/org/apache/oozie/command/coord/CoordSubmitXCommand.java 1424233 
>   branches/hcat-intre/core/src/main/java/org/apache/oozie/coord/CoordELFunctions.java 1424233 
>   branches/hcat-intre/core/src/main/java/org/apache/oozie/coord/HCatELFunctions.java PRE-CREATION 
>   branches/hcat-intre/core/src/main/java/org/apache/oozie/util/HCatURI.java 1424233 
>   branches/hcat-intre/core/src/main/java/org/apache/oozie/util/XmlUtils.java 1424233 
>   branches/hcat-intre/core/src/main/resources/oozie-default.xml 1424233 
>   branches/hcat-intre/core/src/test/java/org/apache/oozie/coord/TestCoordELFunctions.java 1424233 
>   branches/hcat-intre/core/src/test/java/org/apache/oozie/coord/TestHCatELFunctions.java PRE-CREATION 
>   branches/hcat-intre/core/src/test/java/org/apache/oozie/util/TestHCatURI.java 1424233 
>   branches/hcat-intre/core/src/test/java/org/apache/oozie/util/TestXmlUtils.java 1424233 
>   branches/hcat-intre/examples/src/main/apps/hcatalog/README PRE-CREATION 
>   branches/hcat-intre/examples/src/main/apps/hcatalog/coordinator.xml PRE-CREATION 
>   branches/hcat-intre/examples/src/main/apps/hcatalog/id.pig PRE-CREATION 
>   branches/hcat-intre/examples/src/main/apps/hcatalog/job.properties PRE-CREATION 
>   branches/hcat-intre/examples/src/main/apps/hcatalog/workflow.xml PRE-CREATION 
>   branches/hcat-intre/pom.xml 1424233 
>   branches/hcat-intre/sharelib/hive/pom.xml 1424233 
> 
> Diff: https://reviews.apache.org/r/8673/diff/
> 
> 
> Testing
> -------
> 
> unit tests added. end-to-end test will be done using Pig job
> 
> 
> Thanks,
> 
> Mona Chitnis
> 
>


Re: Review Request: OOZIE-1123 EL Functions for hcatalog

Posted by Virag Kothari <vi...@yahoo-inc.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8673/#review14849
-----------------------------------------------------------

Ship it!


Ship It!

- Virag Kothari


On Dec. 21, 2012, 11:25 p.m., Mona Chitnis wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/8673/
> -----------------------------------------------------------
> 
> (Updated Dec. 21, 2012, 11:25 p.m.)
> 
> 
> Review request for oozie.
> 
> 
> Description
> -------
> 
> See https://issues.apache.org/jira/browse/OOZIE-1123
> All HCatalog related EL functions separated out into another class. Changed nomenclature instead of having 'meta'
> WIP - URIHandler to determine EL Functions to FS and HCat URIs and throw error otherwise
> 
> 
> This addresses bug OOZIE-1123.
>     https://issues.apache.org/jira/browse/OOZIE-1123
> 
> 
> Diffs
> -----
> 
>   branches/hcat-intre/core/src/main/java/org/apache/oozie/command/coord/CoordActionUpdatePushMissingDependency.java 1425179 
>   branches/hcat-intre/core/src/main/java/org/apache/oozie/coord/CoordELFunctions.java 1425179 
>   branches/hcat-intre/core/src/main/java/org/apache/oozie/coord/HCatELFunctions.java PRE-CREATION 
>   branches/hcat-intre/core/src/main/java/org/apache/oozie/util/HCatURI.java 1425179 
>   branches/hcat-intre/core/src/main/resources/oozie-default.xml 1425179 
>   branches/hcat-intre/core/src/test/java/org/apache/oozie/coord/TestCoordELFunctions.java 1425179 
>   branches/hcat-intre/core/src/test/java/org/apache/oozie/coord/TestHCatELFunctions.java PRE-CREATION 
>   branches/hcat-intre/core/src/test/java/org/apache/oozie/util/TestHCatURI.java 1425179 
>   branches/hcat-intre/examples/src/main/apps/hcatalog/README PRE-CREATION 
>   branches/hcat-intre/examples/src/main/apps/hcatalog/coordinator.xml PRE-CREATION 
>   branches/hcat-intre/examples/src/main/apps/hcatalog/id.pig PRE-CREATION 
>   branches/hcat-intre/examples/src/main/apps/hcatalog/job.properties PRE-CREATION 
>   branches/hcat-intre/examples/src/main/apps/hcatalog/workflow.xml PRE-CREATION 
>   branches/hcat-intre/pom.xml 1425179 
>   branches/hcat-intre/sharelib/hive/pom.xml 1425179 
> 
> Diff: https://reviews.apache.org/r/8673/diff/
> 
> 
> Testing
> -------
> 
> unit tests added. end-to-end test will be done using Pig job
> 
> 
> Thanks,
> 
> Mona Chitnis
> 
>


Re: Review Request: OOZIE-1123 EL Functions for hcatalog

Posted by Rohini Palaniswamy <ro...@gmail.com>.

> On Dec. 21, 2012, 11:35 p.m., Rohini Palaniswamy wrote:
> > Xmlutils and TestXmlutils need to be removed from the patch before check in.

Sorry. I was seeing the diff with the previous version. They are actually not part of the patch.


- Rohini


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8673/#review14848
-----------------------------------------------------------


On Dec. 21, 2012, 11:25 p.m., Mona Chitnis wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/8673/
> -----------------------------------------------------------
> 
> (Updated Dec. 21, 2012, 11:25 p.m.)
> 
> 
> Review request for oozie.
> 
> 
> Description
> -------
> 
> See https://issues.apache.org/jira/browse/OOZIE-1123
> All HCatalog related EL functions separated out into another class. Changed nomenclature instead of having 'meta'
> WIP - URIHandler to determine EL Functions to FS and HCat URIs and throw error otherwise
> 
> 
> This addresses bug OOZIE-1123.
>     https://issues.apache.org/jira/browse/OOZIE-1123
> 
> 
> Diffs
> -----
> 
>   branches/hcat-intre/core/src/main/java/org/apache/oozie/command/coord/CoordActionUpdatePushMissingDependency.java 1425179 
>   branches/hcat-intre/core/src/main/java/org/apache/oozie/coord/CoordELFunctions.java 1425179 
>   branches/hcat-intre/core/src/main/java/org/apache/oozie/coord/HCatELFunctions.java PRE-CREATION 
>   branches/hcat-intre/core/src/main/java/org/apache/oozie/util/HCatURI.java 1425179 
>   branches/hcat-intre/core/src/main/resources/oozie-default.xml 1425179 
>   branches/hcat-intre/core/src/test/java/org/apache/oozie/coord/TestCoordELFunctions.java 1425179 
>   branches/hcat-intre/core/src/test/java/org/apache/oozie/coord/TestHCatELFunctions.java PRE-CREATION 
>   branches/hcat-intre/core/src/test/java/org/apache/oozie/util/TestHCatURI.java 1425179 
>   branches/hcat-intre/examples/src/main/apps/hcatalog/README PRE-CREATION 
>   branches/hcat-intre/examples/src/main/apps/hcatalog/coordinator.xml PRE-CREATION 
>   branches/hcat-intre/examples/src/main/apps/hcatalog/id.pig PRE-CREATION 
>   branches/hcat-intre/examples/src/main/apps/hcatalog/job.properties PRE-CREATION 
>   branches/hcat-intre/examples/src/main/apps/hcatalog/workflow.xml PRE-CREATION 
>   branches/hcat-intre/pom.xml 1425179 
>   branches/hcat-intre/sharelib/hive/pom.xml 1425179 
> 
> Diff: https://reviews.apache.org/r/8673/diff/
> 
> 
> Testing
> -------
> 
> unit tests added. end-to-end test will be done using Pig job
> 
> 
> Thanks,
> 
> Mona Chitnis
> 
>


Re: Review Request: OOZIE-1123 EL Functions for hcatalog

Posted by Rohini Palaniswamy <ro...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8673/#review14848
-----------------------------------------------------------

Ship it!


Xmlutils and TestXmlutils need to be removed from the patch before check in.

- Rohini Palaniswamy


On Dec. 21, 2012, 11:25 p.m., Mona Chitnis wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/8673/
> -----------------------------------------------------------
> 
> (Updated Dec. 21, 2012, 11:25 p.m.)
> 
> 
> Review request for oozie.
> 
> 
> Description
> -------
> 
> See https://issues.apache.org/jira/browse/OOZIE-1123
> All HCatalog related EL functions separated out into another class. Changed nomenclature instead of having 'meta'
> WIP - URIHandler to determine EL Functions to FS and HCat URIs and throw error otherwise
> 
> 
> This addresses bug OOZIE-1123.
>     https://issues.apache.org/jira/browse/OOZIE-1123
> 
> 
> Diffs
> -----
> 
>   branches/hcat-intre/core/src/main/java/org/apache/oozie/command/coord/CoordActionUpdatePushMissingDependency.java 1425179 
>   branches/hcat-intre/core/src/main/java/org/apache/oozie/coord/CoordELFunctions.java 1425179 
>   branches/hcat-intre/core/src/main/java/org/apache/oozie/coord/HCatELFunctions.java PRE-CREATION 
>   branches/hcat-intre/core/src/main/java/org/apache/oozie/util/HCatURI.java 1425179 
>   branches/hcat-intre/core/src/main/resources/oozie-default.xml 1425179 
>   branches/hcat-intre/core/src/test/java/org/apache/oozie/coord/TestCoordELFunctions.java 1425179 
>   branches/hcat-intre/core/src/test/java/org/apache/oozie/coord/TestHCatELFunctions.java PRE-CREATION 
>   branches/hcat-intre/core/src/test/java/org/apache/oozie/util/TestHCatURI.java 1425179 
>   branches/hcat-intre/examples/src/main/apps/hcatalog/README PRE-CREATION 
>   branches/hcat-intre/examples/src/main/apps/hcatalog/coordinator.xml PRE-CREATION 
>   branches/hcat-intre/examples/src/main/apps/hcatalog/id.pig PRE-CREATION 
>   branches/hcat-intre/examples/src/main/apps/hcatalog/job.properties PRE-CREATION 
>   branches/hcat-intre/examples/src/main/apps/hcatalog/workflow.xml PRE-CREATION 
>   branches/hcat-intre/pom.xml 1425179 
>   branches/hcat-intre/sharelib/hive/pom.xml 1425179 
> 
> Diff: https://reviews.apache.org/r/8673/diff/
> 
> 
> Testing
> -------
> 
> unit tests added. end-to-end test will be done using Pig job
> 
> 
> Thanks,
> 
> Mona Chitnis
> 
>


Re: Review Request: OOZIE-1123 EL Functions for hcatalog

Posted by Mona Chitnis <mo...@yahoo.in>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8673/
-----------------------------------------------------------

(Updated Dec. 21, 2012, 11:25 p.m.)


Review request for oozie.


Changes
-------

Addressed Rohini's comments


Description
-------

See https://issues.apache.org/jira/browse/OOZIE-1123
All HCatalog related EL functions separated out into another class. Changed nomenclature instead of having 'meta'
WIP - URIHandler to determine EL Functions to FS and HCat URIs and throw error otherwise


This addresses bug OOZIE-1123.
    https://issues.apache.org/jira/browse/OOZIE-1123


Diffs (updated)
-----

  branches/hcat-intre/core/src/main/java/org/apache/oozie/command/coord/CoordActionUpdatePushMissingDependency.java 1425179 
  branches/hcat-intre/core/src/main/java/org/apache/oozie/coord/CoordELFunctions.java 1425179 
  branches/hcat-intre/core/src/main/java/org/apache/oozie/coord/HCatELFunctions.java PRE-CREATION 
  branches/hcat-intre/core/src/main/java/org/apache/oozie/util/HCatURI.java 1425179 
  branches/hcat-intre/core/src/main/resources/oozie-default.xml 1425179 
  branches/hcat-intre/core/src/test/java/org/apache/oozie/coord/TestCoordELFunctions.java 1425179 
  branches/hcat-intre/core/src/test/java/org/apache/oozie/coord/TestHCatELFunctions.java PRE-CREATION 
  branches/hcat-intre/core/src/test/java/org/apache/oozie/util/TestHCatURI.java 1425179 
  branches/hcat-intre/examples/src/main/apps/hcatalog/README PRE-CREATION 
  branches/hcat-intre/examples/src/main/apps/hcatalog/coordinator.xml PRE-CREATION 
  branches/hcat-intre/examples/src/main/apps/hcatalog/id.pig PRE-CREATION 
  branches/hcat-intre/examples/src/main/apps/hcatalog/job.properties PRE-CREATION 
  branches/hcat-intre/examples/src/main/apps/hcatalog/workflow.xml PRE-CREATION 
  branches/hcat-intre/pom.xml 1425179 
  branches/hcat-intre/sharelib/hive/pom.xml 1425179 

Diff: https://reviews.apache.org/r/8673/diff/


Testing
-------

unit tests added. end-to-end test will be done using Pig job


Thanks,

Mona Chitnis


Re: Review Request: OOZIE-1123 EL Functions for hcatalog

Posted by Mona Chitnis <mo...@yahoo.in>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8673/
-----------------------------------------------------------

(Updated Dec. 21, 2012, 12:04 a.m.)


Review request for oozie.


Changes
-------

Forgot to remove older HCat testcases from TestCoordELFunctions


Description
-------

See https://issues.apache.org/jira/browse/OOZIE-1123
All HCatalog related EL functions separated out into another class. Changed nomenclature instead of having 'meta'
WIP - URIHandler to determine EL Functions to FS and HCat URIs and throw error otherwise


This addresses bug OOZIE-1123.
    https://issues.apache.org/jira/browse/OOZIE-1123


Diffs (updated)
-----

  branches/hcat-intre/client/src/main/java/org/apache/oozie/cli/OozieCLI.java 1424233 
  branches/hcat-intre/core/src/main/java/org/apache/oozie/command/coord/CoordActionUpdatePushMissingDependency.java 1424233 
  branches/hcat-intre/core/src/main/java/org/apache/oozie/command/coord/CoordSubmitXCommand.java 1424233 
  branches/hcat-intre/core/src/main/java/org/apache/oozie/coord/CoordELFunctions.java 1424233 
  branches/hcat-intre/core/src/main/java/org/apache/oozie/coord/HCatELFunctions.java PRE-CREATION 
  branches/hcat-intre/core/src/main/java/org/apache/oozie/util/HCatURI.java 1424233 
  branches/hcat-intre/core/src/main/java/org/apache/oozie/util/XmlUtils.java 1424233 
  branches/hcat-intre/core/src/main/resources/oozie-default.xml 1424233 
  branches/hcat-intre/core/src/test/java/org/apache/oozie/coord/TestCoordELFunctions.java 1424233 
  branches/hcat-intre/core/src/test/java/org/apache/oozie/coord/TestHCatELFunctions.java PRE-CREATION 
  branches/hcat-intre/core/src/test/java/org/apache/oozie/util/TestHCatURI.java 1424233 
  branches/hcat-intre/core/src/test/java/org/apache/oozie/util/TestXmlUtils.java 1424233 
  branches/hcat-intre/examples/src/main/apps/hcatalog/README PRE-CREATION 
  branches/hcat-intre/examples/src/main/apps/hcatalog/coordinator.xml PRE-CREATION 
  branches/hcat-intre/examples/src/main/apps/hcatalog/id.pig PRE-CREATION 
  branches/hcat-intre/examples/src/main/apps/hcatalog/job.properties PRE-CREATION 
  branches/hcat-intre/examples/src/main/apps/hcatalog/workflow.xml PRE-CREATION 
  branches/hcat-intre/pom.xml 1424233 
  branches/hcat-intre/sharelib/hive/pom.xml 1424233 

Diff: https://reviews.apache.org/r/8673/diff/


Testing
-------

unit tests added. end-to-end test will be done using Pig job


Thanks,

Mona Chitnis


Re: Review Request: OOZIE-1123 EL Functions for hcatalog

Posted by Mona Chitnis <mo...@yahoo.in>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8673/
-----------------------------------------------------------

(Updated Dec. 20, 2012, 11:09 p.m.)


Review request for oozie.


Changes
-------

Addressed Virag's review comments.
Simple utility in XmlUtils to escape '&' in HCat uri-template and used as stop-gap (to be enhanced via OOZIE-1141)
e-2-e Test can be run with example provided


Description
-------

See https://issues.apache.org/jira/browse/OOZIE-1123
All HCatalog related EL functions separated out into another class. Changed nomenclature instead of having 'meta'
WIP - URIHandler to determine EL Functions to FS and HCat URIs and throw error otherwise


This addresses bug OOZIE-1123.
    https://issues.apache.org/jira/browse/OOZIE-1123


Diffs (updated)
-----

  branches/hcat-intre/client/src/main/java/org/apache/oozie/cli/OozieCLI.java 1424233 
  branches/hcat-intre/core/src/main/java/org/apache/oozie/command/coord/CoordActionUpdatePushMissingDependency.java 1424233 
  branches/hcat-intre/core/src/main/java/org/apache/oozie/command/coord/CoordSubmitXCommand.java 1424233 
  branches/hcat-intre/core/src/main/java/org/apache/oozie/coord/CoordELFunctions.java 1424233 
  branches/hcat-intre/core/src/main/java/org/apache/oozie/coord/HCatELFunctions.java PRE-CREATION 
  branches/hcat-intre/core/src/main/java/org/apache/oozie/util/HCatURI.java 1424233 
  branches/hcat-intre/core/src/main/java/org/apache/oozie/util/XmlUtils.java 1424233 
  branches/hcat-intre/core/src/main/resources/oozie-default.xml 1424233 
  branches/hcat-intre/core/src/test/java/org/apache/oozie/coord/TestCoordELFunctions.java 1424233 
  branches/hcat-intre/core/src/test/java/org/apache/oozie/coord/TestHCatELFunctions.java PRE-CREATION 
  branches/hcat-intre/core/src/test/java/org/apache/oozie/util/TestHCatURI.java 1424233 
  branches/hcat-intre/core/src/test/java/org/apache/oozie/util/TestXmlUtils.java 1424233 
  branches/hcat-intre/examples/src/main/apps/hcatalog/README PRE-CREATION 
  branches/hcat-intre/examples/src/main/apps/hcatalog/coordinator.xml PRE-CREATION 
  branches/hcat-intre/examples/src/main/apps/hcatalog/id.pig PRE-CREATION 
  branches/hcat-intre/examples/src/main/apps/hcatalog/job.properties PRE-CREATION 
  branches/hcat-intre/examples/src/main/apps/hcatalog/workflow.xml PRE-CREATION 
  branches/hcat-intre/pom.xml 1424233 
  branches/hcat-intre/sharelib/hive/pom.xml 1424233 

Diff: https://reviews.apache.org/r/8673/diff/


Testing
-------

unit tests added. end-to-end test will be done using Pig job


Thanks,

Mona Chitnis


Re: Review Request: OOZIE-1123 EL Functions for hcatalog

Posted by Mona Chitnis <mo...@yahoo.in>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8673/
-----------------------------------------------------------

(Updated Dec. 20, 2012, 7 p.m.)


Review request for oozie.


Changes
-------

1. Moved '&' encoding logic to common place XmlUtils, added small test-case
2. Oozie CLI -validate command to account for it


Description
-------

See https://issues.apache.org/jira/browse/OOZIE-1123
All HCatalog related EL functions separated out into another class. Changed nomenclature instead of having 'meta'
WIP - URIHandler to determine EL Functions to FS and HCat URIs and throw error otherwise


This addresses bug OOZIE-1123.
    https://issues.apache.org/jira/browse/OOZIE-1123


Diffs (updated)
-----

  branches/hcat-intre/client/src/main/java/org/apache/oozie/cli/OozieCLI.java 1424233 
  branches/hcat-intre/core/src/main/java/org/apache/oozie/command/coord/CoordActionUpdatePushMissingDependency.java 1424233 
  branches/hcat-intre/core/src/main/java/org/apache/oozie/command/coord/CoordSubmitXCommand.java 1424233 
  branches/hcat-intre/core/src/main/java/org/apache/oozie/coord/CoordELFunctions.java 1424233 
  branches/hcat-intre/core/src/main/java/org/apache/oozie/coord/HCatELFunctions.java PRE-CREATION 
  branches/hcat-intre/core/src/main/java/org/apache/oozie/util/HCatURI.java 1424233 
  branches/hcat-intre/core/src/main/java/org/apache/oozie/util/PartitionWrapper.java 1424233 
  branches/hcat-intre/core/src/main/java/org/apache/oozie/util/XmlUtils.java 1424233 
  branches/hcat-intre/core/src/main/resources/oozie-default.xml 1424233 
  branches/hcat-intre/core/src/test/java/org/apache/oozie/coord/TestCoordELFunctions.java 1424233 
  branches/hcat-intre/core/src/test/java/org/apache/oozie/coord/TestHCatELFunctions.java PRE-CREATION 
  branches/hcat-intre/core/src/test/java/org/apache/oozie/util/TestHCatURI.java 1424233 
  branches/hcat-intre/core/src/test/java/org/apache/oozie/util/TestXmlUtils.java 1424233 
  branches/hcat-intre/examples/src/main/apps/hcatalog/README PRE-CREATION 
  branches/hcat-intre/examples/src/main/apps/hcatalog/coordinator.xml PRE-CREATION 
  branches/hcat-intre/examples/src/main/apps/hcatalog/id.pig PRE-CREATION 
  branches/hcat-intre/examples/src/main/apps/hcatalog/job.properties PRE-CREATION 
  branches/hcat-intre/examples/src/main/apps/hcatalog/workflow.xml PRE-CREATION 
  branches/hcat-intre/pom.xml 1424233 
  branches/hcat-intre/sharelib/hive/pom.xml 1424233 

Diff: https://reviews.apache.org/r/8673/diff/


Testing
-------

unit tests added. end-to-end test will be done using Pig job


Thanks,

Mona Chitnis


Re: Review Request: OOZIE-1123 EL Functions for hcatalog

Posted by Mona Chitnis <mo...@yahoo.in>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8673/
-----------------------------------------------------------

(Updated Dec. 20, 2012, 1:18 a.m.)


Review request for oozie.


Changes
-------

EL functions complete
Fix for encoding "ampersand" in the uri-template for HCat URI which fails coordinator XSD parsing otherwise
Added the examples/app/hcatalog for referencing my test example
Addressed minor comments by Rohini

Will work on incorporating a unit test for firing up this test in minicluster itself in another JIRA
Checking for applicable EL functions in URIHandler also scoped out


Description
-------

See https://issues.apache.org/jira/browse/OOZIE-1123
All HCatalog related EL functions separated out into another class. Changed nomenclature instead of having 'meta'
WIP - URIHandler to determine EL Functions to FS and HCat URIs and throw error otherwise


This addresses bug OOZIE-1123.
    https://issues.apache.org/jira/browse/OOZIE-1123


Diffs (updated)
-----

  branches/hcat-intre/core/src/main/java/org/apache/oozie/command/coord/CoordActionUpdatePushMissingDependency.java 1424233 
  branches/hcat-intre/core/src/main/java/org/apache/oozie/command/coord/CoordSubmitXCommand.java 1424233 
  branches/hcat-intre/core/src/main/java/org/apache/oozie/coord/CoordELFunctions.java 1424233 
  branches/hcat-intre/core/src/main/java/org/apache/oozie/coord/HCatELFunctions.java PRE-CREATION 
  branches/hcat-intre/core/src/main/java/org/apache/oozie/util/HCatURI.java 1424233 
  branches/hcat-intre/core/src/main/java/org/apache/oozie/util/PartitionWrapper.java 1424233 
  branches/hcat-intre/core/src/main/resources/oozie-default.xml 1424233 
  branches/hcat-intre/core/src/test/java/org/apache/oozie/coord/TestCoordELFunctions.java 1424233 
  branches/hcat-intre/core/src/test/java/org/apache/oozie/coord/TestHCatELFunctions.java PRE-CREATION 
  branches/hcat-intre/core/src/test/java/org/apache/oozie/util/TestHCatURI.java 1424233 
  branches/hcat-intre/examples/src/main/apps/hcatalog/README PRE-CREATION 
  branches/hcat-intre/examples/src/main/apps/hcatalog/coordinator.xml PRE-CREATION 
  branches/hcat-intre/examples/src/main/apps/hcatalog/id.pig PRE-CREATION 
  branches/hcat-intre/examples/src/main/apps/hcatalog/job.properties PRE-CREATION 
  branches/hcat-intre/examples/src/main/apps/hcatalog/workflow.xml PRE-CREATION 
  branches/hcat-intre/pom.xml 1424233 
  branches/hcat-intre/sharelib/hive/pom.xml 1424233 

Diff: https://reviews.apache.org/r/8673/diff/


Testing
-------

unit tests added. end-to-end test will be done using Pig job


Thanks,

Mona Chitnis


Re: Review Request: OOZIE-1123 EL Functions for hcatalog

Posted by Mona Chitnis <mo...@yahoo.in>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8673/
-----------------------------------------------------------

(Updated Dec. 19, 2012, 9:59 a.m.)


Review request for oozie.


Changes
-------

made changes to EL functions according to how the pig script used with HCatalog actually accepts arguments


Description
-------

See https://issues.apache.org/jira/browse/OOZIE-1123
All HCatalog related EL functions separated out into another class. Changed nomenclature instead of having 'meta'
WIP - URIHandler to determine EL Functions to FS and HCat URIs and throw error otherwise


This addresses bug OOZIE-1123.
    https://issues.apache.org/jira/browse/OOZIE-1123


Diffs (updated)
-----

  branches/hcat-intre/core/src/main/java/org/apache/oozie/command/coord/CoordActionUpdatePushMissingDependency.java 1423676 
  branches/hcat-intre/core/src/main/java/org/apache/oozie/coord/CoordELFunctions.java 1423676 
  branches/hcat-intre/core/src/main/java/org/apache/oozie/coord/HCatELFunctions.java PRE-CREATION 
  branches/hcat-intre/core/src/main/java/org/apache/oozie/util/HCatURI.java 1423676 
  branches/hcat-intre/core/src/main/java/org/apache/oozie/util/PartitionWrapper.java 1423676 
  branches/hcat-intre/core/src/main/resources/oozie-default.xml 1423676 
  branches/hcat-intre/core/src/test/java/org/apache/oozie/coord/TestCoordELFunctions.java 1423676 
  branches/hcat-intre/core/src/test/java/org/apache/oozie/coord/TestHCatELFunctions.java PRE-CREATION 
  branches/hcat-intre/pom.xml 1423676 
  branches/hcat-intre/sharelib/hive/pom.xml 1423676 

Diff: https://reviews.apache.org/r/8673/diff/


Testing
-------

unit tests added. end-to-end test will be done using Pig job


Thanks,

Mona Chitnis


Re: Review Request: OOZIE-1123 EL Functions for hcatalog

Posted by Mona Chitnis <mo...@yahoo.in>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8673/
-----------------------------------------------------------

(Updated Dec. 18, 2012, 11:17 p.m.)


Review request for oozie.


Changes
-------

dataInPartitionMin and dataInPartitionMax implemented


Description (updated)
-------

See https://issues.apache.org/jira/browse/OOZIE-1123
All HCatalog related EL functions separated out into another class. Changed nomenclature instead of having 'meta'
WIP - URIHandler to determine EL Functions to FS and HCat URIs and throw error otherwise


This addresses bug OOZIE-1123.
    https://issues.apache.org/jira/browse/OOZIE-1123


Diffs (updated)
-----

  branches/hcat-intre/core/src/main/java/org/apache/oozie/coord/CoordELFunctions.java 1423676 
  branches/hcat-intre/core/src/main/java/org/apache/oozie/coord/HCatELFunctions.java PRE-CREATION 
  branches/hcat-intre/core/src/main/resources/oozie-default.xml 1423676 
  branches/hcat-intre/core/src/test/java/org/apache/oozie/coord/TestCoordELFunctions.java 1423676 
  branches/hcat-intre/core/src/test/java/org/apache/oozie/coord/TestHCatELFunctions.java PRE-CREATION 
  branches/hcat-intre/pom.xml 1423676 
  branches/hcat-intre/sharelib/hive/pom.xml 1423676 

Diff: https://reviews.apache.org/r/8673/diff/


Testing
-------

unit tests added. end-to-end test will be done using Pig job


Thanks,

Mona Chitnis