You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@datafu.apache.org by wang jian <ki...@gmail.com> on 2014/06/07 17:04:57 UTC
Re: Review Request 21306: Add UDFs to handle map type data.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/21306/
-----------------------------------------------------------
(Updated June 7, 2014, 3:04 p.m.)
Review request for DataFu.
Changes
-------
Update map UDF based on code review.
Problem: I have not tested the javadocs of these new classes. Would like to know how to generate javadocs using gradlew.
i have tried ./gradlew javadoc, but does not find the output of these generated documents.
Please provide some advice on it.
Thanks
Bugs: https://issues.apache.org/jira/browse/DATAFU-34
https://issues.apache.org/jira/browse/https://issues.apache.org/jira/browse/DATAFU-34
Repository: datafu
Description
-------
Add UDFs to handle map type data.
Unit tests done.
One thing left over: validate the byte array output from map
Diffs (updated)
-----
datafu-pig/src/main/java/datafu/pig/maps/AbstractHandleByKeys.java PRE-CREATION
datafu-pig/src/main/java/datafu/pig/maps/AbstractToMap.java PRE-CREATION
datafu-pig/src/main/java/datafu/pig/maps/BagToMap.java PRE-CREATION
datafu-pig/src/main/java/datafu/pig/maps/ContainsAllKeys.java PRE-CREATION
datafu-pig/src/main/java/datafu/pig/maps/ContainsAnyKey.java PRE-CREATION
datafu-pig/src/main/java/datafu/pig/maps/ContainsKeys.java PRE-CREATION
datafu-pig/src/main/java/datafu/pig/maps/GetByKeys.java PRE-CREATION
datafu-pig/src/main/java/datafu/pig/maps/MapPutHandler.java PRE-CREATION
datafu-pig/src/main/java/datafu/pig/maps/MapToBag.java PRE-CREATION
datafu-pig/src/main/java/datafu/pig/maps/PutMapToMap.java PRE-CREATION
datafu-pig/src/main/java/datafu/pig/maps/RemoveByKeys.java PRE-CREATION
datafu-pig/src/main/java/datafu/pig/maps/SchemaValidator.java PRE-CREATION
datafu-pig/src/main/java/datafu/pig/maps/TupleToMap.java PRE-CREATION
datafu-pig/src/test/java/datafu/test/pig/maps/CheckBag.java PRE-CREATION
datafu-pig/src/test/java/datafu/test/pig/maps/CheckMap.java PRE-CREATION
datafu-pig/src/test/java/datafu/test/pig/maps/ContainKeysTests.java PRE-CREATION
datafu-pig/src/test/java/datafu/test/pig/maps/HandleByKeysMapTests.java PRE-CREATION
datafu-pig/src/test/java/datafu/test/pig/maps/ToMapTests.java PRE-CREATION
Diff: https://reviews.apache.org/r/21306/diff/
Testing
-------
ues
Thanks,
wang jian
Re: Review Request 21306: Add UDFs to handle map type data.
Posted by Will Vaughan <wi...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/21306/#review50229
-----------------------------------------------------------
For most of these, if the value of the map is a tuple with a schema, that schema ends up getting lost instead of passed alone.
Should be able to support,e.g.
(B: bag {T: tuple(key:CHARARRAY, value:tuple(v1:int, v2:CHARARRAY)})
datafu-pig/src/main/java/datafu/pig/maps/BagToMap.java
<https://reviews.apache.org/r/21306/#comment87875>
need to handle schema
datafu-pig/src/main/java/datafu/pig/maps/MapToBag.java
<https://reviews.apache.org/r/21306/#comment87874>
if the mapValue has a schema, that schema should be preserved in the outputSchema
- Will Vaughan
On June 7, 2014, 3:16 p.m., wang jian wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/21306/
> -----------------------------------------------------------
>
> (Updated June 7, 2014, 3:16 p.m.)
>
>
> Review request for DataFu.
>
>
> Bugs: https://issues.apache.org/jira/browse/DATAFU-34
> https://issues.apache.org/jira/browse/https://issues.apache.org/jira/browse/DATAFU-34
>
>
> Repository: datafu
>
>
> Description
> -------
>
> Add UDFs to handle map type data.
>
> Unit tests done.
>
> Remaining: verify javadocs
>
>
> Diffs
> -----
>
> datafu-pig/src/main/java/datafu/pig/maps/AbstractHandleByKeys.java PRE-CREATION
> datafu-pig/src/main/java/datafu/pig/maps/AbstractToMap.java PRE-CREATION
> datafu-pig/src/main/java/datafu/pig/maps/BagToMap.java PRE-CREATION
> datafu-pig/src/main/java/datafu/pig/maps/ContainsAllKeys.java PRE-CREATION
> datafu-pig/src/main/java/datafu/pig/maps/ContainsAnyKey.java PRE-CREATION
> datafu-pig/src/main/java/datafu/pig/maps/ContainsKeys.java PRE-CREATION
> datafu-pig/src/main/java/datafu/pig/maps/GetByKeys.java PRE-CREATION
> datafu-pig/src/main/java/datafu/pig/maps/MapPutHandler.java PRE-CREATION
> datafu-pig/src/main/java/datafu/pig/maps/MapToBag.java PRE-CREATION
> datafu-pig/src/main/java/datafu/pig/maps/PutMapToMap.java PRE-CREATION
> datafu-pig/src/main/java/datafu/pig/maps/RemoveByKeys.java PRE-CREATION
> datafu-pig/src/main/java/datafu/pig/maps/SchemaValidator.java PRE-CREATION
> datafu-pig/src/main/java/datafu/pig/maps/TupleToMap.java PRE-CREATION
> datafu-pig/src/test/java/datafu/test/pig/maps/CheckBag.java PRE-CREATION
> datafu-pig/src/test/java/datafu/test/pig/maps/CheckMap.java PRE-CREATION
> datafu-pig/src/test/java/datafu/test/pig/maps/ContainKeysTests.java PRE-CREATION
> datafu-pig/src/test/java/datafu/test/pig/maps/HandleByKeysMapTests.java PRE-CREATION
> datafu-pig/src/test/java/datafu/test/pig/maps/ToMapTests.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/21306/diff/
>
>
> Testing
> -------
>
> ues
>
>
> Thanks,
>
> wang jian
>
>
Re: Review Request 21306: Add UDFs to handle map type data.
Posted by wang jian <ki...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/21306/
-----------------------------------------------------------
(Updated June 7, 2014, 3:16 p.m.)
Review request for DataFu.
Changes
-------
(1) ./gradlew javadocs does not output generated javadocs for the new UDFs. Need to know the location of these generated docs.
(2) Add 2 new UDFs: GetByKeys and RemoveByKeys to support variable number of keys to handle from the map.
(3) Only support PutMapToMap, to put a bag to a map, use BagToMap() to convert the bag to map first.
Bugs: https://issues.apache.org/jira/browse/DATAFU-34
https://issues.apache.org/jira/browse/https://issues.apache.org/jira/browse/DATAFU-34
Repository: datafu
Description (updated)
-------
Add UDFs to handle map type data.
Unit tests done.
Remaining: verify javadocs
Diffs
-----
datafu-pig/src/main/java/datafu/pig/maps/AbstractHandleByKeys.java PRE-CREATION
datafu-pig/src/main/java/datafu/pig/maps/AbstractToMap.java PRE-CREATION
datafu-pig/src/main/java/datafu/pig/maps/BagToMap.java PRE-CREATION
datafu-pig/src/main/java/datafu/pig/maps/ContainsAllKeys.java PRE-CREATION
datafu-pig/src/main/java/datafu/pig/maps/ContainsAnyKey.java PRE-CREATION
datafu-pig/src/main/java/datafu/pig/maps/ContainsKeys.java PRE-CREATION
datafu-pig/src/main/java/datafu/pig/maps/GetByKeys.java PRE-CREATION
datafu-pig/src/main/java/datafu/pig/maps/MapPutHandler.java PRE-CREATION
datafu-pig/src/main/java/datafu/pig/maps/MapToBag.java PRE-CREATION
datafu-pig/src/main/java/datafu/pig/maps/PutMapToMap.java PRE-CREATION
datafu-pig/src/main/java/datafu/pig/maps/RemoveByKeys.java PRE-CREATION
datafu-pig/src/main/java/datafu/pig/maps/SchemaValidator.java PRE-CREATION
datafu-pig/src/main/java/datafu/pig/maps/TupleToMap.java PRE-CREATION
datafu-pig/src/test/java/datafu/test/pig/maps/CheckBag.java PRE-CREATION
datafu-pig/src/test/java/datafu/test/pig/maps/CheckMap.java PRE-CREATION
datafu-pig/src/test/java/datafu/test/pig/maps/ContainKeysTests.java PRE-CREATION
datafu-pig/src/test/java/datafu/test/pig/maps/HandleByKeysMapTests.java PRE-CREATION
datafu-pig/src/test/java/datafu/test/pig/maps/ToMapTests.java PRE-CREATION
Diff: https://reviews.apache.org/r/21306/diff/
Testing
-------
ues
Thanks,
wang jian