You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by Rohini Palaniswamy <ro...@gmail.com> on 2014/01/27 07:34:53 UTC
Review Request 17379: [PIG-3658] Use Tez ObjectRegistry to cache FRJoin map
and WeightedRangePartitioner map
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/17379/
-----------------------------------------------------------
Review request for pig, Alex Bain, Cheolsoo Park, Daniel Dai, and Mark Wagner.
Bugs: PIG-3658
https://issues.apache.org/jira/browse/PIG-3658
Repository: pig
Description
-------
Caching broadcast inputs with vertex scope.
Usage similar to hive.
http://svn.apache.org/viewvc/hive/branches/tez/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ObjectCache.java?view=markup
Diffs
-----
http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/partitioners/WeightedRangePartitioner.java 1561449
http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/ObjectCache.java PRE-CREATION
http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/POFRJoinTez.java 1561449
http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/POPartitionRearrangeTez.java 1561449
http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/PigProcessor.java 1561449
http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/SkewedPartitionerTez.java 1561449
http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/TezCompiler.java 1561449
http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/WeightedRangePartitionerTez.java 1561449
Diff: https://reviews.apache.org/r/17379/diff/
Testing
-------
Operators and Join e2e pass. TEZ-711 is a issue when running on scale.
Thanks,
Rohini Palaniswamy
Re: Review Request 17379: [PIG-3658] Use Tez ObjectRegistry to cache FRJoin
map and WeightedRangePartitioner map
Posted by Cheolsoo Park <pi...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/17379/#review32854
-----------------------------------------------------------
Ship it!
Please go ahead commit this. I verified that my fix for skewed join still works with this change. I will rebase my patch to yours.
- Cheolsoo Park
On Jan. 27, 2014, 6:40 a.m., Rohini Palaniswamy wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/17379/
> -----------------------------------------------------------
>
> (Updated Jan. 27, 2014, 6:40 a.m.)
>
>
> Review request for pig, Alex Bain, Cheolsoo Park, Daniel Dai, and Mark Wagner.
>
>
> Bugs: PIG-3658
> https://issues.apache.org/jira/browse/PIG-3658
>
>
> Repository: pig
>
>
> Description
> -------
>
> Caching broadcast inputs with vertex scope.
>
> Usage similar to hive.
> http://svn.apache.org/viewvc/hive/branches/tez/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ObjectCache.java?view=markup
>
>
> Diffs
> -----
>
> http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/partitioners/WeightedRangePartitioner.java 1561449
> http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/ObjectCache.java PRE-CREATION
> http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/POFRJoinTez.java 1561449
> http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/POPartitionRearrangeTez.java 1561449
> http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/PigProcessor.java 1561449
> http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/SkewedPartitionerTez.java 1561449
> http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/TezCompiler.java 1561449
> http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/WeightedRangePartitionerTez.java 1561449
>
> Diff: https://reviews.apache.org/r/17379/diff/
>
>
> Testing
> -------
>
> Operators and Join e2e pass. TEZ-711 is a issue when running on scale.
>
>
> Thanks,
>
> Rohini Palaniswamy
>
>
Re: Review Request 17379: [PIG-3658] Use Tez ObjectRegistry to cache FRJoin
map and WeightedRangePartitioner map
Posted by Rohini Palaniswamy <ro...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/17379/
-----------------------------------------------------------
(Updated Jan. 27, 2014, 6:40 a.m.)
Review request for pig, Alex Bain, Cheolsoo Park, Daniel Dai, and Mark Wagner.
Bugs: PIG-3658
https://issues.apache.org/jira/browse/PIG-3658
Repository: pig
Description
-------
Caching broadcast inputs with vertex scope.
Usage similar to hive.
http://svn.apache.org/viewvc/hive/branches/tez/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ObjectCache.java?view=markup
Diffs (updated)
-----
http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/partitioners/WeightedRangePartitioner.java 1561449
http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/ObjectCache.java PRE-CREATION
http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/POFRJoinTez.java 1561449
http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/POPartitionRearrangeTez.java 1561449
http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/PigProcessor.java 1561449
http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/SkewedPartitionerTez.java 1561449
http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/TezCompiler.java 1561449
http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/WeightedRangePartitionerTez.java 1561449
Diff: https://reviews.apache.org/r/17379/diff/
Testing
-------
Operators and Join e2e pass. TEZ-711 is a issue when running on scale.
Thanks,
Rohini Palaniswamy