You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by Cheolsoo Park <pi...@gmail.com> on 2014/01/28 03:47:35 UTC
Review Request 17439: PIG-3728: Fix TestSkewedJoin unit test in tez mode
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/17439/
-----------------------------------------------------------
Review request for pig, Alex Bain, Daniel Dai, Mark Wagner, and Rohini Palaniswamy.
Bugs: PIG-3728
https://issues.apache.org/jira/browse/PIG-3728
Repository: pig-git
Description
-------
The patch fixes TestSkewedJoin in tez mode. Note I changed SkewedPartitionerTez and WeightedRangePartitionerTez to allow null sampleMap to make testSkewedJoinEmptyInput pass. It used to throw a runtime exception, but now it assumes empty sample input.
I also fixed MiniCluster in hadoop20. They were missing the getExecType() method.
Diffs
-----
shims/test/hadoop20/org/apache/pig/test/MiniCluster.java 3e64a5d
shims/test/hadoop20/org/apache/pig/test/TezMiniCluster.java 98a580c
src/org/apache/pig/backend/hadoop/executionengine/tez/POPartitionRearrangeTez.java b17b38b
src/org/apache/pig/backend/hadoop/executionengine/tez/PigProcessor.java 2b6c899
src/org/apache/pig/backend/hadoop/executionengine/tez/SkewedPartitionerTez.java 5fe202b
src/org/apache/pig/backend/hadoop/executionengine/tez/WeightedRangePartitionerTez.java 36441fb
test/org/apache/pig/test/TestAccumulator.java 516d5c5
test/org/apache/pig/test/TestSkewedJoin.java be91d6b
test/tez-tests 0d32d23
Diff: https://reviews.apache.org/r/17439/diff/
Testing
-------
* TestSkewedJoin passes in both tez and mr mode.
* ant test-tez passes except TestCustomPartitioner (known).
* All e2e tests pass.
Thanks,
Cheolsoo Park
Re: Review Request 17439: PIG-3728: Fix TestSkewedJoin unit test in tez mode
Posted by Rohini Palaniswamy <ro...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/17439/#review32970
-----------------------------------------------------------
Ship it!
test/org/apache/pig/test/TestSkewedJoin.java
<https://reviews.apache.org/r/17439/#comment62058>
Can you just change the input files to be in build/test/input directory when you commit instead of current directory so that they get removed on ant clean. They are created in the base pig directory and one needs to manually clean them up now.
- Rohini Palaniswamy
On Jan. 28, 2014, 3:44 a.m., Cheolsoo Park wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/17439/
> -----------------------------------------------------------
>
> (Updated Jan. 28, 2014, 3:44 a.m.)
>
>
> Review request for pig, Alex Bain, Daniel Dai, Mark Wagner, and Rohini Palaniswamy.
>
>
> Bugs: PIG-3728
> https://issues.apache.org/jira/browse/PIG-3728
>
>
> Repository: pig-git
>
>
> Description
> -------
>
> The patch fixes TestSkewedJoin in tez mode. Note I changed SkewedPartitionerTez and WeightedRangePartitionerTez to allow null sampleMap to make testSkewedJoinEmptyInput pass. It used to throw a runtime exception, but now it assumes empty sample input.
>
> I also fixed MiniCluster in hadoop20. They were missing the getExecType() method.
>
>
> Diffs
> -----
>
> shims/test/hadoop20/org/apache/pig/test/MiniCluster.java 3e64a5d
> shims/test/hadoop20/org/apache/pig/test/TezMiniCluster.java 98a580c
> src/org/apache/pig/backend/hadoop/executionengine/tez/POPartitionRearrangeTez.java b17b38b
> src/org/apache/pig/backend/hadoop/executionengine/tez/PigProcessor.java 2b6c899
> src/org/apache/pig/backend/hadoop/executionengine/tez/SkewedPartitionerTez.java 5fe202b
> src/org/apache/pig/backend/hadoop/executionengine/tez/WeightedRangePartitionerTez.java 36441fb
> test/org/apache/pig/test/TestAccumulator.java 516d5c5
> test/org/apache/pig/test/TestSkewedJoin.java be91d6b
> test/tez-tests 0d32d23
>
> Diff: https://reviews.apache.org/r/17439/diff/
>
>
> Testing
> -------
>
> * TestSkewedJoin passes in both tez and mr mode.
> * ant test-tez passes except TestCustomPartitioner (known).
> * All e2e tests pass.
>
>
> Thanks,
>
> Cheolsoo Park
>
>
Re: Review Request 17439: PIG-3728: Fix TestSkewedJoin unit test in tez mode
Posted by Cheolsoo Park <pi...@gmail.com>.
> On Jan. 29, 2014, 12:39 p.m., Rohini Palaniswamy wrote:
> > src/org/apache/pig/backend/hadoop/executionengine/tez/WeightedRangePartitionerTez.java, lines 60-61
> > <https://reviews.apache.org/r/17439/diff/2/?file=452658#file452658line60>
> >
> > In what scenario is quantiles map empty? If we don't initialize weightedParts, quantiles, etc won't we encounter NPE later?
If you sort an empty file, quantiles map will be empty. We won't encounter NPE as long as we construct a non-null but empty map in WeightRangePartitioner.
- Cheolsoo
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/17439/#review33117
-----------------------------------------------------------
On Jan. 28, 2014, 3:44 a.m., Cheolsoo Park wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/17439/
> -----------------------------------------------------------
>
> (Updated Jan. 28, 2014, 3:44 a.m.)
>
>
> Review request for pig, Alex Bain, Daniel Dai, Mark Wagner, and Rohini Palaniswamy.
>
>
> Bugs: PIG-3728
> https://issues.apache.org/jira/browse/PIG-3728
>
>
> Repository: pig-git
>
>
> Description
> -------
>
> The patch fixes TestSkewedJoin in tez mode. Note I changed SkewedPartitionerTez and WeightedRangePartitionerTez to allow null sampleMap to make testSkewedJoinEmptyInput pass. It used to throw a runtime exception, but now it assumes empty sample input.
>
> I also fixed MiniCluster in hadoop20. They were missing the getExecType() method.
>
>
> Diffs
> -----
>
> shims/test/hadoop20/org/apache/pig/test/MiniCluster.java 3e64a5d
> shims/test/hadoop20/org/apache/pig/test/TezMiniCluster.java 98a580c
> src/org/apache/pig/backend/hadoop/executionengine/tez/POPartitionRearrangeTez.java b17b38b
> src/org/apache/pig/backend/hadoop/executionengine/tez/PigProcessor.java 2b6c899
> src/org/apache/pig/backend/hadoop/executionengine/tez/SkewedPartitionerTez.java 5fe202b
> src/org/apache/pig/backend/hadoop/executionengine/tez/WeightedRangePartitionerTez.java 36441fb
> test/org/apache/pig/test/TestAccumulator.java 516d5c5
> test/org/apache/pig/test/TestSkewedJoin.java be91d6b
> test/tez-tests 0d32d23
>
> Diff: https://reviews.apache.org/r/17439/diff/
>
>
> Testing
> -------
>
> * TestSkewedJoin passes in both tez and mr mode.
> * ant test-tez passes except TestCustomPartitioner (known).
> * All e2e tests pass.
>
>
> Thanks,
>
> Cheolsoo Park
>
>
Re: Review Request 17439: PIG-3728: Fix TestSkewedJoin unit test in tez mode
Posted by Rohini Palaniswamy <ro...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/17439/#review33117
-----------------------------------------------------------
src/org/apache/pig/backend/hadoop/executionengine/tez/WeightedRangePartitionerTez.java
<https://reviews.apache.org/r/17439/#comment62402>
In what scenario is quantiles map empty? If we don't initialize weightedParts, quantiles, etc won't we encounter NPE later?
- Rohini Palaniswamy
On Jan. 28, 2014, 3:44 a.m., Cheolsoo Park wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/17439/
> -----------------------------------------------------------
>
> (Updated Jan. 28, 2014, 3:44 a.m.)
>
>
> Review request for pig, Alex Bain, Daniel Dai, Mark Wagner, and Rohini Palaniswamy.
>
>
> Bugs: PIG-3728
> https://issues.apache.org/jira/browse/PIG-3728
>
>
> Repository: pig-git
>
>
> Description
> -------
>
> The patch fixes TestSkewedJoin in tez mode. Note I changed SkewedPartitionerTez and WeightedRangePartitionerTez to allow null sampleMap to make testSkewedJoinEmptyInput pass. It used to throw a runtime exception, but now it assumes empty sample input.
>
> I also fixed MiniCluster in hadoop20. They were missing the getExecType() method.
>
>
> Diffs
> -----
>
> shims/test/hadoop20/org/apache/pig/test/MiniCluster.java 3e64a5d
> shims/test/hadoop20/org/apache/pig/test/TezMiniCluster.java 98a580c
> src/org/apache/pig/backend/hadoop/executionengine/tez/POPartitionRearrangeTez.java b17b38b
> src/org/apache/pig/backend/hadoop/executionengine/tez/PigProcessor.java 2b6c899
> src/org/apache/pig/backend/hadoop/executionengine/tez/SkewedPartitionerTez.java 5fe202b
> src/org/apache/pig/backend/hadoop/executionengine/tez/WeightedRangePartitionerTez.java 36441fb
> test/org/apache/pig/test/TestAccumulator.java 516d5c5
> test/org/apache/pig/test/TestSkewedJoin.java be91d6b
> test/tez-tests 0d32d23
>
> Diff: https://reviews.apache.org/r/17439/diff/
>
>
> Testing
> -------
>
> * TestSkewedJoin passes in both tez and mr mode.
> * ant test-tez passes except TestCustomPartitioner (known).
> * All e2e tests pass.
>
>
> Thanks,
>
> Cheolsoo Park
>
>
Re: Review Request 17439: PIG-3728: Fix TestSkewedJoin unit test in tez mode
Posted by Cheolsoo Park <pi...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/17439/
-----------------------------------------------------------
(Updated Jan. 28, 2014, 3:44 a.m.)
Review request for pig, Alex Bain, Daniel Dai, Mark Wagner, and Rohini Palaniswamy.
Changes
-------
Minor clean up.
Bugs: PIG-3728
https://issues.apache.org/jira/browse/PIG-3728
Repository: pig-git
Description
-------
The patch fixes TestSkewedJoin in tez mode. Note I changed SkewedPartitionerTez and WeightedRangePartitionerTez to allow null sampleMap to make testSkewedJoinEmptyInput pass. It used to throw a runtime exception, but now it assumes empty sample input.
I also fixed MiniCluster in hadoop20. They were missing the getExecType() method.
Diffs (updated)
-----
shims/test/hadoop20/org/apache/pig/test/MiniCluster.java 3e64a5d
shims/test/hadoop20/org/apache/pig/test/TezMiniCluster.java 98a580c
src/org/apache/pig/backend/hadoop/executionengine/tez/POPartitionRearrangeTez.java b17b38b
src/org/apache/pig/backend/hadoop/executionengine/tez/PigProcessor.java 2b6c899
src/org/apache/pig/backend/hadoop/executionengine/tez/SkewedPartitionerTez.java 5fe202b
src/org/apache/pig/backend/hadoop/executionengine/tez/WeightedRangePartitionerTez.java 36441fb
test/org/apache/pig/test/TestAccumulator.java 516d5c5
test/org/apache/pig/test/TestSkewedJoin.java be91d6b
test/tez-tests 0d32d23
Diff: https://reviews.apache.org/r/17439/diff/
Testing
-------
* TestSkewedJoin passes in both tez and mr mode.
* ant test-tez passes except TestCustomPartitioner (known).
* All e2e tests pass.
Thanks,
Cheolsoo Park