You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Cheolsoo Park (JIRA)" <ji...@apache.org> on 2014/01/26 00:52:38 UTC
[jira] [Comment Edited] (PIG-3719) Fix skewed join e2e tests
[ https://issues.apache.org/jira/browse/PIG-3719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13882111#comment-13882111 ]
Cheolsoo Park edited comment on PIG-3719 at 1/25/14 11:51 PM:
--------------------------------------------------------------
Attaching 1st patch that fixes NPE in tez.conf e2e tests.
The root cause of NPE was that POShuffleTezLoad was constructing a new instance of NullableWritable for minimum key, but PigNullableWritable.newInstance() didn't set key for NullablePartitionWritable.
Looks like there is no need to construct a new instance in the first place, so I am reverting the change made by PIG-3626 as follows-
{code}
@@ -108,8 +108,7 @@ public class POShuffleTezLoad extends POPackage implements TezLoad {
hasData = true;
cur = readers.get(i).getCurrentKey();
if (min == null || comparator.compare(min, cur) > 0) {
- min = PigNullableWritable.newInstance((PigNullableWritable)cur);
- cur = min;
+ min = (PigNullableWritable) cur;
}
}
}
{code}
I am going to commit this change first so that tez.conf will be fixed. [~rohini], please let me know if you have any concerns.
But nightly.conf SkewedJoin tests still fail because results differ between MR and Tez run. This problem happens when pig.skewedjoin.reduce.maxtuple is set. I will upload another patch for this problem.
was (Author: cheolsoo):
Attaching 1st patch that fixes NPE in tez.conf e2e tests.
The root cause of NPE was that POShuffleTezLoad was constructing a new instance of NullableWritable for minimum key, but PigNullableWritable.newInstance() didn't set key for NullablePartitionWritable.
Looks like there is no need to construct a new instance in the first place, so I am reverting the change made by PIG-3626 as follows-
{code}
@@ -108,8 +108,7 @@ public class POShuffleTezLoad extends POPackage implements TezLoad {
hasData = true;
cur = readers.get(i).getCurrentKey();
if (min == null || comparator.compare(min, cur) > 0) {
- min = PigNullableWritable.newInstance((PigNullableWritable)cur);
- cur = min;
+ min = (PigNullableWritable) cur;
}
}
}
{code}
I am going to commit this change first so that tez.conf will be fixed.
But nightly.conf SkewedJoin tests still fail due to differences in results. I will upload another patch for that problem.
> Fix skewed join e2e tests
> -------------------------
>
> Key: PIG-3719
> URL: https://issues.apache.org/jira/browse/PIG-3719
> Project: Pig
> Issue Type: Sub-task
> Components: tez
> Affects Versions: tez-branch
> Reporter: Cheolsoo Park
> Assignee: Cheolsoo Park
> Fix For: tez-branch
>
> Attachments: PIG-3719-1.patch
>
>
> There are two sets of skewed join e2e tests-
> # tez.conf: Join7 and 8
> # nightly.conf: SkewedJoin
> We need to get both passing.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)