[jira] [Updated] (PIG-3719) Fix skewed join e2e tests

["Cheolsoo Park (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/PIG-3719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Cheolsoo Park updated PIG-3719:
-------------------------------

    Attachment: PIG-3719-1.patch

Attaching 1st patch that fixes NPE in tez.conf e2e tests.

The root cause of NPE was that POShuffleTezLoad was constructing a new instance of NullableWritable for minimum key, but PigNullableWritable.newInstance() didn't set key for NullablePartitionWritable.

Looks like there is no need to construct a new instance in the first place, so I am reverting the change made by PIG-3626 as follows-
{code}
@@ -108,8 +108,7 @@ public class POShuffleTezLoad extends POPackage implements TezLoad {
                         hasData = true;
                         cur = readers.get(i).getCurrentKey();
                         if (min == null || comparator.compare(min, cur) > 0) {
-                            min = PigNullableWritable.newInstance((PigNullableWritable)cur);
-                            cur = min;
+                            min = (PigNullableWritable) cur;
                         }
                     }
                 }
{code}

I am going to commit this change first so that tez.conf will be fixed.

But nightly.conf SkewedJoin tests still fail due to differences in results. I will upload another patch for that problem.

> Fix skewed join e2e tests
> -------------------------
>
>                 Key: PIG-3719
>                 URL: https://issues.apache.org/jira/browse/PIG-3719
>             Project: Pig
>          Issue Type: Sub-task
>          Components: tez
>    Affects Versions: tez-branch
>            Reporter: Cheolsoo Park
>            Assignee: Cheolsoo Park
>             Fix For: tez-branch
>
>         Attachments: PIG-3719-1.patch
>
>
> There are two sets of skewed join e2e tests-
> # tez.conf: Join7 and 8
> # nightly.conf: SkewedJoin
> We need to get both passing.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)