You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Rohini Palaniswamy (JIRA)" <ji...@apache.org> on 2015/11/10 19:34:10 UTC

[jira] [Updated] (PIG-4722) [Pig on Tez] NPE while running Combiner

     [ https://issues.apache.org/jira/browse/PIG-4722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Rohini Palaniswamy updated PIG-4722:
------------------------------------
         Assignee: Rohini Palaniswamy
    Fix Version/s: 0.16.0

Actually my initial analysis was incorrect. Combiner  is called multiple times from SpillThread and also called from ExternalSorter.flush() which is a different thread. But setup(),reduce() and cleanup() is always called as a batch everytime, so there is no cross thread issue there with initialization or cleanup. The actual problem was PigProcessor.close() was being called when spill thread was still running combiner. So even though PhysicalOperator.java had a getReporter() != null check, it became null when calling getReporter().progress() as PigProcessor.close() was resetting it.

{code}
            if (getReporter() != null) {
                getReporter().progress();
            }
{code}

Need to ensure that PigProcessor.close does not cleanup static data when Combiner is running.

> [Pig on Tez] NPE while running Combiner
> ---------------------------------------
>
>                 Key: PIG-4722
>                 URL: https://issues.apache.org/jira/browse/PIG-4722
>             Project: Pig
>          Issue Type: Bug
>            Reporter: Rohini Palaniswamy
>            Assignee: Rohini Palaniswamy
>             Fix For: 0.16.0
>
>
> DefaultSorter in Tez calls Combiner from two different threads - during spill in SpillThread and flush in the main thread. If both run the combiner, one ends up with NPE as Reporter is set on only one thread in initialization. 
> {code}
> java.lang.NullPointerException
>         at org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.getNext(PhysicalOperator.java:366)
>         at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POLocalRearrange.getNextTuple(POLocalRearrange.java:332)
>         at org.apache.pig.backend.hadoop.executionengine.tez.plan.operator.POLocalRearrangeTez.getNextTuple(POLocalRearrangeTez.java:128)
>         at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigCombiner$Combine.processOnePackageOutput(PigCombiner.java:197)
>         at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigCombiner$Combine.reduce(PigCombiner.java:175)
>         at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigCombiner$Combine.reduce(PigCombiner.java:50)
>         at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:171)
>         at org.apache.tez.mapreduce.combine.MRCombiner.runNewCombiner(MRCombiner.java:191)
>         at org.apache.tez.mapreduce.combine.MRCombiner.combine(MRCombiner.java:115)
>         at org.apache.tez.runtime.library.common.sort.impl.ExternalSorter.runCombineProcessor(ExternalSorter.java:279)
>         at org.apache.tez.runtime.library.common.sort.impl.dflt.DefaultSorter.spill(DefaultSorter.java:854)
>         at org.apache.tez.runtime.library.common.sort.impl.dflt.DefaultSorter.sortAndSpill(DefaultSorter.java:780)
>         at org.apache.tez.runtime.library.common.sort.impl.dflt.DefaultSorter$SpillThread.run(DefaultSorter.java:708)
> Caused by: java.lang.NullPointerException
>         at org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.processInput(PhysicalOperator.java:303)
>         at org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POProject.getNextTuple(POProject.java:403)
>         at org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.getNext(PhysicalOperator.java:361)
>         ... 12 more
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)