[jira] [Commented] (PIG-4564) Pig can deadlock in POPartialAgg if there is a bag

["Rohini Palaniswamy (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/PIG-4564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14553355#comment-14553355 ] 

Rohini Palaniswamy commented on PIG-4564:
-----------------------------------------

{code}
"TezChild" daemon prio=10 tid=0xaa753400 nid=0x5a58 waiting for monitor entry [0xaabfe000]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at org.apache.pig.impl.util.SpillableMemoryManager.registerSpillable(SpillableMemoryManager.java:324)
        - waiting to lock <0xb49c1e08> (a java.util.LinkedList)
        at org.apache.pig.data.DefaultAbstractBag.markSpillableIfNecessary(DefaultAbstractBag.java:129)
        at org.apache.pig.data.DefaultAbstractBag.add(DefaultAbstractBag.java:118)
        at org.apache.hive.hcatalog.pig.PigHCatUtil.transformToBag(PigHCatUtil.java:491)
        at org.apache.hive.hcatalog.pig.PigHCatUtil.extractPigObject(PigHCatUtil.java:403)
        at org.apache.hive.hcatalog.pig.PigHCatUtil.transformToTuple(PigHCatUtil.java:458)
        at org.apache.hive.hcatalog.pig.PigHCatUtil.transformToTuple(PigHCatUtil.java:376)
        at org.apache.hive.hcatalog.pig.HCatBaseLoader.getNext(HCatBaseLoader.java:64)
        at org.apache.hive.hcatalog.pig.HCatLoader.getNext(HCatLoader.java:59)
        at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigRecordReader.nextKeyValue(PigRecordReader.java:204)
        at org.apache.tez.mapreduce.lib.MRReaderMapReduce.next(MRReaderMapReduce.java:116)
        at org.apache.pig.backend.hadoop.executionengine.tez.plan.operator.POSimpleTezLoad.getNextTuple(POSimpleTezLoad.java:106)
        at org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.processInput(PhysicalOperator.java:307)
        at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POSplit.getNextTuple(POSplit.java:210)
        at org.apache.pig.backend.hadoop.executionengine.tez.runtime.PigProcessor.runPipeline(PigProcessor.java:317)
        at org.apache.pig.backend.hadoop.executionengine.tez.runtime.PigProcessor.run(PigProcessor.java:196)
        at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:324)
        at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:176)
        at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:168)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1694)
        at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.call(TezTaskRunner.java:168)
        at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.call(TezTaskRunner.java:163)

"Service Thread" daemon prio=10 tid=0xacbfbc00 nid=0x59ff waiting on condition [0xac7fc000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POPartialAgg.spill(POPartialAgg.java:602)
        - locked <0xd89759d8> (a java.lang.Object)
        at org.apache.pig.impl.util.SpillableMemoryManager.handleNotification(SpillableMemoryManager.java:274)
        - locked <0xb49c1e08> (a java.util.LinkedList)
        at sun.management.NotificationEmitterSupport.sendNotification(NotificationEmitterSupport.java:156)
        at sun.management.MemoryImpl.createNotification(MemoryImpl.java:168)
        at sun.management.MemoryPoolImpl$PoolSensor.triggerAction(MemoryPoolImpl.java:301)
        at sun.management.Sensor.trigger(Sensor.java:137)
{code}

> Pig can deadlock in POPartialAgg if there is a bag
> --------------------------------------------------
>
>                 Key: PIG-4564
>                 URL: https://issues.apache.org/jira/browse/PIG-4564
>             Project: Pig
>          Issue Type: Bug
>            Reporter: Rohini Palaniswamy
>            Assignee: Rohini Palaniswamy
>             Fix For: 0.15.0
>
>
> PIG-3979 made spill of POPartialAgg synchronous, but if there is a bag in the data being grouped then it can deadlock when that tries to register the Spillable with SpillableMemoryManager.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)