You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@pig.apache.org by "Eli Collins (JIRA)" <ji...@apache.org> on 2010/06/15 18:13:23 UTC

[jira] Created: (PIG-1450) TestAlgebraicEvalLocal failures due to OOM

TestAlgebraicEvalLocal failures due to OOM
------------------------------------------

                 Key: PIG-1450
                 URL: https://issues.apache.org/jira/browse/PIG-1450
             Project: Pig
          Issue Type: Test
    Affects Versions: 0.7.0, 0.8.0
            Reporter: Eli Collins
             Fix For: 0.8.0


6 test cases in TestAlgebraicEvalLocal fail on trunk and release 0.7 across a number of different machines.

Example failure:

Unable to open iterator for alias myid
org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to open iterator for alias myid
        at org.apache.pig.PigServer.openIterator(PigServer.java:521)
        at org.apache.pig.test.TestAlgebraicEvalLocal.testGroupUniqueColumnCount(TestAlgebraicEvalLocal.java:236)
Caused by: java.io.IOException: Job terminated with anomalous status FAILED
        at org.apache.pig.PigServer.openIterator(PigServer.java:515)

Probably due to OOMs in the log:

10/06/14 19:38:43 WARN mapred.LocalJobRunner: job_local_0002
java.lang.OutOfMemoryError: Java heap space
        at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.<init>(MapTask.java:781)
        at org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.java:524)
        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:613)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
        at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:177)


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (PIG-1450) TestAlgebraicEvalLocal failures due to OOM

Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.

    [ https://issues.apache.org/jira/browse/PIG-1450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12879846#action_12879846 ] 

Daniel Dai commented on PIG-1450:
---------------------------------

Yes, I also see this error couple of times. Sounds like we should increase the heap space.

> TestAlgebraicEvalLocal failures due to OOM
> ------------------------------------------
>
>                 Key: PIG-1450
>                 URL: https://issues.apache.org/jira/browse/PIG-1450
>             Project: Pig
>          Issue Type: Test
>    Affects Versions: 0.7.0, 0.8.0
>            Reporter: Eli Collins
>             Fix For: 0.8.0
>
>         Attachments: TEST-org.apache.pig.test.TestAlgebraicEvalLocal.txt
>
>
> 6 test cases in TestAlgebraicEvalLocal fail on trunk and release 0.7 across a number of different machines.
> Example failure:
> Unable to open iterator for alias myid
> org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to open iterator for alias myid
>         at org.apache.pig.PigServer.openIterator(PigServer.java:521)
>         at org.apache.pig.test.TestAlgebraicEvalLocal.testGroupUniqueColumnCount(TestAlgebraicEvalLocal.java:236)
> Caused by: java.io.IOException: Job terminated with anomalous status FAILED
>         at org.apache.pig.PigServer.openIterator(PigServer.java:515)
> Probably due to OOMs in the log:
> 10/06/14 19:38:43 WARN mapred.LocalJobRunner: job_local_0002
> java.lang.OutOfMemoryError: Java heap space
>         at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.<init>(MapTask.java:781)
>         at org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.java:524)
>         at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:613)
>         at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
>         at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:177)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (PIG-1450) TestAlgebraicEvalLocal failures due to OOM

Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.

     [ https://issues.apache.org/jira/browse/PIG-1450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daniel Dai updated PIG-1450:
----------------------------

    Attachment: PIG-1450.patch

> TestAlgebraicEvalLocal failures due to OOM
> ------------------------------------------
>
>                 Key: PIG-1450
>                 URL: https://issues.apache.org/jira/browse/PIG-1450
>             Project: Pig
>          Issue Type: Test
>    Affects Versions: 0.7.0, 0.8.0
>            Reporter: Eli Collins
>             Fix For: 0.8.0
>
>         Attachments: PIG-1450.patch, TEST-org.apache.pig.test.TestAlgebraicEvalLocal.txt
>
>
> 6 test cases in TestAlgebraicEvalLocal fail on trunk and release 0.7 across a number of different machines.
> Example failure:
> Unable to open iterator for alias myid
> org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to open iterator for alias myid
>         at org.apache.pig.PigServer.openIterator(PigServer.java:521)
>         at org.apache.pig.test.TestAlgebraicEvalLocal.testGroupUniqueColumnCount(TestAlgebraicEvalLocal.java:236)
> Caused by: java.io.IOException: Job terminated with anomalous status FAILED
>         at org.apache.pig.PigServer.openIterator(PigServer.java:515)
> Probably due to OOMs in the log:
> 10/06/14 19:38:43 WARN mapred.LocalJobRunner: job_local_0002
> java.lang.OutOfMemoryError: Java heap space
>         at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.<init>(MapTask.java:781)
>         at org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.java:524)
>         at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:613)
>         at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
>         at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:177)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (PIG-1450) TestAlgebraicEvalLocal failures due to OOM

Posted by "Jeff Zhang (JIRA)" <ji...@apache.org>.

    [ https://issues.apache.org/jira/browse/PIG-1450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12879833#action_12879833 ] 

Jeff Zhang commented on PIG-1450:
---------------------------------

I believe so, hadoop's local mode need much more memory. One solution fo this problem is to increase the heap space for test case by setting the Xmx argument

> TestAlgebraicEvalLocal failures due to OOM
> ------------------------------------------
>
>                 Key: PIG-1450
>                 URL: https://issues.apache.org/jira/browse/PIG-1450
>             Project: Pig
>          Issue Type: Test
>    Affects Versions: 0.7.0, 0.8.0
>            Reporter: Eli Collins
>             Fix For: 0.8.0
>
>         Attachments: TEST-org.apache.pig.test.TestAlgebraicEvalLocal.txt
>
>
> 6 test cases in TestAlgebraicEvalLocal fail on trunk and release 0.7 across a number of different machines.
> Example failure:
> Unable to open iterator for alias myid
> org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to open iterator for alias myid
>         at org.apache.pig.PigServer.openIterator(PigServer.java:521)
>         at org.apache.pig.test.TestAlgebraicEvalLocal.testGroupUniqueColumnCount(TestAlgebraicEvalLocal.java:236)
> Caused by: java.io.IOException: Job terminated with anomalous status FAILED
>         at org.apache.pig.PigServer.openIterator(PigServer.java:515)
> Probably due to OOMs in the log:
> 10/06/14 19:38:43 WARN mapred.LocalJobRunner: job_local_0002
> java.lang.OutOfMemoryError: Java heap space
>         at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.<init>(MapTask.java:781)
>         at org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.java:524)
>         at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:613)
>         at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
>         at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:177)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (PIG-1450) TestAlgebraicEvalLocal failures due to OOM

Posted by "Eli Collins (JIRA)" <ji...@apache.org>.

    [ https://issues.apache.org/jira/browse/PIG-1450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12879830#action_12879830 ] 

Eli Collins commented on PIG-1450:
----------------------------------

Could this be due to the switch to Hadoop's local mode?

> TestAlgebraicEvalLocal failures due to OOM
> ------------------------------------------
>
>                 Key: PIG-1450
>                 URL: https://issues.apache.org/jira/browse/PIG-1450
>             Project: Pig
>          Issue Type: Test
>    Affects Versions: 0.7.0, 0.8.0
>            Reporter: Eli Collins
>             Fix For: 0.8.0
>
>         Attachments: TEST-org.apache.pig.test.TestAlgebraicEvalLocal.txt
>
>
> 6 test cases in TestAlgebraicEvalLocal fail on trunk and release 0.7 across a number of different machines.
> Example failure:
> Unable to open iterator for alias myid
> org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to open iterator for alias myid
>         at org.apache.pig.PigServer.openIterator(PigServer.java:521)
>         at org.apache.pig.test.TestAlgebraicEvalLocal.testGroupUniqueColumnCount(TestAlgebraicEvalLocal.java:236)
> Caused by: java.io.IOException: Job terminated with anomalous status FAILED
>         at org.apache.pig.PigServer.openIterator(PigServer.java:515)
> Probably due to OOMs in the log:
> 10/06/14 19:38:43 WARN mapred.LocalJobRunner: job_local_0002
> java.lang.OutOfMemoryError: Java heap space
>         at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.<init>(MapTask.java:781)
>         at org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.java:524)
>         at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:613)
>         at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
>         at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:177)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (PIG-1450) TestAlgebraicEvalLocal failures due to OOM

Posted by "Eli Collins (JIRA)" <ji...@apache.org>.

    [ https://issues.apache.org/jira/browse/PIG-1450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12879940#action_12879940 ] 

Eli Collins commented on PIG-1450:
----------------------------------

I confirmed this patch fixed the test failures. Thanks Daniel.


> TestAlgebraicEvalLocal failures due to OOM
> ------------------------------------------
>
>                 Key: PIG-1450
>                 URL: https://issues.apache.org/jira/browse/PIG-1450
>             Project: Pig
>          Issue Type: Test
>    Affects Versions: 0.7.0, 0.8.0
>            Reporter: Eli Collins
>             Fix For: 0.8.0
>
>         Attachments: PIG-1450.patch, TEST-org.apache.pig.test.TestAlgebraicEvalLocal.txt
>
>
> 6 test cases in TestAlgebraicEvalLocal fail on trunk and release 0.7 across a number of different machines.
> Example failure:
> Unable to open iterator for alias myid
> org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to open iterator for alias myid
>         at org.apache.pig.PigServer.openIterator(PigServer.java:521)
>         at org.apache.pig.test.TestAlgebraicEvalLocal.testGroupUniqueColumnCount(TestAlgebraicEvalLocal.java:236)
> Caused by: java.io.IOException: Job terminated with anomalous status FAILED
>         at org.apache.pig.PigServer.openIterator(PigServer.java:515)
> Probably due to OOMs in the log:
> 10/06/14 19:38:43 WARN mapred.LocalJobRunner: job_local_0002
> java.lang.OutOfMemoryError: Java heap space
>         at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.<init>(MapTask.java:781)
>         at org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.java:524)
>         at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:613)
>         at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
>         at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:177)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (PIG-1450) TestAlgebraicEvalLocal failures due to OOM

Posted by "Richard Ding (JIRA)" <ji...@apache.org>.

    [ https://issues.apache.org/jira/browse/PIG-1450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12879874#action_12879874 ] 

Richard Ding commented on PIG-1450:
-----------------------------------

+1 for commit. This should fix this OOM problem.

> TestAlgebraicEvalLocal failures due to OOM
> ------------------------------------------
>
>                 Key: PIG-1450
>                 URL: https://issues.apache.org/jira/browse/PIG-1450
>             Project: Pig
>          Issue Type: Test
>    Affects Versions: 0.7.0, 0.8.0
>            Reporter: Eli Collins
>             Fix For: 0.8.0
>
>         Attachments: PIG-1450.patch, TEST-org.apache.pig.test.TestAlgebraicEvalLocal.txt
>
>
> 6 test cases in TestAlgebraicEvalLocal fail on trunk and release 0.7 across a number of different machines.
> Example failure:
> Unable to open iterator for alias myid
> org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to open iterator for alias myid
>         at org.apache.pig.PigServer.openIterator(PigServer.java:521)
>         at org.apache.pig.test.TestAlgebraicEvalLocal.testGroupUniqueColumnCount(TestAlgebraicEvalLocal.java:236)
> Caused by: java.io.IOException: Job terminated with anomalous status FAILED
>         at org.apache.pig.PigServer.openIterator(PigServer.java:515)
> Probably due to OOMs in the log:
> 10/06/14 19:38:43 WARN mapred.LocalJobRunner: job_local_0002
> java.lang.OutOfMemoryError: Java heap space
>         at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.<init>(MapTask.java:781)
>         at org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.java:524)
>         at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:613)
>         at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
>         at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:177)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Resolved: (PIG-1450) TestAlgebraicEvalLocal failures due to OOM

Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.

     [ https://issues.apache.org/jira/browse/PIG-1450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daniel Dai resolved PIG-1450.
-----------------------------

    Hadoop Flags: [Reviewed]
      Resolution: Fixed

> TestAlgebraicEvalLocal failures due to OOM
> ------------------------------------------
>
>                 Key: PIG-1450
>                 URL: https://issues.apache.org/jira/browse/PIG-1450
>             Project: Pig
>          Issue Type: Test
>    Affects Versions: 0.7.0, 0.8.0
>            Reporter: Eli Collins
>             Fix For: 0.8.0
>
>         Attachments: PIG-1450.patch, TEST-org.apache.pig.test.TestAlgebraicEvalLocal.txt
>
>
> 6 test cases in TestAlgebraicEvalLocal fail on trunk and release 0.7 across a number of different machines.
> Example failure:
> Unable to open iterator for alias myid
> org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to open iterator for alias myid
>         at org.apache.pig.PigServer.openIterator(PigServer.java:521)
>         at org.apache.pig.test.TestAlgebraicEvalLocal.testGroupUniqueColumnCount(TestAlgebraicEvalLocal.java:236)
> Caused by: java.io.IOException: Job terminated with anomalous status FAILED
>         at org.apache.pig.PigServer.openIterator(PigServer.java:515)
> Probably due to OOMs in the log:
> 10/06/14 19:38:43 WARN mapred.LocalJobRunner: job_local_0002
> java.lang.OutOfMemoryError: Java heap space
>         at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.<init>(MapTask.java:781)
>         at org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.java:524)
>         at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:613)
>         at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
>         at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:177)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (PIG-1450) TestAlgebraicEvalLocal failures due to OOM

Posted by "Eli Collins (JIRA)" <ji...@apache.org>.

     [ https://issues.apache.org/jira/browse/PIG-1450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated PIG-1450:
-----------------------------

    Attachment: TEST-org.apache.pig.test.TestAlgebraicEvalLocal.txt

Full test log attached.

> TestAlgebraicEvalLocal failures due to OOM
> ------------------------------------------
>
>                 Key: PIG-1450
>                 URL: https://issues.apache.org/jira/browse/PIG-1450
>             Project: Pig
>          Issue Type: Test
>    Affects Versions: 0.7.0, 0.8.0
>            Reporter: Eli Collins
>             Fix For: 0.8.0
>
>         Attachments: TEST-org.apache.pig.test.TestAlgebraicEvalLocal.txt
>
>
> 6 test cases in TestAlgebraicEvalLocal fail on trunk and release 0.7 across a number of different machines.
> Example failure:
> Unable to open iterator for alias myid
> org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to open iterator for alias myid
>         at org.apache.pig.PigServer.openIterator(PigServer.java:521)
>         at org.apache.pig.test.TestAlgebraicEvalLocal.testGroupUniqueColumnCount(TestAlgebraicEvalLocal.java:236)
> Caused by: java.io.IOException: Job terminated with anomalous status FAILED
>         at org.apache.pig.PigServer.openIterator(PigServer.java:515)
> Probably due to OOMs in the log:
> 10/06/14 19:38:43 WARN mapred.LocalJobRunner: job_local_0002
> java.lang.OutOfMemoryError: Java heap space
>         at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.<init>(MapTask.java:781)
>         at org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.java:524)
>         at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:613)
>         at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
>         at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:177)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.