Re: Failing to make sense of an error.

[Dmitriy Ryaboy <dv...@gmail.com>]

Are you running in local mode?
Heap error implies the *local* JVM is running into trouble (so, either
you are doing the compute locally, or something odd is going on with
processing the script or collecting the results).

What is your java Xmx set to on the local (client) machine?

On Wed, Nov 28, 2012 at 8:30 AM, Kris Coward <kr...@melon.org> wrote:
> Hi,
>
> I've been compiling some top 25 lists for the frequency with which
> values appear in certain columns in a relation, and based on some of the
> counts, am curious to see if some of the values occur particularly often
> together. To this end, I've been running the following code:
>
> abandonSpike4pvnUid = GROUP abandonSpike3 BY (pv_num, user_id);
> abandonSpike5pvnUid = FOREACH abandonSpike4pvnUid GENERATE group.pv_num AS pvn, group.user_id AS uid, COUNT(abandonSpike3) AS session_count;
> abandonSpike6pvnUid = FILTER abandonSpike5pvnUid BY (session_count > 20);
> abandonSpike7pvnUid = ORDER abandonSpike6pvnUid BY session_count DESC;
> abandonSpikePvnUid = LIMIT abandonSpike7pvnUid 25;
> dump abandonSpikePvnUid;
>
> Where abandonSpike3 has already been successfully grouped by single
> columns to generate the existing top-25 lists.
>
> What's odd is that I was getting the error:
>
> 2012-11-27 23:17:00,112 [main] ERROR org.apache.pig.tools.grunt.Grunt -
> ERROR 2997: Unable to recreate exception from backed error: Error: Java
> heap space
>
> and figured that I'd probably made the groups too small (and therefore
> the list of groups too big) for ORDER to run properly. My solution was
> to insert the FILTER line above, but even trying to get the output of
> that line without any ORDERing (i.e. dump abandonSpike6pvnUid), I'm
> continuing to get the same error message.
>
> It would seem that I'm misjudging what's over-filling the heap, and
> would appreciate a tip on what else it might be. In case it helps, the
> logfile.stack trace from my most recent attempt follows:
>
> ================================================================================
> Backend error message
> ---------------------
> java.lang.Throwable: Child Error
>         at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:242)
> Caused by: java.io.IOException: Task process exit with nonzero status of 1.
>         at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:229)
>
> Backend error message
> ---------------------
> Error: GC overhead limit exceeded
>
> Pig Stack Trace
> ---------------
> ERROR 6015: During execution, encountered a Hadoop error.
>
> org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to open iterator for alias abandonSpike6pvnUid. Backend error : During execution, encountered a Hadoop error.
>         at org.apache.pig.PigServer.openIterator(PigServer.java:753)
>         at
> org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:615)
>         at
> org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:303)
>         at
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:168)
>         at
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:144)
>         at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:76)
>         at org.apache.pig.Main.run(Main.java:455)
>         at org.apache.pig.Main.main(Main.java:107)
> Caused by: org.apache.pig.backend.executionengine.ExecException: ERROR
> 6015: During execution, encountered a Hadoop error.
>         at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:242)
> Caused by: java.lang.Throwable: Child Error
>         ... 1 more
> Caused by: java.io.IOException: Task process exit with nonzero status of 1.
> ================================================================================
>
> Thanks,
> Kris
>
> --
> Kris Coward                                     http://unripe.melon.org/
> GPG Fingerprint: 2BF3 957D 310A FEEC 4733  830E 21A4 05C7 1FEB 12B3