You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@pig.apache.org by Sotiris Matzanas <so...@gmail.com> on 2011/08/11 10:52:23 UTC
PigUnit example failure
Hi all
first, sorry if this has been asked before, but could not find any reference
in the list archives.
I have tried to run the PigUnit example (top_queries.pig) provided on
http://pig.apache.org/docs/r0.8.1/pigunit.html on my 64-bit MacOS 10.6.
I have built and deployed the 0.9.0 jars via maven but I get:
...
11/08/11 10:43:10 INFO mapred.MapTask: io.sort.mb = 100
11/08/11 10:43:10 INFO mapReduceLayer.MapReduceLauncher: HadoopJobId:
job_local_0001
11/08/11 10:43:10 INFO mapReduceLayer.MapReduceLauncher: 0% complete
11/08/11 10:43:10 WARN mapred.LocalJobRunner: job_local_0001
java.lang.OutOfMemoryError: Java heap space
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.<init>(MapTask.java:781)
at
org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.java:524)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:613)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:177)
11/08/11 10:43:15 INFO mapReduceLayer.MapReduceLauncher: job job_local_0001
has failed! Stop running all dependent jobs
11/08/11 10:43:15 INFO mapReduceLayer.MapReduceLauncher: 100% complete
11/08/11 10:43:15 ERROR pigstats.PigStatsUtil: 1 map reduce job(s) failed!
11/08/11 10:43:15 INFO pigstats.SimplePigStats: Detected Local mode. Stats
reported below may be incomplete
11/08/11 10:43:15 INFO pigstats.SimplePigStats: Script Statistics:
I find it very hard to believe that I have run out of heap space processing
10 lines of input. I have searched and searched online and there was one
entry that seemed to indicate that the issue "may" be MacOS specific but no
solution or work around given.
Any input on this highly appreciated.
Cheers
Sotos
Re: PigUnit example failure
Posted by Sotiris Matzanas <so...@gmail.com>.
Thanks for the response Daniel. I should have mentioned that I have actually
tried increasing the java heap to 2GB but with no success.
E.g
% maven -X test
...
[DEBUG] properties used {java.vendor=Apple Inc.,....
env.MAVEN_OPTS=-Xmx2024m, ...
Also my regular (rather heavy) pig scripts run fine without even having to
bump up the Java heap size from its default setting.
So I am really at a loss here.
On Thu, Aug 11, 2011 at 7:21 PM, Daniel Dai <da...@hortonworks.com> wrote:
> There is some static overhead which may already exceeds your heap
> size, no matter how simple your script is. Try to increase the heap
> size. Here are common scenario:
>
> 1. ant: export ANT_OPTS=-Xmx1024m
> 2. pig script: export PIG_HEAPSIZE=1024
> 3. java: add "-Xmx1024m" to command line
> 4. maven: export MAVEN_OPTS=”-Xmx1024m”
>
> Daniel
>
> On Thu, Aug 11, 2011 at 1:52 AM, Sotiris Matzanas
> <so...@gmail.com> wrote:
> > Hi all
> >
> > first, sorry if this has been asked before, but could not find any
> reference
> > in the list archives.
> >
> > I have tried to run the PigUnit example (top_queries.pig) provided on
> > http://pig.apache.org/docs/r0.8.1/pigunit.html on my 64-bit MacOS 10.6.
> >
> > I have built and deployed the 0.9.0 jars via maven but I get:
> >
> > ...
> > 11/08/11 10:43:10 INFO mapred.MapTask: io.sort.mb = 100
> > 11/08/11 10:43:10 INFO mapReduceLayer.MapReduceLauncher: HadoopJobId:
> > job_local_0001
> > 11/08/11 10:43:10 INFO mapReduceLayer.MapReduceLauncher: 0% complete
> > 11/08/11 10:43:10 WARN mapred.LocalJobRunner: job_local_0001
> > java.lang.OutOfMemoryError: Java heap space
> > at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.<init>(MapTask.java:781)
> > at
> >
> org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.java:524)
> > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:613)
> > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
> > at
> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:177)
> > 11/08/11 10:43:15 INFO mapReduceLayer.MapReduceLauncher: job
> job_local_0001
> > has failed! Stop running all dependent jobs
> > 11/08/11 10:43:15 INFO mapReduceLayer.MapReduceLauncher: 100% complete
> > 11/08/11 10:43:15 ERROR pigstats.PigStatsUtil: 1 map reduce job(s)
> failed!
> > 11/08/11 10:43:15 INFO pigstats.SimplePigStats: Detected Local mode.
> Stats
> > reported below may be incomplete
> > 11/08/11 10:43:15 INFO pigstats.SimplePigStats: Script Statistics:
> >
> > I find it very hard to believe that I have run out of heap space
> processing
> > 10 lines of input. I have searched and searched online and there was one
> > entry that seemed to indicate that the issue "may" be MacOS specific but
> no
> > solution or work around given.
> > Any input on this highly appreciated.
> >
> > Cheers
> > Sotos
> >
>
Re: PigUnit example failure
Posted by Daniel Dai <da...@hortonworks.com>.
There is some static overhead which may already exceeds your heap
size, no matter how simple your script is. Try to increase the heap
size. Here are common scenario:
1. ant: export ANT_OPTS=-Xmx1024m
2. pig script: export PIG_HEAPSIZE=1024
3. java: add "-Xmx1024m" to command line
4. maven: export MAVEN_OPTS=”-Xmx1024m”
Daniel
On Thu, Aug 11, 2011 at 1:52 AM, Sotiris Matzanas
<so...@gmail.com> wrote:
> Hi all
>
> first, sorry if this has been asked before, but could not find any reference
> in the list archives.
>
> I have tried to run the PigUnit example (top_queries.pig) provided on
> http://pig.apache.org/docs/r0.8.1/pigunit.html on my 64-bit MacOS 10.6.
>
> I have built and deployed the 0.9.0 jars via maven but I get:
>
> ...
> 11/08/11 10:43:10 INFO mapred.MapTask: io.sort.mb = 100
> 11/08/11 10:43:10 INFO mapReduceLayer.MapReduceLauncher: HadoopJobId:
> job_local_0001
> 11/08/11 10:43:10 INFO mapReduceLayer.MapReduceLauncher: 0% complete
> 11/08/11 10:43:10 WARN mapred.LocalJobRunner: job_local_0001
> java.lang.OutOfMemoryError: Java heap space
> at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.<init>(MapTask.java:781)
> at
> org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.java:524)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:613)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
> at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:177)
> 11/08/11 10:43:15 INFO mapReduceLayer.MapReduceLauncher: job job_local_0001
> has failed! Stop running all dependent jobs
> 11/08/11 10:43:15 INFO mapReduceLayer.MapReduceLauncher: 100% complete
> 11/08/11 10:43:15 ERROR pigstats.PigStatsUtil: 1 map reduce job(s) failed!
> 11/08/11 10:43:15 INFO pigstats.SimplePigStats: Detected Local mode. Stats
> reported below may be incomplete
> 11/08/11 10:43:15 INFO pigstats.SimplePigStats: Script Statistics:
>
> I find it very hard to believe that I have run out of heap space processing
> 10 lines of input. I have searched and searched online and there was one
> entry that seemed to indicate that the issue "may" be MacOS specific but no
> solution or work around given.
> Any input on this highly appreciated.
>
> Cheers
> Sotos
>