You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@pig.apache.org by Patcharee Thongtra <Pa...@uni.no> on 2014/04/30 14:42:23 UTC

increase memory size used by Pig UDF

Hi,

How can I increase memory size used by Pig UDF? I got OutOfMemoryError 
exception which was thrown before Pig submitted jobs to Hadoop, see 
error log.

426405 [JobControl] ERROR org.apache.pig.backend.hadoop23.PigJobControl  
- Error while trying to run jobs.
java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
     at 
org.apache.pig.backend.hadoop23.PigJobControl.submit(PigJobControl.java:130)
     at 
org.apache.pig.backend.hadoop23.PigJobControl.run(PigJobControl.java:191)
     at java.lang.Thread.run(Thread.java:662)
     at 
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher$1.run(MapReduceLauncher.java:270)
Caused by: java.lang.reflect.InvocationTargetException
     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
     at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
     at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
     at java.lang.reflect.Method.invoke(Method.java:597)
     at 
org.apache.pig.backend.hadoop23.PigJobControl.submit(PigJobControl.java:128)
     ... 3 more
Caused by: java.lang.OutOfMemoryError: Java heap space
     at ucar.nc2.iosp.IospHelper.makePrimitiveArray(IospHelper.java:669)
     at ucar.nc2.iosp.IospHelper.readDataFill(IospHelper.java:74)
     at ucar.nc2.iosp.netcdf3.N3raf.readData(N3raf.java:62)
     at ucar.nc2.iosp.netcdf3.N3iosp.readData(N3iosp.java:496)
     at ucar.nc2.NetcdfFile.readData(NetcdfFile.java:1894)
     at ucar.nc2.Variable.reallyRead(Variable.java:856)
     at ucar.nc2.Variable._read(Variable.java:828)
     at ucar.nc2.Variable.read(Variable.java:706)
     at 
no.uni.computing.pig.io.input.WRFInputFormat.genFileSplits(WRFInputFormat.java:106)
     at 
no.uni.computing.pig.io.input.WRFInputFormat.getSplits(WRFInputFormat.java:57)
     at 
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:274)
     at 
org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter.java:491)
     at 
org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:508)
     at 
org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:392)
     at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1268)
     at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1265)
     at java.security.AccessController.doPrivileged(Native Method)
     at javax.security.auth.Subject.doAs(Subject.java:396)
     at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
     at org.apache.hadoop.mapreduce.Job.submit(Job.java:1265)
     at 
org.apache.hadoop.mapreduce.lib.jobcontrol.ControlledJob.submit(ControlledJob.java:335)
     ... 8 more

Patcharee


Re: increase memory size used by Pig UDF

Posted by Cheolsoo Park <pi...@gmail.com>.
Pig UDF doesn't seem related.

The stack trace shows the Pig client JVM ran out of memory while computing
input splits. You can try to increase the heap by setting PIG_HEAPSIZE to a
larger than the default (1000). See the "bin/pig" script.





On Wed, Apr 30, 2014 at 5:42 AM, Patcharee Thongtra <
Patcharee.Thongtra@uni.no> wrote:

> Hi,
>
> How can I increase memory size used by Pig UDF? I got OutOfMemoryError
> exception which was thrown before Pig submitted jobs to Hadoop, see error
> log.
>
> 426405 [JobControl] ERROR org.apache.pig.backend.hadoop23.PigJobControl
>  - Error while trying to run jobs.
> java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
>     at org.apache.pig.backend.hadoop23.PigJobControl.submit(
> PigJobControl.java:130)
>     at org.apache.pig.backend.hadoop23.PigJobControl.run(
> PigJobControl.java:191)
>     at java.lang.Thread.run(Thread.java:662)
>     at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.
> MapReduceLauncher$1.run(MapReduceLauncher.java:270)
> Caused by: java.lang.reflect.InvocationTargetException
>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>     at sun.reflect.NativeMethodAccessorImpl.invoke(
> NativeMethodAccessorImpl.java:39)
>     at sun.reflect.DelegatingMethodAccessorImpl.invoke(
> DelegatingMethodAccessorImpl.java:25)
>     at java.lang.reflect.Method.invoke(Method.java:597)
>     at org.apache.pig.backend.hadoop23.PigJobControl.submit(
> PigJobControl.java:128)
>     ... 3 more
> Caused by: java.lang.OutOfMemoryError: Java heap space
>     at ucar.nc2.iosp.IospHelper.makePrimitiveArray(IospHelper.java:669)
>     at ucar.nc2.iosp.IospHelper.readDataFill(IospHelper.java:74)
>     at ucar.nc2.iosp.netcdf3.N3raf.readData(N3raf.java:62)
>     at ucar.nc2.iosp.netcdf3.N3iosp.readData(N3iosp.java:496)
>     at ucar.nc2.NetcdfFile.readData(NetcdfFile.java:1894)
>     at ucar.nc2.Variable.reallyRead(Variable.java:856)
>     at ucar.nc2.Variable._read(Variable.java:828)
>     at ucar.nc2.Variable.read(Variable.java:706)
>     at no.uni.computing.pig.io.input.WRFInputFormat.genFileSplits(
> WRFInputFormat.java:106)
>     at no.uni.computing.pig.io.input.WRFInputFormat.getSplits(
> WRFInputFormat.java:57)
>     at org.apache.pig.backend.hadoop.executionengine.
> mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:274)
>     at org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(
> JobSubmitter.java:491)
>     at org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(
> JobSubmitter.java:508)
>     at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(
> JobSubmitter.java:392)
>     at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1268)
>     at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1265)
>     at java.security.AccessController.doPrivileged(Native Method)
>     at javax.security.auth.Subject.doAs(Subject.java:396)
>     at org.apache.hadoop.security.UserGroupInformation.doAs(
> UserGroupInformation.java:1491)
>     at org.apache.hadoop.mapreduce.Job.submit(Job.java:1265)
>     at org.apache.hadoop.mapreduce.lib.jobcontrol.ControlledJob.
> submit(ControlledJob.java:335)
>     ... 8 more
>
> Patcharee
>
>