You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@pig.apache.org by Patcharee Thongtra <Pa...@uni.no> on 2014/04/30 14:42:23 UTC
increase memory size used by Pig UDF
Hi,
How can I increase memory size used by Pig UDF? I got OutOfMemoryError
exception which was thrown before Pig submitted jobs to Hadoop, see
error log.
426405 [JobControl] ERROR org.apache.pig.backend.hadoop23.PigJobControl
- Error while trying to run jobs.
java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
at
org.apache.pig.backend.hadoop23.PigJobControl.submit(PigJobControl.java:130)
at
org.apache.pig.backend.hadoop23.PigJobControl.run(PigJobControl.java:191)
at java.lang.Thread.run(Thread.java:662)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher$1.run(MapReduceLauncher.java:270)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.pig.backend.hadoop23.PigJobControl.submit(PigJobControl.java:128)
... 3 more
Caused by: java.lang.OutOfMemoryError: Java heap space
at ucar.nc2.iosp.IospHelper.makePrimitiveArray(IospHelper.java:669)
at ucar.nc2.iosp.IospHelper.readDataFill(IospHelper.java:74)
at ucar.nc2.iosp.netcdf3.N3raf.readData(N3raf.java:62)
at ucar.nc2.iosp.netcdf3.N3iosp.readData(N3iosp.java:496)
at ucar.nc2.NetcdfFile.readData(NetcdfFile.java:1894)
at ucar.nc2.Variable.reallyRead(Variable.java:856)
at ucar.nc2.Variable._read(Variable.java:828)
at ucar.nc2.Variable.read(Variable.java:706)
at
no.uni.computing.pig.io.input.WRFInputFormat.genFileSplits(WRFInputFormat.java:106)
at
no.uni.computing.pig.io.input.WRFInputFormat.getSplits(WRFInputFormat.java:57)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:274)
at
org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter.java:491)
at
org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:508)
at
org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:392)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1268)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1265)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1265)
at
org.apache.hadoop.mapreduce.lib.jobcontrol.ControlledJob.submit(ControlledJob.java:335)
... 8 more
Patcharee
Re: increase memory size used by Pig UDF
Posted by Cheolsoo Park <pi...@gmail.com>.
Pig UDF doesn't seem related.
The stack trace shows the Pig client JVM ran out of memory while computing
input splits. You can try to increase the heap by setting PIG_HEAPSIZE to a
larger than the default (1000). See the "bin/pig" script.
On Wed, Apr 30, 2014 at 5:42 AM, Patcharee Thongtra <
Patcharee.Thongtra@uni.no> wrote:
> Hi,
>
> How can I increase memory size used by Pig UDF? I got OutOfMemoryError
> exception which was thrown before Pig submitted jobs to Hadoop, see error
> log.
>
> 426405 [JobControl] ERROR org.apache.pig.backend.hadoop23.PigJobControl
> - Error while trying to run jobs.
> java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
> at org.apache.pig.backend.hadoop23.PigJobControl.submit(
> PigJobControl.java:130)
> at org.apache.pig.backend.hadoop23.PigJobControl.run(
> PigJobControl.java:191)
> at java.lang.Thread.run(Thread.java:662)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.
> MapReduceLauncher$1.run(MapReduceLauncher.java:270)
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(
> NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(
> DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.pig.backend.hadoop23.PigJobControl.submit(
> PigJobControl.java:128)
> ... 3 more
> Caused by: java.lang.OutOfMemoryError: Java heap space
> at ucar.nc2.iosp.IospHelper.makePrimitiveArray(IospHelper.java:669)
> at ucar.nc2.iosp.IospHelper.readDataFill(IospHelper.java:74)
> at ucar.nc2.iosp.netcdf3.N3raf.readData(N3raf.java:62)
> at ucar.nc2.iosp.netcdf3.N3iosp.readData(N3iosp.java:496)
> at ucar.nc2.NetcdfFile.readData(NetcdfFile.java:1894)
> at ucar.nc2.Variable.reallyRead(Variable.java:856)
> at ucar.nc2.Variable._read(Variable.java:828)
> at ucar.nc2.Variable.read(Variable.java:706)
> at no.uni.computing.pig.io.input.WRFInputFormat.genFileSplits(
> WRFInputFormat.java:106)
> at no.uni.computing.pig.io.input.WRFInputFormat.getSplits(
> WRFInputFormat.java:57)
> at org.apache.pig.backend.hadoop.executionengine.
> mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:274)
> at org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(
> JobSubmitter.java:491)
> at org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(
> JobSubmitter.java:508)
> at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(
> JobSubmitter.java:392)
> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1268)
> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1265)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(
> UserGroupInformation.java:1491)
> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1265)
> at org.apache.hadoop.mapreduce.lib.jobcontrol.ControlledJob.
> submit(ControlledJob.java:335)
> ... 8 more
>
> Patcharee
>
>