You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by $iddhe$h Divekar <si...@gmail.com> on 2016/08/04 14:34:41 UTC
Spark jobs failing due to java.lang.OutOfMemoryError: PermGen space
Hi,
I am running spark jobs using apache oozie in yarn-client mode.
My job.properties has sparkConf which gets used in workflow.xml.
I have tried increasing MaxPermSize using sparkConf in job.properties
but that is not resolving the issue.
*sparkConf*=--verbose --driver-java-options '-XX:MaxPermSize=8192M' --conf
spark.speculation=false --conf
spark.hadoop.spark.sql.parquet.output.committer.class=\
"org.apache.spark.sql.parquet.DirectParquetOutputCommitter" --conf
spark.hadoop.mapred.output.committer.class="org.apache.hadoop.mapred.DirectFileOutputCommit\
ter.class" --conf spark.hadoop.mapreduce.use.directfileoutputcommitter=true
Am I missing anything ?
I am seeing following errors.
2016-08-03 22:33:43,318 WARN SparkActionExecutor:523 -
SERVER[ip-10-0-0-161.ec2.internal] USER[hadoop] GROUP[-] TOKEN[]
APP[ApprouteOozie] JOB[0000031-160803180548580-oozie-oozi-W]
ACTION[0000031-160803180548580-oozie-oozi-W@spark-approute] Launcher ERROR,
reason: Main class [org.apache.oozie.action.hadoop.SparkMain], main() threw
exception, PermGen space
2016-08-03 22:33:43,319 WARN SparkActionExecutor:523 -
SERVER[ip-10-0-0-161.ec2.internal] USER[hadoop] GROUP[-] TOKEN[]
APP[ApprouteOozie] JOB[0000031-160803180548580-oozie-oozi-W]
ACTION[0000031-160803180548580-oozie-oozi-W@spark-approute] Launcher
exception: PermGen space
java.lang.OutOfMemoryError: PermGen space
oozie-oozi-W@spark-approute] Launcher exception: PermGen space
java.lang.OutOfMemoryError: PermGen space
at java.lang.Class.getDeclaredConstructors0(Native Method)
at java.lang.Class.privateGetDeclaredConstructors(Class.java:2595)
at java.lang.Class.getConstructor0(Class.java:2895)
at java.lang.Class.newInstance(Class.java:354)
at
sun.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:399)
at
sun.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:396)
at java.security.AccessController.doPrivileged(Native Method)
at
sun.reflect.MethodAccessorGenerator.generate(MethodAccessorGenerator.java:395)
at
sun.reflect.MethodAccessorGenerator.generateSerializationConstructor(MethodAccessorGenerator.java:113)
at
sun.reflect.ReflectionFactory.newConstructorForSerialization(ReflectionFactory.java:331)
at
java.io.ObjectStreamClass.getSerializableConstructor(ObjectStreamClass.java:1420)
at java.io.ObjectStreamClass.access$1500(ObjectStreamClass.java:72)
at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:497)
at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:472)
at java.security.AccessController.doPrivileged(Native Method)
at java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:472)
at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:369)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1133)
at
java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547)
at
java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508)
at
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
at
java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547)
at
java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508)
at
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
at
java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547)
at
java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508)
at
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
at java.io.ObjectOutputStream.writeArray(ObjectOutputStream.java:1377)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1173)
--
-$iddhi.
Re: Spark jobs failing due to java.lang.OutOfMemoryError: PermGen space
Posted by Deepak Sharma <de...@gmail.com>.
Yes agreed.It seems to be issue with mapping the text file contents to case
classes, not sure though.
On Thu, Aug 4, 2016 at 8:17 PM, $iddhe$h Divekar <siddhesh.divekar@gmail.com
> wrote:
> Hi Deepak,
>
> My files are always > 50MB.
> I would think there would be a small config to overcome this.
> Tried almost everything i could after searching online.
>
> Any help from the mailing list would be appreciated.
>
> On Thu, Aug 4, 2016 at 7:43 AM, Deepak Sharma <de...@gmail.com>
> wrote:
>
>> I am facing the same issue with spark 1.5.2
>> If the file size that's being processed by spark , is of size 10-12 MB ,
>> it throws out of memory .
>> But if the same file is within 5 MB limit , it runs fine.
>> I am using spark configuration with 7GB of memory and 3 cores for
>> executors in the cluster of 8 executor.
>>
>> Thanks
>> Deepak
>>
>> On 4 Aug 2016 8:04 pm, "$iddhe$h Divekar" <si...@gmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>> I am running spark jobs using apache oozie in yarn-client mode.
>>> My job.properties has sparkConf which gets used in workflow.xml.
>>>
>>> I have tried increasing MaxPermSize using sparkConf in job.properties
>>> but that is not resolving the issue.
>>>
>>> *sparkConf*=--verbose --driver-java-options '-XX:MaxPermSize=8192M'
>>> --conf spark.speculation=false --conf spark.hadoop.spark.sql.
>>> parquet.output.committer.class=\
>>> "org.apache.spark.sql.parquet.DirectParquetOutputCommitter" --conf
>>> spark.hadoop.mapred.output.committer.class="org.apache.hadoop.mapred.
>>> DirectFileOutputCommit\
>>> ter.class" --conf spark.hadoop.mapreduce.use.
>>> directfileoutputcommitter=true
>>>
>>> Am I missing anything ?
>>>
>>> I am seeing following errors.
>>>
>>> 2016-08-03 22:33:43,318 WARN SparkActionExecutor:523 -
>>> SERVER[ip-10-0-0-161.ec2.internal] USER[hadoop] GROUP[-] TOKEN[]
>>> APP[ApprouteOozie] JOB[0000031-160803180548580-oozie-oozi-W]
>>> ACTION[0000031-160803180548580-oozie-oozi-W@spark-approute] Launcher
>>> ERROR, reason: Main class [org.apache.oozie.action.hadoop.SparkMain],
>>> main() threw exception, PermGen space
>>> 2016-08-03 22:33:43,319 WARN SparkActionExecutor:523 -
>>> SERVER[ip-10-0-0-161.ec2.internal] USER[hadoop] GROUP[-] TOKEN[]
>>> APP[ApprouteOozie] JOB[0000031-160803180548580-oozie-oozi-W]
>>> ACTION[0000031-160803180548580-oozie-oozi-W@spark-approute] Launcher
>>> exception: PermGen space
>>> java.lang.OutOfMemoryError: PermGen space
>>>
>>> oozie-oozi-W@spark-approute] Launcher exception: PermGen space
>>> java.lang.OutOfMemoryError: PermGen space
>>> at java.lang.Class.getDeclaredConstructors0(Native Method)
>>> at java.lang.Class.privateGetDeclaredConstructors(Class.java:2595)
>>> at java.lang.Class.getConstructor0(Class.java:2895)
>>> at java.lang.Class.newInstance(Class.java:354)
>>> at sun.reflect.MethodAccessorGenerator$1.run(
>>> MethodAccessorGenerator.java:399)
>>> at sun.reflect.MethodAccessorGenerator$1.run(
>>> MethodAccessorGenerator.java:396)
>>> at java.security.AccessController.doPrivileged(Native Method)
>>> at sun.reflect.MethodAccessorGenerator.generate(
>>> MethodAccessorGenerator.java:395)
>>> at sun.reflect.MethodAccessorGenerator.
>>> generateSerializationConstructor(MethodAccessorGenerator.java:113)
>>> at sun.reflect.ReflectionFactory.newConstructorForSerialization
>>> (ReflectionFactory.java:331)
>>> at java.io.ObjectStreamClass.getSerializableConstructor(
>>> ObjectStreamClass.java:1420)
>>> at java.io.ObjectStreamClass.access$1500(ObjectStreamClass.java:72)
>>> at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:497)
>>> at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:472)
>>> at java.security.AccessController.doPrivileged(Native Method)
>>> at java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:472)
>>> at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:369)
>>> at java.io.ObjectOutputStream.writeObject0(
>>> ObjectOutputStream.java:1133)
>>> at java.io.ObjectOutputStream.defaultWriteFields(
>>> ObjectOutputStream.java:1547)
>>> at java.io.ObjectOutputStream.writeSerialData(
>>> ObjectOutputStream.java:1508)
>>> at java.io.ObjectOutputStream.writeOrdinaryObject(
>>> ObjectOutputStream.java:1431)
>>> at java.io.ObjectOutputStream.writeObject0(
>>> ObjectOutputStream.java:1177)
>>> at java.io.ObjectOutputStream.defaultWriteFields(
>>> ObjectOutputStream.java:1547)
>>> at java.io.ObjectOutputStream.writeSerialData(
>>> ObjectOutputStream.java:1508)
>>> at java.io.ObjectOutputStream.writeOrdinaryObject(
>>> ObjectOutputStream.java:1431)
>>> at java.io.ObjectOutputStream.writeObject0(
>>> ObjectOutputStream.java:1177)
>>> at java.io.ObjectOutputStream.defaultWriteFields(
>>> ObjectOutputStream.java:1547)
>>> at java.io.ObjectOutputStream.writeSerialData(
>>> ObjectOutputStream.java:1508)
>>> at java.io.ObjectOutputStream.writeOrdinaryObject(
>>> ObjectOutputStream.java:1431)
>>> at java.io.ObjectOutputStream.writeObject0(
>>> ObjectOutputStream.java:1177)
>>> at java.io.ObjectOutputStream.writeArray(ObjectOutputStream.
>>> java:1377)
>>> at java.io.ObjectOutputStream.writeObject0(
>>> ObjectOutputStream.java:1173)
>>>
>>> --
>>> -$iddhi.
>>>
>>
>
>
> --
> -$iddhi.
>
--
Thanks
Deepak
www.bigdatabig.com
www.keosha.net
Re: Spark jobs failing due to java.lang.OutOfMemoryError: PermGen space
Posted by $iddhe$h Divekar <si...@gmail.com>.
Hi Deepak,
My files are always > 50MB.
I would think there would be a small config to overcome this.
Tried almost everything i could after searching online.
Any help from the mailing list would be appreciated.
On Thu, Aug 4, 2016 at 7:43 AM, Deepak Sharma <de...@gmail.com> wrote:
> I am facing the same issue with spark 1.5.2
> If the file size that's being processed by spark , is of size 10-12 MB ,
> it throws out of memory .
> But if the same file is within 5 MB limit , it runs fine.
> I am using spark configuration with 7GB of memory and 3 cores for
> executors in the cluster of 8 executor.
>
> Thanks
> Deepak
>
> On 4 Aug 2016 8:04 pm, "$iddhe$h Divekar" <si...@gmail.com>
> wrote:
>
>> Hi,
>>
>> I am running spark jobs using apache oozie in yarn-client mode.
>> My job.properties has sparkConf which gets used in workflow.xml.
>>
>> I have tried increasing MaxPermSize using sparkConf in job.properties
>> but that is not resolving the issue.
>>
>> *sparkConf*=--verbose --driver-java-options '-XX:MaxPermSize=8192M'
>> --conf spark.speculation=false --conf
>> spark.hadoop.spark.sql.parquet.output.committer.class=\
>> "org.apache.spark.sql.parquet.DirectParquetOutputCommitter" --conf
>> spark.hadoop.mapred.output.committer.class="org.apache.hadoop.mapred.DirectFileOutputCommit\
>> ter.class" --conf
>> spark.hadoop.mapreduce.use.directfileoutputcommitter=true
>>
>> Am I missing anything ?
>>
>> I am seeing following errors.
>>
>> 2016-08-03 22:33:43,318 WARN SparkActionExecutor:523 -
>> SERVER[ip-10-0-0-161.ec2.internal] USER[hadoop] GROUP[-] TOKEN[]
>> APP[ApprouteOozie] JOB[0000031-160803180548580-oozie-oozi-W]
>> ACTION[0000031-160803180548580-oozie-oozi-W@spark-approute] Launcher
>> ERROR, reason: Main class [org.apache.oozie.action.hadoop.SparkMain],
>> main() threw exception, PermGen space
>> 2016-08-03 22:33:43,319 WARN SparkActionExecutor:523 -
>> SERVER[ip-10-0-0-161.ec2.internal] USER[hadoop] GROUP[-] TOKEN[]
>> APP[ApprouteOozie] JOB[0000031-160803180548580-oozie-oozi-W]
>> ACTION[0000031-160803180548580-oozie-oozi-W@spark-approute] Launcher
>> exception: PermGen space
>> java.lang.OutOfMemoryError: PermGen space
>>
>> oozie-oozi-W@spark-approute] Launcher exception: PermGen space
>> java.lang.OutOfMemoryError: PermGen space
>> at java.lang.Class.getDeclaredConstructors0(Native Method)
>> at java.lang.Class.privateGetDeclaredConstructors(Class.java:2595)
>> at java.lang.Class.getConstructor0(Class.java:2895)
>> at java.lang.Class.newInstance(Class.java:354)
>> at
>> sun.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:399)
>> at
>> sun.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:396)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at
>> sun.reflect.MethodAccessorGenerator.generate(MethodAccessorGenerator.java:395)
>> at
>> sun.reflect.MethodAccessorGenerator.generateSerializationConstructor(MethodAccessorGenerator.java:113)
>> at
>> sun.reflect.ReflectionFactory.newConstructorForSerialization(ReflectionFactory.java:331)
>> at
>> java.io.ObjectStreamClass.getSerializableConstructor(ObjectStreamClass.java:1420)
>> at java.io.ObjectStreamClass.access$1500(ObjectStreamClass.java:72)
>> at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:497)
>> at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:472)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:472)
>> at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:369)
>> at
>> java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1133)
>> at
>> java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547)
>> at
>> java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508)
>> at
>> java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
>> at
>> java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
>> at
>> java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547)
>> at
>> java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508)
>> at
>> java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
>> at
>> java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
>> at
>> java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547)
>> at
>> java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508)
>> at
>> java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
>> at
>> java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
>> at java.io.ObjectOutputStream.writeArray(ObjectOutputStream.java:1377)
>> at
>> java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1173)
>>
>> --
>> -$iddhi.
>>
>
--
-$iddhi.
Re: Spark jobs failing due to java.lang.OutOfMemoryError: PermGen space
Posted by Deepak Sharma <de...@gmail.com>.
I am facing the same issue with spark 1.5.2
If the file size that's being processed by spark , is of size 10-12 MB , it
throws out of memory .
But if the same file is within 5 MB limit , it runs fine.
I am using spark configuration with 7GB of memory and 3 cores for executors
in the cluster of 8 executor.
Thanks
Deepak
On 4 Aug 2016 8:04 pm, "$iddhe$h Divekar" <si...@gmail.com>
wrote:
> Hi,
>
> I am running spark jobs using apache oozie in yarn-client mode.
> My job.properties has sparkConf which gets used in workflow.xml.
>
> I have tried increasing MaxPermSize using sparkConf in job.properties
> but that is not resolving the issue.
>
> *sparkConf*=--verbose --driver-java-options '-XX:MaxPermSize=8192M'
> --conf spark.speculation=false --conf spark.hadoop.spark.sql.
> parquet.output.committer.class=\
> "org.apache.spark.sql.parquet.DirectParquetOutputCommitter" --conf
> spark.hadoop.mapred.output.committer.class="org.apache.hadoop.mapred.
> DirectFileOutputCommit\
> ter.class" --conf spark.hadoop.mapreduce.use.
> directfileoutputcommitter=true
>
> Am I missing anything ?
>
> I am seeing following errors.
>
> 2016-08-03 22:33:43,318 WARN SparkActionExecutor:523 -
> SERVER[ip-10-0-0-161.ec2.internal] USER[hadoop] GROUP[-] TOKEN[]
> APP[ApprouteOozie] JOB[0000031-160803180548580-oozie-oozi-W]
> ACTION[0000031-160803180548580-oozie-oozi-W@spark-approute] Launcher
> ERROR, reason: Main class [org.apache.oozie.action.hadoop.SparkMain],
> main() threw exception, PermGen space
> 2016-08-03 22:33:43,319 WARN SparkActionExecutor:523 -
> SERVER[ip-10-0-0-161.ec2.internal] USER[hadoop] GROUP[-] TOKEN[]
> APP[ApprouteOozie] JOB[0000031-160803180548580-oozie-oozi-W]
> ACTION[0000031-160803180548580-oozie-oozi-W@spark-approute] Launcher
> exception: PermGen space
> java.lang.OutOfMemoryError: PermGen space
>
> oozie-oozi-W@spark-approute] Launcher exception: PermGen space
> java.lang.OutOfMemoryError: PermGen space
> at java.lang.Class.getDeclaredConstructors0(Native Method)
> at java.lang.Class.privateGetDeclaredConstructors(Class.java:2595)
> at java.lang.Class.getConstructor0(Class.java:2895)
> at java.lang.Class.newInstance(Class.java:354)
> at sun.reflect.MethodAccessorGenerator$1.run(
> MethodAccessorGenerator.java:399)
> at sun.reflect.MethodAccessorGenerator$1.run(
> MethodAccessorGenerator.java:396)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.reflect.MethodAccessorGenerator.generate(
> MethodAccessorGenerator.java:395)
> at sun.reflect.MethodAccessorGenerator.generateSerializationConstruct
> or(MethodAccessorGenerator.java:113)
> at sun.reflect.ReflectionFactory.newConstructorForSerialization
> (ReflectionFactory.java:331)
> at java.io.ObjectStreamClass.getSerializableConstructor(
> ObjectStreamClass.java:1420)
> at java.io.ObjectStreamClass.access$1500(ObjectStreamClass.java:72)
> at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:497)
> at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:472)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:472)
> at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:369)
> at java.io.ObjectOutputStream.writeObject0(
> ObjectOutputStream.java:1133)
> at java.io.ObjectOutputStream.defaultWriteFields(
> ObjectOutputStream.java:1547)
> at java.io.ObjectOutputStream.writeSerialData(
> ObjectOutputStream.java:1508)
> at java.io.ObjectOutputStream.writeOrdinaryObject(
> ObjectOutputStream.java:1431)
> at java.io.ObjectOutputStream.writeObject0(
> ObjectOutputStream.java:1177)
> at java.io.ObjectOutputStream.defaultWriteFields(
> ObjectOutputStream.java:1547)
> at java.io.ObjectOutputStream.writeSerialData(
> ObjectOutputStream.java:1508)
> at java.io.ObjectOutputStream.writeOrdinaryObject(
> ObjectOutputStream.java:1431)
> at java.io.ObjectOutputStream.writeObject0(
> ObjectOutputStream.java:1177)
> at java.io.ObjectOutputStream.defaultWriteFields(
> ObjectOutputStream.java:1547)
> at java.io.ObjectOutputStream.writeSerialData(
> ObjectOutputStream.java:1508)
> at java.io.ObjectOutputStream.writeOrdinaryObject(
> ObjectOutputStream.java:1431)
> at java.io.ObjectOutputStream.writeObject0(
> ObjectOutputStream.java:1177)
> at java.io.ObjectOutputStream.writeArray(ObjectOutputStream.java:1377)
> at java.io.ObjectOutputStream.writeObject0(
> ObjectOutputStream.java:1173)
>
> --
> -$iddhi.
>