You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Haiyang Fu <ha...@gmail.com> on 2014/08/01 07:29:13 UTC
Re: java.lang.OutOfMemoryError: Java heap space
Hi,
here are two tips for you,
1. increase the parallism level
2.increase the driver memory
On Fri, Aug 1, 2014 at 12:58 AM, Sameer Tilak <ss...@live.com> wrote:
> Hi everyone,
> I have the following configuration. I am currently running my app in local
> mode.
>
> val conf = new
> SparkConf().setMaster("local[2]").setAppName("ApproxStrMatch").set("spark.executor.memory",
> "3g").set("spark.storage.memoryFraction", "0.1")
>
> I am getting the following error. I tried setting up spark.executor.memory
> and memory fraction setting, however my UI does not show the increase and I
> still get these errors. I am loading a TSV file from HDFS (around 5 GB).
> Does this mean, I should update these settings and add more memory or is it
> somethign else? Spark master has 24 GB physical memory and workers have 16
> GB, but we are running other services (CDH 5.1) on these nodes as well.
>
> 14/07/31 09:48:09 INFO BlockFetcherIterator$BasicBlockFetcherIterator:
> Getting 2 non-empty blocks out of 2 blocks
> 14/07/31 09:48:09 INFO BlockFetcherIterator$BasicBlockFetcherIterator:
> Getting 2 non-empty blocks out of 2 blocks
> 14/07/31 09:48:09 INFO BlockFetcherIterator$BasicBlockFetcherIterator:
> Started 0 remote fetches in 6 ms
> 14/07/31 09:48:09 INFO BlockFetcherIterator$BasicBlockFetcherIterator:
> Started 0 remote fetches in 6 ms
> 14/07/31 09:48:09 INFO BlockFetcherIterator$BasicBlockFetcherIterator:
> maxBytesInFlight: 50331648, targetRequestSize: 10066329
> 14/07/31 09:48:09 INFO BlockFetcherIterator$BasicBlockFetcherIterator:
> maxBytesInFlight: 50331648, targetRequestSize: 10066329
> 14/07/31 09:48:09 INFO BlockFetcherIterator$BasicBlockFetcherIterator:
> Getting 2 non-empty blocks out of 2 blocks
> 14/07/31 09:48:09 INFO BlockFetcherIterator$BasicBlockFetcherIterator:
> Getting 2 non-empty blocks out of 2 blocks
> 14/07/31 09:48:09 INFO BlockFetcherIterator$BasicBlockFetcherIterator:
> Started 0 remote fetches in 1 ms
> 14/07/31 09:48:09 INFO BlockFetcherIterator$BasicBlockFetcherIterator:
> Started 0 remote fetches in 1 ms
> 14/07/31 09:48:17 ERROR Executor: Exception in task ID 5
> java.lang.OutOfMemoryError: Java heap space
> at java.util.Arrays.copyOf(Arrays.java:2271)
> at
> java.io.ByteArrayOutputStream.toByteArray(ByteArrayOutputStream.java:178)
> at
> org.apache.spark.serializer.JavaSerializerInstance.serialize(JavaSerializer.scala:73)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:197)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
> 14/07/31 09:48:17 ERROR ExecutorUncaughtExceptionHandler: Uncaught
> exception in thread Thread[Executor task launch worker-3,5,main]
> java.lang.OutOfMemoryError: Java heap space
> at java.util.Arrays.copyOf(Arrays.java:2271)
> at
> java.io.ByteArrayOutputStream.toByteArray(ByteArrayOutputStream.java:178)
> at
> org.apache.spark.serializer.JavaSerializerInstance.serialize(JavaSerializer.scala:73)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:197)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
> 14/07/31 09:48:17 WARN TaskSetManager: Lost TID 5 (task 1.0:0)
> 14/07/31 09:48:17 WARN TaskSetManager: Loss was due to
> java.lang.OutOfMemoryError
> java.lang.OutOfMemoryError: Java heap space
> at java.util.Arrays.copyOf(Arrays.java:2271)
> at
> java.io.ByteArrayOutputStream.toByteArray(ByteArrayOutputStream.java:178)
> at
> org.apache.spark.serializer.JavaSerializerInstance.serialize(JavaSerializer.scala:73)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:197)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
> 14/07/31 09:48:17 ERROR TaskSetManager: Task 1.0:0 failed 1 times;
> aborting job
> 14/07/31 09:48:17 INFO TaskSchedulerImpl: Cancelling stage 1
> 14/07/31 09:48:17 INFO DAGScheduler: Failed to run collect at
> ComputeScores.scala:76
> 14/07/31 09:48:17 INFO Executor: Executor is trying to kill task 6
> 14/07/31 09:48:17 INFO TaskSchedulerImpl: Stage 1 was cancelled
>
Re: java.lang.OutOfMemoryError: Java heap space
Posted by Haiyang Fu <ha...@gmail.com>.
http://spark.apache.org/docs/latest/tuning.html#level-of-parallelism
On Fri, Aug 1, 2014 at 1:29 PM, Haiyang Fu <ha...@gmail.com> wrote:
> Hi,
> here are two tips for you,
> 1. increase the parallism level
> 2.increase the driver memory
>
>
> On Fri, Aug 1, 2014 at 12:58 AM, Sameer Tilak <ss...@live.com> wrote:
>
>> Hi everyone,
>> I have the following configuration. I am currently running my app in
>> local mode.
>>
>> val conf = new
>> SparkConf().setMaster("local[2]").setAppName("ApproxStrMatch").set("spark.executor.memory",
>> "3g").set("spark.storage.memoryFraction", "0.1")
>>
>> I am getting the following error. I tried setting up spark.executor.memory
>> and memory fraction setting, however my UI does not show the increase and I
>> still get these errors. I am loading a TSV file from HDFS (around 5 GB).
>> Does this mean, I should update these settings and add more memory or is it
>> somethign else? Spark master has 24 GB physical memory and workers have 16
>> GB, but we are running other services (CDH 5.1) on these nodes as well.
>>
>> 14/07/31 09:48:09 INFO BlockFetcherIterator$BasicBlockFetcherIterator:
>> Getting 2 non-empty blocks out of 2 blocks
>> 14/07/31 09:48:09 INFO BlockFetcherIterator$BasicBlockFetcherIterator:
>> Getting 2 non-empty blocks out of 2 blocks
>> 14/07/31 09:48:09 INFO BlockFetcherIterator$BasicBlockFetcherIterator:
>> Started 0 remote fetches in 6 ms
>> 14/07/31 09:48:09 INFO BlockFetcherIterator$BasicBlockFetcherIterator:
>> Started 0 remote fetches in 6 ms
>> 14/07/31 09:48:09 INFO BlockFetcherIterator$BasicBlockFetcherIterator:
>> maxBytesInFlight: 50331648, targetRequestSize: 10066329
>> 14/07/31 09:48:09 INFO BlockFetcherIterator$BasicBlockFetcherIterator:
>> maxBytesInFlight: 50331648, targetRequestSize: 10066329
>> 14/07/31 09:48:09 INFO BlockFetcherIterator$BasicBlockFetcherIterator:
>> Getting 2 non-empty blocks out of 2 blocks
>> 14/07/31 09:48:09 INFO BlockFetcherIterator$BasicBlockFetcherIterator:
>> Getting 2 non-empty blocks out of 2 blocks
>> 14/07/31 09:48:09 INFO BlockFetcherIterator$BasicBlockFetcherIterator:
>> Started 0 remote fetches in 1 ms
>> 14/07/31 09:48:09 INFO BlockFetcherIterator$BasicBlockFetcherIterator:
>> Started 0 remote fetches in 1 ms
>> 14/07/31 09:48:17 ERROR Executor: Exception in task ID 5
>> java.lang.OutOfMemoryError: Java heap space
>> at java.util.Arrays.copyOf(Arrays.java:2271)
>> at
>> java.io.ByteArrayOutputStream.toByteArray(ByteArrayOutputStream.java:178)
>> at
>> org.apache.spark.serializer.JavaSerializerInstance.serialize(JavaSerializer.scala:73)
>> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:197)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> at java.lang.Thread.run(Thread.java:744)
>> 14/07/31 09:48:17 ERROR ExecutorUncaughtExceptionHandler: Uncaught
>> exception in thread Thread[Executor task launch worker-3,5,main]
>> java.lang.OutOfMemoryError: Java heap space
>> at java.util.Arrays.copyOf(Arrays.java:2271)
>> at
>> java.io.ByteArrayOutputStream.toByteArray(ByteArrayOutputStream.java:178)
>> at
>> org.apache.spark.serializer.JavaSerializerInstance.serialize(JavaSerializer.scala:73)
>> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:197)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> at java.lang.Thread.run(Thread.java:744)
>> 14/07/31 09:48:17 WARN TaskSetManager: Lost TID 5 (task 1.0:0)
>> 14/07/31 09:48:17 WARN TaskSetManager: Loss was due to
>> java.lang.OutOfMemoryError
>> java.lang.OutOfMemoryError: Java heap space
>> at java.util.Arrays.copyOf(Arrays.java:2271)
>> at
>> java.io.ByteArrayOutputStream.toByteArray(ByteArrayOutputStream.java:178)
>> at
>> org.apache.spark.serializer.JavaSerializerInstance.serialize(JavaSerializer.scala:73)
>> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:197)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> at java.lang.Thread.run(Thread.java:744)
>> 14/07/31 09:48:17 ERROR TaskSetManager: Task 1.0:0 failed 1 times;
>> aborting job
>> 14/07/31 09:48:17 INFO TaskSchedulerImpl: Cancelling stage 1
>> 14/07/31 09:48:17 INFO DAGScheduler: Failed to run collect at
>> ComputeScores.scala:76
>> 14/07/31 09:48:17 INFO Executor: Executor is trying to kill task 6
>> 14/07/31 09:48:17 INFO TaskSchedulerImpl: Stage 1 was cancelled
>>
>
>