You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Jean Georges Perrin <jg...@jgp.net> on 2016/07/14 01:14:46 UTC
Memory issue java.lang.OutOfMemoryError: Java heap space
Hi,
I have a Java memory issue with Spark. The same application working on my 8GB Mac crashes on my 72GB Ubuntu server...
I have changed things in the conf file, but it looks like Spark does not care, so I wonder if my issues are with the driver or executor.
I set:
spark.driver.memory 20g
spark.executor.memory 20g
And, whatever I do, the crash is always at the same spot in the app, which makes me think that it is a driver problem.
The exception I get is:
16/07/13 20:36:30 WARN TaskSetManager: Lost task 0.0 in stage 7.0 (TID 208, micha.nc.rr.com): java.lang.OutOfMemoryError: Java heap space
at java.nio.HeapCharBuffer.<init>(HeapCharBuffer.java:57)
at java.nio.CharBuffer.allocate(CharBuffer.java:335)
at java.nio.charset.CharsetDecoder.decode(CharsetDecoder.java:810)
at org.apache.hadoop.io.Text.decode(Text.java:412)
at org.apache.hadoop.io.Text.decode(Text.java:389)
at org.apache.hadoop.io.Text.toString(Text.java:280)
at org.apache.spark.sql.execution.datasources.json.JSONRelation$$anonfun$org$apache$spark$sql$execution$datasources$json$JSONRelation$$createBaseRdd$1.apply(JSONRelation.scala:105)
at org.apache.spark.sql.execution.datasources.json.JSONRelation$$anonfun$org$apache$spark$sql$execution$datasources$json$JSONRelation$$createBaseRdd$1.apply(JSONRelation.scala:105)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
at scala.collection.Iterator$class.foreach(Iterator.scala:727)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
at scala.collection.TraversableOnce$class.foldLeft(TraversableOnce.scala:144)
at scala.collection.AbstractIterator.foldLeft(Iterator.scala:1157)
at scala.collection.TraversableOnce$class.aggregate(TraversableOnce.scala:201)
at scala.collection.AbstractIterator.aggregate(Iterator.scala:1157)
at org.apache.spark.rdd.RDD$$anonfun$treeAggregate$1$$anonfun$23.apply(RDD.scala:1135)
at org.apache.spark.rdd.RDD$$anonfun$treeAggregate$1$$anonfun$23.apply(RDD.scala:1135)
at org.apache.spark.rdd.RDD$$anonfun$treeAggregate$1$$anonfun$24.apply(RDD.scala:1136)
at org.apache.spark.rdd.RDD$$anonfun$treeAggregate$1$$anonfun$24.apply(RDD.scala:1136)
at org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$20.apply(RDD.scala:710)
at org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$20.apply(RDD.scala:710)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
at org.apache.spark.scheduler.Task.run(Task.scala:89)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:227)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
I have set a small memory "dumper" in my app. At the beginning, it says:
** Free ......... 1,413,566
** Allocated .... 1,705,984
** Max .......... 16,495,104
**> Total free ... 16,202,686
Just before the crash, it says:
** Free ......... 1,461,633
** Allocated .... 1,786,880
** Max .......... 16,495,104
**> Total free ... 16,169,857
Re: Memory issue java.lang.OutOfMemoryError: Java heap space
Posted by Jean Georges Perrin <jg...@jgp.net>.
Sure - you see that NC Eatery app is still running, it takes all the cores of the server and 4GB per node, as I have now configured.
Here is the detail of the jobs:
and the storage:
As you can see, there is a lot of things going on...
> On Jul 13, 2016, at 10:53 PM, Chanh Le <gi...@gmail.com> wrote:
>
> Can you show me at Spark UI -> executors tab and storage tab.
> It will show us how many executor was executed and how much memory we use to cache.
>
>
>
>
>> On Jul 14, 2016, at 9:49 AM, Jean Georges Perrin <jgp@jgp.net <ma...@jgp.net>> wrote:
>>
>> I use it as a standalone cluster.
>>
>> I run it through start-master, then start-slave. I only have one slave now, but I will probably have a few soon.
>>
>> The "application" is run on a separate box.
>>
>> When everything was running on my mac, i was in local mode, but i never setup anything in local mode. Going "production" was a little more complex that I thought.
>>
>>> On Jul 13, 2016, at 10:35 PM, Chanh Le <giaosudau@gmail.com <ma...@gmail.com>> wrote:
>>>
>>> Hi Jean,
>>> How do you run your Spark Application? Local Mode, Cluster Mode?
>>> If you run in local mode did you use —driver-memory and —executor-memory because in local mode your setting about executor and driver didn’t work that you expected.
>>>
>>>
>>>
>>>
>>>> On Jul 14, 2016, at 8:43 AM, Jean Georges Perrin <jgp@jgp.net <ma...@jgp.net>> wrote:
>>>>
>>>> Looks like replacing the setExecutorEnv() by set() did the trick... let's see how fast it'll process my 50x 10ˆ15 data points...
>>>>
>>>>> On Jul 13, 2016, at 9:24 PM, Jean Georges Perrin <jgp@jgp.net <ma...@jgp.net>> wrote:
>>>>>
>>>>> I have added:
>>>>>
>>>>> SparkConf conf = new SparkConf().setAppName("app").setExecutorEnv("spark.executor.memory", "8g")
>>>>> .setMaster("spark://10.0.100.120:7077 <spark://10.0.100.120:7077>");
>>>>>
>>>>> but it did not change a thing
>>>>>
>>>>>> On Jul 13, 2016, at 9:14 PM, Jean Georges Perrin <jgp@jgp.net <ma...@jgp.net>> wrote:
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I have a Java memory issue with Spark. The same application working on my 8GB Mac crashes on my 72GB Ubuntu server...
>>>>>>
>>>>>> I have changed things in the conf file, but it looks like Spark does not care, so I wonder if my issues are with the driver or executor.
>>>>>>
>>>>>> I set:
>>>>>>
>>>>>> spark.driver.memory 20g
>>>>>> spark.executor.memory 20g
>>>>>> And, whatever I do, the crash is always at the same spot in the app, which makes me think that it is a driver problem.
>>>>>>
>>>>>> The exception I get is:
>>>>>>
>>>>>> 16/07/13 20:36:30 WARN TaskSetManager: Lost task 0.0 in stage 7.0 (TID 208, micha.nc.rr.com): java.lang.OutOfMemoryError: Java heap space
>>>>>> at java.nio.HeapCharBuffer.<init>(HeapCharBuffer.java:57)
>>>>>> at java.nio.CharBuffer.allocate(CharBuffer.java:335)
>>>>>> at java.nio.charset.CharsetDecoder.decode(CharsetDecoder.java:810)
>>>>>> at org.apache.hadoop.io.Text.decode(Text.java:412)
>>>>>> at org.apache.hadoop.io.Text.decode(Text.java:389)
>>>>>> at org.apache.hadoop.io.Text.toString(Text.java:280)
>>>>>> at org.apache.spark.sql.execution.datasources.json.JSONRelation$$anonfun$org$apache$spark$sql$execution$datasources$json$JSONRelation$$createBaseRdd$1.apply(JSONRelation.scala:105)
>>>>>> at org.apache.spark.sql.execution.datasources.json.JSONRelation$$anonfun$org$apache$spark$sql$execution$datasources$json$JSONRelation$$createBaseRdd$1.apply(JSONRelation.scala:105)
>>>>>> at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
>>>>>> at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
>>>>>> at scala.collection.Iterator$class.foreach(Iterator.scala:727)
>>>>>> at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
>>>>>> at scala.collection.TraversableOnce$class.foldLeft(TraversableOnce.scala:144)
>>>>>> at scala.collection.AbstractIterator.foldLeft(Iterator.scala:1157)
>>>>>> at scala.collection.TraversableOnce$class.aggregate(TraversableOnce.scala:201)
>>>>>> at scala.collection.AbstractIterator.aggregate(Iterator.scala:1157)
>>>>>> at org.apache.spark.rdd.RDD$$anonfun$treeAggregate$1$$anonfun$23.apply(RDD.scala:1135)
>>>>>> at org.apache.spark.rdd.RDD$$anonfun$treeAggregate$1$$anonfun$23.apply(RDD.scala:1135)
>>>>>> at org.apache.spark.rdd.RDD$$anonfun$treeAggregate$1$$anonfun$24.apply(RDD.scala:1136)
>>>>>> at org.apache.spark.rdd.RDD$$anonfun$treeAggregate$1$$anonfun$24.apply(RDD.scala:1136)
>>>>>> at org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$20.apply(RDD.scala:710)
>>>>>> at org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$20.apply(RDD.scala:710)
>>>>>> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
>>>>>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
>>>>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
>>>>>> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
>>>>>> at org.apache.spark.scheduler.Task.run(Task.scala:89)
>>>>>> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:227)
>>>>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>>
>>>>>> I have set a small memory "dumper" in my app. At the beginning, it says:
>>>>>>
>>>>>> ** Free ......... 1,413,566
>>>>>> ** Allocated .... 1,705,984
>>>>>> ** Max .......... 16,495,104
>>>>>> **> Total free ... 16,202,686
>>>>>> Just before the crash, it says:
>>>>>>
>>>>>> ** Free ......... 1,461,633
>>>>>> ** Allocated .... 1,786,880
>>>>>> ** Max .......... 16,495,104
>>>>>> **> Total free ... 16,169,857
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Memory issue java.lang.OutOfMemoryError: Java heap space
Posted by Chanh Le <gi...@gmail.com>.
Can you show me at Spark UI -> executors tab and storage tab.
It will show us how many executor was executed and how much memory we use to cache.
> On Jul 14, 2016, at 9:49 AM, Jean Georges Perrin <jg...@jgp.net> wrote:
>
> I use it as a standalone cluster.
>
> I run it through start-master, then start-slave. I only have one slave now, but I will probably have a few soon.
>
> The "application" is run on a separate box.
>
> When everything was running on my mac, i was in local mode, but i never setup anything in local mode. Going "production" was a little more complex that I thought.
>
>> On Jul 13, 2016, at 10:35 PM, Chanh Le <giaosudau@gmail.com <ma...@gmail.com>> wrote:
>>
>> Hi Jean,
>> How do you run your Spark Application? Local Mode, Cluster Mode?
>> If you run in local mode did you use —driver-memory and —executor-memory because in local mode your setting about executor and driver didn’t work that you expected.
>>
>>
>>
>>
>>> On Jul 14, 2016, at 8:43 AM, Jean Georges Perrin <jgp@jgp.net <ma...@jgp.net>> wrote:
>>>
>>> Looks like replacing the setExecutorEnv() by set() did the trick... let's see how fast it'll process my 50x 10ˆ15 data points...
>>>
>>>> On Jul 13, 2016, at 9:24 PM, Jean Georges Perrin <jgp@jgp.net <ma...@jgp.net>> wrote:
>>>>
>>>> I have added:
>>>>
>>>> SparkConf conf = new SparkConf().setAppName("app").setExecutorEnv("spark.executor.memory", "8g")
>>>> .setMaster("spark://10.0.100.120:7077 <spark://10.0.100.120:7077>");
>>>>
>>>> but it did not change a thing
>>>>
>>>>> On Jul 13, 2016, at 9:14 PM, Jean Georges Perrin <jgp@jgp.net <ma...@jgp.net>> wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> I have a Java memory issue with Spark. The same application working on my 8GB Mac crashes on my 72GB Ubuntu server...
>>>>>
>>>>> I have changed things in the conf file, but it looks like Spark does not care, so I wonder if my issues are with the driver or executor.
>>>>>
>>>>> I set:
>>>>>
>>>>> spark.driver.memory 20g
>>>>> spark.executor.memory 20g
>>>>> And, whatever I do, the crash is always at the same spot in the app, which makes me think that it is a driver problem.
>>>>>
>>>>> The exception I get is:
>>>>>
>>>>> 16/07/13 20:36:30 WARN TaskSetManager: Lost task 0.0 in stage 7.0 (TID 208, micha.nc.rr.com): java.lang.OutOfMemoryError: Java heap space
>>>>> at java.nio.HeapCharBuffer.<init>(HeapCharBuffer.java:57)
>>>>> at java.nio.CharBuffer.allocate(CharBuffer.java:335)
>>>>> at java.nio.charset.CharsetDecoder.decode(CharsetDecoder.java:810)
>>>>> at org.apache.hadoop.io.Text.decode(Text.java:412)
>>>>> at org.apache.hadoop.io.Text.decode(Text.java:389)
>>>>> at org.apache.hadoop.io.Text.toString(Text.java:280)
>>>>> at org.apache.spark.sql.execution.datasources.json.JSONRelation$$anonfun$org$apache$spark$sql$execution$datasources$json$JSONRelation$$createBaseRdd$1.apply(JSONRelation.scala:105)
>>>>> at org.apache.spark.sql.execution.datasources.json.JSONRelation$$anonfun$org$apache$spark$sql$execution$datasources$json$JSONRelation$$createBaseRdd$1.apply(JSONRelation.scala:105)
>>>>> at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
>>>>> at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
>>>>> at scala.collection.Iterator$class.foreach(Iterator.scala:727)
>>>>> at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
>>>>> at scala.collection.TraversableOnce$class.foldLeft(TraversableOnce.scala:144)
>>>>> at scala.collection.AbstractIterator.foldLeft(Iterator.scala:1157)
>>>>> at scala.collection.TraversableOnce$class.aggregate(TraversableOnce.scala:201)
>>>>> at scala.collection.AbstractIterator.aggregate(Iterator.scala:1157)
>>>>> at org.apache.spark.rdd.RDD$$anonfun$treeAggregate$1$$anonfun$23.apply(RDD.scala:1135)
>>>>> at org.apache.spark.rdd.RDD$$anonfun$treeAggregate$1$$anonfun$23.apply(RDD.scala:1135)
>>>>> at org.apache.spark.rdd.RDD$$anonfun$treeAggregate$1$$anonfun$24.apply(RDD.scala:1136)
>>>>> at org.apache.spark.rdd.RDD$$anonfun$treeAggregate$1$$anonfun$24.apply(RDD.scala:1136)
>>>>> at org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$20.apply(RDD.scala:710)
>>>>> at org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$20.apply(RDD.scala:710)
>>>>> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
>>>>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
>>>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
>>>>> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
>>>>> at org.apache.spark.scheduler.Task.run(Task.scala:89)
>>>>> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:227)
>>>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> I have set a small memory "dumper" in my app. At the beginning, it says:
>>>>>
>>>>> ** Free ......... 1,413,566
>>>>> ** Allocated .... 1,705,984
>>>>> ** Max .......... 16,495,104
>>>>> **> Total free ... 16,202,686
>>>>> Just before the crash, it says:
>>>>>
>>>>> ** Free ......... 1,461,633
>>>>> ** Allocated .... 1,786,880
>>>>> ** Max .......... 16,495,104
>>>>> **> Total free ... 16,169,857
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Memory issue java.lang.OutOfMemoryError: Java heap space
Posted by Jean Georges Perrin <jg...@jgp.net>.
I use it as a standalone cluster.
I run it through start-master, then start-slave. I only have one slave now, but I will probably have a few soon.
The "application" is run on a separate box.
When everything was running on my mac, i was in local mode, but i never setup anything in local mode. Going "production" was a little more complex that I thought.
> On Jul 13, 2016, at 10:35 PM, Chanh Le <gi...@gmail.com> wrote:
>
> Hi Jean,
> How do you run your Spark Application? Local Mode, Cluster Mode?
> If you run in local mode did you use —driver-memory and —executor-memory because in local mode your setting about executor and driver didn’t work that you expected.
>
>
>
>
>> On Jul 14, 2016, at 8:43 AM, Jean Georges Perrin <jgp@jgp.net <ma...@jgp.net>> wrote:
>>
>> Looks like replacing the setExecutorEnv() by set() did the trick... let's see how fast it'll process my 50x 10ˆ15 data points...
>>
>>> On Jul 13, 2016, at 9:24 PM, Jean Georges Perrin <jgp@jgp.net <ma...@jgp.net>> wrote:
>>>
>>> I have added:
>>>
>>> SparkConf conf = new SparkConf().setAppName("app").setExecutorEnv("spark.executor.memory", "8g")
>>> .setMaster("spark://10.0.100.120:7077 <spark://10.0.100.120:7077>");
>>>
>>> but it did not change a thing
>>>
>>>> On Jul 13, 2016, at 9:14 PM, Jean Georges Perrin <jgp@jgp.net <ma...@jgp.net>> wrote:
>>>>
>>>> Hi,
>>>>
>>>> I have a Java memory issue with Spark. The same application working on my 8GB Mac crashes on my 72GB Ubuntu server...
>>>>
>>>> I have changed things in the conf file, but it looks like Spark does not care, so I wonder if my issues are with the driver or executor.
>>>>
>>>> I set:
>>>>
>>>> spark.driver.memory 20g
>>>> spark.executor.memory 20g
>>>> And, whatever I do, the crash is always at the same spot in the app, which makes me think that it is a driver problem.
>>>>
>>>> The exception I get is:
>>>>
>>>> 16/07/13 20:36:30 WARN TaskSetManager: Lost task 0.0 in stage 7.0 (TID 208, micha.nc.rr.com): java.lang.OutOfMemoryError: Java heap space
>>>> at java.nio.HeapCharBuffer.<init>(HeapCharBuffer.java:57)
>>>> at java.nio.CharBuffer.allocate(CharBuffer.java:335)
>>>> at java.nio.charset.CharsetDecoder.decode(CharsetDecoder.java:810)
>>>> at org.apache.hadoop.io.Text.decode(Text.java:412)
>>>> at org.apache.hadoop.io.Text.decode(Text.java:389)
>>>> at org.apache.hadoop.io.Text.toString(Text.java:280)
>>>> at org.apache.spark.sql.execution.datasources.json.JSONRelation$$anonfun$org$apache$spark$sql$execution$datasources$json$JSONRelation$$createBaseRdd$1.apply(JSONRelation.scala:105)
>>>> at org.apache.spark.sql.execution.datasources.json.JSONRelation$$anonfun$org$apache$spark$sql$execution$datasources$json$JSONRelation$$createBaseRdd$1.apply(JSONRelation.scala:105)
>>>> at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
>>>> at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
>>>> at scala.collection.Iterator$class.foreach(Iterator.scala:727)
>>>> at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
>>>> at scala.collection.TraversableOnce$class.foldLeft(TraversableOnce.scala:144)
>>>> at scala.collection.AbstractIterator.foldLeft(Iterator.scala:1157)
>>>> at scala.collection.TraversableOnce$class.aggregate(TraversableOnce.scala:201)
>>>> at scala.collection.AbstractIterator.aggregate(Iterator.scala:1157)
>>>> at org.apache.spark.rdd.RDD$$anonfun$treeAggregate$1$$anonfun$23.apply(RDD.scala:1135)
>>>> at org.apache.spark.rdd.RDD$$anonfun$treeAggregate$1$$anonfun$23.apply(RDD.scala:1135)
>>>> at org.apache.spark.rdd.RDD$$anonfun$treeAggregate$1$$anonfun$24.apply(RDD.scala:1136)
>>>> at org.apache.spark.rdd.RDD$$anonfun$treeAggregate$1$$anonfun$24.apply(RDD.scala:1136)
>>>> at org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$20.apply(RDD.scala:710)
>>>> at org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$20.apply(RDD.scala:710)
>>>> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
>>>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
>>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
>>>> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
>>>> at org.apache.spark.scheduler.Task.run(Task.scala:89)
>>>> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:227)
>>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>>
>>>> I have set a small memory "dumper" in my app. At the beginning, it says:
>>>>
>>>> ** Free ......... 1,413,566
>>>> ** Allocated .... 1,705,984
>>>> ** Max .......... 16,495,104
>>>> **> Total free ... 16,202,686
>>>> Just before the crash, it says:
>>>>
>>>> ** Free ......... 1,461,633
>>>> ** Allocated .... 1,786,880
>>>> ** Max .......... 16,495,104
>>>> **> Total free ... 16,169,857
>>>>
>>>>
>>>>
>>>>
>>>
>>
>
Re: Memory issue java.lang.OutOfMemoryError: Java heap space
Posted by Chanh Le <gi...@gmail.com>.
Hi Jean,
How do you run your Spark Application? Local Mode, Cluster Mode?
If you run in local mode did you use —driver-memory and —executor-memory because in local mode your setting about executor and driver didn’t work that you expected.
> On Jul 14, 2016, at 8:43 AM, Jean Georges Perrin <jg...@jgp.net> wrote:
>
> Looks like replacing the setExecutorEnv() by set() did the trick... let's see how fast it'll process my 50x 10ˆ15 data points...
>
>> On Jul 13, 2016, at 9:24 PM, Jean Georges Perrin <jgp@jgp.net <ma...@jgp.net>> wrote:
>>
>> I have added:
>>
>> SparkConf conf = new SparkConf().setAppName("app").setExecutorEnv("spark.executor.memory", "8g")
>> .setMaster("spark://10.0.100.120:7077 <spark://10.0.100.120:7077>");
>>
>> but it did not change a thing
>>
>>> On Jul 13, 2016, at 9:14 PM, Jean Georges Perrin <jgp@jgp.net <ma...@jgp.net>> wrote:
>>>
>>> Hi,
>>>
>>> I have a Java memory issue with Spark. The same application working on my 8GB Mac crashes on my 72GB Ubuntu server...
>>>
>>> I have changed things in the conf file, but it looks like Spark does not care, so I wonder if my issues are with the driver or executor.
>>>
>>> I set:
>>>
>>> spark.driver.memory 20g
>>> spark.executor.memory 20g
>>> And, whatever I do, the crash is always at the same spot in the app, which makes me think that it is a driver problem.
>>>
>>> The exception I get is:
>>>
>>> 16/07/13 20:36:30 WARN TaskSetManager: Lost task 0.0 in stage 7.0 (TID 208, micha.nc.rr.com): java.lang.OutOfMemoryError: Java heap space
>>> at java.nio.HeapCharBuffer.<init>(HeapCharBuffer.java:57)
>>> at java.nio.CharBuffer.allocate(CharBuffer.java:335)
>>> at java.nio.charset.CharsetDecoder.decode(CharsetDecoder.java:810)
>>> at org.apache.hadoop.io.Text.decode(Text.java:412)
>>> at org.apache.hadoop.io.Text.decode(Text.java:389)
>>> at org.apache.hadoop.io.Text.toString(Text.java:280)
>>> at org.apache.spark.sql.execution.datasources.json.JSONRelation$$anonfun$org$apache$spark$sql$execution$datasources$json$JSONRelation$$createBaseRdd$1.apply(JSONRelation.scala:105)
>>> at org.apache.spark.sql.execution.datasources.json.JSONRelation$$anonfun$org$apache$spark$sql$execution$datasources$json$JSONRelation$$createBaseRdd$1.apply(JSONRelation.scala:105)
>>> at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
>>> at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
>>> at scala.collection.Iterator$class.foreach(Iterator.scala:727)
>>> at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
>>> at scala.collection.TraversableOnce$class.foldLeft(TraversableOnce.scala:144)
>>> at scala.collection.AbstractIterator.foldLeft(Iterator.scala:1157)
>>> at scala.collection.TraversableOnce$class.aggregate(TraversableOnce.scala:201)
>>> at scala.collection.AbstractIterator.aggregate(Iterator.scala:1157)
>>> at org.apache.spark.rdd.RDD$$anonfun$treeAggregate$1$$anonfun$23.apply(RDD.scala:1135)
>>> at org.apache.spark.rdd.RDD$$anonfun$treeAggregate$1$$anonfun$23.apply(RDD.scala:1135)
>>> at org.apache.spark.rdd.RDD$$anonfun$treeAggregate$1$$anonfun$24.apply(RDD.scala:1136)
>>> at org.apache.spark.rdd.RDD$$anonfun$treeAggregate$1$$anonfun$24.apply(RDD.scala:1136)
>>> at org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$20.apply(RDD.scala:710)
>>> at org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$20.apply(RDD.scala:710)
>>> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
>>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
>>> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
>>> at org.apache.spark.scheduler.Task.run(Task.scala:89)
>>> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:227)
>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> at java.lang.Thread.run(Thread.java:745)
>>>
>>> I have set a small memory "dumper" in my app. At the beginning, it says:
>>>
>>> ** Free ......... 1,413,566
>>> ** Allocated .... 1,705,984
>>> ** Max .......... 16,495,104
>>> **> Total free ... 16,202,686
>>> Just before the crash, it says:
>>>
>>> ** Free ......... 1,461,633
>>> ** Allocated .... 1,786,880
>>> ** Max .......... 16,495,104
>>> **> Total free ... 16,169,857
>>>
>>>
>>>
>>>
>>
>
Re: Memory issue java.lang.OutOfMemoryError: Java heap space
Posted by Jean Georges Perrin <jg...@jgp.net>.
Looks like replacing the setExecutorEnv() by set() did the trick... let's see how fast it'll process my 50x 10ˆ15 data points...
> On Jul 13, 2016, at 9:24 PM, Jean Georges Perrin <jg...@jgp.net> wrote:
>
> I have added:
>
> SparkConf conf = new SparkConf().setAppName("app").setExecutorEnv("spark.executor.memory", "8g")
> .setMaster("spark://10.0.100.120:7077 <spark://10.0.100.120:7077>");
>
> but it did not change a thing
>
>> On Jul 13, 2016, at 9:14 PM, Jean Georges Perrin <jgp@jgp.net <ma...@jgp.net>> wrote:
>>
>> Hi,
>>
>> I have a Java memory issue with Spark. The same application working on my 8GB Mac crashes on my 72GB Ubuntu server...
>>
>> I have changed things in the conf file, but it looks like Spark does not care, so I wonder if my issues are with the driver or executor.
>>
>> I set:
>>
>> spark.driver.memory 20g
>> spark.executor.memory 20g
>> And, whatever I do, the crash is always at the same spot in the app, which makes me think that it is a driver problem.
>>
>> The exception I get is:
>>
>> 16/07/13 20:36:30 WARN TaskSetManager: Lost task 0.0 in stage 7.0 (TID 208, micha.nc.rr.com): java.lang.OutOfMemoryError: Java heap space
>> at java.nio.HeapCharBuffer.<init>(HeapCharBuffer.java:57)
>> at java.nio.CharBuffer.allocate(CharBuffer.java:335)
>> at java.nio.charset.CharsetDecoder.decode(CharsetDecoder.java:810)
>> at org.apache.hadoop.io.Text.decode(Text.java:412)
>> at org.apache.hadoop.io.Text.decode(Text.java:389)
>> at org.apache.hadoop.io.Text.toString(Text.java:280)
>> at org.apache.spark.sql.execution.datasources.json.JSONRelation$$anonfun$org$apache$spark$sql$execution$datasources$json$JSONRelation$$createBaseRdd$1.apply(JSONRelation.scala:105)
>> at org.apache.spark.sql.execution.datasources.json.JSONRelation$$anonfun$org$apache$spark$sql$execution$datasources$json$JSONRelation$$createBaseRdd$1.apply(JSONRelation.scala:105)
>> at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
>> at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
>> at scala.collection.Iterator$class.foreach(Iterator.scala:727)
>> at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
>> at scala.collection.TraversableOnce$class.foldLeft(TraversableOnce.scala:144)
>> at scala.collection.AbstractIterator.foldLeft(Iterator.scala:1157)
>> at scala.collection.TraversableOnce$class.aggregate(TraversableOnce.scala:201)
>> at scala.collection.AbstractIterator.aggregate(Iterator.scala:1157)
>> at org.apache.spark.rdd.RDD$$anonfun$treeAggregate$1$$anonfun$23.apply(RDD.scala:1135)
>> at org.apache.spark.rdd.RDD$$anonfun$treeAggregate$1$$anonfun$23.apply(RDD.scala:1135)
>> at org.apache.spark.rdd.RDD$$anonfun$treeAggregate$1$$anonfun$24.apply(RDD.scala:1136)
>> at org.apache.spark.rdd.RDD$$anonfun$treeAggregate$1$$anonfun$24.apply(RDD.scala:1136)
>> at org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$20.apply(RDD.scala:710)
>> at org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$20.apply(RDD.scala:710)
>> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
>> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
>> at org.apache.spark.scheduler.Task.run(Task.scala:89)
>> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:227)
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> I have set a small memory "dumper" in my app. At the beginning, it says:
>>
>> ** Free ......... 1,413,566
>> ** Allocated .... 1,705,984
>> ** Max .......... 16,495,104
>> **> Total free ... 16,202,686
>> Just before the crash, it says:
>>
>> ** Free ......... 1,461,633
>> ** Allocated .... 1,786,880
>> ** Max .......... 16,495,104
>> **> Total free ... 16,169,857
>>
>>
>>
>>
>
Re: Memory issue java.lang.OutOfMemoryError: Java heap space
Posted by Jean Georges Perrin <jg...@jgp.net>.
I have added:
SparkConf conf = new SparkConf().setAppName("app").setExecutorEnv("spark.executor.memory", "8g")
.setMaster("spark://10.0.100.120:7077");
but it did not change a thing
> On Jul 13, 2016, at 9:14 PM, Jean Georges Perrin <jg...@jgp.net> wrote:
>
> Hi,
>
> I have a Java memory issue with Spark. The same application working on my 8GB Mac crashes on my 72GB Ubuntu server...
>
> I have changed things in the conf file, but it looks like Spark does not care, so I wonder if my issues are with the driver or executor.
>
> I set:
>
> spark.driver.memory 20g
> spark.executor.memory 20g
> And, whatever I do, the crash is always at the same spot in the app, which makes me think that it is a driver problem.
>
> The exception I get is:
>
> 16/07/13 20:36:30 WARN TaskSetManager: Lost task 0.0 in stage 7.0 (TID 208, micha.nc.rr.com): java.lang.OutOfMemoryError: Java heap space
> at java.nio.HeapCharBuffer.<init>(HeapCharBuffer.java:57)
> at java.nio.CharBuffer.allocate(CharBuffer.java:335)
> at java.nio.charset.CharsetDecoder.decode(CharsetDecoder.java:810)
> at org.apache.hadoop.io.Text.decode(Text.java:412)
> at org.apache.hadoop.io.Text.decode(Text.java:389)
> at org.apache.hadoop.io.Text.toString(Text.java:280)
> at org.apache.spark.sql.execution.datasources.json.JSONRelation$$anonfun$org$apache$spark$sql$execution$datasources$json$JSONRelation$$createBaseRdd$1.apply(JSONRelation.scala:105)
> at org.apache.spark.sql.execution.datasources.json.JSONRelation$$anonfun$org$apache$spark$sql$execution$datasources$json$JSONRelation$$createBaseRdd$1.apply(JSONRelation.scala:105)
> at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
> at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
> at scala.collection.Iterator$class.foreach(Iterator.scala:727)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
> at scala.collection.TraversableOnce$class.foldLeft(TraversableOnce.scala:144)
> at scala.collection.AbstractIterator.foldLeft(Iterator.scala:1157)
> at scala.collection.TraversableOnce$class.aggregate(TraversableOnce.scala:201)
> at scala.collection.AbstractIterator.aggregate(Iterator.scala:1157)
> at org.apache.spark.rdd.RDD$$anonfun$treeAggregate$1$$anonfun$23.apply(RDD.scala:1135)
> at org.apache.spark.rdd.RDD$$anonfun$treeAggregate$1$$anonfun$23.apply(RDD.scala:1135)
> at org.apache.spark.rdd.RDD$$anonfun$treeAggregate$1$$anonfun$24.apply(RDD.scala:1136)
> at org.apache.spark.rdd.RDD$$anonfun$treeAggregate$1$$anonfun$24.apply(RDD.scala:1136)
> at org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$20.apply(RDD.scala:710)
> at org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$20.apply(RDD.scala:710)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
> at org.apache.spark.scheduler.Task.run(Task.scala:89)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:227)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> I have set a small memory "dumper" in my app. At the beginning, it says:
>
> ** Free ......... 1,413,566
> ** Allocated .... 1,705,984
> ** Max .......... 16,495,104
> **> Total free ... 16,202,686
> Just before the crash, it says:
>
> ** Free ......... 1,461,633
> ** Allocated .... 1,786,880
> ** Max .......... 16,495,104
> **> Total free ... 16,169,857
>
>
>
>