You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by StanZhai <ma...@zhaishidan.cn> on 2017/01/22 08:57:52 UTC
Executors exceed maximum memory defined with `--executor-memory` in
Spark 2.1.0
Hi all,
We just upgraded our Spark from 1.6.2 to 2.1.0.
Our Spark application is started by spark-submit with config of
`--executor-memory 35G` in standalone model, but the actual use of memory up
to 65G after a full gc(jmap -histo:live $pid) as follow:
test@c6 ~ $ ps aux | grep CoarseGrainedExecutorBackend
test 181941 181 34.7 94665384 68836752 ? Sl 09:25 711:21
/home/test/service/jdk/bin/java -cp
/home/test/service/hadoop/share/hadoop/common/hadoop-lzo-0.4.20-SNAPSHOT.jar:/home/test/service/hadoop/share/hadoop/common/hadoop-lzo-0.4.20-SNAPSHOT.jar:/home/test/service/spark/conf/:/home/test/service/spark/jars/*:/home/test/service/hadoop/etc/hadoop/
-Xmx35840M -Dspark.driver.port=47781 -XX:+PrintGCDetails
-XX:+PrintGCDateStamps -Xloggc:./gc.log -verbose:gc
org.apache.spark.executor.CoarseGrainedExecutorBackend --driver-url
spark://CoarseGrainedScheduler@xxx.xxx.xxx.xxx:47781 --executor-id 1
--hostname test-192 --cores 36 --app-id app-20170122092509-0017 --worker-url
spark://Worker@test-192:33890
Our Spark jobs are all sql.
The exceed memory looks like off-heap memory, but the default value of
`spark.memory.offHeap.enabled` is `false`.
We didn't find the problem in Spark 1.6.x, what causes this in Spark 2.1.0?
Any help is greatly appreicated!
Best,
Stan
--
View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/Executors-exceed-maximum-memory-defined-with-executor-memory-in-Spark-2-1-0-tp20697.html
Sent from the Apache Spark Developers List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: Executors exceed maximum memory defined with
`--executor-memory` in Spark 2.1.0
Posted by StanZhai <ma...@zhaishidan.cn>.
I've filed a JIRA about this problem.
https://issues.apache.org/jira/browse/SPARK-19532
<https://issues.apache.org/jira/browse/SPARK-19532>
I've tried to set `spark.speculation` to `false`, but the off-heap also
exceed about 10G after triggering a FullGC to the Executor
process(--executor-memory 30G), as follow:
test@test Online ~ $ ps aux | grep CoarseGrainedExecutorBackend
test 105371 106 21.5 67325492 42621992 ? Sl 15:20 55:14
/home/test/service/jdk/bin/java -cp
/home/test/service/hadoop/share/hadoop/common/hadoop-lzo-0.4.20-SNAPSHOT.jar:/home/test/service/hadoop/share/hadoop/common/hadoop-lzo-0.4.20-SNAPSHOT.jar:/home/test/service/spark/conf/:/home/test/service/spark/jars/*:/home/test/service/hadoop/etc/hadoop/
-Xmx30720M -Dspark.driver.port=9835 -Dtag=spark_2_1_test -XX:+PrintGCDetails
-XX:+PrintGCDateStamps -Xloggc:./gc.log -verbose:gc
org.apache.spark.executor.CoarseGrainedExecutorBackend --driver-url
spark://CoarseGrainedScheduler@172.16.34.235:9835 --executor-id 4 --hostname
test-192 --cores 36 --app-id app-20170213152037-0043 --worker-url
spark://Worker@test-192:33890
So, I think these are also other reasons for this problem.
We have been trying to upgrade our Spark from the releasing of Spark 2.1.0.
This version is unstable and not available for us because of the memory
problems, we should pay attention to this.
StanZhai wrote
> From thread dump page of Executor of WebUI, I found that there are about
> 1300 threads named "DataStreamer for file
> /test/data/test_temp/_temporary/0/_temporary/attempt_20170207172435_80750_m_000069_1/part-00069-690407af-0900-46b1-9590-a6d6c696fe68.snappy.parquet"
> in TIMED_WAITING state like this:
<http://apache-spark-developers-list.1001551.n3.nabble.com/file/n20881/QQ20170207-212340.png>
>
> The exceed off-heap memory may be caused by these abnormal threads.
>
> This problem occurs only when writing data to the Hadoop(tasks may be
> killed by Executor during writing).
>
> Could this be related to
> https://issues.apache.org/jira/browse/HDFS-9812
> <https://issues.apache.org/jira/browse/HDFS-9812>
> ?
>
> It's may be a bug of Spark when killing tasks during writing data. What's
> the difference between Spark 1.6.x and 2.1.0 in killing tasks?
>
> This is a critical issue, I've worked on this for days.
>
> Any help?
--
View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/Executors-exceed-maximum-memory-defined-with-executor-memory-in-Spark-2-1-0-tp20697p20935.html
Sent from the Apache Spark Developers List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: Executors exceed maximum memory defined with
`--executor-memory` in Spark 2.1.0
Posted by StanZhai <ma...@zhaishidan.cn>.
From thread dump page of Executor of WebUI, I found that there are about 1300
threads named "DataStreamer for file
/test/data/test_temp/_temporary/0/_temporary/attempt_20170207172435_80750_m_000069_1/part-00069-690407af-0900-46b1-9590-a6d6c696fe68.snappy.parquet"
in TIMED_WAITING state like this:
<http://apache-spark-developers-list.1001551.n3.nabble.com/file/n20881/QQ20170207-212340.png>
The exceed off-heap memory may be caused by these abnormal threads.
This problem occurs only when writing data to the Hadoop(tasks may be killed
by Executor during writing).
Could this be related to https://issues.apache.org/jira/browse/HDFS-9812
<https://issues.apache.org/jira/browse/HDFS-9812> ?
It's may be a bug of Spark when killing tasks during writing data. What's
the difference between Spark 1.6.x and 2.1.0 in killing tasks?
This is a critical issue, I've worked on this for days.
Any help?
--
View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/Executors-exceed-maximum-memory-defined-with-executor-memory-in-Spark-2-1-0-tp20697p20881.html
Sent from the Apache Spark Developers List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: Executors exceed maximum memory defined with `--executor-memory`
in Spark 2.1.0
Posted by Jacek Laskowski <ja...@japila.pl>.
Hi,
Just to throw few zlotys to the conversation, I believe that Spark
Standalone does not enforce any memory checks to limit or even kill
executors beyond requested memory (like YARN). I also found that
memory does not have much of use while scheduling tasks and CPU
matters only.
My understanding of `spark.memory.offHeap.enabled` is `false` is that
it does not disable off heap memory used in Java NIO for buffers in
shuffling, RPC, etc. so the memory is always (?) more than you request
for mx using executor-memory.
Pozdrawiam,
Jacek Laskowski
----
https://medium.com/@jaceklaskowski/
Mastering Apache Spark 2.0 https://bit.ly/mastering-apache-spark
Follow me at https://twitter.com/jaceklaskowski
On Sun, Jan 22, 2017 at 9:57 AM, StanZhai <ma...@zhaishidan.cn> wrote:
> Hi all,
>
> We just upgraded our Spark from 1.6.2 to 2.1.0.
>
> Our Spark application is started by spark-submit with config of
> `--executor-memory 35G` in standalone model, but the actual use of memory up
> to 65G after a full gc(jmap -histo:live $pid) as follow:
>
> test@c6 ~ $ ps aux | grep CoarseGrainedExecutorBackend
> test 181941 181 34.7 94665384 68836752 ? Sl 09:25 711:21
> /home/test/service/jdk/bin/java -cp
> /home/test/service/hadoop/share/hadoop/common/hadoop-lzo-0.4.20-SNAPSHOT.jar:/home/test/service/hadoop/share/hadoop/common/hadoop-lzo-0.4.20-SNAPSHOT.jar:/home/test/service/spark/conf/:/home/test/service/spark/jars/*:/home/test/service/hadoop/etc/hadoop/
> -Xmx35840M -Dspark.driver.port=47781 -XX:+PrintGCDetails
> -XX:+PrintGCDateStamps -Xloggc:./gc.log -verbose:gc
> org.apache.spark.executor.CoarseGrainedExecutorBackend --driver-url
> spark://CoarseGrainedScheduler@xxx.xxx.xxx.xxx:47781 --executor-id 1
> --hostname test-192 --cores 36 --app-id app-20170122092509-0017 --worker-url
> spark://Worker@test-192:33890
>
> Our Spark jobs are all sql.
>
> The exceed memory looks like off-heap memory, but the default value of
> `spark.memory.offHeap.enabled` is `false`.
>
> We didn't find the problem in Spark 1.6.x, what causes this in Spark 2.1.0?
>
> Any help is greatly appreicated!
>
> Best,
> Stan
>
>
>
> --
> View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/Executors-exceed-maximum-memory-defined-with-executor-memory-in-Spark-2-1-0-tp20697.html
> Sent from the Apache Spark Developers List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: Executors exceed maximum memory defined with `--executor-memory`
in Spark 2.1.0
Posted by Koert Kuipers <ko...@tresata.com>.
could this be related to SPARK-18787?
On Sun, Jan 22, 2017 at 1:45 PM, Reynold Xin <rx...@databricks.com> wrote:
> Are you using G1 GC? G1 sometimes uses a lot more memory than the size
> allocated.
>
>
> On Sun, Jan 22, 2017 at 12:58 AM StanZhai <ma...@zhaishidan.cn> wrote:
>
>> Hi all,
>>
>>
>>
>> We just upgraded our Spark from 1.6.2 to 2.1.0.
>>
>>
>>
>> Our Spark application is started by spark-submit with config of
>>
>> `--executor-memory 35G` in standalone model, but the actual use of memory
>> up
>>
>> to 65G after a full gc(jmap -histo:live $pid) as follow:
>>
>>
>>
>> test@c6 ~ $ ps aux | grep CoarseGrainedExecutorBackend
>>
>> test 181941 181 34.7 94665384 68836752 ? Sl 09:25 711:21
>>
>> /home/test/service/jdk/bin/java -cp
>>
>> /home/test/service/hadoop/share/hadoop/common/hadoop-
>> lzo-0.4.20-SNAPSHOT.jar:/home/test/service/hadoop/share/
>> hadoop/common/hadoop-lzo-0.4.20-SNAPSHOT.jar:/home/test/
>> service/spark/conf/:/home/test/service/spark/jars/*:/
>> home/test/service/hadoop/etc/hadoop/
>>
>> -Xmx35840M -Dspark.driver.port=47781 -XX:+PrintGCDetails
>>
>> -XX:+PrintGCDateStamps -Xloggc:./gc.log -verbose:gc
>>
>> org.apache.spark.executor.CoarseGrainedExecutorBackend --driver-url
>>
>> spark://CoarseGrainedScheduler@xxx.xxx.xxx.xxx:47781 --executor-id 1
>>
>> --hostname test-192 --cores 36 --app-id app-20170122092509-0017
>> --worker-url
>>
>> spark://Worker@test-192:33890
>>
>>
>>
>> Our Spark jobs are all sql.
>>
>>
>>
>> The exceed memory looks like off-heap memory, but the default value of
>>
>> `spark.memory.offHeap.enabled` is `false`.
>>
>>
>>
>> We didn't find the problem in Spark 1.6.x, what causes this in Spark
>> 2.1.0?
>>
>>
>>
>> Any help is greatly appreicated!
>>
>>
>>
>> Best,
>>
>> Stan
>>
>>
>>
>>
>>
>>
>>
>> --
>>
>> View this message in context: http://apache-spark-
>> developers-list.1001551.n3.nabble.com/Executors-exceed-
>> maximum-memory-defined-with-executor-memory-in-Spark-2-1-0-tp20697.html
>>
>> Sent from the Apache Spark Developers List mailing list archive at
>> Nabble.com.
>>
>>
>>
>> ---------------------------------------------------------------------
>>
>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>
>>
>>
>>
Re: Executors exceed maximum memory defined with
`--executor-memory` in Spark 2.1.0
Posted by StanZhai <ma...@zhaishidan.cn>.
CentOS 7.1,
Linux version 3.10.0-229.el7.x86_64 (builder@kbuilder.dev.centos.org) (gcc
version 4.8.2 20140120 (Red Hat 4.8.2-16) (GCC) ) #1 SMP Fri Mar 6 11:36:42
UTC 2015
Michael Allman-2 wrote
> Hi Stan,
>
> What OS/version are you using?
>
> Michael
>
>> On Jan 22, 2017, at 11:36 PM, StanZhai <
> mail@
> > wrote:
>>
>> I'm using Parallel GC.
>> rxin wrote
>>> Are you using G1 GC? G1 sometimes uses a lot more memory than the size
>>> allocated.
>>>
>>>
>>> On Sun, Jan 22, 2017 at 12:58 AM StanZhai <
>>
>>> mail@
>>
>>> > wrote:
>>>
>>>> Hi all,
>>>>
>>>>
>>>>
>>>> We just upgraded our Spark from 1.6.2 to 2.1.0.
>>>>
>>>>
>>>>
>>>> Our Spark application is started by spark-submit with config of
>>>>
>>>> `--executor-memory 35G` in standalone model, but the actual use of
>>>> memory
>>>> up
>>>>
>>>> to 65G after a full gc(jmap -histo:live $pid) as follow:
>>>>
>>>>
>>>>
>>>> test@c6 ~ $ ps aux | grep CoarseGrainedExecutorBackend
>>>>
>>>> test 181941 181 34.7 94665384 68836752 ? Sl 09:25 711:21
>>>>
>>>> /home/test/service/jdk/bin/java -cp
>>>>
>>>>
>>>> /home/test/service/hadoop/share/hadoop/common/hadoop-lzo-0.4.20-SNAPSHOT.jar:/home/test/service/hadoop/share/hadoop/common/hadoop-lzo-0.4.20-SNAPSHOT.jar:/home/test/service/spark/conf/:/home/test/service/spark/jars/*:/home/test/service/hadoop/etc/hadoop/
>>>>
>>>> -Xmx35840M -Dspark.driver.port=47781 -XX:+PrintGCDetails
>>>>
>>>> -XX:+PrintGCDateStamps -Xloggc:./gc.log -verbose:gc
>>>>
>>>> org.apache.spark.executor.CoarseGrainedExecutorBackend --driver-url
>>>>
>>>> spark://
>>
>>> CoarseGrainedScheduler@.xxx
>>
>>> :47781 --executor-id 1
>>>>
>>>> --hostname test-192 --cores 36 --app-id app-20170122092509-0017
>>>> --worker-url
>>>>
>>>> spark://Worker@test-192:33890
>>>>
>>>>
>>>>
>>>> Our Spark jobs are all sql.
>>>>
>>>>
>>>>
>>>> The exceed memory looks like off-heap memory, but the default value of
>>>>
>>>> `spark.memory.offHeap.enabled` is `false`.
>>>>
>>>>
>>>>
>>>> We didn't find the problem in Spark 1.6.x, what causes this in Spark
>>>> 2.1.0?
>>>>
>>>>
>>>>
>>>> Any help is greatly appreicated!
>>>>
>>>>
>>>>
>>>> Best,
>>>>
>>>> Stan
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>>
>>>> View this message in context:
>>>> http://apache-spark-developers-list.1001551.n3.nabble.com/Executors-exceed-maximum-memory-defined-with-executor-memory-in-Spark-2-1-0-tp20697.html
>>>>
>>>> Sent from the Apache Spark Developers List mailing list archive at
>>>> Nabble.com <http://nabble.com/>.
>>>>
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>>
>>>> To unsubscribe e-mail:
>>
>>> dev-unsubscribe@.apache
>>
>>>>
>>>>
>>>>
>>>>
>>
>>
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-spark-developers-list.1001551.n3.nabble.com/Executors-exceed-maximum-memory-defined-with-executor-memory-in-Spark-2-1-0-tp20697p20707.html
>> <http://apache-spark-developers-list.1001551.n3.nabble.com/Executors-exceed-maximum-memory-defined-with-executor-memory-in-Spark-2-1-0-tp20697p20707.html>
>> Sent from the Apache Spark Developers List mailing list archive at
>> Nabble.com <http://nabble.com/>.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail:
> dev-unsubscribe@.apache
> <mailto:
> dev-unsubscribe@.apache
> >
--
View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/Executors-exceed-maximum-memory-defined-with-executor-memory-in-Spark-2-1-0-tp20697p20833.html
Sent from the Apache Spark Developers List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: Executors exceed maximum memory defined with `--executor-memory`
in Spark 2.1.0
Posted by Michael Allman <mi...@videoamp.com>.
Hi Stan,
What OS/version are you using?
Michael
> On Jan 22, 2017, at 11:36 PM, StanZhai <ma...@zhaishidan.cn> wrote:
>
> I'm using Parallel GC.
> rxin wrote
>> Are you using G1 GC? G1 sometimes uses a lot more memory than the size
>> allocated.
>>
>>
>> On Sun, Jan 22, 2017 at 12:58 AM StanZhai <
>
>> mail@
>
>> > wrote:
>>
>>> Hi all,
>>>
>>>
>>>
>>> We just upgraded our Spark from 1.6.2 to 2.1.0.
>>>
>>>
>>>
>>> Our Spark application is started by spark-submit with config of
>>>
>>> `--executor-memory 35G` in standalone model, but the actual use of memory
>>> up
>>>
>>> to 65G after a full gc(jmap -histo:live $pid) as follow:
>>>
>>>
>>>
>>> test@c6 ~ $ ps aux | grep CoarseGrainedExecutorBackend
>>>
>>> test 181941 181 34.7 94665384 68836752 ? Sl 09:25 711:21
>>>
>>> /home/test/service/jdk/bin/java -cp
>>>
>>>
>>> /home/test/service/hadoop/share/hadoop/common/hadoop-lzo-0.4.20-SNAPSHOT.jar:/home/test/service/hadoop/share/hadoop/common/hadoop-lzo-0.4.20-SNAPSHOT.jar:/home/test/service/spark/conf/:/home/test/service/spark/jars/*:/home/test/service/hadoop/etc/hadoop/
>>>
>>> -Xmx35840M -Dspark.driver.port=47781 -XX:+PrintGCDetails
>>>
>>> -XX:+PrintGCDateStamps -Xloggc:./gc.log -verbose:gc
>>>
>>> org.apache.spark.executor.CoarseGrainedExecutorBackend --driver-url
>>>
>>> spark://
>
>> CoarseGrainedScheduler@.xxx
>
>> :47781 --executor-id 1
>>>
>>> --hostname test-192 --cores 36 --app-id app-20170122092509-0017
>>> --worker-url
>>>
>>> spark://Worker@test-192:33890
>>>
>>>
>>>
>>> Our Spark jobs are all sql.
>>>
>>>
>>>
>>> The exceed memory looks like off-heap memory, but the default value of
>>>
>>> `spark.memory.offHeap.enabled` is `false`.
>>>
>>>
>>>
>>> We didn't find the problem in Spark 1.6.x, what causes this in Spark
>>> 2.1.0?
>>>
>>>
>>>
>>> Any help is greatly appreicated!
>>>
>>>
>>>
>>> Best,
>>>
>>> Stan
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> --
>>>
>>> View this message in context:
>>> http://apache-spark-developers-list.1001551.n3.nabble.com/Executors-exceed-maximum-memory-defined-with-executor-memory-in-Spark-2-1-0-tp20697.html
>>>
>>> Sent from the Apache Spark Developers List mailing list archive at
>>> Nabble.com <http://nabble.com/>.
>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>>
>>> To unsubscribe e-mail:
>
>> dev-unsubscribe@.apache
>
>>>
>>>
>>>
>>>
>
>
>
>
>
> --
> View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/Executors-exceed-maximum-memory-defined-with-executor-memory-in-Spark-2-1-0-tp20697p20707.html <http://apache-spark-developers-list.1001551.n3.nabble.com/Executors-exceed-maximum-memory-defined-with-executor-memory-in-Spark-2-1-0-tp20697p20707.html>
> Sent from the Apache Spark Developers List mailing list archive at Nabble.com <http://nabble.com/>.
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org <ma...@spark.apache.org>
Re: Executors exceed maximum memory defined with
`--executor-memory` in Spark 2.1.0
Posted by StanZhai <ma...@zhaishidan.cn>.
I'm using Parallel GC.
rxin wrote
> Are you using G1 GC? G1 sometimes uses a lot more memory than the size
> allocated.
>
>
> On Sun, Jan 22, 2017 at 12:58 AM StanZhai <
> mail@
> > wrote:
>
>> Hi all,
>>
>>
>>
>> We just upgraded our Spark from 1.6.2 to 2.1.0.
>>
>>
>>
>> Our Spark application is started by spark-submit with config of
>>
>> `--executor-memory 35G` in standalone model, but the actual use of memory
>> up
>>
>> to 65G after a full gc(jmap -histo:live $pid) as follow:
>>
>>
>>
>> test@c6 ~ $ ps aux | grep CoarseGrainedExecutorBackend
>>
>> test 181941 181 34.7 94665384 68836752 ? Sl 09:25 711:21
>>
>> /home/test/service/jdk/bin/java -cp
>>
>>
>> /home/test/service/hadoop/share/hadoop/common/hadoop-lzo-0.4.20-SNAPSHOT.jar:/home/test/service/hadoop/share/hadoop/common/hadoop-lzo-0.4.20-SNAPSHOT.jar:/home/test/service/spark/conf/:/home/test/service/spark/jars/*:/home/test/service/hadoop/etc/hadoop/
>>
>> -Xmx35840M -Dspark.driver.port=47781 -XX:+PrintGCDetails
>>
>> -XX:+PrintGCDateStamps -Xloggc:./gc.log -verbose:gc
>>
>> org.apache.spark.executor.CoarseGrainedExecutorBackend --driver-url
>>
>> spark://
> CoarseGrainedScheduler@.xxx
> :47781 --executor-id 1
>>
>> --hostname test-192 --cores 36 --app-id app-20170122092509-0017
>> --worker-url
>>
>> spark://Worker@test-192:33890
>>
>>
>>
>> Our Spark jobs are all sql.
>>
>>
>>
>> The exceed memory looks like off-heap memory, but the default value of
>>
>> `spark.memory.offHeap.enabled` is `false`.
>>
>>
>>
>> We didn't find the problem in Spark 1.6.x, what causes this in Spark
>> 2.1.0?
>>
>>
>>
>> Any help is greatly appreicated!
>>
>>
>>
>> Best,
>>
>> Stan
>>
>>
>>
>>
>>
>>
>>
>> --
>>
>> View this message in context:
>> http://apache-spark-developers-list.1001551.n3.nabble.com/Executors-exceed-maximum-memory-defined-with-executor-memory-in-Spark-2-1-0-tp20697.html
>>
>> Sent from the Apache Spark Developers List mailing list archive at
>> Nabble.com.
>>
>>
>>
>> ---------------------------------------------------------------------
>>
>> To unsubscribe e-mail:
> dev-unsubscribe@.apache
>>
>>
>>
>>
--
View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/Executors-exceed-maximum-memory-defined-with-executor-memory-in-Spark-2-1-0-tp20697p20707.html
Sent from the Apache Spark Developers List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: Executors exceed maximum memory defined with `--executor-memory`
in Spark 2.1.0
Posted by Reynold Xin <rx...@databricks.com>.
Are you using G1 GC? G1 sometimes uses a lot more memory than the size
allocated.
On Sun, Jan 22, 2017 at 12:58 AM StanZhai <ma...@zhaishidan.cn> wrote:
> Hi all,
>
>
>
> We just upgraded our Spark from 1.6.2 to 2.1.0.
>
>
>
> Our Spark application is started by spark-submit with config of
>
> `--executor-memory 35G` in standalone model, but the actual use of memory
> up
>
> to 65G after a full gc(jmap -histo:live $pid) as follow:
>
>
>
> test@c6 ~ $ ps aux | grep CoarseGrainedExecutorBackend
>
> test 181941 181 34.7 94665384 68836752 ? Sl 09:25 711:21
>
> /home/test/service/jdk/bin/java -cp
>
>
> /home/test/service/hadoop/share/hadoop/common/hadoop-lzo-0.4.20-SNAPSHOT.jar:/home/test/service/hadoop/share/hadoop/common/hadoop-lzo-0.4.20-SNAPSHOT.jar:/home/test/service/spark/conf/:/home/test/service/spark/jars/*:/home/test/service/hadoop/etc/hadoop/
>
> -Xmx35840M -Dspark.driver.port=47781 -XX:+PrintGCDetails
>
> -XX:+PrintGCDateStamps -Xloggc:./gc.log -verbose:gc
>
> org.apache.spark.executor.CoarseGrainedExecutorBackend --driver-url
>
> spark://CoarseGrainedScheduler@xxx.xxx.xxx.xxx:47781 --executor-id 1
>
> --hostname test-192 --cores 36 --app-id app-20170122092509-0017
> --worker-url
>
> spark://Worker@test-192:33890
>
>
>
> Our Spark jobs are all sql.
>
>
>
> The exceed memory looks like off-heap memory, but the default value of
>
> `spark.memory.offHeap.enabled` is `false`.
>
>
>
> We didn't find the problem in Spark 1.6.x, what causes this in Spark 2.1.0?
>
>
>
> Any help is greatly appreicated!
>
>
>
> Best,
>
> Stan
>
>
>
>
>
>
>
> --
>
> View this message in context:
> http://apache-spark-developers-list.1001551.n3.nabble.com/Executors-exceed-maximum-memory-defined-with-executor-memory-in-Spark-2-1-0-tp20697.html
>
> Sent from the Apache Spark Developers List mailing list archive at
> Nabble.com.
>
>
>
> ---------------------------------------------------------------------
>
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>
>
>
>