You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Valerii Moisieienko <va...@gmail.com> on 2015/05/29 15:56:47 UTC
Spark Executor Memory Usage
Hello!
My name is Valerii. I have noticed strange memory behaivour of Spark's
executor on my cluster. Cluster works in standalone mode with 3 workers.
Application runs in cluster mode.
>From topology configuration
spark.executor.memory 1536m
I checked heap usage via JVisualVM:
http://joxi.ru/Q2KqBMdSvYpDrj
and via htop:
http://joxi.ru/Vm63RWeCvG6L2Z
I have 2 questions regarding Spark's executors memory usage:
1. Why does Max Heap Size change during executor work?
2. Why does Memory usage via htop greater than executor's heap size?
Thank you!
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Executor-Memory-Usage-tp23083.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
RE: Spark Executor Memory Usage
Posted by "HuS.Andy" <hu...@hotmail.com>.
#1 I not sure if I got you point, as I known, Xmx is not turn into physical memory as soon as the process running. it first loaded into virtual memory, if you heap is need more, it will gradually increase in physical memory until to the max heap.
#2 Physical memory contains not only heap, but also stack, direct memory, shared lib, and perm space, and also there have VSS, RSS, PSS, USS concept, you can google.
simple says:Vss = virtual set sizeRss = resident set sizePss = proportional set size Uss = unique set size
Best Regards,Andy Hu（胡 珊）
Date: Fri, 29 May 2015 07:41:41 -0700
Subject: Re: Spark Executor Memory Usage
From: yuzhihong@gmail.com
To: valeramoiseenko@gmail.com
CC: user@spark.apache.org
For #2, see http://unix.stackexchange.com/questions/65835/htop-reporting-much-higher-memory-usage-than-free-or-top
Cheers
On Fri, May 29, 2015 at 6:56 AM, Valerii Moisieienko <va...@gmail.com> wrote:
Hello!
My name is Valerii. I have noticed strange memory behaivour of Spark's
executor on my cluster. Cluster works in standalone mode with 3 workers.
Application runs in cluster mode.
From topology configuration
spark.executor.memory 1536m
I checked heap usage via JVisualVM:
http://joxi.ru/Q2KqBMdSvYpDrj
and via htop:
http://joxi.ru/Vm63RWeCvG6L2Z
I have 2 questions regarding Spark's executors memory usage:
1. Why does Max Heap Size change during executor work?
2. Why does Memory usage via htop greater than executor's heap size?
Thank you!
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Executor-Memory-Usage-tp23083.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Spark Executor Memory Usage
Posted by Ted Yu <yu...@gmail.com>.
For #2, see
http://unix.stackexchange.com/questions/65835/htop-reporting-much-higher-memory-usage-than-free-or-top
Cheers
On Fri, May 29, 2015 at 6:56 AM, Valerii Moisieienko <
valeramoiseenko@gmail.com> wrote:
> Hello!
> My name is Valerii. I have noticed strange memory behaivour of Spark's
> executor on my cluster. Cluster works in standalone mode with 3 workers.
> Application runs in cluster mode.
> From topology configuration
> spark.executor.memory 1536m
> I checked heap usage via JVisualVM:
> http://joxi.ru/Q2KqBMdSvYpDrj
> and via htop:
> http://joxi.ru/Vm63RWeCvG6L2Z
>
> I have 2 questions regarding Spark's executors memory usage:
> 1. Why does Max Heap Size change during executor work?
> 2. Why does Memory usage via htop greater than executor's heap size?
>
> Thank you!
>
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Executor-Memory-Usage-tp23083.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>