You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Larry Xiao <xi...@sjtu.edu.cn> on 2014/10/10 03:40:10 UTC
Executor and BlockManager memory size
Hi all,
I'm confused about Executor and BlockManager, why they have different
memory.
> 14/10/10 08:50:02 INFO AppClient$ClientActor: Executor added:
> app-20141010085001-0000/2 on worker-20141010004933-brick6-35657
> (brick6:35657) with 6 cores
> 14/10/10 08:50:02 INFO SparkDeploySchedulerBackend: Granted executor
> ID app-20141010085001-0000/2 on hostPort brick6:35657 with 6 cores,
> 50.0 GB RAM
>
> 14/10/10 08:50:07 INFO BlockManagerMasterActor: Registering block
> manager brick6:53296 with 26.8 GB RAM
and on the WebUI,
> Executor ID Address RDD Blocks Memory Used Disk Used
> Active Tasks Failed Tasks Complete Tasks Total Tasks Task
> Time Input Shuffle Read Shuffle Write
> 0 brick3:37607 0 0.0 B / 26.8 GB 0.0 B 6 0 0 6
> 0 ms 0.0 B 0.0 B 0.0 B
> 1 brick1:59493 0 0.0 B / 26.8 GB 0.0 B 6 0 0 6
> 0 ms 0.0 B 0.0 B 0.0 B
> 2 brick6:53296 0 0.0 B / 26.8 GB 0.0 B 6 0 0 6
> 0 ms 0.0 B 0.0 B 0.0 B
> 3 brick5:38543 0 0.0 B / 26.8 GB 0.0 B 6 0 0 6
> 0 ms 0.0 B 0.0 B 0.0 B
> 4 brick2:44937 0 0.0 B / 26.8 GB 0.0 B 6 0 0 6
> 0 ms 0.0 B 0.0 B 0.0 B
> 5 brick4:46798 0 0.0 B / 26.8 GB 0.0 B 6 0 0 6
> 0 ms 0.0 B 0.0 B 0.0 B
> <driver> brick0:57692 0 0.0 B / 274.6 MB 0.0 B 0 0
> 0 0 0 ms 0.0 B 0.0 B 0.0 B
As I understand it, a worker consist of a daemon and an executor, and
executor takes charge both execution and storage.
So does it mean that 26.8 GB is saved for storage and the rest is for
execution?
Another question is that, throughout execution, it seems that the
blockmanager is always almost free.
> 14/10/05 14:33:44 INFO BlockManagerInfo: Added broadcast_21_piece0 in
> memory on brick2:57501 (size: 1669.0 B, free: 21.2 GB)
I don't know what I'm missing here.
Best regards,
Larry
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Executor and BlockManager memory size
Posted by Gen <ge...@gmail.com>.
Hi,
I meet the same problem in the context of spark and yarn.
When I open pyspark with the following command:
spark/bin/pyspark --master yarn-client --num-executors 1 --executor-memory
2500m
It turns out *INFO storage.BlockManagerMasterActor: Registering block
manager ip-10-0-6-171.us-west-2.compute.internal:38770 with 1294.1 MB RAM*
So, according to the documentation, just 2156.83m is allocated to executor.
Moreover, according to yarn 3072m memory is used for this container.
Do you have any ideas about this?
Thanks a lot
Cheers
Gen
Boromir Widas wrote
> Hey Larry,
>
> I have been trying to figure this out for standalone clusters as well.
> http://apache-spark-user-list.1001560.n3.nabble.com/What-is-a-Block-Manager-td12833.html
> has an answer as to what block manager is for.
>
> From the documentation, what I understood was if you assign X GB to each
> executor, spark.storage.memoryFraction(default 0.6) * X is assigned to the
> BlockManager and the rest for the JVM itself(?).
> However, as you see, 26.8G is assigned to the BM, and assuming 0.6
> memoryFraction, this means the executor sees ~44.7G of memory, I am not
> sure what happens to the difference(5.3G).
>
>
> On Thu, Oct 9, 2014 at 9:40 PM, Larry Xiao <
> xiaodi@.edu
> > wrote:
>
>> Hi all,
>>
>> I'm confused about Executor and BlockManager, why they have different
>> memory.
>>
>>> 14/10/10 08:50:02 INFO AppClient$ClientActor: Executor added:
>>> app-20141010085001-0000/2 on worker-20141010004933-brick6-35657
>>> (brick6:35657) with 6 cores
>>> 14/10/10 08:50:02 INFO SparkDeploySchedulerBackend: Granted executor ID
>>> app-20141010085001-0000/2 on hostPort brick6:35657 with 6 cores, 50.0 GB
>>> RAM
>>>
>>> 14/10/10 08:50:07 INFO BlockManagerMasterActor: Registering block
>>> manager
>>> brick6:53296 with 26.8 GB RAM
>>>
>> and on the WebUI,
>>
>>> Executor ID Address RDD Blocks Memory Used Disk Used Active
>>> Tasks Failed Tasks Complete Tasks Total Tasks Task Time
>>> Input
>>> Shuffle Read Shuffle Write
>>> 0 brick3:37607 0 0.0 B / 26.8 GB 0.0 B 6 0 0 6
>>> 0
>>> ms 0.0 B 0.0 B 0.0 B
>>> 1 brick1:59493 0 0.0 B / 26.8 GB 0.0 B 6 0 0 6
>>> 0
>>> ms 0.0 B 0.0 B 0.0 B
>>> 2 brick6:53296 0 0.0 B / 26.8 GB 0.0 B 6 0 0 6
>>> 0
>>> ms 0.0 B 0.0 B 0.0 B
>>> 3 brick5:38543 0 0.0 B / 26.8 GB 0.0 B 6 0 0 6
>>> 0
>>> ms 0.0 B 0.0 B 0.0 B
>>> 4 brick2:44937 0 0.0 B / 26.8 GB 0.0 B 6 0 0 6
>>> 0
>>> ms 0.0 B 0.0 B 0.0 B
>>> 5 brick4:46798 0 0.0 B / 26.8 GB 0.0 B 6 0 0 6
>>> 0
>>> ms 0.0 B 0.0 B 0.0 B
>>>
> <driver>
> brick0:57692 0 0.0 B / 274.6 MB 0.0 B 0 0 0
>>> 0 0 ms 0.0 B 0.0 B 0.0 B
>>>
>> As I understand it, a worker consist of a daemon and an executor, and
>> executor takes charge both execution and storage.
>> So does it mean that 26.8 GB is saved for storage and the rest is for
>> execution?
>>
>> Another question is that, throughout execution, it seems that the
>> blockmanager is always almost free.
>>
>>> 14/10/05 14:33:44 INFO BlockManagerInfo: Added broadcast_21_piece0 in
>>> memory on brick2:57501 (size: 1669.0 B, free: 21.2 GB)
>>>
>> I don't know what I'm missing here.
>>
>> Best regards,
>> Larry
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail:
> user-unsubscribe@.apache
>> For additional commands, e-mail:
> user-help@.apache
>>
>>
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Executor-and-BlockManager-memory-size-tp16091p17816.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Executor and BlockManager memory size
Posted by Boromir Widas <vc...@gmail.com>.
Hey Larry,
I have been trying to figure this out for standalone clusters as well.
http://apache-spark-user-list.1001560.n3.nabble.com/What-is-a-Block-Manager-td12833.html
has an answer as to what block manager is for.
>From the documentation, what I understood was if you assign X GB to each
executor, spark.storage.memoryFraction(default 0.6) * X is assigned to the
BlockManager and the rest for the JVM itself(?).
However, as you see, 26.8G is assigned to the BM, and assuming 0.6
memoryFraction, this means the executor sees ~44.7G of memory, I am not
sure what happens to the difference(5.3G).
On Thu, Oct 9, 2014 at 9:40 PM, Larry Xiao <xi...@sjtu.edu.cn> wrote:
> Hi all,
>
> I'm confused about Executor and BlockManager, why they have different
> memory.
>
>> 14/10/10 08:50:02 INFO AppClient$ClientActor: Executor added:
>> app-20141010085001-0000/2 on worker-20141010004933-brick6-35657
>> (brick6:35657) with 6 cores
>> 14/10/10 08:50:02 INFO SparkDeploySchedulerBackend: Granted executor ID
>> app-20141010085001-0000/2 on hostPort brick6:35657 with 6 cores, 50.0 GB RAM
>>
>> 14/10/10 08:50:07 INFO BlockManagerMasterActor: Registering block manager
>> brick6:53296 with 26.8 GB RAM
>>
> and on the WebUI,
>
>> Executor ID Address RDD Blocks Memory Used Disk Used Active
>> Tasks Failed Tasks Complete Tasks Total Tasks Task Time Input
>> Shuffle Read Shuffle Write
>> 0 brick3:37607 0 0.0 B / 26.8 GB 0.0 B 6 0 0 6 0
>> ms 0.0 B 0.0 B 0.0 B
>> 1 brick1:59493 0 0.0 B / 26.8 GB 0.0 B 6 0 0 6 0
>> ms 0.0 B 0.0 B 0.0 B
>> 2 brick6:53296 0 0.0 B / 26.8 GB 0.0 B 6 0 0 6 0
>> ms 0.0 B 0.0 B 0.0 B
>> 3 brick5:38543 0 0.0 B / 26.8 GB 0.0 B 6 0 0 6 0
>> ms 0.0 B 0.0 B 0.0 B
>> 4 brick2:44937 0 0.0 B / 26.8 GB 0.0 B 6 0 0 6 0
>> ms 0.0 B 0.0 B 0.0 B
>> 5 brick4:46798 0 0.0 B / 26.8 GB 0.0 B 6 0 0 6 0
>> ms 0.0 B 0.0 B 0.0 B
>> <driver> brick0:57692 0 0.0 B / 274.6 MB 0.0 B 0 0 0
>> 0 0 ms 0.0 B 0.0 B 0.0 B
>>
> As I understand it, a worker consist of a daemon and an executor, and
> executor takes charge both execution and storage.
> So does it mean that 26.8 GB is saved for storage and the rest is for
> execution?
>
> Another question is that, throughout execution, it seems that the
> blockmanager is always almost free.
>
>> 14/10/05 14:33:44 INFO BlockManagerInfo: Added broadcast_21_piece0 in
>> memory on brick2:57501 (size: 1669.0 B, free: 21.2 GB)
>>
> I don't know what I'm missing here.
>
> Best regards,
> Larry
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>