You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Dev Lakhani (JIRA)" <ji...@apache.org> on 2014/10/16 18:42:33 UTC
[jira] [Commented] (SPARK-3957) Broadcast variable memory usage not
reflected in UI
[ https://issues.apache.org/jira/browse/SPARK-3957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14173926#comment-14173926 ]
Dev Lakhani commented on SPARK-3957:
------------------------------------
Here is my thoughts on a possible approach.
Hi All
The broadcast occurs form the Spark Context to the broadcastmanager and new Broadcast method. In the first instance, the broadcasted data is stored in the Block Manager (see HttpBroadCast) of the executor. Any tracking of broadcast variables must be referenced by the BlockManagerSlaveActor and BlockManagerMasterActor. In particular UpdateBlockInfo and RemoveBroadcast should update the total memory in blocks used when blocks are added and removed.
These can then be hooked up to the UI using a new Page like ExecutorsPage and defining a new methods in the relevant listener such as StorageStatusListener.
These are my initial thoughts for someone new to these components, any other ideas or approaches?
> Broadcast variable memory usage not reflected in UI
> ---------------------------------------------------
>
> Key: SPARK-3957
> URL: https://issues.apache.org/jira/browse/SPARK-3957
> Project: Spark
> Issue Type: Bug
> Components: Block Manager, Web UI
> Affects Versions: 1.0.2, 1.1.0
> Reporter: Shivaram Venkataraman
> Assignee: Nan Zhu
>
> Memory used by broadcast variables are not reflected in the memory usage reported in the WebUI. For example, the executors tab shows memory used in each executor but this number doesn't include memory used by broadcast variables. Similarly the storage tab only shows list of rdds cached and how much memory they use.
> We should add a separate column / tab for broadcast variables to make it easier to debug.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org