You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Hyukjin Kwon (JIRA)" <ji...@apache.org> on 2019/05/21 04:25:27 UTC
[jira] [Updated] (SPARK-11156) Web UI doesn't count or show info
about replicated blocks
[ https://issues.apache.org/jira/browse/SPARK-11156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hyukjin Kwon updated SPARK-11156:
---------------------------------
Labels: bulk-closed (was: )
> Web UI doesn't count or show info about replicated blocks
> ---------------------------------------------------------
>
> Key: SPARK-11156
> URL: https://issues.apache.org/jira/browse/SPARK-11156
> Project: Spark
> Issue Type: Bug
> Components: Web UI
> Affects Versions: 1.5.1
> Reporter: Ryan Williams
> Priority: Major
> Labels: bulk-closed
>
> When executors receive a replica of a block, they [notify the driver with a {{UpdateBlockInfo}} message|https://github.com/apache/spark/blob/4ee2cea2a43f7d04ab8511d9c029f80c5dadd48e/core/src/main/scala/org/apache/spark/storage/BlockManagerMaster.scala#L59-L61] which [sends a {{SparkListenerBlockUpdated}} event to SparkListeners|https://github.com/apache/spark/blob/4ee2cea2a43f7d04ab8511d9c029f80c5dadd48e/core/src/main/scala/org/apache/spark/storage/BlockManagerMasterEndpoint.scala#L67].
> However, the web UI (via its BlockStatusListener) [ignores {{SparkListenerBlockUpdated}} events for non-streaming blocks|https://github.com/apache/spark/blob/4ee2cea2a43f7d04ab8511d9c029f80c5dadd48e/core/src/main/scala/org/apache/spark/storage/BlockStatusListener.scala#L57-L60].
> As a result, in non-streaming apps:
> * The "Executors" column on RDD Page doesn't show executors housing replicas; it can only show the executor that initially computed (and initiated replication of) the block.
> * The executor-memory-usage and related stats displayed throughout the web interface are undercounting due to ignorance of the existence of block replicas.
> For example, here is the Storage tab for a simple app with 3 identical RDDs cached with replication equal to 1, 2, and 3:
> !http://f.cl.ly/items/3m3B2v2k2J23350I3t1c/Screen%20Shot%202015-10-16%20at%2012.30.54%20AM.png!
> These were generated with:
> {code}
> val bar1 = sc.parallelize(1 to 100000000, 100).map(_ % 100 -> 1).reduceByKey(_+_, 100).setName("bar1").persist(StorageLevel(false, true, false, true, 1))
> bar1.count
> val bar2 = sc.parallelize(1 to 100000000, 100).map(_ % 100 -> 1).reduceByKey(_+_, 100).setName("bar2").persist(StorageLevel(false, true, false, true, 2))
> bar2.count
> val bar3 = sc.parallelize(1 to 100000000, 100).map(_ % 100 -> 1).reduceByKey(_+_, 100).setName("bar3").persist(StorageLevel(false, true, false, true, 3))
> bar3.count
> {code}
> Note the identically-reported memory usage across the three.
> Here is the RDD page for the 3x-replicated RDD above:
> !http://f.cl.ly/items/0t0H1o2S2g140s1A0X0k/Screen%20Shot%202015-10-16%20at%2012.31.24%20AM.png!
> Note that only one executor is listed for each partition.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org