You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Jacek Laskowski <ja...@japila.pl> on 2020/04/16 11:25:49 UTC
BlockManager and ShuffleManager = can getLocalBytes be ever used for
shuffle blocks?
Hi,
While trying to understand the relationship of BlockManager
and ShuffleManager I found that ShuffleManager is used for shuffle block
data [1] (and that makes sense).
What I found quite surprising is that BlockManager can call getLocalBytes
for non-shuffle blocks that in turn does...fetching shuffle blocks (!) [2].
That begs the question, who could want this? In other words, what other
components would want to call BlockManager.getLocalBytes for shuffle
blocks? The only caller is TorrentBroadcast which is not for shuffle
blocks, is it?
In other words, getLocalBytes should NOT be bothering itself with fetching
local shuffle blocks as that's a responsibility of someone else,
i.e. ShuffleManager.
Is this correct?
I'd appreciate any help you can provide so I can understand the Spark core
better. Thanks!
[1]
https://github.com/apache/spark/blob/31734399d57f3c128e66b0f97ef83eb4c9165978/core/src/main/scala/org/apache/spark/storage/BlockManager.scala#L382
[2]
https://github.com/apache/spark/blob/31734399d57f3c128e66b0f97ef83eb4c9165978/core/src/main/scala/org/apache/spark/storage/BlockManager.scala#L637
Pozdrawiam,
Jacek Laskowski
----
https://about.me/JacekLaskowski
"The Internals Of" Online Books <https://books.japila.pl/>
Follow me on https://twitter.com/jaceklaskowski
<https://twitter.com/jaceklaskowski>
Re: BlockManager and ShuffleManager = can getLocalBytes be ever used
for shuffle blocks?
Posted by Jacek Laskowski <ja...@japila.pl>.
Thanks Yi Wu!
Pozdrawiam,
Jacek Laskowski
----
https://about.me/JacekLaskowski
"The Internals Of" Online Books <https://books.japila.pl/>
Follow me on https://twitter.com/jaceklaskowski
<https://twitter.com/jaceklaskowski>
On Sat, Apr 18, 2020 at 12:17 PM wuyi <yi...@databricks.com> wrote:
> Hi Jacek,
>
> code in link [2] is the out of date. The commit
>
> https://github.com/apache/spark/commit/32ec528e63cb768f85644282978040157c3c2fb7
> has already removed unreachable branch.
>
>
> Best,
> Yi Wu
>
>
>
> --
> Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>
>
Re: BlockManager and ShuffleManager = can getLocalBytes be ever
used for shuffle blocks?
Posted by wuyi <yi...@databricks.com>.
Hi Jacek,
code in link [2] is the out of date. The commit
https://github.com/apache/spark/commit/32ec528e63cb768f85644282978040157c3c2fb7
has already removed unreachable branch.
Best,
Yi Wu
--
Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org