You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Jacek Laskowski <ja...@japila.pl> on 2016/12/23 14:13:35 UTC
MapOutputTracker.getMapSizesByExecutorId and mutation on the driver?
Hi,
I've been reviewing how MapOutputTracker works and can't understand
the comment [1]:
// Synchronize on the returned array because, on the driver, it gets
mutated in place
How is this possible since "the returned array" is a local value? I'm
stuck and would appreciate help. Thanks!
(It also says "Called from executors" [2] so how could the driver be involved?!)
[1] https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/MapOutputTracker.scala#L145
[2] https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/MapOutputTracker.scala#L133
Pozdrawiam,
Jacek Laskowski
----
https://medium.com/@jaceklaskowski/
Mastering Apache Spark 2.0 https://bit.ly/mastering-apache-spark
Follow me at https://twitter.com/jaceklaskowski
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: MapOutputTracker.getMapSizesByExecutorId and mutation on the
driver?
Posted by Liang-Chi Hsieh <vi...@gmail.com>.
Hi,
I think the comment [1] is only correct for "getStatistics" as it is called
at driver side. It should be added in "getMapSizesByExecutorId" by mistake.
Jacek Laskowski wrote
> Hi,
>
> I've been reviewing how MapOutputTracker works and can't understand
> the comment [1]:
>
> // Synchronize on the returned array because, on the driver, it gets
> mutated in place
>
> How is this possible since "the returned array" is a local value? I'm
> stuck and would appreciate help. Thanks!
>
> (It also says "Called from executors" [2] so how could the driver be
> involved?!)
>
> [1]
> https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/MapOutputTracker.scala#L145
>
> [2]
> https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/MapOutputTracker.scala#L133
>
> Pozdrawiam,
> Jacek Laskowski
> ----
> https://medium.com/@jaceklaskowski/
> Mastering Apache Spark 2.0 https://bit.ly/mastering-apache-spark
> Follow me at https://twitter.com/jaceklaskowski
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail:
> dev-unsubscribe@.apache
-----
Liang-Chi Hsieh | @viirya
Spark Technology Center
http://www.spark.tc/
--
View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/MapOutputTracker-getMapSizesByExecutorId-and-mutation-on-the-driver-tp20342p20349.html
Sent from the Apache Spark Developers List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org