You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Patrick Wendell (JIRA)" <ji...@apache.org> on 2014/10/21 09:17:34 UTC
[jira] [Comment Edited] (SPARK-4030) `destroy` method in Broadcast
should be public
[ https://issues.apache.org/jira/browse/SPARK-4030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14178068#comment-14178068 ]
Patrick Wendell edited comment on SPARK-4030 at 10/21/14 7:17 AM:
------------------------------------------------------------------
Hey Shivaram - IIRC we made this private to protect users from shooting themselves in the foot and trying to use a broadcast variable that has been destroy()'d, potentially by another thread or somewhere difficult to track down. If we expose this it would be good to test it thoroughly (I think we put some logic in there around this already thinking that we might open it up later). Also, it might be good to capture the callsite of the destroy call and echo it back to users if they attempt to use an invalid broadcast variable.
I also notice we have this terminology around "valid" in the Broadcast class, it might make more sense to change that to "destroyed" to make it more explicit - that's just a minor clean-up though and we could do it in a different class.
was (Author: pwendell):
Hey Shivaram - IIRC we made this private to protect users from shooting themselves in the foot and trying to use a broadcast variable that has been destroy()'d, potentially by another thread or somewhere difficult to track down. If we expose this it would be good to test it thoroughly (I think we put some logic in there around this already thinking that we might open it up later). Also, it might be good to capture the callsite of the destroy call and echo it back to users if they attempt to use an invalid broadcast variable.
> `destroy` method in Broadcast should be public
> ----------------------------------------------
>
> Key: SPARK-4030
> URL: https://issues.apache.org/jira/browse/SPARK-4030
> Project: Spark
> Issue Type: Improvement
> Components: Block Manager, Spark Core
> Affects Versions: 1.1.0, 1.2.0
> Reporter: Shivaram Venkataraman
>
> The destroy method in Broadcast.scala [https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/broadcast/Broadcast.scala#L91] is right now marked as private[spark]
> This prevents long-running applications from cleaning up memory used by broadcast variables on the driver. Also as broadcast variables are always created with persistence MEMORY_DISK, this slows down jobs when old broadcast variables are flushed to disk.
> Making `destroy` public can help applications control the lifetime.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org