[jira] [Commented] (SPARK-18098) Broadcast creates 1 instance / core, not 1 instance / executor

["Herman van Hovell (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-18098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15671049#comment-15671049 ] 

Herman van Hovell commented on SPARK-18098:
-------------------------------------------

I think this is caused by how you have structured your code. Could you share an example of what you are doing?

> Broadcast creates 1 instance / core, not 1 instance / executor
> --------------------------------------------------------------
>
>                 Key: SPARK-18098
>                 URL: https://issues.apache.org/jira/browse/SPARK-18098
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Core
>    Affects Versions: 2.0.1
>            Reporter: Anthony Sciola
>
> I've created my spark executors with $SPARK_HOME/sbin/start-slave.sh -c 7 -m 55g
> When I run a job which broadcasts data, it appears each *thread* requests and receives a copy of the broadcast object, not each *executor*. This means I need 7x as much memory for the broadcasted item because I have 7 cores.
> The problem appears to be due to a lack of synchronization around requesting broadcast items.
> The only workaround I've come up with is writing the data out to HDFS, broadcasting the paths, and doing a synchronized load from HDFS.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org