You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Deependra Patel (Jira)" <ji...@apache.org> on 2024/12/11 18:20:00 UTC

[jira] [Created] (SPARK-50550) [Connect] Support sc.broadcast on Connect

Deependra Patel created SPARK-50550:
---------------------------------------

             Summary: [Connect] Support sc.broadcast on Connect 
                 Key: SPARK-50550
                 URL: https://issues.apache.org/jira/browse/SPARK-50550
             Project: Spark
          Issue Type: Improvement
          Components: Connect, PySpark
    Affects Versions: 4.0.0
            Reporter: Deependra Patel


Spark Context methods are not supported in Spark Connect. Broadcast is a common API used by developers to send over some data to all executors. Eg. machine learning model weights can be broadcasted, so no need to fetch individually.
 
This makes migration of workloads to Spark Connect complex.
 
I will try to come up with a design doc & PR after that if approach looks good.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org