[jira] [Updated] (SPARK-42980) lightweight Broadcast implementation

["Ziqi Liu (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-42980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ziqi Liu updated SPARK-42980:
-----------------------------
    Summary: lightweight Broadcast implementation  (was: SmallBroadcast)

> lightweight Broadcast implementation
> ------------------------------------
>
>                 Key: SPARK-42980
>                 URL: https://issues.apache.org/jira/browse/SPARK-42980
>             Project: Spark
>          Issue Type: New Feature
>          Components: Spark Core
>    Affects Versions: 3.4.0
>            Reporter: Ziqi Liu
>            Priority: Major
>
> The current TorrentBroadcast implementation is originally designed for large data transmission where driver might become the bottleneck and memory might also be the concern. Therefore it's kind of heavy and comes with some fixed overhead:
>  * torrent protocol: more round traffic
>  * disk level persistency, BlockManager, extra overhead
> which makes it inefficient for some small data transmission.
> We can have a lightweight broadcast implementation, e.g, SmallBroadcast, which implement star-topology broadcast(which avoids the unnecessary round traffic), and maybe in-memory only.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org