You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Boyang Jerry Peng (Jira)" <ji...@apache.org> on 2026/05/03 00:42:00 UTC
[jira] [Updated] (SPARK-56664) Add a new shuffle implementation to support Real-time Mode (RTM)
[ https://issues.apache.org/jira/browse/SPARK-56664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Boyang Jerry Peng updated SPARK-56664:
--------------------------------------
Description:
The streaming shuffle is an alternative {{ShuffleManager}} implementation designed for low-latency, continuously-running queries (for example, real-time mode in Structured Streaming). Unlike the default sort-based shuffle, it does not materialize map outputs to disk and does not require map tasks to finish before reduce tasks can start. Instead, each map task hosts a network server that pushes records to reduce tasks as they are produced; reduce tasks open clients to those servers and consume records as a stream.
Design doc:
https://github.com/jerrypeng/spark/blob/ecb48211f821842caf717ebdf069da5505f43f68/docs/streaming-shuffle.md
> Add a new shuffle implementation to support Real-time Mode (RTM)
> ----------------------------------------------------------------
>
> Key: SPARK-56664
> URL: https://issues.apache.org/jira/browse/SPARK-56664
> Project: Spark
> Issue Type: Epic
> Components: Structured Streaming
> Affects Versions: 4.2.0
> Reporter: Boyang Jerry Peng
> Priority: Major
>
> The streaming shuffle is an alternative {{ShuffleManager}} implementation designed for low-latency, continuously-running queries (for example, real-time mode in Structured Streaming). Unlike the default sort-based shuffle, it does not materialize map outputs to disk and does not require map tasks to finish before reduce tasks can start. Instead, each map task hosts a network server that pushes records to reduce tasks as they are produced; reduce tasks open clients to those servers and consume records as a stream.
>
> Design doc:
>
> https://github.com/jerrypeng/spark/blob/ecb48211f821842caf717ebdf069da5505f43f68/docs/streaming-shuffle.md
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org