You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Dongjoon Hyun (Jira)" <ji...@apache.org> on 2026/07/09 20:09:00 UTC
[jira] [Resolved] (SPARK-57851) Shuffle-free single-task execution for small queries
[ https://issues.apache.org/jira/browse/SPARK-57851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dongjoon Hyun resolved SPARK-57851.
-----------------------------------
Fix Version/s: 4.3.0
Resolution: Fixed
Issue resolved by pull request 56928
[https://github.com/apache/spark/pull/56928]
> Shuffle-free single-task execution for small queries
> ----------------------------------------------------
>
> Key: SPARK-57851
> URL: https://issues.apache.org/jira/browse/SPARK-57851
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
> Affects Versions: 4.3.0
> Reporter: L. C. Hsieh
> Assignee: L. C. Hsieh
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.3.0
>
>
> Part of the SPIP umbrella SPARK-56978 (Faster queries in local laptop mode), covering the third category: shuffle-free local execution for small queries.
> This adds a conservative optimizer rule (MarkSingleTaskExecution) that marks small single-partition scans, optionally with a shuffle-inducing operator on top (sort, aggregate, distinct, window, limit/offset, expand), as candidates for single-task execution. Such a scan reports a SinglePartition output partitioning, allowing EnsureRequirements to elide the shuffle that would otherwise be inserted before the operator on top. The shuffle is not required for correctness, so removing it reduces scheduling overhead for small, low-latency queries.
> The feature is controlled by a new set of internal configs under spark.sql.optimizer.singleTaskExecution.* and is disabled by default.
> This initial change covers the core operator set; join is intentionally left out and can be added as a follow-up. Union is already covered by the existing spark.sql.unionOutputPartitioning.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org