You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "acvictor (via GitHub)" <gi...@apache.org> on 2026/09/16 13:37:55 UTC
[PR] [CORE] Avoid a per-row copy on row-based shuffle writes [gluten]
acvictor opened a new pull request, #13039:
URL: https://github.com/apache/gluten/pull/13039
## What changes are proposed in this pull request?
spark.shuffle.manager is process-wide, so ColumnarShuffleManager also serves the row-based exchanges Gluten does not offload. Spark decides whether each map task must defensively copy every row in ShuffleExchangeExec.needToCopyObjectsBeforeShuffle, which [gates on a single check, shuffleManager.isInstanceOf[SortShuffleManager]](https://github.com/apache/spark/blob/branch-4.1/sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/ShuffleExchangeExec.scala#L297). Any other implementation falls through to a catch-all `true`.
Because ColumnarShuffleManager extended ShuffleManager directly, that check was false and Spark selected the copying closure, running UnsafeRow.copy() once per row and allocating a tuple per row, instead of the branch that reuses a single MutablePair.
The copy is pure overhead here rather than protection. It exists because ExternalSorter buffers deserialized rows, and the row-based branches in this class never reach that state: registerShuffle uses the same shouldBypassMergeSort and canUseSerializedShuffle predicates as SortShuffleManager, and getWriter builds the same BypassMergeSortShuffleWriter and UnsafeShuffleWriter. Same handle, same writer, same safety property; only the type check differed.
Extending SortShuffleManager restores the zero-copy path. Shuffle output is unchanged.
## How was this patch tested?
UT
## Was this patch authored or co-authored using generative AI tooling?
Co-authored with GitHub Copilot CLI 1.0.84-4
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Avoid a per-row copy on row-based shuffle writes [gluten]
Posted by "marin-ma (via GitHub)" <gi...@apache.org>.
marin-ma commented on PR #13039:
URL: https://github.com/apache/gluten/pull/13039#issuecomment-5757602072
Thanks for the fix!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Avoid a per-row copy on row-based shuffle writes [gluten]
Posted by "acvictor (via GitHub)" <gi...@apache.org>.
acvictor commented on PR #13039:
URL: https://github.com/apache/gluten/pull/13039#issuecomment-5757424954
> @acvictor Do you mind cherry pick this commit to see if it fixes the CH CI failure? https://github.com/apache/gluten/pull/13054/changes
@marin-ma this worked!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Avoid a per-row copy on row-based shuffle writes [gluten]
Posted by "marin-ma (via GitHub)" <gi...@apache.org>.
marin-ma commented on PR #13039:
URL: https://github.com/apache/gluten/pull/13039#issuecomment-5712220741
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Avoid a per-row copy on row-based shuffle writes [gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #13039:
URL: https://github.com/apache/gluten/pull/13039#issuecomment-5698455930
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Avoid a per-row copy on row-based shuffle writes [gluten]
Posted by "acvictor (via GitHub)" <gi...@apache.org>.
acvictor commented on PR #13039:
URL: https://github.com/apache/gluten/pull/13039#issuecomment-5708348970
@marin-ma would appreciate a review, thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Avoid a per-row copy on row-based shuffle writes [gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #13039:
URL: https://github.com/apache/gluten/pull/13039#issuecomment-5728377719
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Avoid a per-row copy on row-based shuffle writes [gluten]
Posted by "marin-ma (via GitHub)" <gi...@apache.org>.
marin-ma commented on PR #13039:
URL: https://github.com/apache/gluten/pull/13039#issuecomment-5728811175
@acvictor Do you mind cherry pick this commit to see if it fixes the CH CI failure? https://github.com/apache/gluten/pull/13054/changes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Avoid a per-row copy on row-based shuffle writes [gluten]
Posted by "marin-ma (via GitHub)" <gi...@apache.org>.
marin-ma commented on PR #13039:
URL: https://github.com/apache/gluten/pull/13039#issuecomment-5728340130
@acvictor Can you please rebase? There's a test failure in CH CI but seems irrelevant.
```
[2026-09-17T16:11:27.803Z] ColumnarShuffleManagerSuite:
[2026-09-17T16:11:27.803Z] - is a SortShuffleManager so row-based exchanges keep Spark's zero-copy write path
[2026-09-17T16:11:27.803Z] SoftAffinityWithRDDInfoSuite:
[2026-09-17T16:11:27.803Z] - Soft Affinity Scheduler with duplicate reading detection *** FAILED ***
[2026-09-17T16:11:27.803Z] Expected Set("executor_host-0_0"), but got Set("host-3") (SoftAffinityWithRDDInfoSuite.scala:102)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Avoid a per-row copy on row-based shuffle writes [gluten]
Posted by "marin-ma (via GitHub)" <gi...@apache.org>.
marin-ma merged PR #13039:
URL: https://github.com/apache/gluten/pull/13039
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Avoid a per-row copy on row-based shuffle writes [gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #13039:
URL: https://github.com/apache/gluten/pull/13039#issuecomment-5755752872
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org