You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "leixm (via GitHub)" <gi...@apache.org> on 2026/04/01 09:33:39 UTC
[PR] [SPARK-56319] ShuffleExternalSorter should reuse write buffer [spark]
leixm opened a new pull request, #55136:
URL: https://github.com/apache/spark/pull/55136
### What changes were proposed in this pull request?
[SPARK-56319] ShuffleExternalSorter should reuse write buffer
### Why are the changes needed?
In ShuffleExternalSorter, the writeSortedFile() method allocates a new byte[] of diskWriteBufferSize (default 1 MB) on every invocation. This method is called once per spill and once more for the final file write, so in a spill-heavy shuffle task the same 1 MB array is allocated and discarded repeatedly, adding unnecessary GC pressure.
This PR promotes the write buffer to a lazily-initialized instance field so that it is allocated on the first call to writeSortedFile() and reused across subsequent spills. Lazy initialization ensures that the buffer is not allocated until it is actually needed, so tasks that insert no records (and therefore never call into the write path) pay no extra cost.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Existing UTs.
### Was this patch authored or co-authored using generative AI tooling?
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56319] ShuffleExternalSorter should reuse write buffer [spark]
Posted by "leixm (via GitHub)" <gi...@apache.org>.
leixm commented on PR #55136:
URL: https://github.com/apache/spark/pull/55136#issuecomment-4169287653
@cloud-fan Can you help review?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56319][SQL] ShuffleExternalSorter should reuse write buffer [spark]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #55136: [SPARK-56319][SQL] ShuffleExternalSorter should reuse write buffer
URL: https://github.com/apache/spark/pull/55136
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56319] ShuffleExternalSorter should reuse write buffer [spark]
Posted by "leixm (via GitHub)" <gi...@apache.org>.
leixm commented on PR #55136:
URL: https://github.com/apache/spark/pull/55136#issuecomment-4175734113
cc @LuciferYang Can you help review this PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56319][SQL] ShuffleExternalSorter should reuse write buffer [spark]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #55136:
URL: https://github.com/apache/spark/pull/55136#issuecomment-4975525070
We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org