You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "wForget (via GitHub)" <gi...@apache.org> on 2025/10/22 05:41:42 UTC
[PR] [GLUTEN-10920][VL] Allow disabling hash/sort shuffle reader buffer [incubator-gluten]
wForget opened a new pull request, #10922:
URL: https://github.com/apache/incubator-gluten/pull/10922
<!--
Thank you for submitting a pull request! Here are some tips:
1. For first-time contributors, please read our contributing guide:
https://github.com/apache/incubator-gluten/blob/main/CONTRIBUTING.md
2. If necessary, create a GitHub issue for discussion beforehand to avoid duplicate work.
3. If the PR is specific to a single backend, include [VL] or [CH] in the PR title to indicate the
Velox or ClickHouse backend, respectively.
4. If the PR is not ready for review, please mark it as a draft.
-->
## What changes are proposed in this pull request?
Add buffer to the shuffle read input stream only if `readerBufferSize` is greater than 0.
## How was this patch tested?
Manually testing internal test case:
add `spark.gluten.sql.columnar.shuffle.readerBufferSize=1MB;` conf:
<img width="1906" height="89" alt="image" src="https://github.com/user-attachments/assets/b9f1f640-27e3-47da-9504-b777cc3a97c5" />
add `spark.gluten.sql.columnar.shuffle.readerBufferSize=0;` conf:
<img width="1885" height="53" alt="image" src="https://github.com/user-attachments/assets/e9221517-c6e2-46df-899f-fc7e37a84a1b" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-10920][VL] Allow disabling hash/sort shuffle reader buffer [incubator-gluten]
Posted by "zuston (via GitHub)" <gi...@apache.org>.
zuston commented on PR #10922:
URL: https://github.com/apache/incubator-gluten/pull/10922#issuecomment-3434793392
> @wForget did you set spark.shuffle.io.preferDirectBufs=False?
The issue occurs when using Uniffle, rather than the vanilla shuffle.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-10920][VL] Allow disabling hash/sort shuffle reader buffer [incubator-gluten]
Posted by "wForget (via GitHub)" <gi...@apache.org>.
wForget commented on PR #10922:
URL: https://github.com/apache/incubator-gluten/pull/10922#issuecomment-3434834473
> @marin-ma why it's onheap copy? shouldn't reducer use netty to load data into direct memory?
I filed #10923 for this issue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-10920][VL] Allow disabling hash/sort shuffle reader buffer [incubator-gluten]
Posted by "FelixYBW (via GitHub)" <gi...@apache.org>.
FelixYBW commented on PR #10922:
URL: https://github.com/apache/incubator-gluten/pull/10922#issuecomment-3434079304
@wForget did you set spark.shuffle.io.preferDirectBufs=False?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-10920][VL] Allow disabling hash/sort shuffle reader buffer [incubator-gluten]
Posted by "FelixYBW (via GitHub)" <gi...@apache.org>.
FelixYBW commented on PR #10922:
URL: https://github.com/apache/incubator-gluten/pull/10922#issuecomment-3434031809
@marin-ma why it's onheap copy? shouldn't reducer use netty to load data into direct memory?
<img width="1377" height="137" alt="image" src="https://github.com/user-attachments/assets/77f79037-5885-4e19-8c81-d6b93bbe8027" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-10920][VL] Allow disabling hash/sort shuffle reader buffer [incubator-gluten]
Posted by "wForget (via GitHub)" <gi...@apache.org>.
wForget commented on PR #10922:
URL: https://github.com/apache/incubator-gluten/pull/10922#issuecomment-3434847984
Thanks @zuston @FelixYBW for the review, merged to main
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-10920][VL] Allow disabling hash/sort shuffle reader buffer [incubator-gluten]
Posted by "wForget (via GitHub)" <gi...@apache.org>.
wForget merged PR #10922:
URL: https://github.com/apache/incubator-gluten/pull/10922
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org