You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@uniffle.apache.org by "zuston (via GitHub)" <gi...@apache.org> on 2025/09/29 07:22:00 UTC
[PR] [#2636] feat(spark): Cache shuffle handle info for reader to reduce RPC cost when partition reassign is enabled [uniffle]
zuston opened a new pull request, #2637:
URL: https://github.com/apache/uniffle/pull/2637
### What changes were proposed in this pull request?
This PR is to introduce the cache mechanism to cache the read shuffle handle info to reduce the RPC cost and driver the GC pressure when the partition reassign is enabled
### Why are the changes needed?
for #2636
### Does this PR introduce _any_ user-facing change?
Yes.
`rss.client.read.shuffleHandleCacheEnabled=false`
### How was this patch tested?
Existing tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#2636] feat(spark): Cache shuffle handle info for reader to reduce RPC cost when partition reassign is enabled [uniffle]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #2637:
URL: https://github.com/apache/uniffle/pull/2637#issuecomment-3345630764
## Test Results
3 130 files ±0 3 130 suites ±0 6h 48m 44s ⏱️ - 2m 8s
1 206 tests +1 1 205 ✅ +1 1 💤 ±0 0 ❌ ±0
15 274 runs +8 15 259 ✅ +8 15 💤 ±0 0 ❌ ±0
Results for commit aeecd09d. ± Comparison against base commit 1d162dcf.
[test-results]:data:application/gzip;base64,H4sIAHZB2mgC/03Myw6DIBCF4VcxrLsYLqPYl2nsDCakXhqElem7F61SV+T/hpxV9H5wi7hXWmq4VWJJPl6bU+iin6csyqAymfI9bj+kgvrMx5KIfoZ/e/n3ZgX6zg8ZoIALYQ6HhDTtq6gac2RZRYVtsWMVT7is7n0dpXkcfcwhOueIoWUNRklWsrbUsgVsmHR+yYLr+WlJfL47GlxtFAEAAA==
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#2636] feat(spark): Cache shuffle handle info for reader to reduce RPC cost when partition reassign is enabled [uniffle]
Posted by "zuston (via GitHub)" <gi...@apache.org>.
zuston merged PR #2637:
URL: https://github.com/apache/uniffle/pull/2637
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#2636] feat(spark): Cache shuffle handle info for reader to reduce RPC cost when partition reassign is enabled [uniffle]
Posted by "zuston (via GitHub)" <gi...@apache.org>.
zuston commented on code in PR #2637:
URL: https://github.com/apache/uniffle/pull/2637#discussion_r2386982643
##########
client-spark/common/src/main/java/org/apache/uniffle/shuffle/manager/RssShuffleManagerBase.java:
##########
@@ -444,6 +453,9 @@ public boolean unregisterShuffle(int shuffleId) {
if (blockIdManager != null) {
blockIdManager.remove(shuffleId);
}
+ if (readShuffleHandleCache != null) {
+ readShuffleHandleCache.remove(shuffleId);
Review Comment:
cc @jerqi here to clear cache
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#2636] feat(spark): Cache shuffle handle info for reader to reduce RPC cost when partition reassign is enabled [uniffle]
Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi commented on code in PR #2637:
URL: https://github.com/apache/uniffle/pull/2637#discussion_r2386993856
##########
client-spark/common/src/main/java/org/apache/uniffle/shuffle/manager/RssShuffleManagerBase.java:
##########
@@ -444,6 +453,9 @@ public boolean unregisterShuffle(int shuffleId) {
if (blockIdManager != null) {
blockIdManager.remove(shuffleId);
}
+ if (readShuffleHandleCache != null) {
+ readShuffleHandleCache.remove(shuffleId);
Review Comment:
OK. Got it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#2636] feat(spark): Cache shuffle handle info for reader to reduce RPC cost when partition reassign is enabled [uniffle]
Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi commented on PR #2637:
URL: https://github.com/apache/uniffle/pull/2637#issuecomment-3345441573
When to clean the cache?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org