You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@uniffle.apache.org by "leixm (via GitHub)" <gi...@apache.org> on 2023/03/23 12:14:43 UTC
[GitHub] [incubator-uniffle] leixm opened a new issue, #756: [Improvement] ShuffleBufferManager.flushSingleBufferIfNecessary unreasonable
leixm opened a new issue, #756:
URL: https://github.com/apache/incubator-uniffle/issues/756
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
### Search before asking
- [X] I have searched in the [issues](https://github.com/apache/incubator-uniffle/issues?q=is%3Aissue) and found no similar issues.
### Describe the bug
After #471 , single buffer flush feature does not work, when rss.server.single.buffer.flush.enabled=true, and the data size of this batch of blocks in the cache of a partition exceeds rss.server.single.buffer.flush.threshold, the current scenario will not To trigger flush, I think it is better to trigger flush instead of waiting to meet the conditions of huge partition
https://github.com/apache/incubator-uniffle/blob/master/server/src/main/java/org/apache/uniffle/server/buffer/ShuffleBufferManager.java#228
`if ((isHugePartition || this.bufferFlushEnabled) && buffer.getSize() > this.bufferFlushThreshold) {`
### Affects Version(s)
master
### Uniffle Server Log Output
_No response_
### Uniffle Engine Log Output
_No response_
### Uniffle Server Configurations
_No response_
### Uniffle Engine Configurations
_No response_
### Additional context
_No response_
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] leixm commented on issue #756: [Improvement] ShuffleBufferManager.flushSingleBufferIfNecessary unreasonable
Posted by "leixm (via GitHub)" <gi...@apache.org>.
leixm commented on issue #756:
URL: https://github.com/apache/incubator-uniffle/issues/756#issuecomment-1481152107
cc @zuston
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] zuston commented on issue #756: [Improvement] ShuffleBufferManager.flushSingleBufferIfNecessary unreasonable
Posted by "zuston (via GitHub)" <gi...@apache.org>.
zuston commented on issue #756:
URL: https://github.com/apache/incubator-uniffle/issues/756#issuecomment-1481104527
It’s my mistake, could you help fix?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] leixm commented on issue #756: [Improvement] ShuffleBufferManager.flushSingleBufferIfNecessary unreasonable
Posted by "leixm (via GitHub)" <gi...@apache.org>.
leixm commented on issue #756:
URL: https://github.com/apache/incubator-uniffle/issues/756#issuecomment-1481150837
I misunderstood just now, the single-partition flush still takes effect, but when the data size of a single partition reaches rss.server.huge-partition.size.threshold, can flush be triggered without waiting for rss.server.single.buffer.flush.threshold satisfy.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] leixm closed issue #756: [Improvement] ShuffleBufferManager.flushSingleBufferIfNecessary unreasonable
Posted by "leixm (via GitHub)" <gi...@apache.org>.
leixm closed issue #756: [Improvement] ShuffleBufferManager.flushSingleBufferIfNecessary unreasonable
URL: https://github.com/apache/incubator-uniffle/issues/756
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] leixm commented on issue #756: [Improvement] ShuffleBufferManager.flushSingleBufferIfNecessary unreasonable
Posted by "leixm (via GitHub)" <gi...@apache.org>.
leixm commented on issue #756:
URL: https://github.com/apache/incubator-uniffle/issues/756#issuecomment-1481089359
@zuston PTAL.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] leixm commented on issue #756: [Improvement] ShuffleBufferManager.flushSingleBufferIfNecessary unreasonable
Posted by "leixm (via GitHub)" <gi...@apache.org>.
leixm commented on issue #756:
URL: https://github.com/apache/incubator-uniffle/issues/756#issuecomment-1482347223
Misunderstood before, seems reasonable, I will close this issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] leixm commented on issue #756: [Improvement] ShuffleBufferManager.flushSingleBufferIfNecessary unreasonable
Posted by "leixm (via GitHub)" <gi...@apache.org>.
leixm commented on issue #756:
URL: https://github.com/apache/incubator-uniffle/issues/756#issuecomment-1481154214
((isHugePartition || this.bufferFlushEnabled) && buffer.getSize() > this.bufferFlushThreshold) -> (isHugePartition || (this.bufferFlushEnabled && buffer.getSize() > this.bufferFlushThreshold))
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] zuston commented on issue #756: [Improvement] ShuffleBufferManager.flushSingleBufferIfNecessary unreasonable
Posted by "zuston (via GitHub)" <gi...@apache.org>.
zuston commented on issue #756:
URL: https://github.com/apache/incubator-uniffle/issues/756#issuecomment-1482148132
> I misunderstood just now, the single-partition flush still takes effect, but when the data size of a single partition reaches rss.server.huge-partition.size.threshold, can flush be triggered without waiting for rss.server.single.buffer.flush.threshold satisfy.
Yes. I have rechecked, this is reasonable. WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org