You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/04/04 12:07:51 UTC
[GitHub] [spark] weixiuli opened a new pull request, #36060: [SPARK-38683][SHUFFLE] It is unnecessary to release the ShuffleManagedBufferIterator or ShuffleChunkManagedBufferIterator or ManagedBufferIterator buffers when the client channel's connection is terminated.
weixiuli opened a new pull request, #36060:
URL: https://github.com/apache/spark/pull/36060
### What changes were proposed in this pull request?
It is unnecessary to release the ShuffleManagedBufferIterator or ShuffleChunkManagedBufferIterator or ManagedBufferIterator buffers when the client channel's connection is terminated.
If a client connection is closed before the iterator is fully drained, then the remaining materialized buffers should all be released, but some buffers like `ShuffleManagedBufferIterator`, `ShuffleChunkManagedBufferIterator`, `ManagedBufferIterator` are not materialized until the iterator is traversed by calling next(), so we should not traverse and release them in order to avoid unnecessary buffer materialization, which could be I/O based.
### Why are the changes needed?
To reduce I/O operations for the External Shuffle Service.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Existing unittests.
Closes #36000 from weixiuli/SPARK-38683-unnecessary-release.
Authored-by: weixiuli <we...@jd.com>
(cherry picked from commit https://github.com/apache/spark/commit/dc995f29592c720b959c877e31133bcd706d9b4e)
Signed-off-by: yi.wu <yi...@databricks.com>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #36060: [SPARK-38683][SHUFFLE] It is unnecessary to release the ShuffleManagedBufferIterator or ShuffleChunkManagedBufferIterator or ManagedBufferIterator buffers when the client channel's connection is terminated.
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on PR #36060:
URL: https://github.com/apache/spark/pull/36060#issuecomment-1087775987
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun closed pull request #36060: [SPARK-38683][SHUFFLE][3.2] It is unnecessary to release the ShuffleManagedBufferIterator or ShuffleChunkManagedBufferIterator or ManagedBufferIterator buffers when the client channel's connection is terminated.
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun closed pull request #36060: [SPARK-38683][SHUFFLE][3.2] It is unnecessary to release the ShuffleManagedBufferIterator or ShuffleChunkManagedBufferIterator or ManagedBufferIterator buffers when the client channel's connection is terminated.
URL: https://github.com/apache/spark/pull/36060
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #36060: [SPARK-38683][SHUFFLE][3.2] It is unnecessary to release the ShuffleManagedBufferIterator or ShuffleChunkManagedBufferIterator or ManagedBufferIterator buffers when the client channel's connection is terminated.
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on PR #36060:
URL: https://github.com/apache/spark/pull/36060#issuecomment-1088144612
Hi, @weixiuli .
We don't backport improvement JIRAs. Let me close this first.
We can continue to discuss if needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org