You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/07/05 09:35:11 UTC
[GitHub] [spark] LuciferYang opened a new pull request, #37086: [SPARK-39680][CORE] Remove `hasSpaceForAnotherRecord` from `UnsafeExternalSorter`
LuciferYang opened a new pull request, #37086:
URL: https://github.com/apache/spark/pull/37086
### What changes were proposed in this pull request?
This pr just remove an unused `@VisibleForTesting` method from `UnsafeExternalSorter` .
### Why are the changes needed?
Code clean.
SPARK-21907 introduce `hasSpaceForAnotherRecord()` method for `UnsafeExternalSorter`, it is identified as `@VisibleForTesting` and only used by `UnsafeExternalSorterSuite#testOOMDuringSpill`. After SPARK-32901, `UnsafeExternalSorterSuite#testOOMDuringSpill` rename to `UnsafeExternalSorterSuite#testNoOOMDuringSpill` and `hasSpaceForAnotherRecord()` method is no longer used in current master now.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Pass GitHub Actions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a diff in pull request #37086: [SPARK-39680][CORE] Remove `hasSpaceForAnotherRecord` from `UnsafeExternalSorter`
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on code in PR #37086:
URL: https://github.com/apache/spark/pull/37086#discussion_r914446291
##########
core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeExternalSorter.java:
##########
@@ -562,10 +562,6 @@ public UnsafeSorterIterator getSortedIterator() throws IOException {
}
}
- @VisibleForTesting boolean hasSpaceForAnotherRecord() {
Review Comment:
Thank you for making a PR, but I'd not remove this. Although this is not used currently, this was added once inevitably. I feel this removal might be too hasty decision because `UnsafeExternalSorter` is designed to be based on `UnsafeInMemorySorter` and `UnsafeInMemorySorter.hasSpaceForAnotherRecord()` is a useful public method. It still looks okay to me to have this `VisibleForTesting` method to expose the underlying structure.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #37086: [SPARK-39680][CORE] Remove `hasSpaceForAnotherRecord` from `UnsafeExternalSorter`
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on PR #37086:
URL: https://github.com/apache/spark/pull/37086#issuecomment-1177109338
> Thank you for closing and sorry for your loss of this PR. If we want to remove this, we can reuse this JIRA and PR.
OK ~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #37086: [SPARK-39680][CORE] Remove `hasSpaceForAnotherRecord` from `UnsafeExternalSorter`
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on PR #37086:
URL: https://github.com/apache/spark/pull/37086#issuecomment-1177092348
Thank you for closing and sorry for your loss of this PR. If we want to remove this, we can reuse this JIRA and PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #37086: [SPARK-39680][CORE] Remove `hasSpaceForAnotherRecord` from `UnsafeExternalSorter`
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on PR #37086:
URL: https://github.com/apache/spark/pull/37086#issuecomment-1175821690
thanks for your review @dongjoon-hyun
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang closed pull request #37086: [SPARK-39680][CORE] Remove `hasSpaceForAnotherRecord` from `UnsafeExternalSorter`
Posted by GitBox <gi...@apache.org>.
LuciferYang closed pull request #37086: [SPARK-39680][CORE] Remove `hasSpaceForAnotherRecord` from `UnsafeExternalSorter`
URL: https://github.com/apache/spark/pull/37086
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a diff in pull request #37086: [SPARK-39680][CORE] Remove `hasSpaceForAnotherRecord` from `UnsafeExternalSorter`
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on code in PR #37086:
URL: https://github.com/apache/spark/pull/37086#discussion_r914446291
##########
core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeExternalSorter.java:
##########
@@ -562,10 +562,6 @@ public UnsafeSorterIterator getSortedIterator() throws IOException {
}
}
- @VisibleForTesting boolean hasSpaceForAnotherRecord() {
Review Comment:
Thank you for making a PR, but I'd not remove this. Although this is not used currently, this was added once inevitably. I feel this removal might become a too hasty decision because `UnsafeExternalSorter` is designed to be based on `UnsafeInMemorySorter` and `UnsafeInMemorySorter.hasSpaceForAnotherRecord()` is a useful public method. It still looks okay to me to have this `VisibleForTesting` method to expose the underlying structure.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #37086: [SPARK-39680][CORE] Remove `hasSpaceForAnotherRecord` from `UnsafeExternalSorter`
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on PR #37086:
URL: https://github.com/apache/spark/pull/37086#issuecomment-1175821199
> May I ask if you are currently some other refactoring on this area? I'm wondering if you want to encapsulate back for some reasons or not.
Not yet, just see this method is no longer used. I will close this pr
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org