You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/04/10 13:53:31 UTC
[GitHub] [spark] LuciferYang opened a new pull request, #36132: [SPARK-38847][CORE][SQL] Introduce a `viewToSeq` function for `KVUtils`
LuciferYang opened a new pull request, #36132:
URL: https://github.com/apache/spark/pull/36132
### What changes were proposed in this pull request?
There are many codes in spark that convert `KVStoreView` into scala `List`, and these codes will not close `KVStoreIterator`, these resources are mainly recycled by `finalize()` method implemented in `LevelDB` and `RockSB`, this makes `KVStoreIterator` resource recycling unpredictable.
This pr introduce a `viewToSeq` function for `KVUtils`, this function will convert all data in the `KVStoreView` into scala `List` and close `KVStoreIterator` in time.
### Why are the changes needed?
Add a general function to convert `KVStoreView` into scala `List` and close `KVStoreIterator` in time.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Pass GA
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen commented on pull request #36132: [SPARK-38847][CORE][SQL][SS] Introduce a `viewToSeq` function for `KVUtils` to close `KVStoreIterator` in time
Posted by GitBox <gi...@apache.org>.
srowen commented on PR #36132:
URL: https://github.com/apache/spark/pull/36132#issuecomment-1096799682
Merged to master
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen closed pull request #36132: [SPARK-38847][CORE][SQL][SS] Introduce a `viewToSeq` function for `KVUtils` to close `KVStoreIterator` in time
Posted by GitBox <gi...@apache.org>.
srowen closed pull request #36132: [SPARK-38847][CORE][SQL][SS] Introduce a `viewToSeq` function for `KVUtils` to close `KVStoreIterator` in time
URL: https://github.com/apache/spark/pull/36132
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a diff in pull request #36132: [SPARK-38847][CORE][SQL][SS] Introduce a `viewToSeq` function for `KVUtils` to close `KVStoreIterator` in time
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on code in PR #36132:
URL: https://github.com/apache/spark/pull/36132#discussion_r846959415
##########
core/src/main/scala/org/apache/spark/status/KVUtils.scala:
##########
@@ -92,6 +92,16 @@ private[spark] object KVUtils extends Logging {
}
}
+ /** Turns a KVStoreView into a Scala sequence. */
+ def viewToSeq[T](view: KVStoreView[T]): Seq[T] = {
+ val iter = view.closeableIterator()
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a diff in pull request #36132: [SPARK-38847][CORE][SQL][SS] Introduce a `viewToSeq` function for `KVUtils` to close `KVStoreIterator` in time
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on code in PR #36132:
URL: https://github.com/apache/spark/pull/36132#discussion_r846956224
##########
core/src/main/scala/org/apache/spark/status/KVUtils.scala:
##########
@@ -92,6 +92,16 @@ private[spark] object KVUtils extends Logging {
}
}
+ /** Turns a KVStoreView into a Scala sequence. */
+ def viewToSeq[T](view: KVStoreView[T]): Seq[T] = {
+ val iter = view.closeableIterator()
Review Comment:
Maybe `Utils.tryWithResource`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #36132: [SPARK-38847][CORE][SQL][SS] Introduce a `viewToSeq` function for `KVUtils` to close `KVStoreIterator` in time
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on PR #36132:
URL: https://github.com/apache/spark/pull/36132#issuecomment-1096800970
thanks @srowen
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org