You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/08/31 03:40:49 UTC
[GitHub] [spark] HeartSaVioR opened a new pull request #33870: [SPARK-36619][SS] Reset prefix-scan iterators on loading RocksDB
HeartSaVioR opened a new pull request #33870:
URL: https://github.com/apache/spark/pull/33870
### What changes were proposed in this pull request?
This PR ensures that resetting prefix-scan iterators is done on loading RocksDB, which was only done in rollback.
### Why are the changes needed?
Prefix-scan iterators are only closed on RocksDB.rollback(), which is only called in RocksDBStateStore.abort().
While `RocksDBStateStore.abort()` method will be called for streaming session window (since it has two physical plans for read and write), other stateful operators which only have read-write physical plan will call either commit or abort, and don't close the iterators on committing.
These unclosed iterators can be "reused" and produce incorrect outputs.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Modified UT which failed without this PR and passes with this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33870: [SPARK-36619][SS] Reset prefix-scan iterators on loading RocksDB
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33870:
URL: https://github.com/apache/spark/pull/33870#issuecomment-908927114
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47377/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on a change in pull request #33870: [SPARK-36619][SS] Fix bugs around prefix-scan for HDFS backed state store and RocksDB state store
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on a change in pull request #33870:
URL: https://github.com/apache/spark/pull/33870#discussion_r699309363
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreMap.scala
##########
@@ -129,7 +129,11 @@ class PrefixScannableHDFSBackedStateStoreMap(
other match {
case o: PrefixScannableHDFSBackedStateStoreMap =>
map.putAll(o.map)
- prefixKeyToKeysMap.putAll(o.prefixKeyToKeysMap)
+ o.prefixKeyToKeysMap.asScala.foreach { case (prefixKey, keySet) =>
+ val newSet = new mutable.HashSet[UnsafeRow]()
Review comment:
This looks confusing to me. Shall we do the following?
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreMap.scala
##########
@@ -129,7 +129,11 @@ class PrefixScannableHDFSBackedStateStoreMap(
other match {
case o: PrefixScannableHDFSBackedStateStoreMap =>
map.putAll(o.map)
- prefixKeyToKeysMap.putAll(o.prefixKeyToKeysMap)
+ o.prefixKeyToKeysMap.asScala.foreach { case (prefixKey, keySet) =>
+ val newSet = new mutable.HashSet[UnsafeRow]()
Review comment:
```
val cloneMap = new java.util.concurrent.ConcurrentHashMap
[UnsafeRow, mutable.Set[UnsafeRow]](o.prefixKeyToKeysMap)
prefixKeyToKeysMap.putAll(cloneMap)
```
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreMap.scala
##########
@@ -129,7 +129,11 @@ class PrefixScannableHDFSBackedStateStoreMap(
other match {
case o: PrefixScannableHDFSBackedStateStoreMap =>
map.putAll(o.map)
- prefixKeyToKeysMap.putAll(o.prefixKeyToKeysMap)
+ o.prefixKeyToKeysMap.asScala.foreach { case (prefixKey, keySet) =>
+ val newSet = new mutable.HashSet[UnsafeRow]()
Review comment:
```
val cloneMap = new java.util.concurrent.ConcurrentHashMap
[UnsafeRow, mutable.Set[UnsafeRow]](o.prefixKeyToKeysMap)
prefixKeyToKeysMap.putAll(cloneMap)
```
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreMap.scala
##########
@@ -129,7 +129,11 @@ class PrefixScannableHDFSBackedStateStoreMap(
other match {
case o: PrefixScannableHDFSBackedStateStoreMap =>
map.putAll(o.map)
- prefixKeyToKeysMap.putAll(o.prefixKeyToKeysMap)
+ o.prefixKeyToKeysMap.asScala.foreach { case (prefixKey, keySet) =>
+ val newSet = new mutable.HashSet[UnsafeRow]()
Review comment:
This looks confusing to me. Shall we do the following?
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreMap.scala
##########
@@ -129,7 +129,11 @@ class PrefixScannableHDFSBackedStateStoreMap(
other match {
case o: PrefixScannableHDFSBackedStateStoreMap =>
map.putAll(o.map)
- prefixKeyToKeysMap.putAll(o.prefixKeyToKeysMap)
+ o.prefixKeyToKeysMap.asScala.foreach { case (prefixKey, keySet) =>
+ val newSet = new mutable.HashSet[UnsafeRow]()
Review comment:
```
val cloneMap = new java.util.concurrent.ConcurrentHashMap
[UnsafeRow, mutable.Set[UnsafeRow]](o.prefixKeyToKeysMap)
prefixKeyToKeysMap.putAll(cloneMap)
```
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreMap.scala
##########
@@ -129,7 +129,11 @@ class PrefixScannableHDFSBackedStateStoreMap(
other match {
case o: PrefixScannableHDFSBackedStateStoreMap =>
map.putAll(o.map)
- prefixKeyToKeysMap.putAll(o.prefixKeyToKeysMap)
Review comment:
Let's add one comment here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang closed pull request #33870: [SPARK-36619][SS] Fix bugs around prefix-scan for HDFS backed state store and RocksDB state store
Posted by GitBox <gi...@apache.org>.
gengliangwang closed pull request #33870:
URL: https://github.com/apache/spark/pull/33870
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a change in pull request #33870: [SPARK-36619][SS] Reset prefix-scan iterators on loading RocksDB
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on a change in pull request #33870:
URL: https://github.com/apache/spark/pull/33870#discussion_r698961491
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala
##########
@@ -120,6 +120,9 @@ class RocksDB(
if (conf.resetStatsOnLoad) {
nativeStats.reset
}
+ // reset resources to prevent side-effects from previous loaded version
Review comment:
We did this in rollback case but not in commit case. This doesn't bring problems in current implementation of session window, but I think we should fix it to avoid future correctness problem.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33870: [SPARK-36619][SS] Fix bugs around prefix-scan for HDFS backed state store and RocksDB state store
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33870:
URL: https://github.com/apache/spark/pull/33870#issuecomment-909188932
**[Test build #142887 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142887/testReport)** for PR 33870 at commit [`1b3213d`](https://github.com/apache/spark/commit/1b3213d7b50d8209578abbfa14311d9e321696ab).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33870: [SPARK-36619][SS] Reset prefix-scan iterators on loading RocksDB
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33870:
URL: https://github.com/apache/spark/pull/33870#issuecomment-908963001
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47378/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33870: [SPARK-36619][SS] Reset prefix-scan iterators on loading RocksDB
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33870:
URL: https://github.com/apache/spark/pull/33870#issuecomment-909055692
**[Test build #142875 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142875/testReport)** for PR 33870 at commit [`a01d733`](https://github.com/apache/spark/commit/a01d73309a23058e452b01b19669131dc212380c).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33870: [SPARK-36619][SS] Fix bugs around prefix-scan for HDFS backed state store and RocksDB state store
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33870:
URL: https://github.com/apache/spark/pull/33870#issuecomment-909425180
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142887/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33870: [SPARK-36619][SS] Fix bugs around prefix-scan for HDFS backed state store and RocksDB state store
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33870:
URL: https://github.com/apache/spark/pull/33870#issuecomment-909373881
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47392/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33870: [SPARK-36619][SS] Reset prefix-scan iterators on loading RocksDB
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33870:
URL: https://github.com/apache/spark/pull/33870#issuecomment-908931363
**[Test build #142874 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142874/testReport)** for PR 33870 at commit [`a01d733`](https://github.com/apache/spark/commit/a01d73309a23058e452b01b19669131dc212380c).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #33870: [SPARK-36619][SS] Reset prefix-scan iterators on loading RocksDB
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #33870:
URL: https://github.com/apache/spark/pull/33870#issuecomment-908935408
retest this, please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on a change in pull request #33870: [SPARK-36619][SS] Fix bugs around prefix-scan for HDFS backed state store and RocksDB state store
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on a change in pull request #33870:
URL: https://github.com/apache/spark/pull/33870#discussion_r699309363
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreMap.scala
##########
@@ -129,7 +129,11 @@ class PrefixScannableHDFSBackedStateStoreMap(
other match {
case o: PrefixScannableHDFSBackedStateStoreMap =>
map.putAll(o.map)
- prefixKeyToKeysMap.putAll(o.prefixKeyToKeysMap)
+ o.prefixKeyToKeysMap.asScala.foreach { case (prefixKey, keySet) =>
+ val newSet = new mutable.HashSet[UnsafeRow]()
Review comment:
This looks confusing to me. Shall we do the following?
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreMap.scala
##########
@@ -129,7 +129,11 @@ class PrefixScannableHDFSBackedStateStoreMap(
other match {
case o: PrefixScannableHDFSBackedStateStoreMap =>
map.putAll(o.map)
- prefixKeyToKeysMap.putAll(o.prefixKeyToKeysMap)
+ o.prefixKeyToKeysMap.asScala.foreach { case (prefixKey, keySet) =>
+ val newSet = new mutable.HashSet[UnsafeRow]()
Review comment:
```
val cloneMap = new java.util.concurrent.ConcurrentHashMap
[UnsafeRow, mutable.Set[UnsafeRow]](o.prefixKeyToKeysMap)
prefixKeyToKeysMap.putAll(cloneMap)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33870: [SPARK-36619][SS] Reset prefix-scan iterators on loading RocksDB
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33870:
URL: https://github.com/apache/spark/pull/33870#issuecomment-908936142
**[Test build #142875 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142875/testReport)** for PR 33870 at commit [`a01d733`](https://github.com/apache/spark/commit/a01d73309a23058e452b01b19669131dc212380c).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33870: [SPARK-36619][SS] Fix bugs around prefix-scan for HDFS backed state store and RocksDB state store
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33870:
URL: https://github.com/apache/spark/pull/33870#issuecomment-909250790
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47390/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33870: [SPARK-36619][SS] Fix bugs around prefix-scan for HDFS backed state store and RocksDB state store
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33870:
URL: https://github.com/apache/spark/pull/33870#issuecomment-909188932
**[Test build #142887 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142887/testReport)** for PR 33870 at commit [`1b3213d`](https://github.com/apache/spark/commit/1b3213d7b50d8209578abbfa14311d9e321696ab).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a change in pull request #33870: [SPARK-36619][SS] Fix bugs around prefix-scan for HDFS backed state store and RocksDB state store
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on a change in pull request #33870:
URL: https://github.com/apache/spark/pull/33870#discussion_r699272824
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreMap.scala
##########
@@ -129,7 +129,11 @@ class PrefixScannableHDFSBackedStateStoreMap(
other match {
case o: PrefixScannableHDFSBackedStateStoreMap =>
map.putAll(o.map)
- prefixKeyToKeysMap.putAll(o.prefixKeyToKeysMap)
Review comment:
This leads the prefix key maps across versions to "share" the values (as mutable Set instances) which it shouldn't. (especially the case of rolling back)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #33870: [SPARK-36619][SS] Reset prefix-scan iterators on loading RocksDB
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #33870:
URL: https://github.com/apache/spark/pull/33870#issuecomment-908881532
cc. @viirya @xuanyuanking
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on a change in pull request #33870: [SPARK-36619][SS] Fix bugs around prefix-scan for HDFS backed state store and RocksDB state store
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on a change in pull request #33870:
URL: https://github.com/apache/spark/pull/33870#discussion_r699309838
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreMap.scala
##########
@@ -129,7 +129,11 @@ class PrefixScannableHDFSBackedStateStoreMap(
other match {
case o: PrefixScannableHDFSBackedStateStoreMap =>
map.putAll(o.map)
- prefixKeyToKeysMap.putAll(o.prefixKeyToKeysMap)
+ o.prefixKeyToKeysMap.asScala.foreach { case (prefixKey, keySet) =>
+ val newSet = new mutable.HashSet[UnsafeRow]()
Review comment:
```
val cloneMap = new java.util.concurrent.ConcurrentHashMap
[UnsafeRow, mutable.Set[UnsafeRow]](o.prefixKeyToKeysMap)
prefixKeyToKeysMap.putAll(cloneMap)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR removed a comment on pull request #33870: [SPARK-36619][SS] Reset prefix-scan iterators on loading RocksDB
Posted by GitBox <gi...@apache.org>.
HeartSaVioR removed a comment on pull request #33870:
URL: https://github.com/apache/spark/pull/33870#issuecomment-909082674
Weird, I can't reproduce the failure locally. (MacOS env.) I tried running RocksDBStateStoreSuite in IDEA.
I will try to run the test suite via sbt, and if it didn't work, I'll try to run several RocksDB test suites altogether to see whether the test is affected by other suites.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #33870: [SPARK-36619][SS] Reset prefix-scan iterators on loading RocksDB
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #33870:
URL: https://github.com/apache/spark/pull/33870#issuecomment-909082674
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33870: [SPARK-36619][SS] Reset prefix-scan iterators on loading RocksDB
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33870:
URL: https://github.com/apache/spark/pull/33870#issuecomment-908961530
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47378/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR edited a comment on pull request #33870: [SPARK-36619][SS] Reset prefix-scan iterators on loading RocksDB
Posted by GitBox <gi...@apache.org>.
HeartSaVioR edited a comment on pull request #33870:
URL: https://github.com/apache/spark/pull/33870#issuecomment-908881532
cc. @zsxwing @viirya @xuanyuanking
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33870: [SPARK-36619][SS] Fix bugs around prefix-scan for HDFS backed state store and RocksDB state store
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33870:
URL: https://github.com/apache/spark/pull/33870#issuecomment-909287667
**[Test build #142889 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142889/testReport)** for PR 33870 at commit [`7913e38`](https://github.com/apache/spark/commit/7913e381e36751e61e405d22dbb1418b00a4135b).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33870: [SPARK-36619][SS] Reset prefix-scan iterators on loading RocksDB
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33870:
URL: https://github.com/apache/spark/pull/33870#issuecomment-908901578
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47374/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #33870: [SPARK-36619][SS] Fix bugs around prefix-scan for HDFS backed state store and RocksDB state store
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #33870:
URL: https://github.com/apache/spark/pull/33870#issuecomment-909193006
@xuanyuanking @viirya @gengliangwang @dongjoon-hyun
I'm sorry, but could you please take a look again? I fixed another bug on HDFS backed state store which led failure on new test.
Thanks in advance!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a change in pull request #33870: [SPARK-36619][SS] Fix bugs around prefix-scan for HDFS backed state store and RocksDB state store
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on a change in pull request #33870:
URL: https://github.com/apache/spark/pull/33870#discussion_r699272824
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreMap.scala
##########
@@ -129,7 +129,11 @@ class PrefixScannableHDFSBackedStateStoreMap(
other match {
case o: PrefixScannableHDFSBackedStateStoreMap =>
map.putAll(o.map)
- prefixKeyToKeysMap.putAll(o.prefixKeyToKeysMap)
Review comment:
This leads the prefix key maps across versions to "share" the values which it shouldn't. (especially the case of rolling back)
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreMap.scala
##########
@@ -129,7 +129,11 @@ class PrefixScannableHDFSBackedStateStoreMap(
other match {
case o: PrefixScannableHDFSBackedStateStoreMap =>
map.putAll(o.map)
- prefixKeyToKeysMap.putAll(o.prefixKeyToKeysMap)
Review comment:
This leads the prefix key maps across versions to "share" the values (as mutable Set instances) which it shouldn't. (especially the case of rolling back)
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreMap.scala
##########
@@ -129,7 +129,11 @@ class PrefixScannableHDFSBackedStateStoreMap(
other match {
case o: PrefixScannableHDFSBackedStateStoreMap =>
map.putAll(o.map)
- prefixKeyToKeysMap.putAll(o.prefixKeyToKeysMap)
Review comment:
Thanks! Addressed via 7913e381e36751e61e405d22dbb1418b00a4135b
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33870: [SPARK-36619][SS] Reset prefix-scan iterators on loading RocksDB
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33870:
URL: https://github.com/apache/spark/pull/33870#issuecomment-908931638
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142874/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33870: [SPARK-36619][SS] Reset prefix-scan iterators on loading RocksDB
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33870:
URL: https://github.com/apache/spark/pull/33870#issuecomment-908997741
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142871/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang closed pull request #33870: [SPARK-36619][SS] Fix bugs around prefix-scan for HDFS backed state store and RocksDB state store
Posted by GitBox <gi...@apache.org>.
gengliangwang closed pull request #33870:
URL: https://github.com/apache/spark/pull/33870
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33870: [SPARK-36619][SS] Reset prefix-scan iterators on loading RocksDB
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33870:
URL: https://github.com/apache/spark/pull/33870#issuecomment-908903853
**[Test build #142874 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142874/testReport)** for PR 33870 at commit [`a01d733`](https://github.com/apache/spark/commit/a01d73309a23058e452b01b19669131dc212380c).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33870: [SPARK-36619][SS] Reset prefix-scan iterators on loading RocksDB
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33870:
URL: https://github.com/apache/spark/pull/33870#issuecomment-908927114
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47377/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a change in pull request #33870: [SPARK-36619][SS] Reset prefix-scan iterators on loading RocksDB
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on a change in pull request #33870:
URL: https://github.com/apache/spark/pull/33870#discussion_r698961491
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala
##########
@@ -120,6 +120,9 @@ class RocksDB(
if (conf.resetStatsOnLoad) {
nativeStats.reset
}
+ // reset resources to prevent side-effects from previous loaded version
Review comment:
We did this in rollback case but not in commit case. This doesn't bring problems in current implementation of session window, but I think we should fix it to avoid future correctness problem.
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala
##########
@@ -120,6 +120,9 @@ class RocksDB(
if (conf.resetStatsOnLoad) {
nativeStats.reset
}
+ // reset resources to prevent side-effects from previous loaded version
+ prefixScanReuseIter.entrySet().asScala.foreach(_.getValue.close())
+ prefixScanReuseIter.clear()
Review comment:
Thanks! Addressed via a01d733
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreMap.scala
##########
@@ -129,7 +129,11 @@ class PrefixScannableHDFSBackedStateStoreMap(
other match {
case o: PrefixScannableHDFSBackedStateStoreMap =>
map.putAll(o.map)
- prefixKeyToKeysMap.putAll(o.prefixKeyToKeysMap)
Review comment:
This leads the prefix key maps across versions to "share" the values which it shouldn't. (especially the case of rolling back)
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreMap.scala
##########
@@ -129,7 +129,11 @@ class PrefixScannableHDFSBackedStateStoreMap(
other match {
case o: PrefixScannableHDFSBackedStateStoreMap =>
map.putAll(o.map)
- prefixKeyToKeysMap.putAll(o.prefixKeyToKeysMap)
Review comment:
This leads the prefix key maps across versions to "share" the values (as mutable Set instances) which it shouldn't. (especially the case of rolling back)
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreMap.scala
##########
@@ -129,7 +129,11 @@ class PrefixScannableHDFSBackedStateStoreMap(
other match {
case o: PrefixScannableHDFSBackedStateStoreMap =>
map.putAll(o.map)
- prefixKeyToKeysMap.putAll(o.prefixKeyToKeysMap)
Review comment:
Thanks! Addressed via 7913e381e36751e61e405d22dbb1418b00a4135b
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33870: [SPARK-36619][SS] Fix bugs around prefix-scan for HDFS backed state store and RocksDB state store
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33870:
URL: https://github.com/apache/spark/pull/33870#issuecomment-909217470
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47390/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33870: [SPARK-36619][SS] Reset prefix-scan iterators on loading RocksDB
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33870:
URL: https://github.com/apache/spark/pull/33870#issuecomment-908901578
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33870: [SPARK-36619][SS] Fix bugs around prefix-scan for HDFS backed state store and RocksDB state store
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33870:
URL: https://github.com/apache/spark/pull/33870#issuecomment-909373881
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47392/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33870: [SPARK-36619][SS] Reset prefix-scan iterators on loading RocksDB
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33870:
URL: https://github.com/apache/spark/pull/33870#issuecomment-908917688
Kubernetes integration test unable to build dist.
exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47377/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #33870: [SPARK-36619][SS] Reset prefix-scan iterators on loading RocksDB
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #33870:
URL: https://github.com/apache/spark/pull/33870#issuecomment-908881532
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33870: [SPARK-36619][SS] Reset prefix-scan iterators on loading RocksDB
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33870:
URL: https://github.com/apache/spark/pull/33870#issuecomment-908894643
Kubernetes integration test unable to build dist.
exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47374/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a change in pull request #33870: [SPARK-36619][SS] Reset prefix-scan iterators on loading RocksDB
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on a change in pull request #33870:
URL: https://github.com/apache/spark/pull/33870#discussion_r698982058
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala
##########
@@ -120,6 +120,9 @@ class RocksDB(
if (conf.resetStatsOnLoad) {
nativeStats.reset
}
+ // reset resources to prevent side-effects from previous loaded version
+ prefixScanReuseIter.entrySet().asScala.foreach(_.getValue.close())
+ prefixScanReuseIter.clear()
Review comment:
Thanks! Addressed via a01d733
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a change in pull request #33870: [SPARK-36619][SS] Fix bugs around prefix-scan for HDFS backed state store and RocksDB state store
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on a change in pull request #33870:
URL: https://github.com/apache/spark/pull/33870#discussion_r699373911
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreMap.scala
##########
@@ -129,7 +129,11 @@ class PrefixScannableHDFSBackedStateStoreMap(
other match {
case o: PrefixScannableHDFSBackedStateStoreMap =>
map.putAll(o.map)
- prefixKeyToKeysMap.putAll(o.prefixKeyToKeysMap)
Review comment:
Thanks! Addressed via 7913e381e36751e61e405d22dbb1418b00a4135b
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR edited a comment on pull request #33870: [SPARK-36619][SS] Reset prefix-scan iterators on loading RocksDB
Posted by GitBox <gi...@apache.org>.
HeartSaVioR edited a comment on pull request #33870:
URL: https://github.com/apache/spark/pull/33870#issuecomment-908881532
cc. @zsxwing @viirya @xuanyuanking
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on a change in pull request #33870: [SPARK-36619][SS] Fix bugs around prefix-scan for HDFS backed state store and RocksDB state store
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on a change in pull request #33870:
URL: https://github.com/apache/spark/pull/33870#discussion_r699309838
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreMap.scala
##########
@@ -129,7 +129,11 @@ class PrefixScannableHDFSBackedStateStoreMap(
other match {
case o: PrefixScannableHDFSBackedStateStoreMap =>
map.putAll(o.map)
- prefixKeyToKeysMap.putAll(o.prefixKeyToKeysMap)
+ o.prefixKeyToKeysMap.asScala.foreach { case (prefixKey, keySet) =>
+ val newSet = new mutable.HashSet[UnsafeRow]()
Review comment:
```
val cloneMap = new java.util.concurrent.ConcurrentHashMap
[UnsafeRow, mutable.Set[UnsafeRow]](o.prefixKeyToKeysMap)
prefixKeyToKeysMap.putAll(cloneMap)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33870: [SPARK-36619][SS] Reset prefix-scan iterators on loading RocksDB
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33870:
URL: https://github.com/apache/spark/pull/33870#issuecomment-908901578
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47374/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33870: [SPARK-36619][SS] Reset prefix-scan iterators on loading RocksDB
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33870:
URL: https://github.com/apache/spark/pull/33870#issuecomment-909055692
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33870: [SPARK-36619][SS] Fix bugs around prefix-scan for HDFS backed state store and RocksDB state store
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33870:
URL: https://github.com/apache/spark/pull/33870#issuecomment-909270480
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47390/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33870: [SPARK-36619][SS] Reset prefix-scan iterators on loading RocksDB
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33870:
URL: https://github.com/apache/spark/pull/33870#issuecomment-909056773
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142875/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33870: [SPARK-36619][SS] Reset prefix-scan iterators on loading RocksDB
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33870:
URL: https://github.com/apache/spark/pull/33870#issuecomment-908963001
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47378/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on a change in pull request #33870: [SPARK-36619][SS] Fix bugs around prefix-scan for HDFS backed state store and RocksDB state store
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on a change in pull request #33870:
URL: https://github.com/apache/spark/pull/33870#discussion_r699330317
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreMap.scala
##########
@@ -129,7 +129,11 @@ class PrefixScannableHDFSBackedStateStoreMap(
other match {
case o: PrefixScannableHDFSBackedStateStoreMap =>
map.putAll(o.map)
- prefixKeyToKeysMap.putAll(o.prefixKeyToKeysMap)
Review comment:
Let's add one comment here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33870: [SPARK-36619][SS] Fix bugs around prefix-scan for HDFS backed state store and RocksDB state store
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33870:
URL: https://github.com/apache/spark/pull/33870#issuecomment-909423601
**[Test build #142887 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142887/testReport)** for PR 33870 at commit [`1b3213d`](https://github.com/apache/spark/commit/1b3213d7b50d8209578abbfa14311d9e321696ab).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33870: [SPARK-36619][SS] Reset prefix-scan iterators on loading RocksDB
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33870:
URL: https://github.com/apache/spark/pull/33870#issuecomment-908882245
**[Test build #142871 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142871/testReport)** for PR 33870 at commit [`2b4f00d`](https://github.com/apache/spark/commit/2b4f00dd0cc6d7feca6c5f7cb548692cb2d98146).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #33870: [SPARK-36619][SS] Reset prefix-scan iterators on loading RocksDB
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #33870:
URL: https://github.com/apache/spark/pull/33870#issuecomment-909082674
Weird, I can't reproduce the failure locally. (MacOS env.) I tried running RocksDBStateStoreSuite in IDEA.
I will try to run the test suite via sbt, and if it didn't work, I'll try to run several RocksDB test suites altogether to see whether the test is affected by other suites.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #33870: [SPARK-36619][SS] Fix bugs around prefix-scan for HDFS backed state store and RocksDB state store
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #33870:
URL: https://github.com/apache/spark/pull/33870#issuecomment-909711314
Thanks all for reviewing and merging actively!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33870: [SPARK-36619][SS] Reset prefix-scan iterators on loading RocksDB
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33870:
URL: https://github.com/apache/spark/pull/33870#issuecomment-909056773
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33870: [SPARK-36619][SS] Reset prefix-scan iterators on loading RocksDB
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33870:
URL: https://github.com/apache/spark/pull/33870#issuecomment-908901578
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on a change in pull request #33870: [SPARK-36619][SS] Fix bugs around prefix-scan for HDFS backed state store and RocksDB state store
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on a change in pull request #33870:
URL: https://github.com/apache/spark/pull/33870#discussion_r699309363
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreMap.scala
##########
@@ -129,7 +129,11 @@ class PrefixScannableHDFSBackedStateStoreMap(
other match {
case o: PrefixScannableHDFSBackedStateStoreMap =>
map.putAll(o.map)
- prefixKeyToKeysMap.putAll(o.prefixKeyToKeysMap)
+ o.prefixKeyToKeysMap.asScala.foreach { case (prefixKey, keySet) =>
+ val newSet = new mutable.HashSet[UnsafeRow]()
Review comment:
This looks confusing to me. Shall we do the following?
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreMap.scala
##########
@@ -129,7 +129,11 @@ class PrefixScannableHDFSBackedStateStoreMap(
other match {
case o: PrefixScannableHDFSBackedStateStoreMap =>
map.putAll(o.map)
- prefixKeyToKeysMap.putAll(o.prefixKeyToKeysMap)
+ o.prefixKeyToKeysMap.asScala.foreach { case (prefixKey, keySet) =>
+ val newSet = new mutable.HashSet[UnsafeRow]()
Review comment:
```
val cloneMap = new java.util.concurrent.ConcurrentHashMap
[UnsafeRow, mutable.Set[UnsafeRow]](o.prefixKeyToKeysMap)
prefixKeyToKeysMap.putAll(cloneMap)
```
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreMap.scala
##########
@@ -129,7 +129,11 @@ class PrefixScannableHDFSBackedStateStoreMap(
other match {
case o: PrefixScannableHDFSBackedStateStoreMap =>
map.putAll(o.map)
- prefixKeyToKeysMap.putAll(o.prefixKeyToKeysMap)
+ o.prefixKeyToKeysMap.asScala.foreach { case (prefixKey, keySet) =>
+ val newSet = new mutable.HashSet[UnsafeRow]()
Review comment:
```
val cloneMap = new java.util.concurrent.ConcurrentHashMap
[UnsafeRow, mutable.Set[UnsafeRow]](o.prefixKeyToKeysMap)
prefixKeyToKeysMap.putAll(cloneMap)
```
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreMap.scala
##########
@@ -129,7 +129,11 @@ class PrefixScannableHDFSBackedStateStoreMap(
other match {
case o: PrefixScannableHDFSBackedStateStoreMap =>
map.putAll(o.map)
- prefixKeyToKeysMap.putAll(o.prefixKeyToKeysMap)
+ o.prefixKeyToKeysMap.asScala.foreach { case (prefixKey, keySet) =>
+ val newSet = new mutable.HashSet[UnsafeRow]()
Review comment:
This looks confusing to me. Shall we do the following?
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreMap.scala
##########
@@ -129,7 +129,11 @@ class PrefixScannableHDFSBackedStateStoreMap(
other match {
case o: PrefixScannableHDFSBackedStateStoreMap =>
map.putAll(o.map)
- prefixKeyToKeysMap.putAll(o.prefixKeyToKeysMap)
+ o.prefixKeyToKeysMap.asScala.foreach { case (prefixKey, keySet) =>
+ val newSet = new mutable.HashSet[UnsafeRow]()
Review comment:
```
val cloneMap = new java.util.concurrent.ConcurrentHashMap
[UnsafeRow, mutable.Set[UnsafeRow]](o.prefixKeyToKeysMap)
prefixKeyToKeysMap.putAll(cloneMap)
```
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreMap.scala
##########
@@ -129,7 +129,11 @@ class PrefixScannableHDFSBackedStateStoreMap(
other match {
case o: PrefixScannableHDFSBackedStateStoreMap =>
map.putAll(o.map)
- prefixKeyToKeysMap.putAll(o.prefixKeyToKeysMap)
Review comment:
Let's add one comment here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on pull request #33870: [SPARK-36619][SS] Fix bugs around prefix-scan for HDFS backed state store and RocksDB state store
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on pull request #33870:
URL: https://github.com/apache/spark/pull/33870#issuecomment-909410233
Merging to master/3.2
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33870: [SPARK-36619][SS] Reset prefix-scan iterators on loading RocksDB
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33870:
URL: https://github.com/apache/spark/pull/33870#issuecomment-908936142
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33870: [SPARK-36619][SS] Reset prefix-scan iterators on loading RocksDB
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33870:
URL: https://github.com/apache/spark/pull/33870#issuecomment-908903853
**[Test build #142874 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142874/testReport)** for PR 33870 at commit [`a01d733`](https://github.com/apache/spark/commit/a01d73309a23058e452b01b19669131dc212380c).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on a change in pull request #33870: [SPARK-36619][SS] Fix bugs around prefix-scan for HDFS backed state store and RocksDB state store
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on a change in pull request #33870:
URL: https://github.com/apache/spark/pull/33870#discussion_r699309363
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreMap.scala
##########
@@ -129,7 +129,11 @@ class PrefixScannableHDFSBackedStateStoreMap(
other match {
case o: PrefixScannableHDFSBackedStateStoreMap =>
map.putAll(o.map)
- prefixKeyToKeysMap.putAll(o.prefixKeyToKeysMap)
+ o.prefixKeyToKeysMap.asScala.foreach { case (prefixKey, keySet) =>
+ val newSet = new mutable.HashSet[UnsafeRow]()
Review comment:
This looks confusing to me. Shall we do the following?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR removed a comment on pull request #33870: [SPARK-36619][SS] Reset prefix-scan iterators on loading RocksDB
Posted by GitBox <gi...@apache.org>.
HeartSaVioR removed a comment on pull request #33870:
URL: https://github.com/apache/spark/pull/33870#issuecomment-909082674
Weird, I can't reproduce the failure locally. (MacOS env.) I tried running RocksDBStateStoreSuite in IDEA.
I will try to run the test suite via sbt, and if it didn't work, I'll try to run several RocksDB test suites altogether to see whether the test is affected by other suites.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xuanyuanking commented on a change in pull request #33870: [SPARK-36619][SS] Reset prefix-scan iterators on loading RocksDB
Posted by GitBox <gi...@apache.org>.
xuanyuanking commented on a change in pull request #33870:
URL: https://github.com/apache/spark/pull/33870#discussion_r698968561
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala
##########
@@ -120,6 +120,9 @@ class RocksDB(
if (conf.resetStatsOnLoad) {
nativeStats.reset
}
+ // reset resources to prevent side-effects from previous loaded version
+ prefixScanReuseIter.entrySet().asScala.foreach(_.getValue.close())
+ prefixScanReuseIter.clear()
Review comment:
nit: How about wrap this 2 lines in a function `closePrefixScan(Reuse?)Iter()`? Since they are already been called in 3 places.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR removed a comment on pull request #33870: [SPARK-36619][SS] Reset prefix-scan iterators on loading RocksDB
Posted by GitBox <gi...@apache.org>.
HeartSaVioR removed a comment on pull request #33870:
URL: https://github.com/apache/spark/pull/33870#issuecomment-909082674
Weird, I can't reproduce the failure locally. (MacOS env.) I tried running RocksDBStateStoreSuite in IDEA.
I will try to run the test suite via sbt, and if it didn't work, I'll try to run several RocksDB test suites altogether to see whether the test is affected by other suites.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33870: [SPARK-36619][SS] Fix bugs around prefix-scan for HDFS backed state store and RocksDB state store
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33870:
URL: https://github.com/apache/spark/pull/33870#issuecomment-909425180
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142887/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33870: [SPARK-36619][SS] Reset prefix-scan iterators on loading RocksDB
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33870:
URL: https://github.com/apache/spark/pull/33870#issuecomment-908982314
**[Test build #142871 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142871/testReport)** for PR 33870 at commit [`2b4f00d`](https://github.com/apache/spark/commit/2b4f00dd0cc6d7feca6c5f7cb548692cb2d98146).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #33870: [SPARK-36619][SS] Reset prefix-scan iterators on loading RocksDB
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #33870:
URL: https://github.com/apache/spark/pull/33870#issuecomment-909142440
Looks like there is another bug in HDFSBackedStateStore as well. I'm looking into it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on pull request #33870: [SPARK-36619][SS] Fix bugs around prefix-scan for HDFS backed state store and RocksDB state store
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on pull request #33870:
URL: https://github.com/apache/spark/pull/33870#issuecomment-909410233
Merging to master/3.2
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33870: [SPARK-36619][SS] Reset prefix-scan iterators on loading RocksDB
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33870:
URL: https://github.com/apache/spark/pull/33870#issuecomment-908931638
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142874/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33870: [SPARK-36619][SS] Reset prefix-scan iterators on loading RocksDB
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33870:
URL: https://github.com/apache/spark/pull/33870#issuecomment-908882245
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33870: [SPARK-36619][SS] Reset prefix-scan iterators on loading RocksDB
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33870:
URL: https://github.com/apache/spark/pull/33870#issuecomment-908936142
**[Test build #142875 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142875/testReport)** for PR 33870 at commit [`a01d733`](https://github.com/apache/spark/commit/a01d73309a23058e452b01b19669131dc212380c).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33870: [SPARK-36619][SS] Reset prefix-scan iterators on loading RocksDB
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33870:
URL: https://github.com/apache/spark/pull/33870#issuecomment-908997741
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142871/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33870: [SPARK-36619][SS] Reset prefix-scan iterators on loading RocksDB
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33870:
URL: https://github.com/apache/spark/pull/33870#issuecomment-908962960
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47378/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33870: [SPARK-36619][SS] Reset prefix-scan iterators on loading RocksDB
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33870:
URL: https://github.com/apache/spark/pull/33870#issuecomment-909056773
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142875/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33870: [SPARK-36619][SS] Fix bugs around prefix-scan for HDFS backed state store and RocksDB state store
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33870:
URL: https://github.com/apache/spark/pull/33870#issuecomment-909270480
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47390/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33870: [SPARK-36619][SS] Reset prefix-scan iterators on loading RocksDB
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33870:
URL: https://github.com/apache/spark/pull/33870#issuecomment-908903853
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a change in pull request #33870: [SPARK-36619][SS] Fix bugs around prefix-scan for HDFS backed state store and RocksDB state store
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on a change in pull request #33870:
URL: https://github.com/apache/spark/pull/33870#discussion_r699272824
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreMap.scala
##########
@@ -129,7 +129,11 @@ class PrefixScannableHDFSBackedStateStoreMap(
other match {
case o: PrefixScannableHDFSBackedStateStoreMap =>
map.putAll(o.map)
- prefixKeyToKeysMap.putAll(o.prefixKeyToKeysMap)
Review comment:
This leads the prefix key maps across versions to "share" the values which it shouldn't. (especially the case of rolling back)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33870: [SPARK-36619][SS] Fix bugs around prefix-scan for HDFS backed state store and RocksDB state store
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33870:
URL: https://github.com/apache/spark/pull/33870#issuecomment-909356831
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47392/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xuanyuanking commented on a change in pull request #33870: [SPARK-36619][SS] Reset prefix-scan iterators on loading RocksDB
Posted by GitBox <gi...@apache.org>.
xuanyuanking commented on a change in pull request #33870:
URL: https://github.com/apache/spark/pull/33870#discussion_r698968561
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala
##########
@@ -120,6 +120,9 @@ class RocksDB(
if (conf.resetStatsOnLoad) {
nativeStats.reset
}
+ // reset resources to prevent side-effects from previous loaded version
+ prefixScanReuseIter.entrySet().asScala.foreach(_.getValue.close())
+ prefixScanReuseIter.clear()
Review comment:
nit: How about wrap this 2 lines in a function `closePrefixScan(Reuse?)Iter()`? Since they are already been called in 3 places.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33870: [SPARK-36619][SS] Reset prefix-scan iterators on loading RocksDB
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33870:
URL: https://github.com/apache/spark/pull/33870#issuecomment-908882245
**[Test build #142871 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142871/testReport)** for PR 33870 at commit [`2b4f00d`](https://github.com/apache/spark/commit/2b4f00dd0cc6d7feca6c5f7cb548692cb2d98146).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33870: [SPARK-36619][SS] Reset prefix-scan iterators on loading RocksDB
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33870:
URL: https://github.com/apache/spark/pull/33870#issuecomment-909056773
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org