You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/08/31 03:40:49 UTC

[GitHub] [spark] HeartSaVioR opened a new pull request #33870: [SPARK-36619][SS] Reset prefix-scan iterators on loading RocksDB

HeartSaVioR opened a new pull request #33870:
URL: https://github.com/apache/spark/pull/33870


   ### What changes were proposed in this pull request?
   
   This PR ensures that resetting prefix-scan iterators is done on loading RocksDB, which was only done in rollback.
   
   ### Why are the changes needed?
   
   Prefix-scan iterators are only closed on RocksDB.rollback(), which is only called in RocksDBStateStore.abort().
   
   While `RocksDBStateStore.abort()` method will be called for streaming session window (since it has two physical plans for read and write), other stateful operators which only have read-write physical plan will call either commit or abort, and don't close the iterators on committing.
   
   These unclosed iterators can be "reused" and produce incorrect outputs.
   
   ### Does this PR introduce _any_ user-facing change?
   
   No.
   
   ### How was this patch tested?
   
   Modified UT which failed without this PR and passes with this PR.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #33870: [SPARK-36619][SS] Reset prefix-scan iterators on loading RocksDB

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33870:
URL: https://github.com/apache/spark/pull/33870#issuecomment-908927114


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47377/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] gengliangwang commented on a change in pull request #33870: [SPARK-36619][SS] Fix bugs around prefix-scan for HDFS backed state store and RocksDB state store

Posted by GitBox <gi...@apache.org>.
gengliangwang commented on a change in pull request #33870:
URL: https://github.com/apache/spark/pull/33870#discussion_r699309363



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreMap.scala
##########
@@ -129,7 +129,11 @@ class PrefixScannableHDFSBackedStateStoreMap(
     other match {
       case o: PrefixScannableHDFSBackedStateStoreMap =>
         map.putAll(o.map)
-        prefixKeyToKeysMap.putAll(o.prefixKeyToKeysMap)
+        o.prefixKeyToKeysMap.asScala.foreach { case (prefixKey, keySet) =>
+          val newSet = new mutable.HashSet[UnsafeRow]()

Review comment:
       This looks confusing to me. Shall we do the following?

##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreMap.scala
##########
@@ -129,7 +129,11 @@ class PrefixScannableHDFSBackedStateStoreMap(
     other match {
       case o: PrefixScannableHDFSBackedStateStoreMap =>
         map.putAll(o.map)
-        prefixKeyToKeysMap.putAll(o.prefixKeyToKeysMap)
+        o.prefixKeyToKeysMap.asScala.foreach { case (prefixKey, keySet) =>
+          val newSet = new mutable.HashSet[UnsafeRow]()

Review comment:
       ```
   val cloneMap = new java.util.concurrent.ConcurrentHashMap
             [UnsafeRow, mutable.Set[UnsafeRow]](o.prefixKeyToKeysMap)
           prefixKeyToKeysMap.putAll(cloneMap)
   ```

##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreMap.scala
##########
@@ -129,7 +129,11 @@ class PrefixScannableHDFSBackedStateStoreMap(
     other match {
       case o: PrefixScannableHDFSBackedStateStoreMap =>
         map.putAll(o.map)
-        prefixKeyToKeysMap.putAll(o.prefixKeyToKeysMap)
+        o.prefixKeyToKeysMap.asScala.foreach { case (prefixKey, keySet) =>
+          val newSet = new mutable.HashSet[UnsafeRow]()

Review comment:
       ```
   val cloneMap = new java.util.concurrent.ConcurrentHashMap
             [UnsafeRow, mutable.Set[UnsafeRow]](o.prefixKeyToKeysMap)
   prefixKeyToKeysMap.putAll(cloneMap)
   ```

##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreMap.scala
##########
@@ -129,7 +129,11 @@ class PrefixScannableHDFSBackedStateStoreMap(
     other match {
       case o: PrefixScannableHDFSBackedStateStoreMap =>
         map.putAll(o.map)
-        prefixKeyToKeysMap.putAll(o.prefixKeyToKeysMap)
+        o.prefixKeyToKeysMap.asScala.foreach { case (prefixKey, keySet) =>
+          val newSet = new mutable.HashSet[UnsafeRow]()

Review comment:
       This looks confusing to me. Shall we do the following?

##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreMap.scala
##########
@@ -129,7 +129,11 @@ class PrefixScannableHDFSBackedStateStoreMap(
     other match {
       case o: PrefixScannableHDFSBackedStateStoreMap =>
         map.putAll(o.map)
-        prefixKeyToKeysMap.putAll(o.prefixKeyToKeysMap)
+        o.prefixKeyToKeysMap.asScala.foreach { case (prefixKey, keySet) =>
+          val newSet = new mutable.HashSet[UnsafeRow]()

Review comment:
       ```
   val cloneMap = new java.util.concurrent.ConcurrentHashMap
             [UnsafeRow, mutable.Set[UnsafeRow]](o.prefixKeyToKeysMap)
   prefixKeyToKeysMap.putAll(cloneMap)
   ```

##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreMap.scala
##########
@@ -129,7 +129,11 @@ class PrefixScannableHDFSBackedStateStoreMap(
     other match {
       case o: PrefixScannableHDFSBackedStateStoreMap =>
         map.putAll(o.map)
-        prefixKeyToKeysMap.putAll(o.prefixKeyToKeysMap)

Review comment:
       Let's add one comment here.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] gengliangwang closed pull request #33870: [SPARK-36619][SS] Fix bugs around prefix-scan for HDFS backed state store and RocksDB state store

Posted by GitBox <gi...@apache.org>.
gengliangwang closed pull request #33870:
URL: https://github.com/apache/spark/pull/33870


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HeartSaVioR commented on a change in pull request #33870: [SPARK-36619][SS] Reset prefix-scan iterators on loading RocksDB

Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on a change in pull request #33870:
URL: https://github.com/apache/spark/pull/33870#discussion_r698961491



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala
##########
@@ -120,6 +120,9 @@ class RocksDB(
       if (conf.resetStatsOnLoad) {
         nativeStats.reset
       }
+      // reset resources to prevent side-effects from previous loaded version

Review comment:
       We did this in rollback case but not in commit case. This doesn't bring problems in current implementation of session window, but I think we should fix it to avoid future correctness problem.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #33870: [SPARK-36619][SS] Fix bugs around prefix-scan for HDFS backed state store and RocksDB state store

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33870:
URL: https://github.com/apache/spark/pull/33870#issuecomment-909188932


   **[Test build #142887 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142887/testReport)** for PR 33870 at commit [`1b3213d`](https://github.com/apache/spark/commit/1b3213d7b50d8209578abbfa14311d9e321696ab).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #33870: [SPARK-36619][SS] Reset prefix-scan iterators on loading RocksDB

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33870:
URL: https://github.com/apache/spark/pull/33870#issuecomment-908963001


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47378/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #33870: [SPARK-36619][SS] Reset prefix-scan iterators on loading RocksDB

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33870:
URL: https://github.com/apache/spark/pull/33870#issuecomment-909055692


   **[Test build #142875 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142875/testReport)** for PR 33870 at commit [`a01d733`](https://github.com/apache/spark/commit/a01d73309a23058e452b01b19669131dc212380c).
    * This patch **fails Spark unit tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #33870: [SPARK-36619][SS] Fix bugs around prefix-scan for HDFS backed state store and RocksDB state store

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33870:
URL: https://github.com/apache/spark/pull/33870#issuecomment-909425180


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142887/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #33870: [SPARK-36619][SS] Fix bugs around prefix-scan for HDFS backed state store and RocksDB state store

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33870:
URL: https://github.com/apache/spark/pull/33870#issuecomment-909373881


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47392/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #33870: [SPARK-36619][SS] Reset prefix-scan iterators on loading RocksDB

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33870:
URL: https://github.com/apache/spark/pull/33870#issuecomment-908931363


   **[Test build #142874 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142874/testReport)** for PR 33870 at commit [`a01d733`](https://github.com/apache/spark/commit/a01d73309a23058e452b01b19669131dc212380c).
    * This patch **fails Spark unit tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HeartSaVioR commented on pull request #33870: [SPARK-36619][SS] Reset prefix-scan iterators on loading RocksDB

Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #33870:
URL: https://github.com/apache/spark/pull/33870#issuecomment-908935408


   retest this, please


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] gengliangwang commented on a change in pull request #33870: [SPARK-36619][SS] Fix bugs around prefix-scan for HDFS backed state store and RocksDB state store

Posted by GitBox <gi...@apache.org>.
gengliangwang commented on a change in pull request #33870:
URL: https://github.com/apache/spark/pull/33870#discussion_r699309363



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreMap.scala
##########
@@ -129,7 +129,11 @@ class PrefixScannableHDFSBackedStateStoreMap(
     other match {
       case o: PrefixScannableHDFSBackedStateStoreMap =>
         map.putAll(o.map)
-        prefixKeyToKeysMap.putAll(o.prefixKeyToKeysMap)
+        o.prefixKeyToKeysMap.asScala.foreach { case (prefixKey, keySet) =>
+          val newSet = new mutable.HashSet[UnsafeRow]()

Review comment:
       This looks confusing to me. Shall we do the following?

##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreMap.scala
##########
@@ -129,7 +129,11 @@ class PrefixScannableHDFSBackedStateStoreMap(
     other match {
       case o: PrefixScannableHDFSBackedStateStoreMap =>
         map.putAll(o.map)
-        prefixKeyToKeysMap.putAll(o.prefixKeyToKeysMap)
+        o.prefixKeyToKeysMap.asScala.foreach { case (prefixKey, keySet) =>
+          val newSet = new mutable.HashSet[UnsafeRow]()

Review comment:
       ```
   val cloneMap = new java.util.concurrent.ConcurrentHashMap
             [UnsafeRow, mutable.Set[UnsafeRow]](o.prefixKeyToKeysMap)
   prefixKeyToKeysMap.putAll(cloneMap)
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #33870: [SPARK-36619][SS] Reset prefix-scan iterators on loading RocksDB

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33870:
URL: https://github.com/apache/spark/pull/33870#issuecomment-908936142


   **[Test build #142875 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142875/testReport)** for PR 33870 at commit [`a01d733`](https://github.com/apache/spark/commit/a01d73309a23058e452b01b19669131dc212380c).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #33870: [SPARK-36619][SS] Fix bugs around prefix-scan for HDFS backed state store and RocksDB state store

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33870:
URL: https://github.com/apache/spark/pull/33870#issuecomment-909250790


   Kubernetes integration test status failure
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47390/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #33870: [SPARK-36619][SS] Fix bugs around prefix-scan for HDFS backed state store and RocksDB state store

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33870:
URL: https://github.com/apache/spark/pull/33870#issuecomment-909188932


   **[Test build #142887 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142887/testReport)** for PR 33870 at commit [`1b3213d`](https://github.com/apache/spark/commit/1b3213d7b50d8209578abbfa14311d9e321696ab).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HeartSaVioR commented on a change in pull request #33870: [SPARK-36619][SS] Fix bugs around prefix-scan for HDFS backed state store and RocksDB state store

Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on a change in pull request #33870:
URL: https://github.com/apache/spark/pull/33870#discussion_r699272824



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreMap.scala
##########
@@ -129,7 +129,11 @@ class PrefixScannableHDFSBackedStateStoreMap(
     other match {
       case o: PrefixScannableHDFSBackedStateStoreMap =>
         map.putAll(o.map)
-        prefixKeyToKeysMap.putAll(o.prefixKeyToKeysMap)

Review comment:
       This leads the prefix key maps across versions to "share" the values (as mutable Set instances) which it shouldn't. (especially the case of rolling back)




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HeartSaVioR commented on pull request #33870: [SPARK-36619][SS] Reset prefix-scan iterators on loading RocksDB

Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #33870:
URL: https://github.com/apache/spark/pull/33870#issuecomment-908881532


   cc. @viirya @xuanyuanking 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] gengliangwang commented on a change in pull request #33870: [SPARK-36619][SS] Fix bugs around prefix-scan for HDFS backed state store and RocksDB state store

Posted by GitBox <gi...@apache.org>.
gengliangwang commented on a change in pull request #33870:
URL: https://github.com/apache/spark/pull/33870#discussion_r699309838



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreMap.scala
##########
@@ -129,7 +129,11 @@ class PrefixScannableHDFSBackedStateStoreMap(
     other match {
       case o: PrefixScannableHDFSBackedStateStoreMap =>
         map.putAll(o.map)
-        prefixKeyToKeysMap.putAll(o.prefixKeyToKeysMap)
+        o.prefixKeyToKeysMap.asScala.foreach { case (prefixKey, keySet) =>
+          val newSet = new mutable.HashSet[UnsafeRow]()

Review comment:
       ```
   val cloneMap = new java.util.concurrent.ConcurrentHashMap
             [UnsafeRow, mutable.Set[UnsafeRow]](o.prefixKeyToKeysMap)
           prefixKeyToKeysMap.putAll(cloneMap)
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HeartSaVioR removed a comment on pull request #33870: [SPARK-36619][SS] Reset prefix-scan iterators on loading RocksDB

Posted by GitBox <gi...@apache.org>.
HeartSaVioR removed a comment on pull request #33870:
URL: https://github.com/apache/spark/pull/33870#issuecomment-909082674


   Weird, I can't reproduce the failure locally. (MacOS env.) I tried running RocksDBStateStoreSuite in IDEA.
   
    I will try to run the test suite via sbt, and if it didn't work, I'll try to run several RocksDB test suites altogether to see whether the test is affected by other suites.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HeartSaVioR commented on pull request #33870: [SPARK-36619][SS] Reset prefix-scan iterators on loading RocksDB

Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #33870:
URL: https://github.com/apache/spark/pull/33870#issuecomment-909082674






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #33870: [SPARK-36619][SS] Reset prefix-scan iterators on loading RocksDB

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33870:
URL: https://github.com/apache/spark/pull/33870#issuecomment-908961530


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47378/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HeartSaVioR edited a comment on pull request #33870: [SPARK-36619][SS] Reset prefix-scan iterators on loading RocksDB

Posted by GitBox <gi...@apache.org>.
HeartSaVioR edited a comment on pull request #33870:
URL: https://github.com/apache/spark/pull/33870#issuecomment-908881532


   cc. @zsxwing  @viirya @xuanyuanking 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #33870: [SPARK-36619][SS] Fix bugs around prefix-scan for HDFS backed state store and RocksDB state store

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33870:
URL: https://github.com/apache/spark/pull/33870#issuecomment-909287667


   **[Test build #142889 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142889/testReport)** for PR 33870 at commit [`7913e38`](https://github.com/apache/spark/commit/7913e381e36751e61e405d22dbb1418b00a4135b).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #33870: [SPARK-36619][SS] Reset prefix-scan iterators on loading RocksDB

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33870:
URL: https://github.com/apache/spark/pull/33870#issuecomment-908901578


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47374/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HeartSaVioR commented on pull request #33870: [SPARK-36619][SS] Fix bugs around prefix-scan for HDFS backed state store and RocksDB state store

Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #33870:
URL: https://github.com/apache/spark/pull/33870#issuecomment-909193006


   @xuanyuanking @viirya @gengliangwang @dongjoon-hyun 
   I'm sorry, but could you please take a look again? I fixed another bug on HDFS backed state store which led failure on new test.
   Thanks in advance!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HeartSaVioR commented on a change in pull request #33870: [SPARK-36619][SS] Fix bugs around prefix-scan for HDFS backed state store and RocksDB state store

Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on a change in pull request #33870:
URL: https://github.com/apache/spark/pull/33870#discussion_r699272824



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreMap.scala
##########
@@ -129,7 +129,11 @@ class PrefixScannableHDFSBackedStateStoreMap(
     other match {
       case o: PrefixScannableHDFSBackedStateStoreMap =>
         map.putAll(o.map)
-        prefixKeyToKeysMap.putAll(o.prefixKeyToKeysMap)

Review comment:
       This leads the prefix key maps across versions to "share" the values which it shouldn't. (especially the case of rolling back)

##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreMap.scala
##########
@@ -129,7 +129,11 @@ class PrefixScannableHDFSBackedStateStoreMap(
     other match {
       case o: PrefixScannableHDFSBackedStateStoreMap =>
         map.putAll(o.map)
-        prefixKeyToKeysMap.putAll(o.prefixKeyToKeysMap)

Review comment:
       This leads the prefix key maps across versions to "share" the values (as mutable Set instances) which it shouldn't. (especially the case of rolling back)

##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreMap.scala
##########
@@ -129,7 +129,11 @@ class PrefixScannableHDFSBackedStateStoreMap(
     other match {
       case o: PrefixScannableHDFSBackedStateStoreMap =>
         map.putAll(o.map)
-        prefixKeyToKeysMap.putAll(o.prefixKeyToKeysMap)

Review comment:
       Thanks! Addressed via 7913e381e36751e61e405d22dbb1418b00a4135b




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #33870: [SPARK-36619][SS] Reset prefix-scan iterators on loading RocksDB

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33870:
URL: https://github.com/apache/spark/pull/33870#issuecomment-908931638


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142874/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #33870: [SPARK-36619][SS] Reset prefix-scan iterators on loading RocksDB

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33870:
URL: https://github.com/apache/spark/pull/33870#issuecomment-908997741


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142871/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] gengliangwang closed pull request #33870: [SPARK-36619][SS] Fix bugs around prefix-scan for HDFS backed state store and RocksDB state store

Posted by GitBox <gi...@apache.org>.
gengliangwang closed pull request #33870:
URL: https://github.com/apache/spark/pull/33870


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #33870: [SPARK-36619][SS] Reset prefix-scan iterators on loading RocksDB

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33870:
URL: https://github.com/apache/spark/pull/33870#issuecomment-908903853


   **[Test build #142874 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142874/testReport)** for PR 33870 at commit [`a01d733`](https://github.com/apache/spark/commit/a01d73309a23058e452b01b19669131dc212380c).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #33870: [SPARK-36619][SS] Reset prefix-scan iterators on loading RocksDB

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33870:
URL: https://github.com/apache/spark/pull/33870#issuecomment-908927114


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47377/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HeartSaVioR commented on a change in pull request #33870: [SPARK-36619][SS] Reset prefix-scan iterators on loading RocksDB

Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on a change in pull request #33870:
URL: https://github.com/apache/spark/pull/33870#discussion_r698961491



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala
##########
@@ -120,6 +120,9 @@ class RocksDB(
       if (conf.resetStatsOnLoad) {
         nativeStats.reset
       }
+      // reset resources to prevent side-effects from previous loaded version

Review comment:
       We did this in rollback case but not in commit case. This doesn't bring problems in current implementation of session window, but I think we should fix it to avoid future correctness problem.

##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala
##########
@@ -120,6 +120,9 @@ class RocksDB(
       if (conf.resetStatsOnLoad) {
         nativeStats.reset
       }
+      // reset resources to prevent side-effects from previous loaded version
+      prefixScanReuseIter.entrySet().asScala.foreach(_.getValue.close())
+      prefixScanReuseIter.clear()

Review comment:
       Thanks! Addressed via a01d733

##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreMap.scala
##########
@@ -129,7 +129,11 @@ class PrefixScannableHDFSBackedStateStoreMap(
     other match {
       case o: PrefixScannableHDFSBackedStateStoreMap =>
         map.putAll(o.map)
-        prefixKeyToKeysMap.putAll(o.prefixKeyToKeysMap)

Review comment:
       This leads the prefix key maps across versions to "share" the values which it shouldn't. (especially the case of rolling back)

##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreMap.scala
##########
@@ -129,7 +129,11 @@ class PrefixScannableHDFSBackedStateStoreMap(
     other match {
       case o: PrefixScannableHDFSBackedStateStoreMap =>
         map.putAll(o.map)
-        prefixKeyToKeysMap.putAll(o.prefixKeyToKeysMap)

Review comment:
       This leads the prefix key maps across versions to "share" the values (as mutable Set instances) which it shouldn't. (especially the case of rolling back)

##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreMap.scala
##########
@@ -129,7 +129,11 @@ class PrefixScannableHDFSBackedStateStoreMap(
     other match {
       case o: PrefixScannableHDFSBackedStateStoreMap =>
         map.putAll(o.map)
-        prefixKeyToKeysMap.putAll(o.prefixKeyToKeysMap)

Review comment:
       Thanks! Addressed via 7913e381e36751e61e405d22dbb1418b00a4135b




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #33870: [SPARK-36619][SS] Fix bugs around prefix-scan for HDFS backed state store and RocksDB state store

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33870:
URL: https://github.com/apache/spark/pull/33870#issuecomment-909217470


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47390/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #33870: [SPARK-36619][SS] Reset prefix-scan iterators on loading RocksDB

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33870:
URL: https://github.com/apache/spark/pull/33870#issuecomment-908901578






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #33870: [SPARK-36619][SS] Fix bugs around prefix-scan for HDFS backed state store and RocksDB state store

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33870:
URL: https://github.com/apache/spark/pull/33870#issuecomment-909373881


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47392/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #33870: [SPARK-36619][SS] Reset prefix-scan iterators on loading RocksDB

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33870:
URL: https://github.com/apache/spark/pull/33870#issuecomment-908917688


   Kubernetes integration test unable to build dist.
   
   exiting with code: 1
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47377/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HeartSaVioR commented on pull request #33870: [SPARK-36619][SS] Reset prefix-scan iterators on loading RocksDB

Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #33870:
URL: https://github.com/apache/spark/pull/33870#issuecomment-908881532






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #33870: [SPARK-36619][SS] Reset prefix-scan iterators on loading RocksDB

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33870:
URL: https://github.com/apache/spark/pull/33870#issuecomment-908894643


   Kubernetes integration test unable to build dist.
   
   exiting with code: 1
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47374/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HeartSaVioR commented on a change in pull request #33870: [SPARK-36619][SS] Reset prefix-scan iterators on loading RocksDB

Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on a change in pull request #33870:
URL: https://github.com/apache/spark/pull/33870#discussion_r698982058



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala
##########
@@ -120,6 +120,9 @@ class RocksDB(
       if (conf.resetStatsOnLoad) {
         nativeStats.reset
       }
+      // reset resources to prevent side-effects from previous loaded version
+      prefixScanReuseIter.entrySet().asScala.foreach(_.getValue.close())
+      prefixScanReuseIter.clear()

Review comment:
       Thanks! Addressed via a01d733




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HeartSaVioR commented on a change in pull request #33870: [SPARK-36619][SS] Fix bugs around prefix-scan for HDFS backed state store and RocksDB state store

Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on a change in pull request #33870:
URL: https://github.com/apache/spark/pull/33870#discussion_r699373911



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreMap.scala
##########
@@ -129,7 +129,11 @@ class PrefixScannableHDFSBackedStateStoreMap(
     other match {
       case o: PrefixScannableHDFSBackedStateStoreMap =>
         map.putAll(o.map)
-        prefixKeyToKeysMap.putAll(o.prefixKeyToKeysMap)

Review comment:
       Thanks! Addressed via 7913e381e36751e61e405d22dbb1418b00a4135b




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HeartSaVioR edited a comment on pull request #33870: [SPARK-36619][SS] Reset prefix-scan iterators on loading RocksDB

Posted by GitBox <gi...@apache.org>.
HeartSaVioR edited a comment on pull request #33870:
URL: https://github.com/apache/spark/pull/33870#issuecomment-908881532


   cc. @zsxwing  @viirya @xuanyuanking 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] gengliangwang commented on a change in pull request #33870: [SPARK-36619][SS] Fix bugs around prefix-scan for HDFS backed state store and RocksDB state store

Posted by GitBox <gi...@apache.org>.
gengliangwang commented on a change in pull request #33870:
URL: https://github.com/apache/spark/pull/33870#discussion_r699309838



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreMap.scala
##########
@@ -129,7 +129,11 @@ class PrefixScannableHDFSBackedStateStoreMap(
     other match {
       case o: PrefixScannableHDFSBackedStateStoreMap =>
         map.putAll(o.map)
-        prefixKeyToKeysMap.putAll(o.prefixKeyToKeysMap)
+        o.prefixKeyToKeysMap.asScala.foreach { case (prefixKey, keySet) =>
+          val newSet = new mutable.HashSet[UnsafeRow]()

Review comment:
       ```
   val cloneMap = new java.util.concurrent.ConcurrentHashMap
             [UnsafeRow, mutable.Set[UnsafeRow]](o.prefixKeyToKeysMap)
   prefixKeyToKeysMap.putAll(cloneMap)
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #33870: [SPARK-36619][SS] Reset prefix-scan iterators on loading RocksDB

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33870:
URL: https://github.com/apache/spark/pull/33870#issuecomment-908901578


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47374/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #33870: [SPARK-36619][SS] Reset prefix-scan iterators on loading RocksDB

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33870:
URL: https://github.com/apache/spark/pull/33870#issuecomment-909055692






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #33870: [SPARK-36619][SS] Fix bugs around prefix-scan for HDFS backed state store and RocksDB state store

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33870:
URL: https://github.com/apache/spark/pull/33870#issuecomment-909270480


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47390/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #33870: [SPARK-36619][SS] Reset prefix-scan iterators on loading RocksDB

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33870:
URL: https://github.com/apache/spark/pull/33870#issuecomment-909056773


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142875/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #33870: [SPARK-36619][SS] Reset prefix-scan iterators on loading RocksDB

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33870:
URL: https://github.com/apache/spark/pull/33870#issuecomment-908963001


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47378/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] gengliangwang commented on a change in pull request #33870: [SPARK-36619][SS] Fix bugs around prefix-scan for HDFS backed state store and RocksDB state store

Posted by GitBox <gi...@apache.org>.
gengliangwang commented on a change in pull request #33870:
URL: https://github.com/apache/spark/pull/33870#discussion_r699330317



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreMap.scala
##########
@@ -129,7 +129,11 @@ class PrefixScannableHDFSBackedStateStoreMap(
     other match {
       case o: PrefixScannableHDFSBackedStateStoreMap =>
         map.putAll(o.map)
-        prefixKeyToKeysMap.putAll(o.prefixKeyToKeysMap)

Review comment:
       Let's add one comment here.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #33870: [SPARK-36619][SS] Fix bugs around prefix-scan for HDFS backed state store and RocksDB state store

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33870:
URL: https://github.com/apache/spark/pull/33870#issuecomment-909423601


   **[Test build #142887 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142887/testReport)** for PR 33870 at commit [`1b3213d`](https://github.com/apache/spark/commit/1b3213d7b50d8209578abbfa14311d9e321696ab).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #33870: [SPARK-36619][SS] Reset prefix-scan iterators on loading RocksDB

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33870:
URL: https://github.com/apache/spark/pull/33870#issuecomment-908882245


   **[Test build #142871 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142871/testReport)** for PR 33870 at commit [`2b4f00d`](https://github.com/apache/spark/commit/2b4f00dd0cc6d7feca6c5f7cb548692cb2d98146).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HeartSaVioR commented on pull request #33870: [SPARK-36619][SS] Reset prefix-scan iterators on loading RocksDB

Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #33870:
URL: https://github.com/apache/spark/pull/33870#issuecomment-909082674


   Weird, I can't reproduce the failure locally. (MacOS env.) I tried running RocksDBStateStoreSuite in IDEA.
   
    I will try to run the test suite via sbt, and if it didn't work, I'll try to run several RocksDB test suites altogether to see whether the test is affected by other suites.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HeartSaVioR commented on pull request #33870: [SPARK-36619][SS] Fix bugs around prefix-scan for HDFS backed state store and RocksDB state store

Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #33870:
URL: https://github.com/apache/spark/pull/33870#issuecomment-909711314


   Thanks all for reviewing and merging actively!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #33870: [SPARK-36619][SS] Reset prefix-scan iterators on loading RocksDB

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33870:
URL: https://github.com/apache/spark/pull/33870#issuecomment-909056773






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #33870: [SPARK-36619][SS] Reset prefix-scan iterators on loading RocksDB

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33870:
URL: https://github.com/apache/spark/pull/33870#issuecomment-908901578






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] gengliangwang commented on a change in pull request #33870: [SPARK-36619][SS] Fix bugs around prefix-scan for HDFS backed state store and RocksDB state store

Posted by GitBox <gi...@apache.org>.
gengliangwang commented on a change in pull request #33870:
URL: https://github.com/apache/spark/pull/33870#discussion_r699309363



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreMap.scala
##########
@@ -129,7 +129,11 @@ class PrefixScannableHDFSBackedStateStoreMap(
     other match {
       case o: PrefixScannableHDFSBackedStateStoreMap =>
         map.putAll(o.map)
-        prefixKeyToKeysMap.putAll(o.prefixKeyToKeysMap)
+        o.prefixKeyToKeysMap.asScala.foreach { case (prefixKey, keySet) =>
+          val newSet = new mutable.HashSet[UnsafeRow]()

Review comment:
       This looks confusing to me. Shall we do the following?

##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreMap.scala
##########
@@ -129,7 +129,11 @@ class PrefixScannableHDFSBackedStateStoreMap(
     other match {
       case o: PrefixScannableHDFSBackedStateStoreMap =>
         map.putAll(o.map)
-        prefixKeyToKeysMap.putAll(o.prefixKeyToKeysMap)
+        o.prefixKeyToKeysMap.asScala.foreach { case (prefixKey, keySet) =>
+          val newSet = new mutable.HashSet[UnsafeRow]()

Review comment:
       ```
   val cloneMap = new java.util.concurrent.ConcurrentHashMap
             [UnsafeRow, mutable.Set[UnsafeRow]](o.prefixKeyToKeysMap)
           prefixKeyToKeysMap.putAll(cloneMap)
   ```

##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreMap.scala
##########
@@ -129,7 +129,11 @@ class PrefixScannableHDFSBackedStateStoreMap(
     other match {
       case o: PrefixScannableHDFSBackedStateStoreMap =>
         map.putAll(o.map)
-        prefixKeyToKeysMap.putAll(o.prefixKeyToKeysMap)
+        o.prefixKeyToKeysMap.asScala.foreach { case (prefixKey, keySet) =>
+          val newSet = new mutable.HashSet[UnsafeRow]()

Review comment:
       ```
   val cloneMap = new java.util.concurrent.ConcurrentHashMap
             [UnsafeRow, mutable.Set[UnsafeRow]](o.prefixKeyToKeysMap)
   prefixKeyToKeysMap.putAll(cloneMap)
   ```

##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreMap.scala
##########
@@ -129,7 +129,11 @@ class PrefixScannableHDFSBackedStateStoreMap(
     other match {
       case o: PrefixScannableHDFSBackedStateStoreMap =>
         map.putAll(o.map)
-        prefixKeyToKeysMap.putAll(o.prefixKeyToKeysMap)
+        o.prefixKeyToKeysMap.asScala.foreach { case (prefixKey, keySet) =>
+          val newSet = new mutable.HashSet[UnsafeRow]()

Review comment:
       This looks confusing to me. Shall we do the following?

##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreMap.scala
##########
@@ -129,7 +129,11 @@ class PrefixScannableHDFSBackedStateStoreMap(
     other match {
       case o: PrefixScannableHDFSBackedStateStoreMap =>
         map.putAll(o.map)
-        prefixKeyToKeysMap.putAll(o.prefixKeyToKeysMap)
+        o.prefixKeyToKeysMap.asScala.foreach { case (prefixKey, keySet) =>
+          val newSet = new mutable.HashSet[UnsafeRow]()

Review comment:
       ```
   val cloneMap = new java.util.concurrent.ConcurrentHashMap
             [UnsafeRow, mutable.Set[UnsafeRow]](o.prefixKeyToKeysMap)
   prefixKeyToKeysMap.putAll(cloneMap)
   ```

##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreMap.scala
##########
@@ -129,7 +129,11 @@ class PrefixScannableHDFSBackedStateStoreMap(
     other match {
       case o: PrefixScannableHDFSBackedStateStoreMap =>
         map.putAll(o.map)
-        prefixKeyToKeysMap.putAll(o.prefixKeyToKeysMap)

Review comment:
       Let's add one comment here.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] gengliangwang commented on pull request #33870: [SPARK-36619][SS] Fix bugs around prefix-scan for HDFS backed state store and RocksDB state store

Posted by GitBox <gi...@apache.org>.
gengliangwang commented on pull request #33870:
URL: https://github.com/apache/spark/pull/33870#issuecomment-909410233


   Merging to master/3.2


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #33870: [SPARK-36619][SS] Reset prefix-scan iterators on loading RocksDB

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33870:
URL: https://github.com/apache/spark/pull/33870#issuecomment-908936142






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #33870: [SPARK-36619][SS] Reset prefix-scan iterators on loading RocksDB

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33870:
URL: https://github.com/apache/spark/pull/33870#issuecomment-908903853


   **[Test build #142874 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142874/testReport)** for PR 33870 at commit [`a01d733`](https://github.com/apache/spark/commit/a01d73309a23058e452b01b19669131dc212380c).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] gengliangwang commented on a change in pull request #33870: [SPARK-36619][SS] Fix bugs around prefix-scan for HDFS backed state store and RocksDB state store

Posted by GitBox <gi...@apache.org>.
gengliangwang commented on a change in pull request #33870:
URL: https://github.com/apache/spark/pull/33870#discussion_r699309363



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreMap.scala
##########
@@ -129,7 +129,11 @@ class PrefixScannableHDFSBackedStateStoreMap(
     other match {
       case o: PrefixScannableHDFSBackedStateStoreMap =>
         map.putAll(o.map)
-        prefixKeyToKeysMap.putAll(o.prefixKeyToKeysMap)
+        o.prefixKeyToKeysMap.asScala.foreach { case (prefixKey, keySet) =>
+          val newSet = new mutable.HashSet[UnsafeRow]()

Review comment:
       This looks confusing to me. Shall we do the following?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HeartSaVioR removed a comment on pull request #33870: [SPARK-36619][SS] Reset prefix-scan iterators on loading RocksDB

Posted by GitBox <gi...@apache.org>.
HeartSaVioR removed a comment on pull request #33870:
URL: https://github.com/apache/spark/pull/33870#issuecomment-909082674


   Weird, I can't reproduce the failure locally. (MacOS env.) I tried running RocksDBStateStoreSuite in IDEA.
   
    I will try to run the test suite via sbt, and if it didn't work, I'll try to run several RocksDB test suites altogether to see whether the test is affected by other suites.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] xuanyuanking commented on a change in pull request #33870: [SPARK-36619][SS] Reset prefix-scan iterators on loading RocksDB

Posted by GitBox <gi...@apache.org>.
xuanyuanking commented on a change in pull request #33870:
URL: https://github.com/apache/spark/pull/33870#discussion_r698968561



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala
##########
@@ -120,6 +120,9 @@ class RocksDB(
       if (conf.resetStatsOnLoad) {
         nativeStats.reset
       }
+      // reset resources to prevent side-effects from previous loaded version
+      prefixScanReuseIter.entrySet().asScala.foreach(_.getValue.close())
+      prefixScanReuseIter.clear()

Review comment:
       nit: How about wrap this 2 lines in a function `closePrefixScan(Reuse?)Iter()`? Since they are already been called in 3 places.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HeartSaVioR removed a comment on pull request #33870: [SPARK-36619][SS] Reset prefix-scan iterators on loading RocksDB

Posted by GitBox <gi...@apache.org>.
HeartSaVioR removed a comment on pull request #33870:
URL: https://github.com/apache/spark/pull/33870#issuecomment-909082674


   Weird, I can't reproduce the failure locally. (MacOS env.) I tried running RocksDBStateStoreSuite in IDEA.
   
    I will try to run the test suite via sbt, and if it didn't work, I'll try to run several RocksDB test suites altogether to see whether the test is affected by other suites.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #33870: [SPARK-36619][SS] Fix bugs around prefix-scan for HDFS backed state store and RocksDB state store

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33870:
URL: https://github.com/apache/spark/pull/33870#issuecomment-909425180


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142887/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #33870: [SPARK-36619][SS] Reset prefix-scan iterators on loading RocksDB

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33870:
URL: https://github.com/apache/spark/pull/33870#issuecomment-908982314


   **[Test build #142871 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142871/testReport)** for PR 33870 at commit [`2b4f00d`](https://github.com/apache/spark/commit/2b4f00dd0cc6d7feca6c5f7cb548692cb2d98146).
    * This patch **fails Spark unit tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HeartSaVioR commented on pull request #33870: [SPARK-36619][SS] Reset prefix-scan iterators on loading RocksDB

Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #33870:
URL: https://github.com/apache/spark/pull/33870#issuecomment-909142440


   Looks like there is another bug in HDFSBackedStateStore as well. I'm looking into it.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] gengliangwang commented on pull request #33870: [SPARK-36619][SS] Fix bugs around prefix-scan for HDFS backed state store and RocksDB state store

Posted by GitBox <gi...@apache.org>.
gengliangwang commented on pull request #33870:
URL: https://github.com/apache/spark/pull/33870#issuecomment-909410233


   Merging to master/3.2


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #33870: [SPARK-36619][SS] Reset prefix-scan iterators on loading RocksDB

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33870:
URL: https://github.com/apache/spark/pull/33870#issuecomment-908931638


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142874/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #33870: [SPARK-36619][SS] Reset prefix-scan iterators on loading RocksDB

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33870:
URL: https://github.com/apache/spark/pull/33870#issuecomment-908882245






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #33870: [SPARK-36619][SS] Reset prefix-scan iterators on loading RocksDB

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33870:
URL: https://github.com/apache/spark/pull/33870#issuecomment-908936142


   **[Test build #142875 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142875/testReport)** for PR 33870 at commit [`a01d733`](https://github.com/apache/spark/commit/a01d73309a23058e452b01b19669131dc212380c).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #33870: [SPARK-36619][SS] Reset prefix-scan iterators on loading RocksDB

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33870:
URL: https://github.com/apache/spark/pull/33870#issuecomment-908997741


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142871/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #33870: [SPARK-36619][SS] Reset prefix-scan iterators on loading RocksDB

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33870:
URL: https://github.com/apache/spark/pull/33870#issuecomment-908962960


   Kubernetes integration test status failure
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47378/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #33870: [SPARK-36619][SS] Reset prefix-scan iterators on loading RocksDB

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33870:
URL: https://github.com/apache/spark/pull/33870#issuecomment-909056773


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142875/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #33870: [SPARK-36619][SS] Fix bugs around prefix-scan for HDFS backed state store and RocksDB state store

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33870:
URL: https://github.com/apache/spark/pull/33870#issuecomment-909270480


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47390/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #33870: [SPARK-36619][SS] Reset prefix-scan iterators on loading RocksDB

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33870:
URL: https://github.com/apache/spark/pull/33870#issuecomment-908903853






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HeartSaVioR commented on a change in pull request #33870: [SPARK-36619][SS] Fix bugs around prefix-scan for HDFS backed state store and RocksDB state store

Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on a change in pull request #33870:
URL: https://github.com/apache/spark/pull/33870#discussion_r699272824



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreMap.scala
##########
@@ -129,7 +129,11 @@ class PrefixScannableHDFSBackedStateStoreMap(
     other match {
       case o: PrefixScannableHDFSBackedStateStoreMap =>
         map.putAll(o.map)
-        prefixKeyToKeysMap.putAll(o.prefixKeyToKeysMap)

Review comment:
       This leads the prefix key maps across versions to "share" the values which it shouldn't. (especially the case of rolling back)




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #33870: [SPARK-36619][SS] Fix bugs around prefix-scan for HDFS backed state store and RocksDB state store

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33870:
URL: https://github.com/apache/spark/pull/33870#issuecomment-909356831


   Kubernetes integration test status failure
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47392/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] xuanyuanking commented on a change in pull request #33870: [SPARK-36619][SS] Reset prefix-scan iterators on loading RocksDB

Posted by GitBox <gi...@apache.org>.
xuanyuanking commented on a change in pull request #33870:
URL: https://github.com/apache/spark/pull/33870#discussion_r698968561



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala
##########
@@ -120,6 +120,9 @@ class RocksDB(
       if (conf.resetStatsOnLoad) {
         nativeStats.reset
       }
+      // reset resources to prevent side-effects from previous loaded version
+      prefixScanReuseIter.entrySet().asScala.foreach(_.getValue.close())
+      prefixScanReuseIter.clear()

Review comment:
       nit: How about wrap this 2 lines in a function `closePrefixScan(Reuse?)Iter()`? Since they are already been called in 3 places.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #33870: [SPARK-36619][SS] Reset prefix-scan iterators on loading RocksDB

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33870:
URL: https://github.com/apache/spark/pull/33870#issuecomment-908882245


   **[Test build #142871 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142871/testReport)** for PR 33870 at commit [`2b4f00d`](https://github.com/apache/spark/commit/2b4f00dd0cc6d7feca6c5f7cb548692cb2d98146).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #33870: [SPARK-36619][SS] Reset prefix-scan iterators on loading RocksDB

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33870:
URL: https://github.com/apache/spark/pull/33870#issuecomment-909056773






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org