[jira] [Commented] (SPARK-36619) Iterators on prefix scan in RocksDB are not cleaned up unless aborting

["Apache Spark (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-36619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17407052#comment-17407052 ] 

Apache Spark commented on SPARK-36619:
--------------------------------------

User 'HeartSaVioR' has created a pull request for this issue:
https://github.com/apache/spark/pull/33870

> Iterators on prefix scan in RocksDB are not cleaned up unless aborting
> ----------------------------------------------------------------------
>
>                 Key: SPARK-36619
>                 URL: https://issues.apache.org/jira/browse/SPARK-36619
>             Project: Spark
>          Issue Type: Bug
>          Components: Structured Streaming
>    Affects Versions: 3.2.0
>            Reporter: Jungtaek Lim
>            Priority: Blocker
>
> In RocksDB state store provider implementation, we leverage iterators on prefix scan, which are being closed in rollback() method.
> While this works now for session window since state store instance in read physical plan will always call abort, it could bring correctness issue for stateful operator which doesn't instantiate two different physical plans on read and write.
> We should make sure these iterators get closed to let these iterators don't affect multiple micro-batches (plays as side-effect).



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org