You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Apache Spark (Jira)" <ji...@apache.org> on 2022/03/11 08:09:00 UTC

[jira] [Commented] (SPARK-38522) Strengthen the contract on iterator method in StateStore

    [ https://issues.apache.org/jira/browse/SPARK-38522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17504778#comment-17504778 ] 

Apache Spark commented on SPARK-38522:
--------------------------------------

User 'HeartSaVioR' has created a pull request for this issue:
https://github.com/apache/spark/pull/35816

> Strengthen the contract on iterator method in StateStore
> --------------------------------------------------------
>
>                 Key: SPARK-38522
>                 URL: https://issues.apache.org/jira/browse/SPARK-38522
>             Project: Spark
>          Issue Type: Improvement
>          Components: Structured Streaming
>    Affects Versions: 3.3.0
>            Reporter: Jungtaek Lim
>            Priority: Major
>
> The root cause of SPARK-38320 was that the logic initialized the iterator first, and performed some updates against state store, and iterated through iterator expecting that all updates in between should be visible in iterator.
> That is not guaranteed in RocksDB state store, and the contract of Java ConcurrentHashMap which is used in HDFSBackedStateStore does not also guarantee it.
> It would be clearer if we update the contract to draw a line on behavioral guarantee to callers so that callers don't get such expectation.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org