You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Cheng Su (Jira)" <ji...@apache.org> on 2020/10/21 20:18:00 UTC

[jira] [Created] (SPARK-33211) Early state store eviction for left semi stream-stream join

Cheng Su created SPARK-33211:
--------------------------------

             Summary: Early state store eviction for left semi stream-stream join
                 Key: SPARK-33211
                 URL: https://issues.apache.org/jira/browse/SPARK-33211
             Project: Spark
          Issue Type: Improvement
          Components: Structured Streaming
    Affects Versions: 3.1.0
            Reporter: Cheng Su


As a followup from discussion [https://github.com/apache/spark/pull/30076/files/3918727a08c8d0d4c65ccc8ea902f77051b78b1d#r508926034] and [https://github.com/apache/spark/pull/30076#discussion_r509222802] , for left semi stream-stream join, the matched left side rows can be evicted from left state store immediately, without waiting for to be below watermark. However it needs more thought for how to implement efficiently to not iterate all values if watermark predicate is on key.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org