You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Cheng Su (Jira)" <ji...@apache.org> on 2020/10/21 20:18:00 UTC
[jira] [Updated] (SPARK-33211) Early state store eviction for left
semi stream-stream join
[ https://issues.apache.org/jira/browse/SPARK-33211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Cheng Su updated SPARK-33211:
-----------------------------
Parent: SPARK-32883
Issue Type: Sub-task (was: Improvement)
> Early state store eviction for left semi stream-stream join
> -----------------------------------------------------------
>
> Key: SPARK-33211
> URL: https://issues.apache.org/jira/browse/SPARK-33211
> Project: Spark
> Issue Type: Sub-task
> Components: Structured Streaming
> Affects Versions: 3.1.0
> Reporter: Cheng Su
> Priority: Minor
>
> As a followup from discussion [https://github.com/apache/spark/pull/30076/files/3918727a08c8d0d4c65ccc8ea902f77051b78b1d#r508926034] and [https://github.com/apache/spark/pull/30076#discussion_r509222802] , for left semi stream-stream join, the matched left side rows can be evicted from left state store immediately, without waiting for to be below watermark. However it needs more thought for how to implement efficiently to not iterate all values if watermark predicate is on key.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org