[jira] [Created] (SPARK-32863) Full outer stream-stream join

["Cheng Su (Jira)" <ji...@apache.org>]

Cheng Su created SPARK-32863:
--------------------------------

             Summary: Full outer stream-stream join
                 Key: SPARK-32863
                 URL: https://issues.apache.org/jira/browse/SPARK-32863
             Project: Spark
          Issue Type: New Feature
          Components: Structured Streaming
    Affects Versions: 3.1.0
            Reporter: Cheng Su


Current stream-stream join supports inner, left outer and right outer join ([https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StreamingSymmetricHashJoinExec.scala#L166] ). With current design of stream-stream join (which marks whether the row is matched or not in state store), it would be very easy to support full outer join as well.

 

Full outer stream-stream join will work as followed:

(1).for left side input row, check if there's a match on right side state store. If there's a match, output all matched rows. Put the row in left side state store.

(2).for right side input row, check if there's a match on left side state store. If there's a match, output all matched rows and update left side rows state with "matched" field to set to true. Put the right side row in right side state store.

(3).for left side row needs to be evicted from state store, output the row if "matched" field is false.

(4).for right side row needs to be evicted from state store, output the row if "matched" field is false.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org