[jira] [Created] (SPARK-59698) ASOF JOIN isn't properly blocked for stream on the right side

["Jungtaek Lim (Jira)" <ji...@apache.org>]

Jungtaek Lim created SPARK-59698:
------------------------------------

             Summary: ASOF JOIN isn't properly blocked for stream on the right side
                 Key: SPARK-59698
                 URL: https://issues.apache.org/jira/browse/SPARK-59698
             Project: Spark
          Issue Type: Bug
          Components: Structured Streaming
    Affects Versions: 4.3.0
            Reporter: Jungtaek Lim


ASOF JOIN is going to be introduced in Apache Spark 4.3.0. It's feature complete for static-static ASOF JOIN, but it doesn't look like we audited with streaming.

From the definition of ASOF JOIN, the join should find the best match among multiple matches (by time). This requires the left side to wait for the right side to have all the data for a certain time, and when stream is involved, this requires event time processing semantic. This requires watermark to be set, and the operator implementation must be stateful - left side has to be buffered and waited for the time watermark passes the event time, so that we know right side has all the data for that event time.

This is NOT considered and Spark allows *-stream ASOF JOIN which is incorrect (it only matches with data within microbatch). We must disallow the case.

In addition, stream-static ASOF JOIN can be accepted with the same semantic of stream-static join but we do not have any test with it. We should also address test coverage.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org