[GitHub] [spark] HeartSaVioR commented on a change in pull request #26108: [SPARK-26154][SS] Streaming left/right outer join should not return outer nulls for already matched rows

[GitBox <gi...@apache.org>]

HeartSaVioR commented on a change in pull request #26108: [SPARK-26154][SS] Streaming left/right outer join should not return outer nulls for already matched rows
URL: https://github.com/apache/spark/pull/26108#discussion_r399569220
 
 

 ##########
 File path: sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingJoinSuite.scala
 ##########
 @@ -418,6 +420,63 @@ class StreamingInnerJoinSuite extends StreamTest with StateStoreMetricsTest with
       AddData(input2, 1.to(1000): _*),
       CheckAnswer(1.to(1000): _*))
   }
+
+  test("SPARK-26187 restore the stream-stream inner join query from Spark 2.4") {
 
 Review comment:
   Please find `SPARK-26187 restore the stream-stream outer join query from Spark 2.4` in this suite. (Yeah twos are unintentionally a bit far away, my bad. If moving test will help readability I'll do it as follow-up PR. Please let me know how you think.)
   
   One thing, the format is changed for stream-stream joins instead of only outer joins, to allow changing the type of join inner <-> left/right outer, which is technically possible (with such correctness issue) in previous state and now possible without correctness issue.
   
   Spark will fail the query if end users try to read state version 1 for stream-stream outer join, but will still allow state version 1 for stream-stream inner join to reduce the impact. Once they try to change the join type to outer then they'll have error message as covered in the test I mentioned above.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org