[GitHub] [spark] c21 commented on a change in pull request #30076: [SPARK-32862][SS] Left semi stream-stream join

[GitBox <gi...@apache.org>]

c21 commented on a change in pull request #30076:
URL: https://github.com/apache/spark/pull/30076#discussion_r507946803



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StreamingSymmetricHashJoinExec.scala
##########
@@ -151,7 +151,7 @@ case class StreamingSymmetricHashJoinExec(
       stateWatermarkPredicates = JoinStateWatermarkPredicates(), stateFormatVersion, left, right)
   }
 
-  if (stateFormatVersion < 2 && joinType != Inner) {
+  if (stateFormatVersion < 2 && (joinType == LeftOuter || joinType == RightOuter)) {

Review comment:
       @HeartSaVioR - sorry I was under impression that we don't allow state format version 1 after 3.0 without looking into the relevant code too much. But I found it seems that we are having a config [spark.sql.streaming.join.stateFormatVersion](https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala#L1345), that allows user to use version 1.
   
   I will add the check to fail the left semi join query with state format 1 as outer joins here. Thanks.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org