[jira] [Created] (SPARK-23869) Spark 2.3.0 left outer join not emitting null values instead waiting for the record in other stream

["bharath kumar avusherla (JIRA)" <ji...@apache.org>]

bharath kumar avusherla created SPARK-23869:
-----------------------------------------------

             Summary: Spark 2.3.0 left outer join not emitting null values instead waiting for the record in other stream
                 Key: SPARK-23869
                 URL: https://issues.apache.org/jira/browse/SPARK-23869
             Project: Spark
          Issue Type: Bug
          Components: Spark Core
    Affects Versions: 2.3.0
            Reporter: bharath kumar avusherla


Left outer join on two streams not emitting the null outputs. It is just waiting for the record to be added to other stream. Using socketstream to test this.

Details of the watermarks and intervals are:

val ds1Map = ds1
 .selectExpr("Id AS ds1_Id", "ds1_timestamp")
 .withWatermark("ds1_timestamp","10 seconds")

val ds2Map = ds2
 .selectExpr("Id AS ds2_Id", "ds2_timestamp")
 .withWatermark("ds2_timestamp", "20 seconds")

val output = ds1Map.join( ds2Map,
 expr(
 """ ds1_Id = ds2_Id AND ds2_timestamp >= ds1_timestamp AND  ds2_timestamp <= ds1_timestamp + interval 1 minutes """),
 "leftOuter")

val query = output.select("*")
.writeStream

.outputMode(OutputMode.Append)
.format("console")
.option("checkpointLocation", "./ewe-spark-checkpoints/")
.start()

 

query.awaitTermination()

 

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org