[jira] [Commented] (SPARK-23855) Performing a Join after a CrossJoin can lead to data corruption

["Erik Selin (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-23855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16437327#comment-16437327 ] 

Erik Selin commented on SPARK-23855:
------------------------------------

+1, from our investigations it looks like we've also hit this issue on 2.2

> Performing a Join after a CrossJoin can lead to data corruption
> ---------------------------------------------------------------
>
>                 Key: SPARK-23855
>                 URL: https://issues.apache.org/jira/browse/SPARK-23855
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 2.2.0, 2.2.1
>            Reporter: Martin Junghanns
>            Priority: Major
>
> The following tests produces the wrong result for the join operation. The error only occurs when joining on the first column of the crossed dataframe. However, a subsequent select fixes the data (which is of course not a solution).
> It works on 2.3.0 though. It would be nice to get this fixed on the 2.2.x releases, too. Maybe someone can point me to the issue that has been fixed? Would be nice to see the solution in code.
> {code}
> it("should correctly perform a join after a cross") {
>     val df1 = sparkSession.createDataFrame(Seq(Tuple1(0L)))
>       .toDF("a")
>     val df2 = sparkSession.createDataFrame(Seq(Tuple1(1L)))
>       .toDF("b")
>     val df3 = sparkSession.createDataFrame(Seq(Tuple1(0L)))
>       .toDF("c")
>     val cross = df1.crossJoin(df2)
>     cross.show()
>     val joined = cross
>       .join(df3, cross.col("a") === df3.col("c"))
>     joined.show()
>     val selected = joined.select("*")
>     selected.show
>   }
> {code}
> prints:
> {code:java}
> +---+---+
> |  a|  b|
> +---+---+
> |  0|  1|
> +---+---+
> +---+---+---+
> |  a|  b|  c|
> +---+---+---+
> |  0|  0|  1|
> +---+---+---+
> +---+---+---+
> |  a|  b|  c|
> +---+---+---+
> |  0|  1|  0|
> +---+---+---+
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org