You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "amaliujia (via GitHub)" <gi...@apache.org> on 2023/06/20 20:05:22 UTC

[GitHub] [spark] amaliujia commented on a diff in pull request #41633: [SPARK-44082][SQL] Generate operator does not update reference set properly

amaliujia commented on code in PR #41633:
URL: https://github.com/apache/spark/pull/41633#discussion_r1235771598


##########
sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala:
##########
@@ -3613,6 +3613,21 @@ class DataFrameSuite extends QueryTest
     val df = Seq("0.5944910").toDF("a")
     checkAnswer(df.selectExpr("cast(a as decimal(7,7)) div 100"), Row(0))
   }
+
+  test("SPARK-44082: Generate operator does not update reference set properly") {
+    val schema = StructType(Array(
+      StructField("col1", StringType, true),
+      StructField("col2", StringType, true)
+    ))
+    val rowData = Seq(Row("100", "20000"),
+      Row("100", "100000"),
+      Row("300", "3000"))
+    val df1 = spark.createDataFrame(spark.sparkContext.parallelize(rowData), schema)
+    val df2 = spark.createDataFrame(spark.sparkContext.parallelize(rowData), schema)
+    df1.dropDuplicates(Seq("col1")).exceptAll(df2).collect.foreach { row =>

Review Comment:
   This is not because of the `Generate` operator but because of `ReplaceDeduplicateWithAggregate` rule which replace duplicate with aggregate which contains columns that current Generator operator does not know.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org