You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Apache Spark (Jira)" <ji...@apache.org> on 2021/12/09 11:10:00 UTC
[jira] [Assigned] (SPARK-37596) Add the support for struct type column in the DropDuplicate in spark
[ https://issues.apache.org/jira/browse/SPARK-37596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Apache Spark reassigned SPARK-37596:
------------------------------------
Assignee: Apache Spark
> Add the support for struct type column in the DropDuplicate in spark
> --------------------------------------------------------------------
>
> Key: SPARK-37596
> URL: https://issues.apache.org/jira/browse/SPARK-37596
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 3.2.0
> Reporter: Saurabh Chawla
> Assignee: Apache Spark
> Priority: Major
>
> Add the support for struct type coulmn in the DropDuplicate in spark
> Currently on using the struct col in the DropDuplicate we will get the below exception
> case class StructDropDup(c1: Int, c2: Int)
> val df = Seq(("d1", StructDropDup(1, 2)),
> ("d1", StructDropDup(1, 2))).toDF("a", "b")
> df.dropDuplicates("b.c1")
> {code:java}
> org.apache.spark.sql.AnalysisException: Cannot resolve column name "b.c1" among (a, b)
> at org.apache.spark.sql.Dataset.$anonfun$dropDuplicates$1(Dataset.scala:2576)
> at scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:245)
> at scala.collection.Iterator.foreach(Iterator.scala:941)
> at scala.collection.Iterator.foreach$(Iterator.scala:941)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1429){code}
> As workAround inorder to find the the duplicate using the struct column
> df1.withColumn("b.c1", col("b.c1")).dropDuplicates("b.c1").drop("b.c1").collect
> {code:java}
> df.withColumn("b.c1", col("b.c1")).dropDuplicates("b.c1").drop("b.c1").collect
> res25: Array[org.apache.spark.sql.Row] = Array([d1,[1,2]]){code}
> There is need to add the support for the dropDuplicates
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org