[jira] [Resolved] (SPARK-23477) Misleading exception message when union fails due to metadata

["Hyukjin Kwon (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-23477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hyukjin Kwon resolved SPARK-23477.
----------------------------------
    Resolution: Cannot Reproduce

I am resolving this as it's not reproduced in the master. It'd be nicer if we could find the Jira fixing this and backports if applicable.

> Misleading exception message when union fails due to metadata 
> --------------------------------------------------------------
>
>                 Key: SPARK-23477
>                 URL: https://issues.apache.org/jira/browse/SPARK-23477
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 2.2.1
>            Reporter: Tomasz Bartczak
>            Priority: Minor
>
> When I have two DF's that are different only in terms of metadata in fields inside a struct - I cannot union them but the error message shows that they are the same:
> {code:java}
> df = spark.createDataFrame([{'a':1}])
> a = df.select(struct('a').alias('x'))
> b = df.select(col('a').alias('a',metadata={'description':'xxx'})).select(struct(col('a')).alias('x'))
> a.union(b).printSchema(){code}
> gives:
> {code:java}
> An error occurred while calling o1076.union.
> : org.apache.spark.sql.AnalysisException: Union can only be performed on tables with the compatible column types. struct<a:bigint> <> struct<a:bigint> at the first column of the second table{code}
> and this part:
> {code:java}
> struct<a:bigint> <> struct<a:bigint>{code}
> does not make any sense because those are the same.
>  
> Since metadata must be the same for union -> it should be incuded in the error message



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org