[jira] [Created] (SPARK-23477) Misleading exception message when union fails due to metadata

["Tomasz Bartczak (JIRA)" <ji...@apache.org>]

Tomasz Bartczak created SPARK-23477:
---------------------------------------

             Summary: Misleading exception message when union fails due to metadata 
                 Key: SPARK-23477
                 URL: https://issues.apache.org/jira/browse/SPARK-23477
             Project: Spark
          Issue Type: Bug
          Components: SQL
    Affects Versions: 2.2.1
            Reporter: Tomasz Bartczak


When I have two DF's that are different only in terms of metadata in fields inside a struct - I cannot union them but the error message shows that they are the same:
{code:java}
df = spark.createDataFrame([{'a':1}])
a = df.select(struct('a').alias('x'))
b = df.select(col('a').alias('a',metadata={'description':'xxx'})).select(struct(col('a')).alias('x'))
a.union(b).printSchema(){code}
gives:
{code:java}
An error occurred while calling o1076.union.
: org.apache.spark.sql.AnalysisException: Union can only be performed on tables with the compatible column types. struct<a:bigint> <> struct<a:bigint> at the first column of the second table{code}
and this part:
{code:java}
struct<a:bigint> <> struct<a:bigint>{code}
does not make any sense because those are the same.

 

Since metadata must be the same for union -> it should be incuded in the error message



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org