[jira] [Resolved] (SPARK-28438) Ignore metadata's(comments) difference when comparing datasource's schema and user-specific schema

["Hyukjin Kwon (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-28438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hyukjin Kwon resolved SPARK-28438.
----------------------------------
    Resolution: Duplicate

> Ignore metadata's(comments) difference when comparing datasource's schema and user-specific schema
> --------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-28438
>                 URL: https://issues.apache.org/jira/browse/SPARK-28438
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 2.3.0
>            Reporter: ShuMing Li
>            Priority: Minor
>
> When users register a datasource table to Spark,  Spark only support complete schema equality of datasource's origin schema  and user-specific's schema now.
> However datasource's origin schema may be little different with user-specific's schema: the diff maybe `column's comment` or other metadata info.
> Can we ignore column's comment or metadata info when comparing?
> {code:java}
> // DataSource.scala
> case (dataSource: RelationProvider, Some(schema)) =>
> val baseRelation =
> dataSource.createRelation(sparkSession.sqlContext, caseInsensitiveOptions)
> if (baseRelation.schema != schema) {
> throw new AnalysisException(s"$className does not allow user-specified schemas, " +
> s"source schema: ${baseRelation.schema}, user-specific schema: ${schema}")
> }
> // StructType.scala
> override def equals(that: Any): Boolean = {
> that match
> { case StructType(otherFields) => java.util.Arrays.equals( fields.asInstanceOf[Array[AnyRef]], otherFields.asInstanceOf[Array[AnyRef]]) case _ => false }
> }
> {code}
>  



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org