You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Hyukjin Kwon (JIRA)" <ji...@apache.org> on 2019/07/19 03:26:00 UTC
[jira] [Updated] (SPARK-28438) Ignore metadata's(comments)
difference when comparing datasource's schema and user-specific schema
[ https://issues.apache.org/jira/browse/SPARK-28438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hyukjin Kwon updated SPARK-28438:
---------------------------------
Summary: Ignore metadata's(comments) difference when comparing datasource's schema and user-specific schema (was: [SQL] Ignore metadata's(comments) difference when comparing datasource's schema and user-specific schema)
> Ignore metadata's(comments) difference when comparing datasource's schema and user-specific schema
> --------------------------------------------------------------------------------------------------
>
> Key: SPARK-28438
> URL: https://issues.apache.org/jira/browse/SPARK-28438
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.3.0
> Reporter: ShuMing Li
> Priority: Minor
>
> When users register a datasource table to Spark, Spark only support complete schema equality of datasource's origin schema and user-specific's schema now.
> However datasource's origin schema may be little different with user-specific's schema: the diff maybe `column's comment` or other metadata info.
> Can we ignore column's comment or metadata info when comparing?
> {code:java}
> // DataSource.scala
> case (dataSource: RelationProvider, Some(schema)) =>
> val baseRelation =
> dataSource.createRelation(sparkSession.sqlContext, caseInsensitiveOptions)
> if (baseRelation.schema != schema) {
> throw new AnalysisException(s"$className does not allow user-specified schemas, " +
> s"source schema: ${baseRelation.schema}, user-specific schema: ${schema}")
> }
> // StructType.scala
> override def equals(that: Any): Boolean = {
> that match
> { case StructType(otherFields) => java.util.Arrays.equals( fields.asInstanceOf[Array[AnyRef]], otherFields.asInstanceOf[Array[AnyRef]]) case _ => false }
> }
> {code}
>
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org