You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "chirag aggarwal (JIRA)" <ji...@apache.org> on 2015/03/10 08:18:39 UTC
[jira] [Updated] (SPARK-6242) Support replace (drop) column for
parquet table
[ https://issues.apache.org/jira/browse/SPARK-6242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
chirag aggarwal updated SPARK-6242:
-----------------------------------
Description:
SPARK-5528 provides a easy way of support for add column to parquet tables. This is done by using the native parquet capability of merging the schema from all the part-files and _common_metadata files.
But, if someone wants to drop a column from the parquet table, this still does not work. This happens because, the merged schema shall still show the dropped column, but the column is no more there in metastore. So, the schema's obtained from the two sources do not match, and hence any subsequent query on this table fails.
Instead of checking for exact match between the two schemas, spark should only check if the schema obtained from metastore is subset of parquet merged schema or not. If this check passes, then the columns present in metastore should be allowed to be referred in the query.
> Support replace (drop) column for parquet table
> -----------------------------------------------
>
> Key: SPARK-6242
> URL: https://issues.apache.org/jira/browse/SPARK-6242
> Project: Spark
> Issue Type: Bug
> Affects Versions: 1.3.0
> Reporter: chirag aggarwal
>
> SPARK-5528 provides a easy way of support for add column to parquet tables. This is done by using the native parquet capability of merging the schema from all the part-files and _common_metadata files.
> But, if someone wants to drop a column from the parquet table, this still does not work. This happens because, the merged schema shall still show the dropped column, but the column is no more there in metastore. So, the schema's obtained from the two sources do not match, and hence any subsequent query on this table fails.
> Instead of checking for exact match between the two schemas, spark should only check if the schema obtained from metastore is subset of parquet merged schema or not. If this check passes, then the columns present in metastore should be allowed to be referred in the query.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org