You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "Todd Lipcon (JIRA)" <ji...@apache.org> on 2018/08/01 18:57:00 UTC
[jira] [Updated] (IMPALA-7309) Prevent the addition of Avro
partitions to non-Avro tables with incompatible schema
[ https://issues.apache.org/jira/browse/IMPALA-7309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated IMPALA-7309:
--------------------------------
Summary: Prevent the addition of Avro partitions to non-Avro tables with incompatible schema (was: Prevent the addition of Avro schemas to non-Avro tables with incompatible schema)
> Prevent the addition of Avro partitions to non-Avro tables with incompatible schema
> -----------------------------------------------------------------------------------
>
> Key: IMPALA-7309
> URL: https://issues.apache.org/jira/browse/IMPALA-7309
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog, Frontend
> Reporter: Todd Lipcon
> Priority: Major
>
> Per a recent [mailing list thread|https://lists.apache.org/thread.html/fb68c54bd66a40982ee17f9f16f87a4112220a5df035a311bda310f1@<dev.impala.apache.org>] the behavior of Avro partitions within non-Avro tables is inconsistent with Hive, and somewhat suprising. For example, the addition of a partition can cause the results of "describe" on the table to change, but only after a refresh or invalidate. In the mailing list thread, we decided to change the behavior to:
> 1. Schema handling:
> - if a table's properties indicate it's an avro table, parse and adopt the
> external avro schema as the table schema, or infer an avro-compatible schema from the existing columns
> - if a table's properties indicate it's _not_ an avro table, but there is
> an external avro schema defined in the table properties, then parse the
> avro schema and include it in the TableDescriptor (for use by avro
> partitions) but *do not* adopt it as the table schema.
> 2. Handling incompatible schemas:
> - If the table-level format is non-Avro,
> - AND the table contains column types incompatible with Avro (eg tinyint),
> - AND the table has an existing avro partition,
> - THEN the query will yield an error about incompatible types
> 3. Try to prevent shooting in the foot
> - If the table-level format is non-Avro,
> - AND the table contains column types incompatible with Avro (eg tinyint),
> - THEN disallow changing the file format of an existing partition to Avro
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org