[jira] [Work logged] (HIVE-24987) hive.metastore.disallow.incompatible.col.type.changes is too restrictive for some storage formats

["ASF GitHub Bot (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HIVE-24987?focusedWorklogId=604694&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-604694 ]

ASF GitHub Bot logged work on HIVE-24987:
-----------------------------------------

                Author: ASF GitHub Bot
            Created on: 01/Jun/21 18:36
            Start Date: 01/Jun/21 18:36
    Worklog Time Spent: 10m 
      Work Description: vihangk1 opened a new pull request #2336:
URL: https://github.com/apache/hive/pull/2336


   
   
   ### What changes were proposed in this pull request?
   hive.metastore.disallow.incompatible.col.type.check config currently checks if a alter table operation is making a incompatible schema change to the table. By default it is set to true which would error out the alter table call when such a change is detected. However, this change is too restrictive for certain file-formats like Kudu. In case of Kudu, it is allowed to drop a column which could result in the schema to be incompatible according to current implementation of this check. This causes a bad user-experience for Kudu users and there is no real work-around than to disable this check all together. Disabling the check is not an option since for file-formats like Parquet this is should be true to avoid data corruption/incorrect results.
   
   This change introduces a new config which can be used by users to provide a exception list based on table serde library name. If a table belongs to such a serde, the check is skipped. By default currently, only Kudu tables are added to this config.
   
   ### Why are the changes needed?
   See above.
   
   ### Does this PR introduce _any_ user-facing change?
   This introduces a new configuration option for metastore.
   
   ### How was this patch tested?
   A new unit-test was added to exercise the specific use-case. Existing tests make sure that previous behavior is not changed.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Issue Time Tracking
-------------------

            Worklog Id:     (was: 604694)
    Remaining Estimate: 0h
            Time Spent: 10m

> hive.metastore.disallow.incompatible.col.type.changes is too restrictive for some storage formats
> -------------------------------------------------------------------------------------------------
>
>                 Key: HIVE-24987
>                 URL: https://issues.apache.org/jira/browse/HIVE-24987
>             Project: Hive
>          Issue Type: Improvement
>            Reporter: Vihang Karajgaonkar
>            Assignee: Vihang Karajgaonkar
>            Priority: Major
>          Time Spent: 10m
>  Remaining Estimate: 0h
>
> Currently when {{hive.metastore.disallow.incompatible.col.type.changes}} is set to true it disallows any schema changes which are deemed as backwards incompatible e.g dropping a column of a table. While this may be a correct thing to do for Parquet or Orc tables, it is too restrictive for storage formats like Kudu. 
> Currently, for Kudu tables, Impala supports dropping a column. But if we set this config to true metastore disallows changing the schema of the metastore table. I am assuming this would be problematic for Iceberg tables too which supports such schema changes.
> The proposal is to have a new configuration which provided a exclusion list of the table fileformat where this check will be skipped. Currently, we will only include Kudu tables to skip this check.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)