[jira] [Commented] (HIVE-15995) Syncing metastore table with serde schema

["Hive QA (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HIVE-15995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15883545#comment-15883545 ] 

Hive QA commented on HIVE-15995:
--------------------------------



Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12854469/HIVE-15995.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 10226 tests executed
*Failed tests:*
{noformat}
TestDerbyConnector - did not produce a TEST-*.xml file (likely timed out) (batchId=235)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver (batchId=161)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14] (batchId=223)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query23] (batchId=223)
org.apache.hadoop.hive.cli.TestSparkCliDriver.org.apache.hadoop.hive.cli.TestSparkCliDriver (batchId=129)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/3762/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/3762/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-3762/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12854469 - PreCommit-HIVE-Build

> Syncing metastore table with serde schema
> -----------------------------------------
>
>                 Key: HIVE-15995
>                 URL: https://issues.apache.org/jira/browse/HIVE-15995
>             Project: Hive
>          Issue Type: Bug
>          Components: Metastore
>    Affects Versions: 1.2.1, 2.1.0
>            Reporter: Michal Ferlinski
>            Assignee: Michal Ferlinski
>             Fix For: 2.1.0
>
>         Attachments: cx1.avsc, cx2.avsc, HIVE-15995.patch
>
>
> Hive enables table schema evolution via properties. For avro e.g. we can alter the 'avro.schema.url' property to update table schema to the next version. Updating properties however doesn't affect column list stored in metastore DB so the table is not in the newest version when returned from metastore API. This is problem for tools working with metastore (e.g. Presto).
> To solve this issue I suggest to introduce new DDL statement syncing metastore columns with those from serde:
> {code}
> ALTER TABLE user_test1 UPDATE COLUMNS
> {code}
> Note that this is format independent solution. 
> To reproduce, follow the instructions below:
> - Create table based on avro schema version 1 (cxv1.avsc)
> {code}
> CREATE EXTERNAL TABLE user_test1
>   PARTITIONED BY (dt string)
>   ROW FORMAT SERDE
>   'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
>   STORED AS INPUTFORMAT
>   'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
>   OUTPUTFORMAT
>   'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
>   LOCATION
>   '/tmp/schema-evolution/user_test1'
>   TBLPROPERTIES ('avro.schema.url'='/tmp/schema-evolution/cx1.avsc');
> {code}
> - Update schema to version 2 (cx2.avsc)
> {code}
> ALTER TABLE user_test1 SET TBLPROPERTIES ('avro.schema.url' = '/tmp/schema-evolution/cx2.avsc');
> {code}
> - Print serde columns (top info) and metastore columns (Detailed Table Information):
> {code}
> DESCRIBE EXTENDED user_test1
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)