[jira] [Updated] (HUDI-5585) After flink creates and writes the table, the spark alter table reports an error

["sivabalan narayanan (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HUDI-5585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

sivabalan narayanan updated HUDI-5585:
--------------------------------------
    Fix Version/s: 0.12.3

> After flink creates and writes the table, the spark alter table reports an error
> --------------------------------------------------------------------------------
>
>                 Key: HUDI-5585
>                 URL: https://issues.apache.org/jira/browse/HUDI-5585
>             Project: Apache Hudi
>          Issue Type: Bug
>          Components: flink-sql, spark-sql
>    Affects Versions: 0.12.2
>            Reporter: waywtdcc
>            Assignee: Danny Chen
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 0.13.0, 0.12.3
>
>
> After flink creates and writes the table, the spark alter table reports an error
>  
> *  flink
> {code:java}
> CREATE CATALOG myhudi WITH(
> 'type' = 'hudi',
> 'default-database' = 'default',
> 'mode' = 'hms',
> 'hive.conf.dir' = 'hdfs:///gdp-dev-3/flink/flink-conf'
> );
> create database if not exists myhudi.test_hudi3;
> drop table if exists  myhudi.test_hudi3.hudi_test19;
> create table if not exists myhudi.test_hudi3.hudi_test19
> (id bigint not null comment 'comment_test', `name` string  not null  comment 'comment_test2', ts3 bigint
>     ,PRIMARY KEY (`id`) NOT ENFORCED
>     )
>     with('connector' = 'hudi',
>         'table.type' = 'COPY_ON_WRITE',
>         'index.type'= 'BUCKET',
>         'precombine.field' = 'ts3',
>         'hoodie.datasource.hive_sync.sync_comment' = 'true');
> show create table myhudi.test_hudi3.hudi_test19;
> insert into myhudi.test_hudi3.hudi_test19
> values
> (1, 'cc1', 1); {code}
>  
> * spark
> {code:java}
> alter table test_hudi3.hudi_test19 add column c string; {code}
> *  error
> {code:java}
> org.apache.kyuubi.KyuubiSQLException: org.apache.kyuubi.KyuubiSQLException: Error operating ExecuteStatement: org.apache.spark.sql.AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to alter table. The following columns have types incompatible with the existing columns in their respective positions :
> col
>    at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:110)
>    at org.apache.spark.sql.hive.HiveExternalCatalog.alterTableDataSchema(HiveExternalCatalog.scala:679)
>    at org.apache.spark.sql.catalyst.catalog.ExternalCatalogWithListener.alterTableDataSchema(ExternalCatalogWithListener.scala:124)
>    at org.apache.spark.sql.catalyst.catalog.SessionCatalog.alterTableDataSchema(SessionCatalog.scala:459)
>    at org.apache.spark.sql.hudi.command.AlterHoodieTableAddColumnsCommand.refreshSchemaInMeta(AlterHoodieTableAddColumnsCommand.scala:90)
>    at org.apache.spark.sql.hudi.command.AlterHoodieTableAddColumnsCommand.run(AlterHoodieTableAddColumnsCommand.scala:70)
>    at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:75)
>    at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:73)
>    at org.apache.spark.sql.execution.command.ExecutedCommandExec.executeCollect(commands.scala:84)
>    at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.$anonfun$applyOrElse$1(QueryExecution.scala:98)
>    at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$6(SQLExecution.scala:109)
>    at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:169)
>    at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:95)
>    at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:779)
>    at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:64)
>    at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:98)
>    at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:94)
>    at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDownWithPruning$1(TreeNode.scala:584)
>    at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:176)
>    at org.apache.spark.sql.catalyst.trees.TreeNode.transformDownWithPruning(TreeNode.scala:584) {code}
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)