You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "zyl891229 (via GitHub)" <gi...@apache.org> on 2023/11/24 03:38:51 UTC
[I] [SUPPORT] Error when overwrite and synchronize hive metastore with 0.14.0 [hudi]
zyl891229 opened a new issue, #10170:
URL: https://github.com/apache/hudi/issues/10170
**Describe the problem you faced**
Error reported when overwrite and synchronize hms
**To Reproduce**
Steps to reproduce the behavior:
1. use spark dataframe , configure the synchronous hms
df.write.format("hudi")
.option(DataSourceWriteOptions.TABLE_TYPE.key(), COW_TABLE_TYPE_OPT_VAL)
.options(getQuickstartWriteConfigs)
.option(PRECOMBINE_FIELD_OPT_KEY, "name")
.option(RECORDKEY_FIELD_OPT_KEY, "name")
.option(PARTITIONPATH_FIELD_OPT_KEY, "city")
.option(HoodieWriteConfig.TBL_NAME.key(), tableName)
.option(DataSourceWriteOptions.OPERATION.key(), DataSourceWriteOptions.BULK_INSERT_OPERATION_OPT_VAL)
.option("hoodie.bulkinsert.overwrite.operation.type","insert_overwrite")
.option(DataSourceWriteOptions.HIVE_STYLE_PARTITIONING.key(), "true")
.option(HoodieTableConfig.POPULATE_META_FIELDS.key(), "false")
.option(HoodieMetadataConfig.ENABLE.key(), "true")
.option("hoodie.metadata.index.column.stats.enable", "true")
.option("hoodie.datasource.hive_sync.enable","true")
.option("hoodie.datasource.hive_sync.mode","hms")
.option("hoodie.datasource.hive_sync.metastore.uris","thrift://127.0.0.1:9083")
.mode(SaveMode.Append)
.save(basePath)
3.
When the second overwrite is written, the following error is reported during the synchronization logic
According to the hms service log, database is an empty string,
But according to the source code default value should be ”default“
<img width="922" alt="image" src="https://github.com/apache/hudi/assets/13060417/e699403b-e3a7-4909-bf0c-f34032a7ff7f">
<img width="899" alt="image" src="https://github.com/apache/hudi/assets/13060417/ef622afb-baf8-4c0a-a203-624e0629f1f7">
**Expected behavior**
The default value for hoodie.datasource.hive_sync.database is ”default“ not empty string
**Environment Description**
* Hudi version :
* 0.14.0
* Spark version :
* 3.1.1
* Storage (HDFS/S3/GCS..) :
* OSS
* Running on Docker? (yes/no) :
* no
**Additional context**
Add any other context about the problem here.
**Stacktrace**
spark log
2023-11-21 21:25:37,195 ERROR org.apache.spark.deploy.yarn.ApplicationMaster - User class threw exception: org.apache.hudi.exception.HoodieMetaSyncException: Could not sync using the meta sync class org.apache.hudi.hive.HiveSyncTool
org.apache.hudi.exception.HoodieMetaSyncException: Could not sync using the meta sync class org.apache.hudi.hive.HiveSyncTool
at org.apache.hudi.sync.common.util.SyncUtilHelpers.runHoodieMetaSync(SyncUtilHelpers.java:81) ~[_f1c60536b59a86c4fb55c4ab4a94cd56.jar:?]
at org.apache.hudi.HoodieSparkSqlWriter$.$anonfun$metaSync$2(HoodieSparkSqlWriter.scala:993) ~[_f1c60536b59a86c4fb55c4ab4a94cd56.jar:?]
at scala.collection.mutable.HashSet.foreach(HashSet.scala:79) ~[scala-library-2.12.10.jar:?]
at org.apache.hudi.HoodieSparkSqlWriter$.metaSync(HoodieSparkSqlWriter.scala:991) ~[_f1c60536b59a86c4fb55c4ab4a94cd56.jar:?]
at org.apache.hudi.HoodieSparkSqlWriter$.commitAndPerformPostOperations(HoodieSparkSqlWriter.scala:1089) ~[_f1c60536b59a86c4fb55c4ab4a94cd56.jar:?]
at org.apache.hudi.HoodieSparkSqlWriter$.bulkInsertAsRow(HoodieSparkSqlWriter.scala:920) ~[_f1c60536b59a86c4fb55c4ab4a94cd56.jar:?]
at org.apache.hudi.HoodieSparkSqlWriter$.writeInternal(HoodieSparkSqlWriter.scala:409) ~[_f1c60536b59a86c4fb55c4ab4a94cd56.jar:?]
at org.apache.hudi.HoodieSparkSqlWriter$.write(HoodieSparkSqlWriter.scala:132) ~[_f1c60536b59a86c4fb55c4ab4a94cd56.jar:?]
at org.apache.hudi.DefaultSource.createRelation(DefaultSource.scala:150) ~[_f1c60536b59a86c4fb55c4ab4a94cd56.jar:?]
at org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:46) ~[spark-sql_2.12-3.1.1.jar:3.1.1]
at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:70) ~[spark-sql_2.12-3.1.1.jar:3.1.1]
at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:68) ~[spark-sql_2.12-3.1.1.jar:3.1.1]
at org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:90) ~[spark-sql_2.12-3.1.1.jar:3.1.1]
at org.apache.spark.sql.execution.SparkPlan.$anonfun$execute$1(SparkPlan.scala:180) ~[spark-sql_2.12-3.1.1.jar:3.1.1]
at org.apache.spark.sql.execution.SparkPlan.$anonfun$executeQuery$1(SparkPlan.scala:218) ~[spark-sql_2.12-3.1.1.jar:3.1.1]
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151) ~[spark-core_2.12-3.1.1.jar:3.1.1]
at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:215) ~[spark-sql_2.12-3.1.1.jar:3.1.1]
at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:176) ~[spark-sql_2.12-3.1.1.jar:3.1.1]
at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:132) ~[spark-sql_2.12-3.1.1.jar:3.1.1]
at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:131) ~[spark-sql_2.12-3.1.1.jar:3.1.1]
at org.apache.spark.sql.DataFrameWriter.$anonfun$runCommand$1(DataFrameWriter.scala:989) ~[spark-sql_2.12-3.1.1.jar:3.1.1]
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$5(SQLExecution.scala:103) ~[spark-sql_2.12-3.1.1.jar:3.1.1]
at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:163) ~[spark-sql_2.12-3.1.1.jar:3.1.1]
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:90) ~[spark-sql_2.12-3.1.1.jar:3.1.1]
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:772) ~[spark-sql_2.12-3.1.1.jar:3.1.1]
at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:64) ~[spark-sql_2.12-3.1.1.jar:3.1.1]
at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:989) ~[spark-sql_2.12-3.1.1.jar:3.1.1]
at org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:438) ~[spark-sql_2.12-3.1.1.jar:3.1.1]
at org.apache.spark.sql.DataFrameWriter.saveInternal(DataFrameWriter.scala:415) ~[spark-sql_2.12-3.1.1.jar:3.1.1]
at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:293) ~[spark-sql_2.12-3.1.1.jar:3.1.1]
at com.aliyun.odps.spark.examples.hudi.SparkHudiJson$.main(SparkHudiJson.scala:253) ~[_f1c60536b59a86c4fb55c4ab4a94cd56.jar:?]
at com.aliyun.odps.spark.examples.hudi.SparkHudiJson.main(SparkHudiJson.scala) ~[_f1c60536b59a86c4fb55c4ab4a94cd56.jar:?]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_65-AliJVM]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_65-AliJVM]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_65-AliJVM]
at java.lang.reflect.Method.invoke(Method.java:497) ~[?:1.8.0_65-AliJVM]
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:732) [spark-yarn_2.12-3.1.1.jar:3.1.1]
Caused by: org.apache.hudi.exception.HoodieException: Got runtime exception when hive syncing hudi_dwd_aplus_pub_log_event_1d_tenant1_project47532305069060_v50_extend_odps_2201_12000_bulkoverwrite
at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:168) ~[_f1c60536b59a86c4fb55c4ab4a94cd56.jar:?]
at org.apache.hudi.sync.common.util.SyncUtilHelpers.runHoodieMetaSync(SyncUtilHelpers.java:79) ~[_f1c60536b59a86c4fb55c4ab4a94cd56.jar:?]
... 36 more
Caused by: org.apache.hudi.hive.HoodieHiveSyncException: failed to create table hudi_dwd_aplus_pub_log_event_1d_tenant1_project47532305069060_v50_extend_odps_2201_12000_bulkoverwrite
at org.apache.hudi.hive.ddl.HMSDDLExecutor.createTable(HMSDDLExecutor.java:140) ~[_f1c60536b59a86c4fb55c4ab4a94cd56.jar:?]
at org.apache.hudi.hive.HoodieHiveSyncClient.createTable(HoodieHiveSyncClient.java:235) ~[_f1c60536b59a86c4fb55c4ab4a94cd56.jar:?]
at org.apache.hudi.hive.HiveSyncTool.syncFirstTime(HiveSyncTool.java:329) ~[_f1c60536b59a86c4fb55c4ab4a94cd56.jar:?]
at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:251) ~[_f1c60536b59a86c4fb55c4ab4a94cd56.jar:?]
at org.apache.hudi.hive.HiveSyncTool.doSync(HiveSyncTool.java:177) ~[_f1c60536b59a86c4fb55c4ab4a94cd56.jar:?]
at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:165) ~[_f1c60536b59a86c4fb55c4ab4a94cd56.jar:?]
at org.apache.hudi.sync.common.util.SyncUtilHelpers.runHoodieMetaSync(SyncUtilHelpers.java:79) ~[_f1c60536b59a86c4fb55c4ab4a94cd56.jar:?]
... 36 more
Caused by: org.apache.hadoop.hive.metastore.api.InvalidObjectException:
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$create_table_with_environment_context_result$create_table_with_environment_context_resultStandardScheme.read(ThriftHiveMetastore.java:42216) ~[hive-metastore-2.3.7.jar:2.3.7]
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$create_table_with_environment_context_result$create_table_with_environment_context_resultStandardScheme.read(ThriftHiveMetastore.java:42193) ~[hive-metastore-2.3.7.jar:2.3.7]
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$create_table_with_environment_context_result.read(ThriftHiveMetastore.java:42119) ~[hive-metastore-2.3.7.jar:2.3.7]
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:88) ~[_f1c60536b59a86c4fb55c4ab4a94cd56.jar:?]
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_create_table_with_environment_context(ThriftHiveMetastore.java:1203) ~[hive-metastore-2.3.7.jar:2.3.7]
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.create_table_with_environment_context(ThriftHiveMetastore.java:1189) ~[hive-metastore-2.3.7.jar:2.3.7]
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.create_table_with_environment_context(HiveMetaStoreClient.java:2405) ~[hive-metastore-2.3.7.jar:2.3.7]
at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.create_table_with_environment_context(SessionHiveMetaStoreClient.java:93) ~[hive-exec-2.3.7-core.jar:2.3.7]
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createTable(HiveMetaStoreClient.java:752) ~[hive-metastore-2.3.7.jar:2.3.7]
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createTable(HiveMetaStoreClient.java:740) ~[hive-metastore-2.3.7.jar:2.3.7]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_65-AliJVM]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_65-AliJVM]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_65-AliJVM]
at java.lang.reflect.Method.invoke(Method.java:497) ~[?:1.8.0_65-AliJVM]
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:173) ~[hive-metastore-2.3.7.jar:2.3.7]
at com.sun.proxy.$Proxy73.createTable(Unknown Source) ~[?:?]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_65-AliJVM]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_65-AliJVM]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_65-AliJVM]
at java.lang.reflect.Method.invoke(Method.java:497) ~[?:1.8.0_65-AliJVM]
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler.invoke(HiveMetaStoreClient.java:2336) ~[hive-metastore-2.3.7.jar:2.3.7]
at com.sun.proxy.$Proxy73.createTable(Unknown Source) ~[?:?]
at org.apache.hudi.hive.ddl.HMSDDLExecutor.createTable(HMSDDLExecutor.java:137) ~[_f1c60536b59a86c4fb55c4ab4a94cd56.jar:?]
at org.apache.hudi.hive.HoodieHiveSyncClient.createTable(HoodieHiveSyncClient.java:235) ~[_f1c60536b59a86c4fb55c4ab4a94cd56.jar:?]
at org.apache.hudi.hive.HiveSyncTool.syncFirstTime(HiveSyncTool.java:329) ~[_f1c60536b59a86c4fb55c4ab4a94cd56.jar:?]
at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:251) ~[_f1c60536b59a86c4fb55c4ab4a94cd56.jar:?]
at org.apache.hudi.hive.HiveSyncTool.doSync(HiveSyncTool.java:177) ~[_f1c60536b59a86c4fb55c4ab4a94cd56.jar:?]
at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:165) ~[_f1c60536b59a86c4fb55c4ab4a94cd56.jar:?]
at org.apache.hudi.sync.common.util.SyncUtilHelpers.runHoodieMetaSync(SyncUtilHelpers.java:79) ~[_f1c60536b59a86c4fb55c4ab4a94cd56.jar:?]
... 36 more
2023-11-21 21:25:37,203 INFO org.apache.spark.deploy.yarn.ApplicationMaster - Final app status: FAILED, exitCode: 15, (reason: User class threw exception: org.apache.hudi.exception.HoodieMetaSyncException: Could not sync using the meta sync class org.apache.hudi.hive.HiveSyncTool
at org.apache.hudi.sync.common.util.SyncUtilHelpers.runHoodieMetaSync(SyncUtilHelpers.java:81)
at org.apache.hudi.HoodieSparkSqlWriter$.$anonfun$metaSync$2(HoodieSparkSqlWriter.scala:993)
at scala.collection.mutable.HashSet.foreach(HashSet.scala:79)
at org.apache.hudi.HoodieSparkSqlWriter$.metaSync(HoodieSparkSqlWriter.scala:991)
at org.apache.hudi.HoodieSparkSqlWriter$.commitAndPerformPostOperations(HoodieSparkSqlWriter.scala:1089)
at org.apache.hudi.HoodieSparkSqlWriter$.bulkInsertAsRow(HoodieSparkSqlWriter.scala:920)
at org.apache.hudi.HoodieSparkSqlWriter$.writeInternal(HoodieSparkSqlWriter.scala:409)
at org.apache.hudi.HoodieSparkSqlWriter$.write(HoodieSparkSqlWriter.scala:132)
at org.apache.hudi.DefaultSource.createRelation(DefaultSource.scala:150)
at org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:46)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:70)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:68)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:90)
at org.apache.spark.sql.execution.SparkPlan.$anonfun$execute$1(SparkPlan.scala:180)
at org.apache.spark.sql.execution.SparkPlan.$anonfun$executeQuery$1(SparkPlan.scala:218)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:215)
at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:176)
at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:132)
at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:131)
at org.apache.spark.sql.DataFrameWriter.$anonfun$runCommand$1(DataFrameWriter.scala:989)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$5(SQLExecution.scala:103)
at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:163)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:90)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:772)
at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:64)
at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:989)
at org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:438)
at org.apache.spark.sql.DataFrameWriter.saveInternal(DataFrameWriter.scala:415)
at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:293)
at com.aliyun.odps.spark.examples.hudi.SparkHudiJson$.main(SparkHudiJson.scala:253)
at com.aliyun.odps.spark.examples.hudi.SparkHudiJson.main(SparkHudiJson.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:732)
Caused by: org.apache.hudi.exception.HoodieException: Got runtime exception when hive syncing hudi_dwd_aplus_pub_log_event_1d_tenant1_project47532305069060_v50_extend_odps_2201_12000_bulkoverwrite
at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:168)
at org.apache.hudi.sync.common.util.SyncUtilHelpers.runHoodieMetaSync(SyncUtilHelpers.java:79)
... 36 more
Caused by: org.apache.hudi.hive.HoodieHiveSyncException: failed to create table hudi_dwd_aplus_pub_log_event_1d_tenant1_project47532305069060_v50_extend_odps_2201_12000_bulkoverwrite
at org.apache.hudi.hive.ddl.HMSDDLExecutor.createTable(HMSDDLExecutor.java:140)
at org.apache.hudi.hive.HoodieHiveSyncClient.createTable(HoodieHiveSyncClient.java:235)
at org.apache.hudi.hive.HiveSyncTool.syncFirstTime(HiveSyncTool.java:329)
at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:251)
at org.apache.hudi.hive.HiveSyncTool.doSync(HiveSyncTool.java:177)
at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:165)
... 37 more
Caused by: InvalidObjectException(message:)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$create_table_with_environment_context_result$create_table_with_environment_context_resultStandardScheme.read(ThriftHiveMetastore.java:42216)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$create_table_with_environment_context_result$create_table_with_environment_context_resultStandardScheme.read(ThriftHiveMetastore.java:42193)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$create_table_with_environment_context_result.read(ThriftHiveMetastore.java:42119)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:88)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_create_table_with_environment_context(ThriftHiveMetastore.java:1203)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.create_table_with_environment_context(ThriftHiveMetastore.java:1189)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.create_table_with_environment_context(HiveMetaStoreClient.java:2405)
at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.create_table_with_environment_context(SessionHiveMetaStoreClient.java:93)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createTable(HiveMetaStoreClient.java:752)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createTable(HiveMetaStoreClient.java:740)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:173)
at com.sun.proxy.$Proxy73.createTable(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler.invoke(HiveMetaStoreClient.java:2336)
at com.sun.proxy.$Proxy73.createTable(Unknown Source)
at org.apache.hudi.hive.ddl.HMSDDLExecutor.createTable(HMSDDLExecutor.java:137)
... 42 more
hive metastore log
2023-11-21T13:25:33,821 INFO [pool-7-thread-193] HiveMetaStore.audit: ugi=aaa ip=100.121.110.144 cmd=source:100.121.110.144 create_database: Database(name:, description:automatically created by hoodie, locationUri:null, parameters:null)
2023-11-21T13:25:33,827 WARN [pool-7-thread-193] metastore.ObjectStore: Failed to get database hive., returning NoSuchObjectException
2023-11-21T13:25:33,827 ERROR [pool-7-thread-193] metastore.RetryingHMSHandler: InvalidObjectException(message: is not a valid database name)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_database_core(HiveMetaStore.java:1185)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_database(HiveMetaStore.java:1267)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
at com.sun.proxy.$Proxy32.create_database(Unknown Source)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_database.getResult(ThriftHiveMetastore.java:14298)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_database.getResult(ThriftHiveMetastore.java:14282)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:111)
at org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:107)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899)
at org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:119)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Error when overwrite and synchronize hive metastore with 0.14.0 [hudi]
Posted by "xicm (via GitHub)" <gi...@apache.org>.
xicm commented on issue #10170:
URL: https://github.com/apache/hudi/issues/10170#issuecomment-1840178727
The infer function has been fixed, https://github.com/apache/hudi/pull/9816
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Error when overwrite and synchronize hive metastore with 0.14.0 [hudi]
Posted by "xicm (via GitHub)" <gi...@apache.org>.
xicm commented on issue #10170:
URL: https://github.com/apache/hudi/issues/10170#issuecomment-1840157102
`META_SYNC_DATABASE_NAME` is inferred from `hoodie.database.name`, You can check that the property `hoodie.table.database` in hoodie.properties is empty.
Nice avatar. :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Error when overwrite and synchronize hive metastore with 0.14.0 [hudi]
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #10170:
URL: https://github.com/apache/hudi/issues/10170#issuecomment-1845639675
Thanks @xicm . Can you confirm the fix and close the issue. @zyl891229 . Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Error when overwrite and synchronize hive metastore with 0.14.0 [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope closed issue #10170: [SUPPORT] Error when overwrite and synchronize hive metastore with 0.14.0
URL: https://github.com/apache/hudi/issues/10170
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Error when overwrite and synchronize hive metastore with 0.14.0 [hudi]
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #10170:
URL: https://github.com/apache/hudi/issues/10170#issuecomment-1884541633
@zyl891229 Closing the issue as we have a fix. Please reopen in case you see this again. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org