You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "bigdata-spec (via GitHub)" <gi...@apache.org> on 2023/04/03 10:49:13 UTC
[GitHub] [hudi] bigdata-spec opened a new issue, #8368: Could not sync using the meta sync class org.apache.hudi.hive.HiveSyncTool[SUPPORT]
bigdata-spec opened a new issue, #8368:
URL: https://github.com/apache/hudi/issues/8368
Hi, I have meet a question:
Spark version is 3.1.1,and copy hudi-spark3.1-bundle_2.12-0.11.1.jar to spark/jars.
when I run create table is ok,
```
> create table hudi_cow_nonpcf_tbl (
> uuid int,
> name string,
> price double
> ) using hudi;
```
but when I run
`insert into hudi_cow_nonpcf_tbl select 1, 'a1', 20;`
```
spark-sql> insert into hudi_cow_nonpcf_tbl select 1, 'a1', 20;
23/04/03 18:23:53 WARN DFSPropertiesConfiguration: Cannot find HUDI_CONF_DIR, please set it as the dir of hudi-defaults.conf
23/04/03 18:23:53 WARN DFSPropertiesConfiguration: Properties file file:/etc/hudi/conf/hudi-defaults.conf not found. Ignoring to load props file
23/04/03 18:23:56 WARN MetricsConfig: Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
00:07 WARN: Timeline-server-based markers are not supported for HDFS: base path hdfs://nameservice1/user/hive/warehouse/hudi_cow_nonpcf_tbl. Falling back to direct markers.
00:13 WARN: Timeline-server-based markers are not supported for HDFS: base path hdfs://nameservice1/user/hive/warehouse/hudi_cow_nonpcf_tbl. Falling back to direct markers.
00:27 WARN: Timeline-server-based markers are not supported for HDFS: base path hdfs://nameservice1/user/hive/warehouse/hudi_cow_nonpcf_tbl. Falling back to direct markers.
23/04/03 18:24:19 WARN HiveConf: HiveConf of name hive.vectorized.use.checked.expressions does not exist
23/04/03 18:24:19 WARN HiveConf: HiveConf of name hive.strict.checks.no.partition.filter does not exist
23/04/03 18:24:19 WARN HiveConf: HiveConf of name hive.stats.jdbc.timeout does not exist
23/04/03 18:24:19 WARN HiveConf: HiveConf of name hive.strict.checks.orderby.no.limit does not exist
23/04/03 18:24:19 WARN HiveConf: HiveConf of name hive.vectorized.input.format.excludes does not exist
23/04/03 18:24:19 WARN HiveConf: HiveConf of name hive.stats.retries.wait does not exist
23/04/03 18:24:19 ERROR SparkSQLDriver: Failed in [insert into hudi_cow_nonpcf_tbl select 1, 'a1', 20]
org.apache.hudi.exception.HoodieException: Could not sync using the meta sync class org.apache.hudi.hive.HiveSyncTool
at org.apache.hudi.sync.common.util.SyncUtilHelpers.runHoodieMetaSync(SyncUtilHelpers.java:61)
at org.apache.hudi.HoodieSparkSqlWriter$.$anonfun$metaSync$2(HoodieSparkSqlWriter.scala:623)
at org.apache.hudi.HoodieSparkSqlWriter$.$anonfun$metaSync$2$adapted(HoodieSparkSqlWriter.scala:622)
at scala.collection.mutable.HashSet.foreach(HashSet.scala:79)
at org.apache.hudi.HoodieSparkSqlWriter$.metaSync(HoodieSparkSqlWriter.scala:622)
at org.apache.hudi.HoodieSparkSqlWriter$.commitAndPerformPostOperations(HoodieSparkSqlWriter.scala:681)
at org.apache.hudi.HoodieSparkSqlWriter$.write(HoodieSparkSqlWriter.scala:315)
at org.apache.spark.sql.hudi.command.InsertIntoHoodieTableCommand$.run(InsertIntoHoodieTableCommand.scala:94)
at org.apache.spark.sql.hudi.command.InsertIntoHoodieTableCommand.run(InsertIntoHoodieTableCommand.scala:47)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:70)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:68)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.executeCollect(commands.scala:79)
at org.apache.spark.sql.Dataset.$anonfun$logicalPlan$1(Dataset.scala:228)
at org.apache.spark.sql.Dataset.$anonfun$withAction$1(Dataset.scala:3687)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$5(SQLExecution.scala:103)
at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:163)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:90)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:772)
at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:64)
at org.apache.spark.sql.Dataset.withAction(Dataset.scala:3685)
at org.apache.spark.sql.Dataset.<init>(Dataset.scala:228)
at org.apache.spark.sql.Dataset$.$anonfun$ofRows$2(Dataset.scala:99)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:772)
at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:96)
at org.apache.spark.sql.SparkSession.$anonfun$sql$1(SparkSession.scala:615)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:772)
at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:610)
at org.apache.spark.sql.SQLContext.sql(SQLContext.scala:650)
at org.apache.spark.sql.hive.thriftserver.SparkSQLDriver.run(SparkSQLDriver.scala:67)
at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.processCmd(SparkSQLCLIDriver.scala:381)
at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.$anonfun$processLine$1(SparkSQLCLIDriver.scala:500)
at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.$anonfun$processLine$1$adapted(SparkSQLCLIDriver.scala:494)
at scala.collection.Iterator.foreach(Iterator.scala:941)
at scala.collection.Iterator.foreach$(Iterator.scala:941)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1429)
at scala.collection.IterableLike.foreach(IterableLike.scala:74)
at scala.collection.IterableLike.foreach$(IterableLike.scala:73)
at scala.collection.AbstractIterable.foreach(Iterable.scala:56)
at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.processLine(SparkSQLCLIDriver.scala:494)
at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver$.main(SparkSQLCLIDriver.scala:284)
at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.main(SparkSQLCLIDriver.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:951)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1030)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1039)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: org.apache.hudi.exception.HoodieException: Got runtime exception when hive syncing hudi_cow_nonpcf_tbl
at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:143)
at org.apache.hudi.sync.common.util.SyncUtilHelpers.runHoodieMetaSync(SyncUtilHelpers.java:59)
... 52 more
Caused by: org.apache.hudi.hive.HoodieHiveSyncException: Failed to check if table exists hudi_cow_nonpcf_tbl
at org.apache.hudi.hive.HoodieHiveClient.tableExists(HoodieHiveClient.java:194)
at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:200)
at org.apache.hudi.hive.HiveSyncTool.doSync(HiveSyncTool.java:152)
at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:140)
... 53 more
Caused by: org.apache.thrift.TApplicationException: Invalid method name: 'get_table_req'
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_table_req(ThriftHiveMetastore.java:1567)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_table_req(ThriftHiveMetastore.java:1554)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.tableExists(HiveMetaStoreClient.java:1464)
at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.tableExists(SessionHiveMetaStoreClient.java:266)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:173)
at com.sun.proxy.$Proxy53.tableExists(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler.invoke(HiveMetaStoreClient.java:2336)
at com.sun.proxy.$Proxy53.tableExists(Unknown Source)
at org.apache.hudi.hive.HoodieHiveClient.tableExists(HoodieHiveClient.java:192)
... 56 more
org.apache.hudi.exception.HoodieException: Could not sync using the meta sync class org.apache.hudi.hive.HiveSyncTool
at org.apache.hudi.sync.common.util.SyncUtilHelpers.runHoodieMetaSync(SyncUtilHelpers.java:61)
at org.apache.hudi.HoodieSparkSqlWriter$.$anonfun$metaSync$2(HoodieSparkSqlWriter.scala:623)
at org.apache.hudi.HoodieSparkSqlWriter$.$anonfun$metaSync$2$adapted(HoodieSparkSqlWriter.scala:622)
at scala.collection.mutable.HashSet.foreach(HashSet.scala:79)
at org.apache.hudi.HoodieSparkSqlWriter$.metaSync(HoodieSparkSqlWriter.scala:622)
at org.apache.hudi.HoodieSparkSqlWriter$.commitAndPerformPostOperations(HoodieSparkSqlWriter.scala:681)
at org.apache.hudi.HoodieSparkSqlWriter$.write(HoodieSparkSqlWriter.scala:315)
at org.apache.spark.sql.hudi.command.InsertIntoHoodieTableCommand$.run(InsertIntoHoodieTableCommand.scala:94)
at org.apache.spark.sql.hudi.command.InsertIntoHoodieTableCommand.run(InsertIntoHoodieTableCommand.scala:47)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:70)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:68)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.executeCollect(commands.scala:79)
at org.apache.spark.sql.Dataset.$anonfun$logicalPlan$1(Dataset.scala:228)
at org.apache.spark.sql.Dataset.$anonfun$withAction$1(Dataset.scala:3687)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$5(SQLExecution.scala:103)
at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:163)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:90)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:772)
at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:64)
at org.apache.spark.sql.Dataset.withAction(Dataset.scala:3685)
at org.apache.spark.sql.Dataset.<init>(Dataset.scala:228)
at org.apache.spark.sql.Dataset$.$anonfun$ofRows$2(Dataset.scala:99)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:772)
at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:96)
at org.apache.spark.sql.SparkSession.$anonfun$sql$1(SparkSession.scala:615)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:772)
at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:610)
at org.apache.spark.sql.SQLContext.sql(SQLContext.scala:650)
at org.apache.spark.sql.hive.thriftserver.SparkSQLDriver.run(SparkSQLDriver.scala:67)
at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.processCmd(SparkSQLCLIDriver.scala:381)
at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.$anonfun$processLine$1(SparkSQLCLIDriver.scala:500)
at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.$anonfun$processLine$1$adapted(SparkSQLCLIDriver.scala:494)
at scala.collection.Iterator.foreach(Iterator.scala:941)
at scala.collection.Iterator.foreach$(Iterator.scala:941)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1429)
at scala.collection.IterableLike.foreach(IterableLike.scala:74)
at scala.collection.IterableLike.foreach$(IterableLike.scala:73)
at scala.collection.AbstractIterable.foreach(Iterable.scala:56)
at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.processLine(SparkSQLCLIDriver.scala:494)
at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver$.main(SparkSQLCLIDriver.scala:284)
at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.main(SparkSQLCLIDriver.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:951)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1030)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1039)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: org.apache.hudi.exception.HoodieException: Got runtime exception when hive syncing hudi_cow_nonpcf_tbl
at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:143)
at org.apache.hudi.sync.common.util.SyncUtilHelpers.runHoodieMetaSync(SyncUtilHelpers.java:59)
... 52 more
Caused by: org.apache.hudi.hive.HoodieHiveSyncException: Failed to check if table exists hudi_cow_nonpcf_tbl
at org.apache.hudi.hive.HoodieHiveClient.tableExists(HoodieHiveClient.java:194)
at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:200)
at org.apache.hudi.hive.HiveSyncTool.doSync(HiveSyncTool.java:152)
at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:140)
... 53 more
Caused by: org.apache.thrift.TApplicationException: Invalid method name: 'get_table_req'
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_table_req(ThriftHiveMetastore.java:1567)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_table_req(ThriftHiveMetastore.java:1554)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.tableExists(HiveMetaStoreClient.java:1464)
at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.tableExists(SessionHiveMetaStoreClient.java:266)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:173)
at com.sun.proxy.$Proxy53.tableExists(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler.invoke(HiveMetaStoreClient.java:2336)
at com.sun.proxy.$Proxy53.tableExists(Unknown Source)
at org.apache.hudi.hive.HoodieHiveClient.tableExists(HoodieHiveClient.java:192)
... 56 more
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] bigdata-spec commented on issue #8368: Could not sync using the meta sync class org.apache.hudi.hive.HiveSyncTool[SUPPORT]
Posted by "bigdata-spec (via GitHub)" <gi...@apache.org>.
bigdata-spec commented on issue #8368:
URL: https://github.com/apache/hudi/issues/8368#issuecomment-1495349137
> `spark.sql.hive.metastore.version` is not supported in hudi. hudi not compatible with all hive metastore version like Spark.
So，What can I do deal with this error?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] bigdata-spec commented on issue #8368: Could not sync using the meta sync class org.apache.hudi.hive.HiveSyncTool[SUPPORT]
Posted by "bigdata-spec (via GitHub)" <gi...@apache.org>.
bigdata-spec commented on issue #8368:
URL: https://github.com/apache/hudi/issues/8368#issuecomment-1501415295
> `spark.sql.hive.metastore.version` is not supported in hudi. hudi not compatible with all hive metastore version like Spark.
@huangxiaopingRD @ad1happy2go
hudi is 0.11.1 ，just I want to use spark3+， I find spark3.2.3 is work fine in hive and hudi table without set
```
spark.sql.hive.metastore.version=2.1.1
spark.sql.hive.metastore.jars=/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/hive/lib/*
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] bigdata-spec commented on issue #8368: Could not sync using the meta sync class org.apache.hudi.hive.HiveSyncTool[SUPPORT]
Posted by "bigdata-spec (via GitHub)" <gi...@apache.org>.
bigdata-spec commented on issue #8368:
URL: https://github.com/apache/hudi/issues/8368#issuecomment-1495208401
@huangxiaopingRD @ad1happy2go Thank you for your kindness.
HMS version is 2.1.1-cdh6.3.2.
our environment is cdh6.3.2, we want to use **Apache Spark3.1.1** to replace **2.4.0-cdh6.3.2 for spark**
so I use command：
`./dev/make-distribution.sh --name 3.0.0-cdh6.3.2 --tgz -Pyarn -Phive-thriftserver -Dhadoop.version=3.0.0-cdh6.3.2 `
and I get **spark-3.1.1-bin-3.0.0-cdh6.3.2.tgz**
spark-defaults.conf I set
```
spark.sql.hive.metastore.version=2.1.1
spark.sql.hive.metastore.jars=/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/hive/lib/*
```
it work well for common hive table. but hudi table can create ,but can't insert.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] cnfree0355 commented on issue #8368: Could not sync using the meta sync class org.apache.hudi.hive.HiveSyncTool[SUPPORT]
Posted by "cnfree0355 (via GitHub)" <gi...@apache.org>.
cnfree0355 commented on issue #8368:
URL: https://github.com/apache/hudi/issues/8368#issuecomment-1578039056
> > `spark.sql.hive.metastore.version` is not supported in hudi. hudi not compatible with all hive metastore version like Spark.
> > @huangxiaopingRD @ad1happy2go
> > hudi is 0.11.1 ，just I want to use spark3+， I find spark3.2.3 is work fine in hive and hudi table without set
>
> ```
> spark.sql.hive.metastore.version=2.1.1
> spark.sql.hive.metastore.jars=/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/hive/lib/*
> ```
## Hello, I have a same problem with you . when I use the sql as below
```sql
create table h2 using hudi
options (type = 'cow', primaryKey = 'id')
partitioned by (dt)
as
select 1 as id, 'a1' as name, 10 as price, 1000 as dt;
```
And the error is
```log
org.apache.thrift.TApplicationException: Required field 'client_protocol' is unset! Struct:TOpenSessionReq(client_protocol:null, configuration:{use:database=default})
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79)
at org.apache.hive.service.rpc.thrift.TCLIService$Client.recv_OpenSession(TCLIService.java:176)
at org.apache.hive.service.rpc.thrift.TCLIService$Client.OpenSession(TCLIService.java:163)
at org.apache.hive.jdbc.HiveConnection.openSession(HiveConnection.java:578)
at org.apache.hive.jdbc.HiveConnection.<init>(HiveConnection.java:188)
at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:107)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:247)
at org.apache.hudi.hive.ddl.JDBCExecutor.createHiveConnection(JDBCExecutor.java:104)
at org.apache.hudi.hive.ddl.JDBCExecutor.<init>(JDBCExecutor.java:59)
at org.apache.hudi.hive.HoodieHiveSyncClient.<init>(HoodieHiveSyncClient.java:93)
at org.apache.hudi.hive.HiveSyncTool.initSyncClient(HiveSyncTool.java:119)
at org.apache.hudi.hive.HiveSyncTool.<init>(HiveSyncTool.java:113)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
.......
... 74 more
Caused by: java.sql.SQLException: Could not establish connection to jdbc:hive2://hadoop7809.xx.com:10000: Required field 'client_protocol' is unset! Struct:TOpenSessionReq(client_protocol:null, configuration:{use:database=default})
at org.apache.hive.jdbc.HiveConnection.openSession(HiveConnection.java:590)
at org.apache.hive.jdbc.HiveConnection.<init>(HiveConnection.java:188)
... 78 more
Caused by: org.apache.thrift.TApplicationException: Required field 'client_protocol' is unset! Struct:TOpenSessionReq(client_protocol:null, configuration:{use:database=default})
```
My enviroment is
```
- hudi 0.13
- spark 3.2.2
- hive hive2.1.1-cdh6.3.2
```
By the way ,**spark.sql.hive.metastore.version** can't work for me
As you mentioned above you solved it ,please help me what is the reason for this problem of mine .tanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] ad1happy2go commented on issue #8368: Could not sync using the meta sync class org.apache.hudi.hive.HiveSyncTool[SUPPORT]
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #8368:
URL: https://github.com/apache/hudi/issues/8368#issuecomment-1494265653
@bigdata-spec It worked fine for me for spark 3.1 version.
may be mismatch in Hadoop/hive version or your setup. Can you please recheck and also the results I shared below which worked for me.
<img width="1724" alt="image" src="https://user-images.githubusercontent.com/63430370/229514103-fc647b4c-23e7-4ca5-91c4-5b0a0c2bf87d.png">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] cnfree0355 commented on issue #8368: Could not sync using the meta sync class org.apache.hudi.hive.HiveSyncTool[SUPPORT]
Posted by "cnfree0355 (via GitHub)" <gi...@apache.org>.
cnfree0355 commented on issue #8368:
URL: https://github.com/apache/hudi/issues/8368#issuecomment-1609141500
>
Thank you so much! My problem is hive version mismatch, I guess spark's version doesn't match hudi built-in version. So I updated my hive version to 2.3.9 and it's working now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] huangxiaopingRD commented on issue #8368: Could not sync using the meta sync class org.apache.hudi.hive.HiveSyncTool[SUPPORT]
Posted by "huangxiaopingRD (via GitHub)" <gi...@apache.org>.
huangxiaopingRD commented on issue #8368:
URL: https://github.com/apache/hudi/issues/8368#issuecomment-1494364520
What's the version of your HMS? It should be cause by the version of HMS is too lower.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] bigdata-spec commented on issue #8368: Could not sync using the meta sync class org.apache.hudi.hive.HiveSyncTool[SUPPORT]
Posted by "bigdata-spec (via GitHub)" <gi...@apache.org>.
bigdata-spec commented on issue #8368:
URL: https://github.com/apache/hudi/issues/8368#issuecomment-1578360228
@cnfree0355 can you try spark 3.3.2？ I use this version work well temporarily.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope closed issue #8368: Could not sync using the meta sync class org.apache.hudi.hive.HiveSyncTool[SUPPORT]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope closed issue #8368: Could not sync using the meta sync class org.apache.hudi.hive.HiveSyncTool[SUPPORT]
URL: https://github.com/apache/hudi/issues/8368
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] bigdata-spec commented on issue #8368: Could not sync using the meta sync class org.apache.hudi.hive.HiveSyncTool[SUPPORT]
Posted by "bigdata-spec (via GitHub)" <gi...@apache.org>.
bigdata-spec commented on issue #8368:
URL: https://github.com/apache/hudi/issues/8368#issuecomment-1497325475
@ad1happy2go @huangxiaopingRD
If I want to use hudi via spark3.1，Do I have to use hive version 2.3 or later ？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] huangxiaopingRD commented on issue #8368: Could not sync using the meta sync class org.apache.hudi.hive.HiveSyncTool[SUPPORT]
Posted by "huangxiaopingRD (via GitHub)" <gi...@apache.org>.
huangxiaopingRD commented on issue #8368:
URL: https://github.com/apache/hudi/issues/8368#issuecomment-1496107356
Our HMS version is 2.3.8 and Spark is 3.1. It works fine with hudi.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] cnfree0355 commented on issue #8368: Could not sync using the meta sync class org.apache.hudi.hive.HiveSyncTool[SUPPORT]
Posted by "cnfree0355 (via GitHub)" <gi...@apache.org>.
cnfree0355 commented on issue #8368:
URL: https://github.com/apache/hudi/issues/8368#issuecomment-1609142807
> @cnfree0355 can you try spark 3.3.2？ I use this version work well temporarily. but you error is not same problem
Thank you so much! My problem is hive version mismatch, I guess spark's version doesn't match hudi built-in version. So I updated my hive version to 2.3.9 and it's working now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] huangxiaopingRD commented on issue #8368: Could not sync using the meta sync class org.apache.hudi.hive.HiveSyncTool[SUPPORT]
Posted by "huangxiaopingRD (via GitHub)" <gi...@apache.org>.
huangxiaopingRD commented on issue #8368:
URL: https://github.com/apache/hudi/issues/8368#issuecomment-1495252052
`spark.sql.hive.metastore.version` is not supported in hudi. hudi not compatible with all hive metastore version like Spark.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] ad1happy2go commented on issue #8368: Could not sync using the meta sync class org.apache.hudi.hive.HiveSyncTool[SUPPORT]
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #8368:
URL: https://github.com/apache/hudi/issues/8368#issuecomment-1495363471
@bigdata-spec I guess at this point you have to use the supported HMS version only. @huangxiaopingRD can comment more.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] bigdata-spec commented on issue #8368: Could not sync using the meta sync class org.apache.hudi.hive.HiveSyncTool[SUPPORT]
Posted by "bigdata-spec (via GitHub)" <gi...@apache.org>.
bigdata-spec commented on issue #8368:
URL: https://github.com/apache/hudi/issues/8368#issuecomment-1498982067
> @huangxiaopingRD @ad1happy2go Thank you for your kindness. HMS version is 2.1.1-cdh6.3.2. our environment is cdh6.3.2, we want to use **Apache Spark3.1.1** to replace **2.4.0-cdh6.3.2 for spark** so I use command：
>
> `./dev/make-distribution.sh --name 3.0.0-cdh6.3.2 --tgz -Pyarn -Phive-thriftserver -Dhadoop.version=3.0.0-cdh6.3.2 ` and I get **spark-3.1.1-bin-3.0.0-cdh6.3.2.tgz**
>
> spark-defaults.conf I set
>
> ```
> spark.sql.hive.metastore.version=2.1.1
> spark.sql.hive.metastore.jars=/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/hive/lib/*
> ```
>
> it work well for common hive table. but hudi table can create ,but can't insert.
@yihua Hello,Can you help me?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] bigdata-spec commented on issue #8368: Could not sync using the meta sync class org.apache.hudi.hive.HiveSyncTool[SUPPORT]
Posted by "bigdata-spec (via GitHub)" <gi...@apache.org>.
bigdata-spec commented on issue #8368:
URL: https://github.com/apache/hudi/issues/8368#issuecomment-1495385255
> @bigdata-spec我想此时您必须只使用受支持的 HMS 版本。@huangxiaopingRD可以评论更多。
@ad1happy2go what HMS version means？ is fit hudi or spark？ hudi support HMS version for 2.1.1-cdh6.3.2.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org