You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2021/04/19 09:19:28 UTC
[GitHub] [hudi] wk888 edited a comment on issue #2834: [SUPPORT] Help~~~org.apache.hudi.exception.TableNotFoundException
wk888 edited a comment on issue #2834:
URL: https://github.com/apache/hudi/issues/2834#issuecomment-822313322
> @wk888 Did you sync table information of hoodie to HMS?
@yanghua yes .tables hudu_test_1_rt and hudu_test_1_ro is create success by run_sync_tool.sh
./hudi-hive-sync/run_sync_tool.sh \
> --jdbc-url jdbc:hive2://11.11.11.11:30000 \
> --user hive \
> --pass hive \
> --partitioned-by dt \
> --partition-value-extractor org.apache.hudi.hive.MultiPartKeysValueExtractor \
> --base-path ofs://qqqqqqqqq/user/wangkai/test_hudi_1 \
> --database test \
> --table hudu_test_1
-----------------------------------------logs-------------------------
2021-04-19 16:26:18,871 INFO [main] hive.HoodieHiveClient (HoodieHiveClient.java:updateHiveSQL(366)) - Executing SQL create database if not exists test
2021-04-19 16:26:18,995 WARN [main] hive.HiveSyncTool (HiveSyncTool.java:syncHoodieTable(140)) - Unable to create database
org.apache.hudi.hive.HoodieHiveSyncException: Failed in executing SQL create database if not exists test
at org.apache.hudi.hive.HoodieHiveClient.updateHiveSQL(HoodieHiveClient.java:369)
at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:137)
at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:112)
at org.apache.hudi.hive.HiveSyncTool.main(HiveSyncTool.java:249)
Caused by: org.apache.hive.service.cli.HiveSQLException: Error while compiling statement: No privilege 'Create' found for outputs { }
at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:267)
at org.apache.hive.jdbc.Utils.verifySuccessWithInfo(Utils.java:253)
at org.apache.hive.jdbc.HiveStatement.runAsyncOnServer(HiveStatement.java:313)
at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:253)
at org.apache.hudi.hive.HoodieHiveClient.updateHiveSQL(HoodieHiveClient.java:367)
... 3 more
Caused by: org.apache.hive.service.cli.HiveSQLException: Error while compiling statement: No privilege 'Create' found for outputs { }
at org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:380)
at org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:206)
at org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:290)
at org.apache.hive.service.cli.operation.Operation.run(Operation.java:320)
at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:530)
at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:517)
at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78)
at org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36)
at org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1924)
at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59)
at com.sun.proxy.$Proxy30.executeStatementAsync(Unknown Source)
at org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:310)
at org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:535)
at org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1437)
at org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1422)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2021-04-19 16:26:19,050 INFO [main] fs.CHDFSInputStream (CHDFSInputStream.java:close(73)) - close input stream for file: /user/wangkai/test_hudi_1/.hoodie/20210419160955.deltacommit, inode: 336302421, fd: 0, time-range-ms:[1618820779011 - 1618820779050], used-time-ms:39
2021-04-19 16:26:19,210 INFO [main] hive.HiveSyncTool (HiveSyncTool.java:syncSchema(182)) - Hive table hudu_test_1_ro is not found. Creating it
2021-04-19 16:26:19,215 INFO [main] hive.HoodieHiveClient (HoodieHiveClient.java:createTable(262)) - Creating table with CREATE EXTERNAL TABLE IF NOT EXISTS `test`.`hudu_test_1_ro`( `_hoodie_commit_time` string, `_hoodie_commit_seqno` string, `_hoodie_record_key` string, `_hoodie_partition_path` string, `_hoodie_file_name` string, `uuid` string, `name` string, `ts` bigint) PARTITIONED BY (`dt` string) ROW FORMAT SERDE 'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe' STORED AS INPUTFORMAT 'org.apache.hudi.hadoop.HoodieParquetInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat' LOCATION 'ofs://qqqqqqq/user/wangkai/test_hudi_1'
2021-04-19 16:26:19,215 INFO [main] hive.HoodieHiveClient (HoodieHiveClient.java:updateHiveSQL(366)) - Executing SQL CREATE EXTERNAL TABLE IF NOT EXISTS `test`.`hudu_test_1_ro`( `_hoodie_commit_time` string, `_hoodie_commit_seqno` string, `_hoodie_record_key` string, `_hoodie_partition_path` string, `_hoodie_file_name` string, `uuid` string, `name` string, `ts` bigint) PARTITIONED BY (`dt` string) ROW FORMAT SERDE 'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe' STORED AS INPUTFORMAT 'org.apache.hudi.hadoop.HoodieParquetInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat' LOCATION 'ofs://qqqqqqq/user/wangkai/test_hudi_1'
2021-04-19 16:26:19,864 INFO [main] hive.HiveSyncTool (HiveSyncTool.java:syncHoodieTable(156)) - Schema sync complete. Syncing partitions for hudu_test_1_ro
2021-04-19 16:26:19,864 INFO [main] hive.HiveSyncTool (HiveSyncTool.java:syncHoodieTable(162)) - Last commit time synced was found to be null
2021-04-19 16:26:19,864 INFO [main] common.AbstractSyncHoodieClient (AbstractSyncHoodieClient.java:getPartitionsWrittenToSince(128)) - Last commit time synced is not known, listing all partitions in ofs://qqqqqqq/user/wangkai/test_hudi_1,FS :com.qcloud.chdfs.fs.CHDFSHadoopFileSystemAdapter@3adcc812
2021-04-19 16:26:19,900 INFO [main] hive.HiveSyncTool (HiveSyncTool.java:syncHoodieTable(164)) - Storage partitions scan complete. Found 2
2021-04-19 16:26:19,921 INFO [main] hive.HiveSyncTool (HiveSyncTool.java:syncPartitions(223)) - New Partitions [20210415, 20210419]
2021-04-19 16:26:19,921 INFO [main] hive.HoodieHiveClient (HoodieHiveClient.java:addPartitionsToTable(120)) - Adding partitions 2 to table hudu_test_1_ro
2021-04-19 16:26:19,922 INFO [main] hive.HoodieHiveClient (HoodieHiveClient.java:updateHiveSQL(366)) - Executing SQL ALTER TABLE `test`.`hudu_test_1_ro` ADD IF NOT EXISTS PARTITION (`dt`='20210415') LOCATION 'ofs://qqqqqqq/user/wangkai/test_hudi_1/20210415' PARTITION (`dt`='20210419') LOCATION 'ofs://qqqqqqq/user/wangkai/test_hudi_1/20210419'
2021-04-19 16:26:20,285 INFO [main] hive.HiveSyncTool (HiveSyncTool.java:syncPartitions(226)) - Changed Partitions []
2021-04-19 16:26:20,285 INFO [main] hive.HoodieHiveClient (HoodieHiveClient.java:updatePartitionsToTable(131)) - No partitions to change for hudu_test_1_ro
2021-04-19 16:26:20,394 INFO [main] hive.HiveSyncTool (HiveSyncTool.java:syncHoodieTable(169)) - Sync complete for hudu_test_1_ro
2021-04-19 16:26:20,394 INFO [main] hive.HiveSyncTool (HiveSyncTool.java:syncHoodieTable(131)) - Trying to sync hoodie table hudu_test_1_rt with base path ofs://qqqqqqq/user/wangkai/test_hudi_1 of type MERGE_ON_READ
2021-04-19 16:26:20,394 INFO [main] hive.HoodieHiveClient (HoodieHiveClient.java:updateHiveSQL(366)) - Executing SQL create database if not exists test
2021-04-19 16:26:20,472 WARN [main] hive.HiveSyncTool (HiveSyncTool.java:syncHoodieTable(140)) - Unable to create database
org.apache.hudi.hive.HoodieHiveSyncException: Failed in executing SQL create database if not exists test
at org.apache.hudi.hive.HoodieHiveClient.updateHiveSQL(HoodieHiveClient.java:369)
at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:137)
at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:114)
at org.apache.hudi.hive.HiveSyncTool.main(HiveSyncTool.java:249)
Caused by: org.apache.hive.service.cli.HiveSQLException: Error while compiling statement: No privilege 'Create' found for outputs { }
at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:267)
at org.apache.hive.jdbc.Utils.verifySuccessWithInfo(Utils.java:253)
at org.apache.hive.jdbc.HiveStatement.runAsyncOnServer(HiveStatement.java:313)
at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:253)
at org.apache.hudi.hive.HoodieHiveClient.updateHiveSQL(HoodieHiveClient.java:367)
... 3 more
Caused by: org.apache.hive.service.cli.HiveSQLException: Error while compiling statement: No privilege 'Create' found for outputs { }
at org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:380)
at org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:206)
at org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:290)
at org.apache.hive.service.cli.operation.Operation.run(Operation.java:320)
at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:530)
at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:517)
at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78)
at org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36)
at org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1924)
at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59)
at com.sun.proxy.$Proxy30.executeStatementAsync(Unknown Source)
at org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:310)
at org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:535)
at org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1437)
at org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1422)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2021-04-19 16:26:20,516 INFO [main] fs.CHDFSInputStream (CHDFSInputStream.java:close(73)) - close input stream for file: /user/wangkai/test_hudi_1/.hoodie/20210419160955.deltacommit, inode: 336302421, fd: 0, time-range-ms:[1618820780481 - 1618820780516], used-time-ms:35
2021-04-19 16:26:20,522 INFO [main] hive.HiveSyncTool (HiveSyncTool.java:syncSchema(182)) - Hive table hudu_test_1_rt is not found. Creating it
2021-04-19 16:26:20,522 INFO [main] hive.HoodieHiveClient (HoodieHiveClient.java:createTable(262)) - Creating table with CREATE EXTERNAL TABLE IF NOT EXISTS `test`.`hudu_test_1_rt`( `_hoodie_commit_time` string, `_hoodie_commit_seqno` string, `_hoodie_record_key` string, `_hoodie_partition_path` string, `_hoodie_file_name` string, `uuid` string, `name` string, `ts` bigint) PARTITIONED BY (`dt` string) ROW FORMAT SERDE 'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe' STORED AS INPUTFORMAT 'org.apache.hudi.hadoop.realtime.HoodieParquetRealtimeInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat' LOCATION 'ofs://qqqqqqq/user/wangkai/test_hudi_1'
2021-04-19 16:26:20,522 INFO [main] hive.HoodieHiveClient (HoodieHiveClient.java:updateHiveSQL(366)) - Executing SQL CREATE EXTERNAL TABLE IF NOT EXISTS `test`.`hudu_test_1_rt`( `_hoodie_commit_time` string, `_hoodie_commit_seqno` string, `_hoodie_record_key` string, `_hoodie_partition_path` string, `_hoodie_file_name` string, `uuid` string, `name` string, `ts` bigint) PARTITIONED BY (`dt` string) ROW FORMAT SERDE 'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe' STORED AS INPUTFORMAT 'org.apache.hudi.hadoop.realtime.HoodieParquetRealtimeInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat' LOCATION 'ofs://qqqqqqq/user/wangkai/test_hudi_1'
2021-04-19 16:26:21,154 INFO [main] hive.HiveSyncTool (HiveSyncTool.java:syncHoodieTable(156)) - Schema sync complete. Syncing partitions for hudu_test_1_rt
2021-04-19 16:26:21,154 INFO [main] hive.HiveSyncTool (HiveSyncTool.java:syncHoodieTable(162)) - Last commit time synced was found to be null
2021-04-19 16:26:21,154 INFO [main] common.AbstractSyncHoodieClient (AbstractSyncHoodieClient.java:getPartitionsWrittenToSince(128)) - Last commit time synced is not known, listing all partitions in ofs://qqqqqqq/user/wangkai/test_hudi_1,FS :com.qcloud.chdfs.fs.CHDFSHadoopFileSystemAdapter@3adcc812
2021-04-19 16:26:21,169 INFO [main] hive.HiveSyncTool (HiveSyncTool.java:syncHoodieTable(164)) - Storage partitions scan complete. Found 2
2021-04-19 16:26:21,187 INFO [main] hive.HiveSyncTool (HiveSyncTool.java:syncPartitions(223)) - New Partitions [20210415, 20210419]
2021-04-19 16:26:21,187 INFO [main] hive.HoodieHiveClient (HoodieHiveClient.java:addPartitionsToTable(120)) - Adding partitions 2 to table hudu_test_1_rt
2021-04-19 16:26:21,187 INFO [main] hive.HoodieHiveClient (HoodieHiveClient.java:updateHiveSQL(366)) - Executing SQL ALTER TABLE `test`.`hudu_test_1_rt` ADD IF NOT EXISTS PARTITION (`dt`='20210415') LOCATION 'ofs://qqqqqqq/user/wangkai/test_hudi_1/20210415' PARTITION (`dt`='20210419') LOCATION 'ofs://qqqqqqq/user/wangkai/test_hudi_1/20210419'
2021-04-19 16:26:21,574 INFO [main] hive.HiveSyncTool (HiveSyncTool.java:syncPartitions(226)) - Changed Partitions []
2021-04-19 16:26:21,574 INFO [main] hive.HoodieHiveClient (HoodieHiveClient.java:updatePartitionsToTable(131)) - No partitions to change for hudu_test_1_rt
2021-04-19 16:26:21,648 INFO [main] hive.HiveSyncTool (HiveSyncTool.java:syncHoodieTable(169)) - Sync complete for hudu_test_1_rt
2021-04-19 16:26:21,680 INFO [main] hive.metastore (HiveMetaStoreClient.java:close(564)) - Closed a connection to metastore, current connections: 0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org