You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "Trevorzhang (Jira)" <ji...@apache.org> on 2021/01/14 12:26:00 UTC
[jira] [Commented] (HUDI-1528) hudi-sync-tool error
[ https://issues.apache.org/jira/browse/HUDI-1528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17264859#comment-17264859 ]
Trevorzhang commented on HUDI-1528:
-----------------------------------
{code:java}
//代码占位符[lingqu@xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01 jars]$ sh /data/zyx/hudi-hive-sync/run_sync_tool.sh --base-path hdfs://xx.3.98.183:8020/data_lake/jqxx/data/xx_biz_operation.t_city --database datalake_taxi --table xx_biz_operation_t_city2 --jdbc-url jdbc:hive2://xx-test-cq-ecs-dtpbu-datalake-cdh-edge-01:xx000 --partition-value-extractor org.apache.hudi.hive.SlashEncodedDayPartitionValueExtractor --user lingqu --pass lingqu@bigdata --partitioned-by ds[lingqu@xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01 jars]$ sh /data/zyx/hudi-hive-sync/run_sync_tool.sh --base-path hdfs://xx.3.98.183:8020/data_lake/jqxx/data/xx_biz_operation.t_city --database datalake_taxi --table xx_biz_operation_t_city2 --jdbc-url jdbc:hive2://xx-test-cq-ecs-dtpbu-datalake-cdh-edge-01:xx000 --partition-value-extractor org.apache.hudi.hive.SlashEncodedDayPartitionValueExtractor --user lingqu --pass lingqu@bigdata --partitioned-by dsRunning Command : java -cp /opt/cloudera/parcels/CDH/lib/hive/lib/hive-metastore-2.1.1-cdh6.3.0.jar::/opt/cloudera/parcels/CDH/lib/hive/lib/hive-service-2.1.1-cdh6.3.0.jar::/opt/cloudera/parcels/CDH/lib/hive/lib/hive-exec-2.1.1-cdh6.3.0.jar::/opt/cloudera/parcels/CDH/lib/hive/lib/hive-jdbc-2.1.1-cdh6.3.0.jar::/opt/cloudera/parcels/CDH/lib/hive/lib/jackson-annotations-2.9.9.jar:/opt/cloudera/parcels/CDH/lib/hive/lib/jackson-core-2.9.9.jar:/opt/cloudera/parcels/CDH/lib/hive/lib/jackson-core-asl-1.9.13.jar:/opt/cloudera/parcels/CDH/lib/hive/lib/jackson-databind-2.9.9.jar:/opt/cloudera/parcels/CDH/lib/hive/lib/jackson-jaxrs-1.9.13.jar:/opt/cloudera/parcels/CDH/lib/hive/lib/jackson-mapper-asl-1.9.13-cloudera.1.jar:/opt/cloudera/parcels/CDH/lib/hive/lib/jackson-xc-1.9.13.jar::/opt/cloudera/parcels/CDH/lib/hadoop/client/*:/data/zyx/hudi-hive-sync/jars/*:/etc/hadoop/conf:/data/zyx/hudi-hive-sync/jars/hudi-hive-sync-bundle-0.6.0.jar org.apache.hudi.hive.HiveSyncTool --base-path hdfs://xx.3.98.183:8020/data_lake/jqxx/data/xx_biz_operation.t_city --database datalake_taxi --table xx_biz_operation_t_city2 --jdbc-url jdbc:hive2://xx-test-cq-ecs-dtpbu-datalake-cdh-edge-01:xx000 --partition-value-extractor org.apache.hudi.hive.SlashEncodedDayPartitionValueExtractor --user lingqu --pass lingqu@bigdata --partitioned-by dsSLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".SLF4J: Defaulting to no-operation (NOP) logger implementationSLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.21/01/14 20:20:58 INFO fs.FSUtils: Hadoop Configuration: fs.defaultFS: [hdfs://nameservice1], Config:[Configuration: core-default.xml, core-site.xml, mapred-default.xml, mapred-site.xml, yarn-default.xml, yarn-site.xml, hdfs-default.xml, hdfs-site.xml], FileSystem: [DFS[DFSClient[clientName=DFSClient_NONMAPREDUCE_1717621797_1, ugi=lingqu (auth:SIMPLE)]]]21/01/14 20:20:59 INFO table.HoodieTableMetaClient: Loading HoodieTableMetaClient from hdfs://xx.3.98.183:8020/data_lake/jqxx/data/xx_biz_operation.t_city21/01/14 20:20:59 INFO fs.FSUtils: Hadoop Configuration: fs.defaultFS: [hdfs://nameservice1], Config:[Configuration: core-default.xml, core-site.xml, mapred-default.xml, mapred-site.xml, yarn-default.xml, yarn-site.xml, hdfs-default.xml, hdfs-site.xml], FileSystem: [DFS[DFSClient[clientName=DFSClient_NONMAPREDUCE_1717621797_1, ugi=lingqu (auth:SIMPLE)]]]21/01/14 20:20:59 INFO table.HoodieTableConfig: Loading table properties from hdfs://xx.3.98.183:8020/data_lake/jqxx/data/xx_biz_operation.t_city/.hoodie/hoodie.properties21/01/14 20:20:59 INFO table.HoodieTableMetaClient: Finished Loading Table of type COPY_ON_WRITE(version=1, baseFileFormat=PARQUET) from hdfs://xx.3.98.183:8020/data_lake/jqxx/data/xx_biz_operation.t_city21/01/14 20:20:59 INFO table.HoodieTableMetaClient: Loading Active commit timeline for hdfs://xx.3.98.183:8020/data_lake/jqxx/data/xx_biz_operation.t_city21/01/14 20:20:59 INFO timeline.HoodieActiveTimeline: Loaded instants [[20201xx6164936__commit__COMPLETED]]21/01/14 20:20:59 INFO hive.HoodieHiveClient: Creating hive connection jdbc:hive2://xx-test-cq-ecs-dtpbu-datalake-cdh-edge-01:xx00021/01/14 20:20:59 INFO hive.HoodieHiveClient: Successfully established Hive connection to jdbc:hive2://xx-test-cq-ecs-dtpbu-datalake-cdh-edge-01:xx00021/01/14 20:21:00 INFO hive.HiveSyncTool: Trying to sync hoodie table xx_biz_operation_t_city2 with base path hdfs://xx.3.98.183:8020/data_lake/jqxx/data/xx_biz_operation.t_city of type COPY_ON_WRITE21/01/14 20:21:00 INFO hive.HoodieHiveClient: Executing SQL create database if not exists datalake_taxi21/01/14 20:21:00 INFO hive.HiveSyncTool: Hive table xx_biz_operation_t_city2 is not found. Creating it21/01/14 20:21:00 INFO hive.HoodieHiveClient: Creating table with CREATE EXTERNAL TABLE IF NOT EXISTS `datalake_taxi`.`xx_biz_operation_t_city2`( `_hoodie_commit_time` string, `_hoodie_commit_seqno` string, `_hoodie_record_key` string, `_hoodie_partition_path` string, `_hoodie_file_name` string, `ums_id_` string, `ums_ts_` string, `ums_op_` string, `uuid` string, `org_code` string, `org_name` string, `parent_code` string, `center` string, `flag` int, `create_time` string, `update_time` string) PARTITIONED BY (`ds` string) ROW FORMAT SERDE 'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe' STORED AS INPUTFORMAT 'org.apache.hudi.hadoop.HoodieParquetInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat' LOCATION 'hdfs://xx.3.98.183:8020/data_lake/jqxx/data/xx_biz_operation.t_city'21/01/14 20:21:00 INFO hive.HoodieHiveClient: Executing SQL CREATE EXTERNAL TABLE IF NOT EXISTS `datalake_taxi`.`xx_biz_operation_t_city2`( `_hoodie_commit_time` string, `_hoodie_commit_seqno` string, `_hoodie_record_key` string, `_hoodie_partition_path` string, `_hoodie_file_name` string, `ums_id_` string, `ums_ts_` string, `ums_op_` string, `uuid` string, `org_code` string, `org_name` string, `parent_code` string, `center` string, `flag` int, `create_time` string, `update_time` string) PARTITIONED BY (`ds` string) ROW FORMAT SERDE 'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe' STORED AS INPUTFORMAT 'org.apache.hudi.hadoop.HoodieParquetInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat' LOCATION 'hdfs://xx.3.98.183:8020/data_lake/jqxx/data/xx_biz_operation.t_city'21/01/14 20:21:00 INFO hive.HiveSyncTool: Schema sync complete. Syncing partitions for xx_biz_operation_t_city221/01/14 20:21:00 INFO hive.HiveSyncTool: Last commit time synced was found to be null21/01/14 20:21:00 INFO common.AbstractSyncHoodieClient: Last commit time synced is not known, listing all partitions in hdfs://xx.3.98.183:8020/data_lake/jqxx/data/xx_biz_operation.t_city,FS :DFS[DFSClient[clientName=DFSClient_NONMAPREDUCE_1717621797_1, ugi=lingqu (auth:SIMPLE)]]21/01/14 20:21:00 INFO hive.HiveSyncTool: Storage partitions scan complete. Found 121/01/14 20:21:00 ERROR hive.HiveSyncTool: Got runtime exception when hive syncingorg.apache.hudi.hive.HoodieHiveSyncException: Failed to sync partitions for table xx_biz_operation_t_city2 at org.apache.hudi.hive.HiveSyncTool.syncPartitions(HiveSyncTool.java:206) at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:142) at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:94) at org.apache.hudi.hive.HiveSyncTool.main(HiveSyncTool.java:226)Caused by: NoSuchObjectException(message:datalake_taxi.xx_biz_operation_t_city2 table not found) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_partitions_result$get_partitions_resultStandardScheme.read(ThriftHiveMetastore.java) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_partitions_result$get_partitions_resultStandardScheme.read(ThriftHiveMetastore.java) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_partitions_result.read(ThriftHiveMetastore.java) at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:86) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_partitions(ThriftHiveMetastore.java:2289) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_partitions(ThriftHiveMetastore.java:2274) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.listPartitions(HiveMetaStoreClient.java:1384) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:154) at com.sun.proxy.$Proxy18.listPartitions(Unknown Source) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler.invoke(HiveMetaStoreClient.java:2562) at com.sun.proxy.$Proxy18.listPartitions(Unknown Source) at org.apache.hudi.hive.HoodieHiveClient.scanTablePartitions(HoodieHiveClient.java:236) at org.apache.hudi.hive.HiveSyncTool.syncPartitions(HiveSyncTool.java:196) ... 3 more
{code}
> hudi-sync-tool error
> --------------------
>
> Key: HUDI-1528
> URL: https://issues.apache.org/jira/browse/HUDI-1528
> Project: Apache Hudi
> Issue Type: Bug
> Components: Hive Integration
> Reporter: Trevorzhang
> Assignee: Trevorzhang
> Priority: Major
> Fix For: 0.8.0
>
>
> When using hudi-sync-tools to synchronize to a remote hive, hivemetastore will throw exceptions.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)