You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/09/02 09:14:48 UTC
[GitHub] [hudi] Archie-selfless commented on issue #6572: [SUPPORT] DeltaStreamer sync hive-table called NPE when adding patition(s)
Archie-selfless commented on issue #6572:
URL: https://github.com/apache/hudi/issues/6572#issuecomment-1235265477
When I switch the sycn_mode to jdbc, I got the same Exception. However, I can run the HQL below which caught from spark logs to complete the hive table.
```sql
ALTER TABLE `default`.`hudi_mysql_table2`
ADD IF NOT EXISTS
PARTITION (`u_name`='A') LOCATION 'cosn://bigdata-1252394470/hudi/cdc/mysql/table2_001/u_name=A'
PARTITION (`u_name`='B') LOCATION 'cosn://bigdata-1252394470/hudi/cdc/mysql/table2_001/u_name=B'
PARTITION (`u_name`='C') LOCATION 'cosn://bigdata-1252394470/hudi/cdc/mysql/table2_001/u_name=C' ;
```
After the HQL, everything goes smoothly! (i.e. Query the hudi-table by Presto...)
** ↓↓ The core logs syncing as jdbc mode ↓↓ **
```bash
22/09/02 16:57:27 INFO HiveSyncTool: Storage partitions scan complete. Found 3
22/09/02 16:57:27 INFO HiveSyncTool: New Partitions [u_name=A, u_name=B, u_name=C]
22/09/02 16:57:27 INFO QueryBasedDDLExecutor: Adding partitions 3 to table hudi_mysql_table001
22/09/02 16:57:27 INFO QueryBasedDDLExecutor: Executing SQL ALTER TABLE `default`.`hudi_mysql_table001` ADD IF NOT EXISTS PARTITION (`u_name`='A') LOCATION 'cosn://bigdata-1252394470/hudi/cdc/mysql/table2_001/u_name=A' PARTITION (`u_name`='B') LOCATION 'cosn://bigdata-1252394470/hudi/cdc/mysql/table2_001/u_name=B' PARTITION (`u_name`='C') LOCATION 'cosn://bigdata-1252394470/hudi/cdc/mysql/table2_001/u_name=C'
22/09/02 16:57:28 INFO metastore: Closed a connection to metastore, current connections: 0
22/09/02 16:57:28 ERROR HoodieDeltaStreamer: Shutting down delta-sync due to exception
org.apache.hudi.exception.HoodieException: Could not sync using the meta sync class org.apache.hudi.hive.HiveSyncTool
at org.apache.hudi.sync.common.util.SyncUtilHelpers.runHoodieMetaSync(SyncUtilHelpers.java:58)
at org.apache.hudi.utilities.deltastreamer.DeltaSync.runMetaSync(DeltaSync.java:716)
at org.apache.hudi.utilities.deltastreamer.DeltaSync.writeToSink(DeltaSync.java:634)
at org.apache.hudi.utilities.deltastreamer.DeltaSync.syncOnce(DeltaSync.java:335)
at org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer$DeltaSyncService.lambda$startService$0(HoodieDeltaStreamer.java:687)
at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.hudi.exception.HoodieException: Got runtime exception when hive syncing hudi_mysql_table001
at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:144)
at org.apache.hudi.sync.common.util.SyncUtilHelpers.runHoodieMetaSync(SyncUtilHelpers.java:56)
... 8 more
Caused by: org.apache.hudi.hive.HoodieHiveSyncException: Failed to sync partitions for table hudi_mysql_table001
at org.apache.hudi.hive.HiveSyncTool.syncPartitions(HiveSyncTool.java:340)
at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:232)
at org.apache.hudi.hive.HiveSyncTool.doSync(HiveSyncTool.java:153)
at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:141)
... 9 more
...
...
Caused by: MetaException(message:java.io.IOException: Got exception: java.io.IOException java.lang.NullPointerException)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$add_partitions_req_result$add_partitions_req_resultStandardScheme.read(ThriftHiveMetastore.java)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$add_partitions_req_result$add_partitions_req_resultStandardScheme.read(ThriftHiveMetastore.java)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$add_partitions_req_result.read(ThriftHiveMetastore.java)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:86)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_add_partitions_req(ThriftHiveMetastore.java:1911)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.add_partitions_req(ThriftHiveMetastore.java:1898)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.add_partitions(HiveMetaStoreClient.java:625)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:173)
at com.sun.proxy.$Proxy26.add_partitions(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler.invoke(HiveMetaStoreClient.java:2327)
at com.sun.proxy.$Proxy26.add_partitions(Unknown Source)
at org.apache.hadoop.hive.ql.metadata.Hive.createPartitions(Hive.java:2103)
... 21 more
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org