You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/09/02 09:14:48 UTC

[GitHub] [hudi] Archie-selfless commented on issue #6572: [SUPPORT] DeltaStreamer sync hive-table called NPE when adding patition(s)

Archie-selfless commented on issue #6572:
URL: https://github.com/apache/hudi/issues/6572#issuecomment-1235265477

   When I switch the sycn_mode to jdbc, I got the same Exception. However, I can run the HQL below which caught from spark logs to complete the hive table.
   ```sql
   ALTER TABLE `default`.`hudi_mysql_table2`
       ADD IF NOT EXISTS
           PARTITION (`u_name`='A') LOCATION 'cosn://bigdata-1252394470/hudi/cdc/mysql/table2_001/u_name=A'
           PARTITION (`u_name`='B') LOCATION 'cosn://bigdata-1252394470/hudi/cdc/mysql/table2_001/u_name=B'
           PARTITION (`u_name`='C') LOCATION 'cosn://bigdata-1252394470/hudi/cdc/mysql/table2_001/u_name=C' ;
   ```
   After the HQL, everything goes smoothly! (i.e. Query the hudi-table by Presto...)
   
   ** ↓↓ The core logs syncing as jdbc mode ↓↓ **
   ```bash
   22/09/02 16:57:27 INFO HiveSyncTool: Storage partitions scan complete. Found 3
   22/09/02 16:57:27 INFO HiveSyncTool: New Partitions [u_name=A, u_name=B, u_name=C]
   22/09/02 16:57:27 INFO QueryBasedDDLExecutor: Adding partitions 3 to table hudi_mysql_table001
   22/09/02 16:57:27 INFO QueryBasedDDLExecutor: Executing SQL ALTER TABLE `default`.`hudi_mysql_table001` ADD IF NOT EXISTS   PARTITION (`u_name`='A') LOCATION 'cosn://bigdata-1252394470/hudi/cdc/mysql/table2_001/u_name=A'   PARTITION (`u_name`='B') LOCATION 'cosn://bigdata-1252394470/hudi/cdc/mysql/table2_001/u_name=B'   PARTITION (`u_name`='C') LOCATION 'cosn://bigdata-1252394470/hudi/cdc/mysql/table2_001/u_name=C' 
   22/09/02 16:57:28 INFO metastore: Closed a connection to metastore, current connections: 0
   22/09/02 16:57:28 ERROR HoodieDeltaStreamer: Shutting down delta-sync due to exception
   org.apache.hudi.exception.HoodieException: Could not sync using the meta sync class org.apache.hudi.hive.HiveSyncTool
   	at org.apache.hudi.sync.common.util.SyncUtilHelpers.runHoodieMetaSync(SyncUtilHelpers.java:58)
   	at org.apache.hudi.utilities.deltastreamer.DeltaSync.runMetaSync(DeltaSync.java:716)
   	at org.apache.hudi.utilities.deltastreamer.DeltaSync.writeToSink(DeltaSync.java:634)
   	at org.apache.hudi.utilities.deltastreamer.DeltaSync.syncOnce(DeltaSync.java:335)
   	at org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer$DeltaSyncService.lambda$startService$0(HoodieDeltaStreamer.java:687)
   	at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
   	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
   	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   	at java.lang.Thread.run(Thread.java:748)
   Caused by: org.apache.hudi.exception.HoodieException: Got runtime exception when hive syncing hudi_mysql_table001
   	at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:144)
   	at org.apache.hudi.sync.common.util.SyncUtilHelpers.runHoodieMetaSync(SyncUtilHelpers.java:56)
   	... 8 more
   Caused by: org.apache.hudi.hive.HoodieHiveSyncException: Failed to sync partitions for table hudi_mysql_table001
   	at org.apache.hudi.hive.HiveSyncTool.syncPartitions(HiveSyncTool.java:340)
   	at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:232)
   	at org.apache.hudi.hive.HiveSyncTool.doSync(HiveSyncTool.java:153)
   	at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:141)
   	... 9 more
   ...
   ...
   Caused by: MetaException(message:java.io.IOException: Got exception: java.io.IOException java.lang.NullPointerException)
   	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$add_partitions_req_result$add_partitions_req_resultStandardScheme.read(ThriftHiveMetastore.java)
   	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$add_partitions_req_result$add_partitions_req_resultStandardScheme.read(ThriftHiveMetastore.java)
   	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$add_partitions_req_result.read(ThriftHiveMetastore.java)
   	at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:86)
   	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_add_partitions_req(ThriftHiveMetastore.java:1911)
   	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.add_partitions_req(ThriftHiveMetastore.java:1898)
   	at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.add_partitions(HiveMetaStoreClient.java:625)
   	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   	at java.lang.reflect.Method.invoke(Method.java:498)
   	at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:173)
   	at com.sun.proxy.$Proxy26.add_partitions(Unknown Source)
   	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   	at java.lang.reflect.Method.invoke(Method.java:498)
   	at org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler.invoke(HiveMetaStoreClient.java:2327)
   	at com.sun.proxy.$Proxy26.add_partitions(Unknown Source)
   	at org.apache.hadoop.hive.ql.metadata.Hive.createPartitions(Hive.java:2103)
   	... 21 more
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org