You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2022/08/11 10:18:00 UTC

[jira] [Updated] (HUDI-4603) Improve HMS Catalog function in flink

     [ https://issues.apache.org/jira/browse/HUDI-4603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

ASF GitHub Bot updated HUDI-4603:
---------------------------------
    Labels: pull-request-available  (was: )

> Improve HMS Catalog function in flink
> -------------------------------------
>
>                 Key: HUDI-4603
>                 URL: https://issues.apache.org/jira/browse/HUDI-4603
>             Project: Apache Hudi
>          Issue Type: Improvement
>            Reporter: xiaozhongcheng
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 0.12.0
>
>
> 1. Make users can choose to sync ro and rt table or not when using hms catalog in flink.
> Currently, if users use hms catalog in flink, ro and rt table will be also sync to the hms.
> But If I just want to sync the metadata of hudi table, I don't want to sync ro and rt table.
> So, I think users should be able to choose to sync the ro and rt table or not.
>  
> 2. Make users can sync ro and rt table correctly when the table is partitioned table.
> Currently, If users create the partitioned table in hms catalog, but the partition field is not in the form of yyyy/mm/dd. If they don't make hive_sync.partition_extractor_class to org.apache.hudi.hive.HiveStylePartitionValueExtractor, ro and rt table will not be synced correctly
> The stacks is list below:
>  
> {code:java}
> org.apache.hudi.exception.HoodieException: Got runtime exception when hive syncing student22
>   at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:144) ~[hudi-flink1.15-bundle-0.12.0-rc2.jar:0.12.0-rc2]
>   at org.apache.hudi.sink.StreamWriteOperatorCoordinator.doSyncHive(StreamWriteOperatorCoordinator.java:335) ~[hudi-flink1.15-bundle-0.12.0-rc2.jar:0.12.0-rc2]
>   at org.apache.hudi.sink.StreamWriteOperatorCoordinator.syncHive(StreamWriteOperatorCoordinator.java:326) ~[hudi-flink1.15-bundle-0.12.0-rc2.jar:0.12.0-rc2]
>   at org.apache.hudi.sink.StreamWriteOperatorCoordinator.handleEndInputEvent(StreamWriteOperatorCoordinator.java:426) ~[hudi-flink1.15-bundle-0.12.0-rc2.jar:0.12.0-rc2]
>   at org.apache.hudi.sink.StreamWriteOperatorCoordinator.lambda$handleEventFromOperator$3(StreamWriteOperatorCoordinator.java:278) ~[hudi-flink1.15-bundle-0.12.0-rc2.jar:0.12.0-rc2]
>   at org.apache.hudi.sink.utils.NonThrownExecutor.lambda$wrapAction$0(NonThrownExecutor.java:130) ~[hudi-flink1.15-bundle-0.12.0-rc2.jar:0.12.0-rc2]
>   at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_241]
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_241]
>   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_241]
>   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_241]
>   at java.lang.Thread.run(Thread.java:748) [?:1.8.0_241]
> Caused by: org.apache.hudi.hive.HoodieHiveSyncException: Failed to sync partitions for table student22_ro
>   at org.apache.hudi.hive.HiveSyncTool.syncPartitions(HiveSyncTool.java:340) ~[hudi-flink1.15-bundle-0.12.0-rc2.jar:0.12.0-rc2]
>   at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:232) ~[hudi-flink1.15-bundle-0.12.0-rc2.jar:0.12.0-rc2]
>   at org.apache.hudi.hive.HiveSyncTool.doSync(HiveSyncTool.java:157) ~[hudi-flink1.15-bundle-0.12.0-rc2.jar:0.12.0-rc2]
>   at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:141) ~[hudi-flink1.15-bundle-0.12.0-rc2.jar:0.12.0-rc2]
>   ... 10 more
> Caused by: java.lang.IllegalArgumentException: Partition path school=beida is not in the form yyyy/mm/dd 
>   at org.apache.hudi.hive.SlashEncodedDayPartitionValueExtractor.extractPartitionValuesInPath(SlashEncodedDayPartitionValueExtractor.java:58) ~[hudi-flink1.15-bundle-0.12.0-rc2.jar:0.12.0-rc2]
>   at org.apache.hudi.sync.common.HoodieSyncClient.getPartitionEvents(HoodieSyncClient.java:144) ~[hudi-flink1.15-bundle-0.12.0-rc2.jar:0.12.0-rc2]
>   at org.apache.hudi.hive.HiveSyncTool.syncPartitions(HiveSyncTool.java:318) ~[hudi-flink1.15-bundle-0.12.0-rc2.jar:0.12.0-rc2]
>   at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:232) ~[hudi-flink1.15-bundle-0.12.0-rc2.jar:0.12.0-rc2]
>   at org.apache.hudi.hive.HiveSyncTool.doSync(HiveSyncTool.java:157) ~[hudi-flink1.15-bundle-0.12.0-rc2.jar:0.12.0-rc2]
>   at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:141) ~[hudi-flink1.15-bundle-0.12.0-rc2.jar:0.12.0-rc2]
>   ... 10 more {code}
>  
>  
>  
>  
>  
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)