You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "rex xiong (Jira)" <ji...@apache.org> on 2022/07/22 02:16:00 UTC
[jira] [Updated] (HUDI-4447) Hive Sync fails fails when performing delete table data operation
[ https://issues.apache.org/jira/browse/HUDI-4447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
rex xiong updated HUDI-4447:
----------------------------
Description:
currently, sync meta would fail when performing delete table data operation, because sync without setting database name/fields/extractor class, etc.This causes two problems：
1）sync schema will recreate table in default database
2）sync will use wrong extractor class(HiveStylePartitionValueExtractor) when syncing non partitioned table with metadata table enabled
{code:java}
spark-sql> delete from test_db.hudi_mor_none_part_table_321_0111 where id=1;
org.apache.hudi.exception.HoodieException: Could not sync using the meta sync class org.apache.hudi.hive.HiveSyncTool
at org.apache.hudi.sync.common.util.SyncUtilHelpers.runHoodieMetaSync(SyncUtilHelpers.java:61)
at org.apache.hudi.HoodieSparkSqlWriter$.$anonfun$metaSync$2(HoodieSparkSqlWriter.scala:626)
at org.apache.hudi.HoodieSparkSqlWriter$.$anonfun$metaSync$2$adapted(HoodieSparkSqlWriter.scala:625)
at scala.collection.mutable.HashSet.foreach(HashSet.scala:79)
....
Caused by: org.apache.hudi.exception.HoodieException: Got runtime exception when hive syncing hudi_mor_none_part_table_321_0111
at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:143)
at org.apache.hudi.sync.common.util.SyncUtilHelpers.runHoodieMetaSync(SyncUtilHelpers.java:59)
... 93 more
Caused by: org.apache.hudi.hive.HoodieHiveSyncException: Failed to sync partitions for table hudi_mor_none_part_table_321_0111_ro
at org.apache.hudi.hive.HiveSyncTool.syncPartitions(HiveSyncTool.java:418)
at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:232)
at org.apache.hudi.hive.HiveSyncTool.doSync(HiveSyncTool.java:156)
at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:140)
... 94 more
Caused by: java.lang.IllegalArgumentException: Partition path is not in the form partition_key=partition_value.
at org.apache.hudi.hive.HiveStylePartitionValueExtractor.extractPartitionValuesInPath(HiveStylePartitionValueExtractor.java:37)
at org.apache.hudi.hive.AbstractHiveSyncHoodieClient.getPartitionEvents(AbstractHiveSyncHoodieClient.java:81)
at org.apache.hudi.hive.HiveSyncTool.syncPartitions(HiveSyncTool.java:396)
... 97 more
org.apache.hudi.exception.HoodieException: Could not sync using the meta sync class org.apache.hudi.hive.HiveSyncTool
at org.apache.hudi.sync.common.util.SyncUtilHelpers.runHoodieMetaSync(SyncUtilHelpers.java:61)
at org.apache.hudi.HoodieSparkSqlWriter$.$anonfun$metaSync$2(HoodieSparkSqlWriter.scala:626)
at org.apache.hudi.HoodieSparkSqlWriter$.$anonfun$metaSync$2$adapted(HoodieSparkSqlWriter.scala:625)
at scala.collection.mutable.HashSet.foreach(HashSet.scala:79)
at org.apache.hudi.HoodieSparkSqlWriter$.metaSync(HoodieSparkSqlWriter.scala:625)
{code}
was:
currently, sync meta would fail if performing delete table data operation, because sync without setting database name/fields/extractor class, etc.This causes two problems：
1）sync schema will recreate table in default database
2）sync will use wrong extractor class(HiveStylePartitionValueExtractor) when syncing non partitioned table with metadata table enabled
{code:java}
spark-sql> delete from test_db.hudi_mor_none_part_table_321_0111 where id=1;
org.apache.hudi.exception.HoodieException: Could not sync using the meta sync class org.apache.hudi.hive.HiveSyncTool
at org.apache.hudi.sync.common.util.SyncUtilHelpers.runHoodieMetaSync(SyncUtilHelpers.java:61)
at org.apache.hudi.HoodieSparkSqlWriter$.$anonfun$metaSync$2(HoodieSparkSqlWriter.scala:626)
at org.apache.hudi.HoodieSparkSqlWriter$.$anonfun$metaSync$2$adapted(HoodieSparkSqlWriter.scala:625)
at scala.collection.mutable.HashSet.foreach(HashSet.scala:79)
....
Caused by: org.apache.hudi.exception.HoodieException: Got runtime exception when hive syncing hudi_mor_none_part_table_321_0111
at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:143)
at org.apache.hudi.sync.common.util.SyncUtilHelpers.runHoodieMetaSync(SyncUtilHelpers.java:59)
... 93 more
Caused by: org.apache.hudi.hive.HoodieHiveSyncException: Failed to sync partitions for table hudi_mor_none_part_table_321_0111_ro
at org.apache.hudi.hive.HiveSyncTool.syncPartitions(HiveSyncTool.java:418)
at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:232)
at org.apache.hudi.hive.HiveSyncTool.doSync(HiveSyncTool.java:156)
at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:140)
... 94 more
Caused by: java.lang.IllegalArgumentException: Partition path is not in the form partition_key=partition_value.
at org.apache.hudi.hive.HiveStylePartitionValueExtractor.extractPartitionValuesInPath(HiveStylePartitionValueExtractor.java:37)
at org.apache.hudi.hive.AbstractHiveSyncHoodieClient.getPartitionEvents(AbstractHiveSyncHoodieClient.java:81)
at org.apache.hudi.hive.HiveSyncTool.syncPartitions(HiveSyncTool.java:396)
... 97 more
org.apache.hudi.exception.HoodieException: Could not sync using the meta sync class org.apache.hudi.hive.HiveSyncTool
at org.apache.hudi.sync.common.util.SyncUtilHelpers.runHoodieMetaSync(SyncUtilHelpers.java:61)
at org.apache.hudi.HoodieSparkSqlWriter$.$anonfun$metaSync$2(HoodieSparkSqlWriter.scala:626)
at org.apache.hudi.HoodieSparkSqlWriter$.$anonfun$metaSync$2$adapted(HoodieSparkSqlWriter.scala:625)
at scala.collection.mutable.HashSet.foreach(HashSet.scala:79)
at org.apache.hudi.HoodieSparkSqlWriter$.metaSync(HoodieSparkSqlWriter.scala:625)
{code}
> Hive Sync fails fails when performing delete table data operation
> -----------------------------------------------------------------
>
> Key: HUDI-4447
> URL: https://issues.apache.org/jira/browse/HUDI-4447
> Project: Apache Hudi
> Issue Type: Bug
> Components: meta-sync
> Environment: Spark3.2.1 & Hudi 0.11.1
> Reporter: rex xiong
> Assignee: rex xiong
> Priority: Minor
> Fix For: 0.12.0
>
>
> currently, sync meta would fail when performing delete table data operation, because sync without setting database name/fields/extractor class, etc.This causes two problems：
> 1）sync schema will recreate table in default database
> 2）sync will use wrong extractor class(HiveStylePartitionValueExtractor) when syncing non partitioned table with metadata table enabled
>
> {code:java}
> spark-sql> delete from test_db.hudi_mor_none_part_table_321_0111 where id=1;
> org.apache.hudi.exception.HoodieException: Could not sync using the meta sync class org.apache.hudi.hive.HiveSyncTool
> at org.apache.hudi.sync.common.util.SyncUtilHelpers.runHoodieMetaSync(SyncUtilHelpers.java:61)
> at org.apache.hudi.HoodieSparkSqlWriter$.$anonfun$metaSync$2(HoodieSparkSqlWriter.scala:626)
> at org.apache.hudi.HoodieSparkSqlWriter$.$anonfun$metaSync$2$adapted(HoodieSparkSqlWriter.scala:625)
> at scala.collection.mutable.HashSet.foreach(HashSet.scala:79)
> ....
> Caused by: org.apache.hudi.exception.HoodieException: Got runtime exception when hive syncing hudi_mor_none_part_table_321_0111
> at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:143)
> at org.apache.hudi.sync.common.util.SyncUtilHelpers.runHoodieMetaSync(SyncUtilHelpers.java:59)
> ... 93 more
> Caused by: org.apache.hudi.hive.HoodieHiveSyncException: Failed to sync partitions for table hudi_mor_none_part_table_321_0111_ro
> at org.apache.hudi.hive.HiveSyncTool.syncPartitions(HiveSyncTool.java:418)
> at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:232)
> at org.apache.hudi.hive.HiveSyncTool.doSync(HiveSyncTool.java:156)
> at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:140)
> ... 94 more
> Caused by: java.lang.IllegalArgumentException: Partition path is not in the form partition_key=partition_value.
> at org.apache.hudi.hive.HiveStylePartitionValueExtractor.extractPartitionValuesInPath(HiveStylePartitionValueExtractor.java:37)
> at org.apache.hudi.hive.AbstractHiveSyncHoodieClient.getPartitionEvents(AbstractHiveSyncHoodieClient.java:81)
> at org.apache.hudi.hive.HiveSyncTool.syncPartitions(HiveSyncTool.java:396)
> ... 97 more
> org.apache.hudi.exception.HoodieException: Could not sync using the meta sync class org.apache.hudi.hive.HiveSyncTool
> at org.apache.hudi.sync.common.util.SyncUtilHelpers.runHoodieMetaSync(SyncUtilHelpers.java:61)
> at org.apache.hudi.HoodieSparkSqlWriter$.$anonfun$metaSync$2(HoodieSparkSqlWriter.scala:626)
> at org.apache.hudi.HoodieSparkSqlWriter$.$anonfun$metaSync$2$adapted(HoodieSparkSqlWriter.scala:625)
> at scala.collection.mutable.HashSet.foreach(HashSet.scala:79)
> at org.apache.hudi.HoodieSparkSqlWriter$.metaSync(HoodieSparkSqlWriter.scala:625)
> {code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)