You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "Vinoth Chandar (Jira)" <ji...@apache.org> on 2021/01/21 06:25:00 UTC
[jira] [Closed] (HUDI-1427) Throw a FileAlreadyExistsException when
set HOODIE_AUTO_COMMIT_PROP to true
[ https://issues.apache.org/jira/browse/HUDI-1427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinoth Chandar closed HUDI-1427.
--------------------------------
Resolution: Fixed
> Throw a FileAlreadyExistsException when set HOODIE_AUTO_COMMIT_PROP to true
> ---------------------------------------------------------------------------
>
> Key: HUDI-1427
> URL: https://issues.apache.org/jira/browse/HUDI-1427
> Project: Apache Hudi
> Issue Type: Bug
> Components: Spark Integration
> Reporter: pengzhiwei
> Assignee: pengzhiwei
> Priority: Major
> Labels: pull-request-available
> Fix For: 0.7.0
>
>
> I have test write hudi with COW & MOR write type. When set HOODIE_AUTO_COMMIT_PROP to true, a FileAlreadyExistsException throw out:
> {code:java}
> Exception in thread "main" org.apache.hudi.exception.HoodieIOException: Failed to create file /tmp/hudi/tbl_price_cow1/.hoodie/20201202104150.commitException in thread "main" org.apache.hudi.exception.HoodieIOException: Failed to create file /tmp/hudi/tbl_price_cow1/.hoodie/20201202104150.commit at org.apache.hudi.common.table.timeline.HoodieActiveTimeline.createImmutableFileInPath(HoodieActiveTimeline.java:474) at org.apache.hudi.common.table.timeline.HoodieActiveTimeline.transitionState(HoodieActiveTimeline.java:350) at org.apache.hudi.common.table.timeline.HoodieActiveTimeline.transitionState(HoodieActiveTimeline.java:325) at org.apache.hudi.common.table.timeline.HoodieActiveTimeline.saveAsComplete(HoodieActiveTimeline.java:144) at org.apache.hudi.client.AbstractHoodieWriteClient.commitStats(AbstractHoodieWriteClient.java:181) at org.apache.hudi.client.SparkRDDWriteClient.commit(SparkRDDWriteClient.java:101) at org.apache.hudi.HoodieSparkSqlWriter$.commitAndPerformPostOperations(HoodieSparkSqlWriter.scala:413) at org.apache.hudi.HoodieSparkSqlWriter$.write(HoodieSparkSqlWriter.scala:210) at org.apache.hudi.DefaultSource.createRelation(DefaultSource.scala:125) at org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:46) at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:70) at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:68) at org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:86) at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:137) at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:133) at org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:161) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151) at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:158) at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:133)
> {code}
> It seems that _../.hoodie/20201202104150.commit_ has commit twice and result in this exception.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)