[jira] [Closed] (HUDI-8942) Upgrade fails from 0.15 when OverwriteWithLatestPayload and DEFAULT_MERGER_STRATEGY_UUID set in table config

["Y Ethan Guo (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HUDI-8942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Y Ethan Guo closed HUDI-8942.
-----------------------------
    Resolution: Fixed

> Upgrade fails from 0.15 when OverwriteWithLatestPayload and DEFAULT_MERGER_STRATEGY_UUID set in table config
> ------------------------------------------------------------------------------------------------------------
>
>                 Key: HUDI-8942
>                 URL: https://issues.apache.org/jira/browse/HUDI-8942
>             Project: Apache Hudi
>          Issue Type: Sub-task
>            Reporter: Sagar Sumit
>            Assignee: Y Ethan Guo
>            Priority: Blocker
>              Labels: pull-request-available
>             Fix For: 1.0.1
>
>          Time Spent: 2h
>  Remaining Estimate: 0h
>
> # Create a table with 0.15.0 with following configs (precombine needs to be set): 
> Map(
> hoodie.clustering.inline.max.commits -> 6, hoodie.write.auto.upgrade -> true, hoodie.datasource.write.precombine.field  -> ts, hoodie.metadata.enable -> false, hoodie.datasource.write.operation -> upsert, hoodie.parquet.compression.codec -> snappy, hoodie.datasource.write.recordkey.field -> uuid, hoodie.table.name -> table_comp_test_0_15_0_1_1_0-SNAPSHOT_1738255863, hoodie.datasource.write.table.type -> MERGE_ON_READ, hoodie.datasource.write.hive_style_partitioning -> true, hoodie.clustering.inline -> false, hoodie.compact.inline -> true, hoodie.keep.max.commits -> 15, hoodie.cleaner.commits.retained -> 8, hoodie.write.lock.provider -> org.apache.hudi.client.transaction.lock.InProcessLockProvider, hoodie.keep.min.commits -> 10, hoodie.datasource.write.partitionpath.field -> partitionpath, hoodie.compact.inline.max.delta.commits -> 1)
> 2. Load some data
> 3. Now load another batch with master build (1.1.0-SNAPSHOT). It fails with below exception:
> {code:java}
> java.lang.IllegalArgumentException: Configured payload class (org.apache.hudi.common.model.OverwriteWithLatestAvroPayload) and record merge strategy ID (eeb8d96f-b1e4-49fd-bbf8-28ac514178e5) conflict with each other. Please only set one of them in the write config.
>   at org.apache.hudi.common.util.ValidationUtils.checkArgument(ValidationUtils.java:42)
>   at org.apache.hudi.common.table.HoodieTableConfig.inferCorrectMergingBehavior(HoodieTableConfig.java:785)
>   at org.apache.hudi.HoodieSparkSqlWriterInternal.mergeParamsAndGetHoodieConfig(HoodieSparkSqlWriter.scala:1136)
>   at org.apache.hudi.HoodieSparkSqlWriterInternal.writeInternal(HoodieSparkSqlWriter.scala:253)
>   at org.apache.hudi.HoodieSparkSqlWriterInternal.$anonfun$write$1(HoodieSparkSqlWriter.scala:193)
>   at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$6(SQLExecution.scala:109)
>   at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:169)
>   at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:95)
>   at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:779)
>   at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:64)
>   at org.apache.spark.sql.adapter.BaseSpark3Adapter.sqlExecutionWithNewExecutionId(BaseSpark3Adapter.scala:105)
>   at org.apache.hudi.HoodieSparkSqlWriterInternal.write(HoodieSparkSqlWriter.scala:215)
>   at org.apache.hudi.HoodieSparkSqlWriter$.write(HoodieSparkSqlWriter.scala:130)
>   at org.apache.hudi.DefaultSource.createRelation(DefaultSource.scala:170)
>   at org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:47) {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)