You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/06/27 03:37:06 UTC
[GitHub] [hudi] JoshuaZhuCN opened a new issue, #5981: [SUPPORT] How to change the precombine field of a created Hudi table
JoshuaZhuCN opened a new issue, #5981:
URL: https://github.com/apache/hudi/issues/5981
How to change the precombine field of a created Hudi table ?
1、I tried to use the alter table of spark SQL to change it, but it didn't take effect. Hoodie.properties is still the original field
2、When I try to use datasource to write, I directly specify the option ("hoodie.table.precombine.field", "new file"), and an error will occur
```org.apache.hudi.exception.HoodieException: Config conflict(key current value existing value):
PreCombineKey: new filed old filed
at org.apache.hudi.HoodieWriterUtils$.validateTableConfig(HoodieWriterUtils.scala:162)
at org.apache.hudi.HoodieSparkSqlWriter$.write(HoodieSparkSqlWriter.scala:89)
at org.apache.hudi.DefaultSource.createRelation(DefaultSource.scala:164)
at org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:46)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:70)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:68)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:90)
at org.apache.spark.sql.execution.SparkPlan.$anonfun$execute$1(SparkPlan.scala:180)
at org.apache.spark.sql.execution.SparkPlan.$anonfun$executeQuery$1(SparkPlan.scala:218)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:215)
at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:176)
at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:132)
at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:131)
at org.apache.spark.sql.DataFrameWriter.$anonfun$runCommand$1(DataFrameWriter.scala:989)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$5(SQLExecution.scala:103)
at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:163)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:90)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:775)
at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:64)
at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:989)
at org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:438)
at org.apache.spark.sql.DataFrameWriter.saveInternal(DataFrameWriter.scala:415)
at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:293)
at com.leqee.sparktool.hoodie.HoodieData.saveData(HoodieData.scala:382)
at com.leqee.sparktool.hoodie.HoodieData.upsert(HoodieData.scala:181)
at com.leqee.sparktool.hoodie.Hoodie.upsert(Hoodie.scala:242)
at com.leqee.sparktool.hoodie.Hoodie.upsert(Hoodie.scala:232)
... 51 elided```
**Environment Description**
* Hudi version : 0.10.1
* Spark version : 3.1.3
* Hive version : 3.1.0
* Hadoop version : 3.1.1
* Storage (HDFS/S3/GCS..) : HDFS
* Running on Docker? (yes/no) : no
**Additional context**
Add any other context about the problem here.
**Stacktrace**
```Add the stacktrace of the error.```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on issue #5981: [SUPPORT] How to change the precombine field of a created Hudi table
Posted by GitBox <gi...@apache.org>.
yihua commented on issue #5981:
URL: https://github.com/apache/hudi/issues/5981#issuecomment-1174650807
@JoshuaZhuCN To change the pre-combine field, the only way is to create a new Hudi table based on the new configurations. Hope that helps. Feel free to reopen the issue if you have more questions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua closed issue #5981: [SUPPORT] How to change the precombine field of a created Hudi table
Posted by GitBox <gi...@apache.org>.
yihua closed issue #5981: [SUPPORT] How to change the precombine field of a created Hudi table
URL: https://github.com/apache/hudi/issues/5981
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on issue #5981: [SUPPORT] How to change the precombine field of a created Hudi table
Posted by GitBox <gi...@apache.org>.
yihua commented on issue #5981:
URL: https://github.com/apache/hudi/issues/5981#issuecomment-1169522103
cc @YannByron
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on issue #5981: [SUPPORT] How to change the precombine field of a created Hudi table
Posted by GitBox <gi...@apache.org>.
yihua commented on issue #5981:
URL: https://github.com/apache/hudi/issues/5981#issuecomment-1169336299
@JoshuaZhuCN Given that the pre-combine field name is written to the table config in `hoodie.properties`, it cannot be changed after being set once in the current settings. @nsivabalan @xushiyan is it a problem to relax this restriction?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] YannByron commented on issue #5981: [SUPPORT] How to change the precombine field of a created Hudi table
Posted by GitBox <gi...@apache.org>.
YannByron commented on issue #5981:
URL: https://github.com/apache/hudi/issues/5981#issuecomment-1174532884
@yihua @JoshuaZhuCN for now the `recordKey` and `preCombineFiled` can not be changed. and IMO, we should not to relax the restriction that will guarantee our correctness of data.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org