You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "xushiyan (via GitHub)" <gi...@apache.org> on 2023/02/05 18:01:25 UTC
[GitHub] [hudi] xushiyan commented on a diff in pull request #7856: [HUDI-5704] De-coupling column drop flag and schema validation flag (0.13.0)
xushiyan commented on code in PR #7856:
URL: https://github.com/apache/hudi/pull/7856#discussion_r1096769218
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieSparkSqlWriter.scala:
##########
@@ -480,6 +486,7 @@ object HoodieSparkSqlWriter {
|Table's schema ${latestTableSchema.toString(true)}
|""".stripMargin)
throw new SchemaCompatibilityException("Incoming batch schema is not compatible with the table's one")
+ }
Review Comment:
/nit fix indentation
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieSparkSqlWriter.scala:
##########
@@ -415,6 +415,7 @@ object HoodieSparkSqlWriter {
val allowAutoEvolutionColumnDrop = opts.getOrDefault(HoodieWriteConfig.SCHEMA_ALLOW_AUTO_EVOLUTION_COLUMN_DROP.key,
HoodieWriteConfig.SCHEMA_ALLOW_AUTO_EVOLUTION_COLUMN_DROP.defaultValue).toBoolean
+ val mergeIntoWrites = opts.getOrDefault(DataSourceWriteOptions.MERGE_INTO_WRITES, DataSourceWriteOptions.DEFAULT_MERGE_INTO_WRITES).toBoolean
Review Comment:
we need to infer this when user is using merge into. it's gonna be pretty bad exp if someone uses merge into and still need to set the config = true.
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/DataSourceOptions.scala:
##########
@@ -463,6 +463,9 @@ object DataSourceWriteOptions {
val RECONCILE_SCHEMA: ConfigProperty[Boolean] = HoodieCommonConfig.RECONCILE_SCHEMA
+ val MERGE_INTO_WRITES = "hoodie.merge.into.writes"
+ val DEFAULT_MERGE_INTO_WRITES = "false"
Review Comment:
why not make it a ConfigProperty
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieSparkSqlWriter.scala:
##########
@@ -415,6 +415,7 @@ object HoodieSparkSqlWriter {
val allowAutoEvolutionColumnDrop = opts.getOrDefault(HoodieWriteConfig.SCHEMA_ALLOW_AUTO_EVOLUTION_COLUMN_DROP.key,
HoodieWriteConfig.SCHEMA_ALLOW_AUTO_EVOLUTION_COLUMN_DROP.defaultValue).toBoolean
+ val mergeIntoWrites = opts.getOrDefault(DataSourceWriteOptions.MERGE_INTO_WRITES, DataSourceWriteOptions.DEFAULT_MERGE_INTO_WRITES).toBoolean
Review Comment:
look into `org.apache.spark.sql.hudi.command.MergeIntoHoodieTableCommand#buildMergeIntoConfig`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org