[jira] [Closed] (HUDI-7017) Prevent full schema evolution from wrongly falling back to OOB

["Danny Chen (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HUDI-7017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Danny Chen closed HUDI-7017.
----------------------------
    Resolution: Fixed

Fixed via master branch: d149d787e9d3dbf425c0dd5ca0265bed5fe2795f

> Prevent full schema evolution from wrongly falling back to OOB
> --------------------------------------------------------------
>
>                 Key: HUDI-7017
>                 URL: https://issues.apache.org/jira/browse/HUDI-7017
>             Project: Apache Hudi
>          Issue Type: Bug
>            Reporter: voon
>            Assignee: voon
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 1.0.0
>
>         Attachments: image-2023-11-01-11-41-25-604.png, image-2023-11-01-11-43-14-149.png
>
>
> For MOR tables that have these 2 configurations enabled:
>  
> {code:java}
> hoodie.schema.on.read.enable=true
> hoodie.datasource.read.extract.partition.values.from.path=true{code}
>  
>  
> BaseFileReader will use a *requiredSchemaReader* when reading some of the parquet files. This BaseFileReader will have an empty *internalSchemaStr* causing *Spark3XLegacyHoodieParquetInputFormat* to fall back to OOB schema evolution.
>  
> Although there are required safeguards that are added in HUDI-5400 to force the code execution path to use Hudi Full Schema Evolution, we should still fix this so that future changes that may deprecate the use of *Spark3XLegacyHoodieParquetInputFormat* will not cause issues.
>  
> A sample test to invoke this:
> {code:java}
> test("Test wrong fallback to OOB schema evolution") {
>   withRecordType()(withTempDir { tmp =>
>     Seq("mor").foreach { tableType =>
>       val tableName = generateTableName
>       val tablePath = s"${new Path(tmp.getCanonicalPath, tableName).toUri.toString}"
>       if (HoodieSparkUtils.gteqSpark3_1) {
>         spark.sql("set " + SPARK_SQL_INSERT_INTO_OPERATION.key + "=upsert")
>         spark.sql("set hoodie.schema.on.read.enable=true")
>         spark.sql("hoodie.datasource.read.extract.partition.values.from.path=true")
>         // NOTE: This is required since as this tests use type coercions which were only permitted in Spark 2.x
>         //       and are disallowed now by default in Spark 3.x
>         spark.sql("set spark.sql.storeAssignmentPolicy=legacy")
>         createAndPreparePartitionTable(spark, tableName, tablePath, tableType)
>         // date -> string
>         spark.sql(s"alter table $tableName alter column col6 type String")
>         checkAnswer(spark.sql(s"select col6 from $tableName where id = 1").collect())(
>           Seq("2021-12-25")
>         )
>       }
>     }
>   })
> } {code}
>  
> Debugger snapshots:
> !image-2023-11-01-11-41-25-604.png|width=1197,height=596!
> As can be seen, *requiredSchema* (used as pruning input) has internalSchema string, but *requiredDataSchema* does has a null internalSchema string.
>  
> !image-2023-11-01-11-43-14-149.png|width=1257,height=672!
> As a result, the internalSchemaStr that is passed into Spark3XLegacyHoodieParquetFileFormat is null (which should not be the case)
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)