You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "hudi-bot (via GitHub)" <gi...@apache.org> on 2025/11/30 10:52:30 UTC

[I] Schema on read schema evolution does not support dropping a column and then renaming another one to the original columns name [hudi]

hudi-bot opened a new issue, #16665:
URL: https://github.com/apache/hudi/issues/16665

   Try out this test by adding it to hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/common/TestHoodieInternalRowUtils.scala:
   {code:java}
     private val twoFieldSchema = StructType(Seq(
       StructField("col1", StringType),
       StructField("col2", StringType)
     ))  private val oneFieldSchema = StructType(Seq(
       StructField("col1", StringType)
     ))
     test("Test simple col rename overwrite") {
       val rows = Seq(Row("aaa", "bbb"))
       val data = sparkSession.sparkContext.parallelize(rows)
       val oldRow = sparkSession.createDataFrame(data, twoFieldSchema).queryExecution.toRdd.first()    val rowWriter1 = HoodieInternalRowUtils.genUnsafeRowWriter(twoFieldSchema, oneFieldSchema, JCollections.singletonMap("col1", "col2"))
       val newRow1 = rowWriter1(oldRow)    //This does not work, see "Test alter column multiple times"
       //val serDe1 = sparkAdapter.createSparkRowSerDe(oneFieldSchema)
       //assertEquals(serDe1.deserializeRow(newRow1), Row("bbb"))
     } {code}
   The assert equals that is commented out will fail.
   
   
   
   This is not the expected behavior. We have a test in TestSpark3DDL "Test alter column multiple times":
   {code:java}
   test("Test alter column multiple times") {
     withTempDir { tmp =>
       Seq("cow", "mor").foreach { tableType =>
         val tableName = generateTableName
         val tablePath = s"${new Path(tmp.getCanonicalPath, tableName).toUri.toString}"
         if (HoodieSparkUtils.gteqSpark3_3) {
           spark.sql("set hoodie.schema.on.read.enable=true")
           spark.sql(
             s"""
                |create table $tableName (
                |  id int,
                |  col1 string,
                |  col2 string,
                |  ts long
                |) using hudi
                | location '$tablePath'
                | options (
                |  type = '$tableType',
                |  primaryKey = 'id',
                |  preCombineField = 'ts'
                | )
              """.stripMargin)
           spark.sql(s"show create table ${tableName}").show(false)
           spark.sql(s"insert into ${tableName} values (1, 'aaa', 'bbb', 1000)")
   
           // Rename to a previously existing column name + insert
           spark.sql(s"alter table ${tableName} drop column col1")
           spark.sql(s"alter table ${tableName} rename column col2 to col1")
   
           spark.sql(s"insert into ${tableName} values (2, 'aaa', 1000)")
           checkAnswer(spark.sql(s"select col1 from ${tableName} order by id").collect())(
             Seq("bbb"), Seq("aaa")
           )
         }
       }
     }
   } {code}
   If you use spark merger for this test, it will fail
   
   ## JIRA info
   
   - Link: https://issues.apache.org/jira/browse/HUDI-8320
   - Type: Bug
   - Fix version(s):
     - 1.1.0


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Schema on read schema evolution does not support dropping a column and then renaming another one to the original columns name [hudi]

Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan closed issue #16665: Schema on read schema evolution does not support dropping a column and then renaming another one to the original columns name
URL: https://github.com/apache/hudi/issues/16665


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Schema on read schema evolution does not support dropping a column and then renaming another one to the original columns name [hudi]

Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on issue #16665:
URL: https://github.com/apache/hudi/issues/16665#issuecomment-4795629603

   This appears to be fixed on master by [HUDI-9683 / PR #13672](https://github.com/apache/hudi/commit/f51c36fd64398718c039b26c15767e793b82904e), which reworks `HoodieInternalRowUtils.genUnsafeRowWriter` to consult the rename map first (via `lookupRenamedField`) before looking up the field name in the previous schema. That prevents the case where the new schema's `col1` (renamed from `col2`) was matching the dropped previous `col1`. Coverage was added in `TestHoodieInternalRowUtils` (`Test rewrite row with renamed columns`, `Test rewrite row with columns swap`). Closing as fixed — please reopen if you can still reproduce on master.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org