You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "hudi-bot (via GitHub)" <gi...@apache.org> on 2025/11/30 10:52:30 UTC
[I] Schema on read schema evolution does not support dropping a column and then renaming another one to the original columns name [hudi]
hudi-bot opened a new issue, #16665:
URL: https://github.com/apache/hudi/issues/16665
Try out this test by adding it to hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/common/TestHoodieInternalRowUtils.scala:
{code:java}
private val twoFieldSchema = StructType(Seq(
StructField("col1", StringType),
StructField("col2", StringType)
)) private val oneFieldSchema = StructType(Seq(
StructField("col1", StringType)
))
test("Test simple col rename overwrite") {
val rows = Seq(Row("aaa", "bbb"))
val data = sparkSession.sparkContext.parallelize(rows)
val oldRow = sparkSession.createDataFrame(data, twoFieldSchema).queryExecution.toRdd.first() val rowWriter1 = HoodieInternalRowUtils.genUnsafeRowWriter(twoFieldSchema, oneFieldSchema, JCollections.singletonMap("col1", "col2"))
val newRow1 = rowWriter1(oldRow) //This does not work, see "Test alter column multiple times"
//val serDe1 = sparkAdapter.createSparkRowSerDe(oneFieldSchema)
//assertEquals(serDe1.deserializeRow(newRow1), Row("bbb"))
} {code}
The assert equals that is commented out will fail.
This is not the expected behavior. We have a test in TestSpark3DDL "Test alter column multiple times":
{code:java}
test("Test alter column multiple times") {
withTempDir { tmp =>
Seq("cow", "mor").foreach { tableType =>
val tableName = generateTableName
val tablePath = s"${new Path(tmp.getCanonicalPath, tableName).toUri.toString}"
if (HoodieSparkUtils.gteqSpark3_3) {
spark.sql("set hoodie.schema.on.read.enable=true")
spark.sql(
s"""
|create table $tableName (
| id int,
| col1 string,
| col2 string,
| ts long
|) using hudi
| location '$tablePath'
| options (
| type = '$tableType',
| primaryKey = 'id',
| preCombineField = 'ts'
| )
""".stripMargin)
spark.sql(s"show create table ${tableName}").show(false)
spark.sql(s"insert into ${tableName} values (1, 'aaa', 'bbb', 1000)")
// Rename to a previously existing column name + insert
spark.sql(s"alter table ${tableName} drop column col1")
spark.sql(s"alter table ${tableName} rename column col2 to col1")
spark.sql(s"insert into ${tableName} values (2, 'aaa', 1000)")
checkAnswer(spark.sql(s"select col1 from ${tableName} order by id").collect())(
Seq("bbb"), Seq("aaa")
)
}
}
}
} {code}
If you use spark merger for this test, it will fail
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-8320
- Type: Bug
- Fix version(s):
- 1.1.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Schema on read schema evolution does not support dropping a column and then renaming another one to the original columns name [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan closed issue #16665: Schema on read schema evolution does not support dropping a column and then renaming another one to the original columns name
URL: https://github.com/apache/hudi/issues/16665
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Schema on read schema evolution does not support dropping a column and then renaming another one to the original columns name [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on issue #16665:
URL: https://github.com/apache/hudi/issues/16665#issuecomment-4795629603
This appears to be fixed on master by [HUDI-9683 / PR #13672](https://github.com/apache/hudi/commit/f51c36fd64398718c039b26c15767e793b82904e), which reworks `HoodieInternalRowUtils.genUnsafeRowWriter` to consult the rename map first (via `lookupRenamedField`) before looking up the field name in the previous schema. That prevents the case where the new schema's `col1` (renamed from `col2`) was matching the dropped previous `col1`. Coverage was added in `TestHoodieInternalRowUtils` (`Test rewrite row with renamed columns`, `Test rewrite row with columns swap`). Closing as fixed — please reopen if you can still reproduce on master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org