[jira] [Created] (HUDI-9683) Fix spark rename columns for HoodieInternalRowUtils.genUnsafeRowWriter

["Jonathan Vexler (Jira)" <ji...@apache.org>]

Jonathan Vexler created HUDI-9683:
-------------------------------------

             Summary: Fix spark rename columns for HoodieInternalRowUtils.genUnsafeRowWriter
                 Key: HUDI-9683
                 URL: https://issues.apache.org/jira/browse/HUDI-9683
             Project: Apache Hudi
          Issue Type: Bug
          Components: spark
            Reporter: Jonathan Vexler
            Assignee: Jonathan Vexler
             Fix For: 1.1.0


{code:java}
val (fieldWriter, prevFieldPos): (RowFieldUpdater, Int) =
  prevStructType.getFieldIndex(newField.name) match {
    case Some(prevFieldPos) =>
      val prevField = prevStructType(prevFieldPos)
      (newWriterRenaming(prevField.dataType, newField.dataType, renamedColumnsMap, fieldNamesStack), prevFieldPos)

    case None =>
      val newFieldQualifiedName = createFullName(fieldNamesStack)
      val prevFieldName: String = lookupRenamedField(newFieldQualifiedName, renamedColumnsMap)

      // Handle rename
      prevStructType.getFieldIndex(prevFieldName) match {
        case Some(prevFieldPos) =>
          val prevField = prevStructType.fields(prevFieldPos)
          (newWriterRenaming(prevField.dataType, newField.dataType, renamedColumnsMap, fieldNamesStack), prevFieldPos)

        case None =>
          val updater: RowFieldUpdater = (fieldUpdater, ordinal, _) => fieldUpdater.setNullAt(ordinal)
          (updater, -1)
      }
  } {code}
We only look up the rename columns if the column is not found. If a new column has been renamed to the name of an old column then we will not respect the rename



--
This message was sent by Atlassian Jira
(v8.20.10#820010)