[I] MOR upserts silently drop records when hoodie.allow.operation.metadata.field is enabled [hudi]

["yihua (via GitHub)" <gi...@apache.org>]

yihua opened a new issue, #20260:
URL: https://github.com/apache/hudi/issues/20260

   ### Bug Description
   
   **What happened:**
   On a MOR table (table versions 6, 8 and 10) with `hoodie.allow.operation.metadata.field=true`, upserts and deletes through the Spark datasource, Spark SQL or the Java client lose every record routed to a log file, and the write still reports success. These writers do not set a `HoodieOperation` on their records, but `HoodieAppendHandle.populateMetadataFields` calls `hoodieRecord.getOperation().getName()` whenever the flag is on, so every record fails with a `NullPointerException`. With `hoodie.write.ignore.failed` at its default `true`, the failure is only recorded on the handle's write status, and `close()` drops that status when no log block was appended. The deltacommit completes without the records, and the next compaction finds no log files. Flink is not affected because it always sets the operation.
   
   **What you expected:**
   The upsert lands in the log file and is visible in snapshot reads, or the write fails.
   
   **Steps to reproduce:**
   1. In spark-shell, write two rows to a new MOR table with the operation field enabled:
      ```scala
      val opts = Map(
        "hoodie.table.name" -> "t",
        "hoodie.datasource.write.table.type" -> "MERGE_ON_READ",
        "hoodie.datasource.write.operation" -> "upsert",
        "hoodie.datasource.write.recordkey.field" -> "id",
        "hoodie.datasource.write.partitionpath.field" -> "part",
        "hoodie.table.ordering.fields" -> "ts",
        "hoodie.allow.operation.metadata.field" -> "true")
      Seq(("1", "a", 1L, "p1"), ("2", "b", 1L, "p1")).toDF("id", "name", "ts", "part")
        .write.format("hudi").options(opts).mode("overwrite").save(path)
      ```
   2. Upsert one of them: `Seq(("1", "a2", 2L, "p1")).toDF("id", "name", "ts", "part").write.format("hudi").options(opts).mode("append").save(path)`. The write succeeds.
   3. Read it back: `spark.read.format("hudi").load(path).select("id", "name").show()` returns `1:a, 2:b` instead of `1:a2, 2:b`. With `hoodie.allow.operation.metadata.field=false` it returns `1:a2, 2:b`.
   
   ### Environment
   
   **Hudi version:** master (1.3.0-SNAPSHOT); the null dereference exists since 0.9.0
   **Query engine:** Spark 3.5 (also the Java client)
   **Relevant configs:** `hoodie.allow.operation.metadata.field=true`, `hoodie.write.ignore.failed=true` (default), MOR table, table versions 6, 8 and 10
   
   ### Logs and Stack Trace
   
   ```
   ERROR HoodieAppendHandle: Error writing record ...
   java.lang.NullPointerException
   	at org.apache.hudi.io.HoodieAppendHandle.populateMetadataFields(HoodieAppendHandle.java:453)
   ```
   
   **Parent issue:** #20257
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org