Re: [I] Do writing stage should shutdown with error when insert failed to reduce user execute time and show error details [hudi]

["nsivabalan (via GitHub)" <gi...@apache.org>]

nsivabalan commented on issue #16340:
URL: https://github.com/apache/hudi/issues/16340#issuecomment-4795586812

   This appears to be addressed in current master by [HUDI-9417 / PR #13307](https://github.com/apache/hudi/pull/13307) (commit `328e259d4a7b`, merged Jun 2025), which introduced a `WriteStatusValidator` framework.
   
   Flow now:
   - `WriteStatus.markFailure(...)` increments `totalErrorRecords` for every failed record.
   - `SparkRDDWriteClient.commit(...)` sums `totalErrorRecords` across statuses and invokes `SparkDataSourceWriteStatusValidator`, which returns `false` whenever `totalErroredRecords > 0` and logs an error summary with the operation type and error count (`DataSourceUtils.java`).
   - `SparkRDDWriteClient` then logs `Exiting early due to errors with write operation` and returns `false` from `commit(...)`.
   - `HoodieSparkSqlWriter.commitAndPerformPostOperations` returns `success=false`, and `InsertIntoHoodieTableCommand` / `MergeIntoHoodieTableCommand` explicitly throw `HoodieException("Insert Into to Hudi table failed")` / `"Merge into Hoodie table command failed"`, which fails the SQL command and the Spark job/stage instead of completing silently.
   
   If you can still reproduce silent success on a recent master/0.16+ build, please share a minimal repro (table props, source/target schema, exact INSERT/MERGE statement, Spark UI screenshot) and we can reopen. Otherwise, this can likely be closed.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org