You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@hudi.apache.org by "linliu-code (via GitHub)" <gi...@apache.org> on 2026/08/12 22:58:04 UTC

[PR] [WIP][HUDI] MERGE INTO silently no-ops when the source omits the partition column [hudi]

linliu-code opened a new pull request, #19610:
URL: https://github.com/apache/hudi/pull/19610

   ### Change Logs
   
   **Draft / placeholder** — opened to track the issue upstream and to serve as the companion PR for an internal one. **The fix approach in this branch is known to be incomplete; see "Status" below.**
   
   `MERGE INTO` against a **partitioned** table whose source query does not project the partition column **reports success while writing an empty commit** — 0 bytes, 0 files, 0 records, 0 errors. The target data is left untouched, with no error and no warning.
   
   Reproduced on Spark 3.5, on `hoodie.table.version` both 6 and 9.
   
   ```sql
   CREATE TABLE t (id BIGINT, name STRING, amount DOUBLE, ts BIGINT, dt STRING)
   USING hudi PARTITIONED BY (dt)
   TBLPROPERTIES (type='cow', primaryKey='id', preCombineField='ts');
   INSERT INTO t VALUES (1,'a',10.0,1,'2026-08-11');
   
   -- source projects the record key and ordering field, but NOT `dt`
   MERGE INTO t AS tgt
   USING (SELECT 1L AS id, 15.0 AS amount, 200L AS ts) AS s
   ON tgt.id = s.id
   WHEN MATCHED THEN UPDATE SET tgt.amount = s.amount, tgt.ts = s.ts;
   -- statement succeeds; id=1 is still amount=10.0 / ts=1
   ```
   
   `show_commits` afterwards:
   
   ```
   commit_time         action  total_bytes  files_added  files_updated  partitions  records  errors
   20260811073208749   commit  0            0            0              0           0        0
   ```
   
   Adding `dt` to the **source projection** alone — leaving the `ON` clause matching on the record key only — makes the merge apply correctly. The same statement on a **non-partitioned** table works unmodified. So the trigger is the source projection, not the join condition.
   
   ### Impact
   
   Data-correctness / silent-no-op. A user's `MERGE INTO` reports success and changes nothing; there is no diagnostic distinguishing it from a legitimate no-match.
   
   Note the asymmetry that makes this surprising: omitting the **ordering** field from the source raises a clear `MergeIntoFieldResolutionException`, whereas omitting the **partition** field is silent. In `MergeIntoHoodieTableCommand.checkSchemaMergeIntoCompatibility` the partition-column resolution is wrapped in a `try/catch` that discards `MergeIntoFieldResolutionException`, while the identical failure for the primary key is allowed to propagate.
   
   ### Status — why this is a draft
   
   The change in this branch makes the partition column a hard error, mirroring the primary key. Internal CI shows that is **too strict**: it also rejects the *documented, supported* partial-update pattern, where the source omits the partition column and the update clause does not assign it, e.g. the existing test `Test merge into Allowed-patterns of assignment clauses` (table partitioned by `value`, `USING (SELECT 1 AS id, 1003 AS ts)`, `UPDATE SET h0.id = s0.id`) which runs with `hoodie.datasource.write.merge.into.partial.updates=true`.
   
   So there are evidently (at least) two paths through this code — one where a missing partition column is handled by taking the partition from the matched target record, and one where it silently produces an empty commit. **Which condition separates them is still being investigated**; the partial-updates flag is the leading candidate, since the passing upstream test enables it and the failing production case did not.
   
   Repointing this PR once that is settled. The likely shape is either narrowing the validation to the configuration where the no-op actually occurs, or fixing the write path so the partition is resolved from the target in that configuration too.
   
   ### Impact
   
   Behaviour change for `MERGE INTO` on partitioned tables — currently silent data loss becomes either a clear error or a correct write, depending on the final approach.
   
   ### Risk level: medium
   
   Touches `MergeIntoHoodieTableCommand` analysis-time validation on both the `hudi-spark3-common` and `hudi-spark4-common` copies. Existing `TestMergeIntoTable*` suites are the guard; internal CI has already surfaced one over-strictness regression, which is being addressed.
   
   ### Documentation Update
   
   None required.
   
   ### Contributor's checklist
   
   - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [x] Change Logs and Impact were stated clearly
   - [ ] Adequate tests were added if applicable — tests added (`TestMergeIntoPartitionFieldResolution`, CoW + MOR) but **not yet green**; see Status
   - [ ] CI passed
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org