Re: [I] Spark SQL MIT global index change partition paths will not work with custom merger [hudi]

["rangareddy (via GitHub)" <gi...@apache.org>]

rangareddy commented on issue #17273:
URL: https://github.com/apache/hudi/issues/17273#issuecomment-5365810478

   This issue was reviewed as part of the JIRA-migrated backlog triage (HUDI-8241).
   
   **Findings: needs a reproduction on current `master`; the premise may have shifted.**
   
   The description reads as a known-limitation note rather than a defect report: `MERGE INTO` uses the Avro record merger so `ExpressionPayload` works, merger implementations get overwritten, and therefore a user-supplied custom merger is not honoured when records are read during a cross-partition (global index) update. It also explicitly defers the fix - *"There might be some changes to how we do merger impls so I don't want to address this issue yet."*
   
   Those changes have since happened. The merge layer was rewritten around table versions 8 and 9, with the `BufferedRecordMerger` family replacing payload-based merging, and merge mode became the primary merging config (#11943). Whether the limitation survives in the same shape is not determinable by reading the current code, and asserting either way would be a guess.
   
   To settle it, a `MERGE INTO` test on current `master` with a global index, an update that moves a record across partitions, and a custom `HoodieRecordMerger` - checking whether the custom merger is invoked on the read of the existing record.
   
   Keeping this open.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org