You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "hudi-bot (via GitHub)" <gi...@apache.org> on 2025/11/30 03:23:58 UTC
[I] preCombine all HoodieRecords and update all fields(which is not DefaultValue) according to orderingVal [hudi]
hudi-bot opened a new issue, #14472:
URL: https://github.com/apache/hudi/issues/14472
When more than one HoodieRecord have the same HoodieKey, this function combines all fields(which is not DefaultValue)
before attempting to insert/upsert (if combining turned on in HoodieClientConfig).
eg: 1)
Before:
id name age ts
1 Karl null 0.0
1 null 18 0.0
After:
id name age ts
1 Karl 18 0.0
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-1284
- Type: New Feature
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] preCombine all HoodieRecords and update all fields(which is not DefaultValue) according to orderingVal [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on issue #14472:
URL: https://github.com/apache/hudi/issues/14472#issuecomment-3630167006
### Linked PR(s)
- https://github.com/apache/hudi/pull/2096
- https://github.com/apache/hudi/pull/2106
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] preCombine all HoodieRecords and update all fields(which is not DefaultValue) according to orderingVal [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on issue #14472:
URL: https://github.com/apache/hudi/issues/14472#issuecomment-4795167300
This feature request appears to be addressed in current master. Two payload classes implement field-level merging across records with the same key:
- `OverwriteNonDefaultsWithLatestAvroPayload` (added in [a1cff8a](https://github.com/apache/hudi/commit/a1cff8abae9d) / HUDI-1255, PR #2056) — picks the latest record by ordering, then merges non-default fields when writing to storage (`combineAndGetUpdateValue`/`mergeRecords`).
- `PartialUpdateAvroPayload` (extends the above) — performs field-level merging directly in `preCombine` (`mergeOldRecord`/`mergeRecords`), so two same-key records in a batch are combined by picking each non-null field.
Applying `PartialUpdateAvroPayload.preCombine` to the example in the issue:
```
before: (id=1, name=Karl, age=null, ts=0.0) + (id=1, name=null, age=18, ts=0.0)
after: (id=1, name=Karl, age=18, ts=0.0)
```
If this matches the intent, the issue can likely be closed. If a different semantic (e.g., merging across more than two values, or treating zero-valued numerics as 'default' the way the original `OverwriteNonDefaultsWithLatestAvroPayload` does via `field.defaultVal()`) is needed, please clarify and we can keep it open.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] preCombine all HoodieRecords and update all fields(which is not DefaultValue) according to orderingVal [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan closed issue #14472: preCombine all HoodieRecords and update all fields(which is not DefaultValue) according to orderingVal
URL: https://github.com/apache/hudi/issues/14472
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org