You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "the-other-tim-brown (via GitHub)" <gi...@apache.org> on 2025/08/20 19:45:32 UTC
[PR] [DRAFT] Custom merger update [hudi]
the-other-tim-brown opened a new pull request, #13742:
URL: https://github.com/apache/hudi/pull/13742
### Change Logs
_Describe context and summary for this change. Highlight if any code was copied._
### Impact
_Describe any public API or user-facing feature change or any performance impact._
### Risk level (write none, low medium or high below)
_If medium or high, explain what verification was done to mitigate the risks._
### Documentation Update
_Describe any necessary documentation update if there is any new feature, config, or user-facing change. If not, put "none"._
- _The config description must be updated if new configs are added or the default value of the configs are changed_
- _Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
ticket number here and follow the [instruction](https://hudi.apache.org/contribute/developer-setup#website) to make
changes to the website._
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9715] Custom merger update [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13742:
URL: https://github.com/apache/hudi/pull/13742#issuecomment-3235473818
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7988",
"triggerID" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"triggerType" : "PUSH"
}, {
"hash" : "ecb694295f7a88371f33df6de60370058670439e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7992",
"triggerID" : "ecb694295f7a88371f33df6de60370058670439e",
"triggerType" : "PUSH"
}, {
"hash" : "efd7462406be4f6d90dd5e42f152d66f683ebdf4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "efd7462406be4f6d90dd5e42f152d66f683ebdf4",
"triggerType" : "PUSH"
}, {
"hash" : "3c67160d5ceec883db5d826ff655865a3209f2f3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7994",
"triggerID" : "3c67160d5ceec883db5d826ff655865a3209f2f3",
"triggerType" : "PUSH"
}, {
"hash" : "f4f84d9f6a480923df27cfe5553d8aa65b2bbb7a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7996",
"triggerID" : "f4f84d9f6a480923df27cfe5553d8aa65b2bbb7a",
"triggerType" : "PUSH"
}, {
"hash" : "521b26bc2c60b7921e2f7aa5fffa620ab7f9a096",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "521b26bc2c60b7921e2f7aa5fffa620ab7f9a096",
"triggerType" : "PUSH"
}, {
"hash" : "22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50",
"triggerType" : "PUSH"
}, {
"hash" : "02a319ba884595a4e3b5453f366b660c1c11f767",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8007",
"triggerID" : "02a319ba884595a4e3b5453f366b660c1c11f767",
"triggerType" : "PUSH"
}, {
"hash" : "3e59b6f8b486016911c9ad14c583f6538cc036ac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8009",
"triggerID" : "3e59b6f8b486016911c9ad14c583f6538cc036ac",
"triggerType" : "PUSH"
}, {
"hash" : "003f1c81efedafa0ddd78868ce83357d8803c5e5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "003f1c81efedafa0ddd78868ce83357d8803c5e5",
"triggerType" : "PUSH"
}, {
"hash" : "40c04768d5233a38be1b4c01ce895189a4af4c3a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8013",
"triggerID" : "40c04768d5233a38be1b4c01ce895189a4af4c3a",
"triggerType" : "PUSH"
}, {
"hash" : "4a699a8dcf79e61ea3bb754e67e9cc4a32e86aa1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8084",
"triggerID" : "4a699a8dcf79e61ea3bb754e67e9cc4a32e86aa1",
"triggerType" : "PUSH"
}, {
"hash" : "fe55d8ec4923cb6c64e83ae4f32217c72599a398",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "fe55d8ec4923cb6c64e83ae4f32217c72599a398",
"triggerType" : "PUSH"
}, {
"hash" : "87cd79787dc8126b6877ba0d3407cfe8f343933a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8087",
"triggerID" : "87cd79787dc8126b6877ba0d3407cfe8f343933a",
"triggerType" : "PUSH"
}, {
"hash" : "594874ffdca647cd268b3947fa4680c57c53d679",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8088",
"triggerID" : "594874ffdca647cd268b3947fa4680c57c53d679",
"triggerType" : "PUSH"
}, {
"hash" : "7b95754f0df2c1a087a6a28722a7471ee12cecf3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8090",
"triggerID" : "7b95754f0df2c1a087a6a28722a7471ee12cecf3",
"triggerType" : "PUSH"
}, {
"hash" : "789c2da879ca66f6b9be71d1814be38b5d8cfb58",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8102",
"triggerID" : "789c2da879ca66f6b9be71d1814be38b5d8cfb58",
"triggerType" : "PUSH"
}, {
"hash" : "63f33fae3e22a9f80de51b2a6e1ed0427f631d0b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8103",
"triggerID" : "63f33fae3e22a9f80de51b2a6e1ed0427f631d0b",
"triggerType" : "PUSH"
}, {
"hash" : "e04c466fa81e9bcfaabe36825127098f5d1bb890",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8224",
"triggerID" : "e04c466fa81e9bcfaabe36825127098f5d1bb890",
"triggerType" : "PUSH"
}, {
"hash" : "cf574f46a1783a39b2074542aad9539ec3c4f36a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8235",
"triggerID" : "cf574f46a1783a39b2074542aad9539ec3c4f36a",
"triggerType" : "PUSH"
}, {
"hash" : "2eb81e923adfa77176f137c442143deaac9e9916",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2eb81e923adfa77176f137c442143deaac9e9916",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* efd7462406be4f6d90dd5e42f152d66f683ebdf4 UNKNOWN
* 521b26bc2c60b7921e2f7aa5fffa620ab7f9a096 UNKNOWN
* 22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50 UNKNOWN
* 003f1c81efedafa0ddd78868ce83357d8803c5e5 UNKNOWN
* fe55d8ec4923cb6c64e83ae4f32217c72599a398 UNKNOWN
* cf574f46a1783a39b2074542aad9539ec3c4f36a Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8235)
* 2eb81e923adfa77176f137c442143deaac9e9916 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9715] Custom merger update [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13742:
URL: https://github.com/apache/hudi/pull/13742#issuecomment-3235545285
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7988",
"triggerID" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"triggerType" : "PUSH"
}, {
"hash" : "ecb694295f7a88371f33df6de60370058670439e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7992",
"triggerID" : "ecb694295f7a88371f33df6de60370058670439e",
"triggerType" : "PUSH"
}, {
"hash" : "efd7462406be4f6d90dd5e42f152d66f683ebdf4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "efd7462406be4f6d90dd5e42f152d66f683ebdf4",
"triggerType" : "PUSH"
}, {
"hash" : "3c67160d5ceec883db5d826ff655865a3209f2f3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7994",
"triggerID" : "3c67160d5ceec883db5d826ff655865a3209f2f3",
"triggerType" : "PUSH"
}, {
"hash" : "f4f84d9f6a480923df27cfe5553d8aa65b2bbb7a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7996",
"triggerID" : "f4f84d9f6a480923df27cfe5553d8aa65b2bbb7a",
"triggerType" : "PUSH"
}, {
"hash" : "521b26bc2c60b7921e2f7aa5fffa620ab7f9a096",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "521b26bc2c60b7921e2f7aa5fffa620ab7f9a096",
"triggerType" : "PUSH"
}, {
"hash" : "22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50",
"triggerType" : "PUSH"
}, {
"hash" : "02a319ba884595a4e3b5453f366b660c1c11f767",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8007",
"triggerID" : "02a319ba884595a4e3b5453f366b660c1c11f767",
"triggerType" : "PUSH"
}, {
"hash" : "3e59b6f8b486016911c9ad14c583f6538cc036ac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8009",
"triggerID" : "3e59b6f8b486016911c9ad14c583f6538cc036ac",
"triggerType" : "PUSH"
}, {
"hash" : "003f1c81efedafa0ddd78868ce83357d8803c5e5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "003f1c81efedafa0ddd78868ce83357d8803c5e5",
"triggerType" : "PUSH"
}, {
"hash" : "40c04768d5233a38be1b4c01ce895189a4af4c3a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8013",
"triggerID" : "40c04768d5233a38be1b4c01ce895189a4af4c3a",
"triggerType" : "PUSH"
}, {
"hash" : "4a699a8dcf79e61ea3bb754e67e9cc4a32e86aa1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8084",
"triggerID" : "4a699a8dcf79e61ea3bb754e67e9cc4a32e86aa1",
"triggerType" : "PUSH"
}, {
"hash" : "fe55d8ec4923cb6c64e83ae4f32217c72599a398",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "fe55d8ec4923cb6c64e83ae4f32217c72599a398",
"triggerType" : "PUSH"
}, {
"hash" : "87cd79787dc8126b6877ba0d3407cfe8f343933a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8087",
"triggerID" : "87cd79787dc8126b6877ba0d3407cfe8f343933a",
"triggerType" : "PUSH"
}, {
"hash" : "594874ffdca647cd268b3947fa4680c57c53d679",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8088",
"triggerID" : "594874ffdca647cd268b3947fa4680c57c53d679",
"triggerType" : "PUSH"
}, {
"hash" : "7b95754f0df2c1a087a6a28722a7471ee12cecf3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8090",
"triggerID" : "7b95754f0df2c1a087a6a28722a7471ee12cecf3",
"triggerType" : "PUSH"
}, {
"hash" : "789c2da879ca66f6b9be71d1814be38b5d8cfb58",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8102",
"triggerID" : "789c2da879ca66f6b9be71d1814be38b5d8cfb58",
"triggerType" : "PUSH"
}, {
"hash" : "63f33fae3e22a9f80de51b2a6e1ed0427f631d0b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8103",
"triggerID" : "63f33fae3e22a9f80de51b2a6e1ed0427f631d0b",
"triggerType" : "PUSH"
}, {
"hash" : "e04c466fa81e9bcfaabe36825127098f5d1bb890",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8224",
"triggerID" : "e04c466fa81e9bcfaabe36825127098f5d1bb890",
"triggerType" : "PUSH"
}, {
"hash" : "cf574f46a1783a39b2074542aad9539ec3c4f36a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8235",
"triggerID" : "cf574f46a1783a39b2074542aad9539ec3c4f36a",
"triggerType" : "PUSH"
}, {
"hash" : "2eb81e923adfa77176f137c442143deaac9e9916",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2eb81e923adfa77176f137c442143deaac9e9916",
"triggerType" : "PUSH"
}, {
"hash" : "12dc3e3eaf746bb1f2bc3f4e93fa14d9adb0f6fd",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8244",
"triggerID" : "12dc3e3eaf746bb1f2bc3f4e93fa14d9adb0f6fd",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* efd7462406be4f6d90dd5e42f152d66f683ebdf4 UNKNOWN
* 521b26bc2c60b7921e2f7aa5fffa620ab7f9a096 UNKNOWN
* 22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50 UNKNOWN
* 003f1c81efedafa0ddd78868ce83357d8803c5e5 UNKNOWN
* fe55d8ec4923cb6c64e83ae4f32217c72599a398 UNKNOWN
* cf574f46a1783a39b2074542aad9539ec3c4f36a Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8235)
* 2eb81e923adfa77176f137c442143deaac9e9916 UNKNOWN
* 12dc3e3eaf746bb1f2bc3f4e93fa14d9adb0f6fd Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8244)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9715] Custom merger update [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13742:
URL: https://github.com/apache/hudi/pull/13742#discussion_r2299957029
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/BufferedRecordMergerFactory.java:
##########
@@ -465,46 +451,45 @@ public CustomPayloadRecordMerger(
}
@Override
- public Option<BufferedRecord<T>> deltaMergeNonDeleteRecord(BufferedRecord<T> newRecord, BufferedRecord<T> existingRecord) throws IOException {
- Option<Pair<HoodieRecord, Schema>> mergedRecordAndSchema = getMergedRecord(existingRecord, newRecord, false);
- if (mergedRecordAndSchema.isEmpty()) {
- // An empty Option indicates that the output represents a delete.
- return Option.of(new BufferedRecord<>(newRecord.getRecordKey(), OrderingValues.getDefault(), null, null, HoodieOperation.DELETE));
- }
- HoodieRecord mergedRecord = mergedRecordAndSchema.get().getLeft();
- Schema mergeResultSchema = mergedRecordAndSchema.get().getRight();
+ public Option<BufferedRecord<T>> deltaMergeRecords(BufferedRecord<T> newRecord, BufferedRecord<T> existingRecord) throws IOException {
+ Pair<HoodieRecord, Schema> mergedRecordAndSchema = getMergedRecord(existingRecord, newRecord, false);
+ HoodieRecord mergedRecord = mergedRecordAndSchema.getLeft();
+ Schema mergeResultSchema = mergedRecordAndSchema.getRight();
// Special handling for SENTINEL record in Expression Payload. This is returned if the condition does not match.
if (mergedRecord.getData() == HoodieRecord.SENTINEL) {
return Option.empty();
}
- T combinedRecordData = recordContext.convertAvroRecord(mergedRecord.toIndexedRecord(mergeResultSchema, props).get().getData());
+ Option<T> combinedRecordData = (mergedRecord.toIndexedRecord(mergeResultSchema, props).map(indexedRecord -> recordContext.convertAvroRecord(indexedRecord.getData())));
// If pre-combine does not return existing record, update it
- if (combinedRecordData != existingRecord.getRecord()) {
+ if (combinedRecordData.map(record -> record != existingRecord.getRecord()).orElse(true)) {
// For pkless we need to use record key from existing record
- return Option.of(BufferedRecords.fromEngineRecord(combinedRecordData, mergeResultSchema, recordContext, orderingFieldNames,
- existingRecord.getRecordKey(), mergedRecord.isDelete(mergeResultSchema, props)));
+ boolean isDelete = mergedRecord.isDelete(readerSchema, props);
+ Comparable orderingValue = mergedRecord.getOrderingValue(mergeResultSchema, props, orderingFieldNames);
+ T mergedEngineRecord = mergedRecord.toIndexedRecord(mergeResultSchema, props)
Review Comment:
why convert to avro record then to engine record? `#convertAvroRecord` already returns engine record?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9715] Custom merger update [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13742:
URL: https://github.com/apache/hudi/pull/13742#discussion_r2302483908
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/BufferedRecordMergerFactory.java:
##########
@@ -465,46 +451,45 @@ public CustomPayloadRecordMerger(
}
@Override
- public Option<BufferedRecord<T>> deltaMergeNonDeleteRecord(BufferedRecord<T> newRecord, BufferedRecord<T> existingRecord) throws IOException {
- Option<Pair<HoodieRecord, Schema>> mergedRecordAndSchema = getMergedRecord(existingRecord, newRecord, false);
- if (mergedRecordAndSchema.isEmpty()) {
- // An empty Option indicates that the output represents a delete.
- return Option.of(new BufferedRecord<>(newRecord.getRecordKey(), OrderingValues.getDefault(), null, null, HoodieOperation.DELETE));
- }
- HoodieRecord mergedRecord = mergedRecordAndSchema.get().getLeft();
- Schema mergeResultSchema = mergedRecordAndSchema.get().getRight();
+ public Option<BufferedRecord<T>> deltaMergeRecords(BufferedRecord<T> newRecord, BufferedRecord<T> existingRecord) throws IOException {
+ Pair<HoodieRecord, Schema> mergedRecordAndSchema = getMergedRecord(existingRecord, newRecord, false);
+ HoodieRecord mergedRecord = mergedRecordAndSchema.getLeft();
+ Schema mergeResultSchema = mergedRecordAndSchema.getRight();
// Special handling for SENTINEL record in Expression Payload. This is returned if the condition does not match.
if (mergedRecord.getData() == HoodieRecord.SENTINEL) {
return Option.empty();
}
- T combinedRecordData = recordContext.convertAvroRecord(mergedRecord.toIndexedRecord(mergeResultSchema, props).get().getData());
+ Option<T> combinedRecordData = (mergedRecord.toIndexedRecord(mergeResultSchema, props).map(indexedRecord -> recordContext.convertAvroRecord(indexedRecord.getData())));
// If pre-combine does not return existing record, update it
- if (combinedRecordData != existingRecord.getRecord()) {
+ if (combinedRecordData.map(record -> record != existingRecord.getRecord()).orElse(true)) {
// For pkless we need to use record key from existing record
- return Option.of(BufferedRecords.fromEngineRecord(combinedRecordData, mergeResultSchema, recordContext, orderingFieldNames,
- existingRecord.getRecordKey(), mergedRecord.isDelete(mergeResultSchema, props)));
+ boolean isDelete = mergedRecord.isDelete(readerSchema, props);
+ Comparable orderingValue = mergedRecord.getOrderingValue(mergeResultSchema, props, orderingFieldNames);
+ T mergedEngineRecord = mergedRecord.toIndexedRecord(mergeResultSchema, props)
+ .map(hoodieAvroIndexedRecord -> recordContext.convertAvroRecord(hoodieAvroIndexedRecord.getData()))
+ .orElse(null);
+ return Option.of(BufferedRecords.fromEngineRecord(mergedEngineRecord, readerSchema, recordContext, orderingValue, existingRecord.getRecordKey(), isDelete));
+
}
return Option.empty();
}
@Override
- public BufferedRecord<T> mergeNonDeleteRecord(BufferedRecord<T> olderRecord, BufferedRecord<T> newerRecord) throws IOException {
- Option<Pair<HoodieRecord, Schema>> mergedRecordAndSchema = getMergedRecord(olderRecord, newerRecord, true);
- if (mergedRecordAndSchema.isEmpty()) {
- return BufferedRecords.createDelete(newerRecord.getRecordKey());
- }
- HoodieRecord mergedRecord = mergedRecordAndSchema.get().getLeft();
- Schema mergeResultSchema = mergedRecordAndSchema.get().getRight();
+ public BufferedRecord<T> mergeRecords(BufferedRecord<T> olderRecord, BufferedRecord<T> newerRecord) throws IOException {
+ Pair<HoodieRecord, Schema> mergedRecordAndSchema = getMergedRecord(olderRecord, newerRecord, true);
+
+ HoodieRecord mergedRecord = mergedRecordAndSchema.getLeft();
+ Schema mergeResultSchema = mergedRecordAndSchema.getRight();
// Special handling for SENTINEL record in Expression Payload
if (mergedRecord.getData() == HoodieRecord.SENTINEL) {
return olderRecord;
}
- if (!mergedRecord.isDelete(mergeResultSchema, props)) {
- IndexedRecord indexedRecord = (IndexedRecord) mergedRecord.getData();
- return BufferedRecords.fromEngineRecord(
- recordContext.convertAvroRecord(indexedRecord), mergeResultSchema, recordContext, orderingFieldNames, newerRecord.getRecordKey(), false);
- }
- return BufferedRecords.createDelete(newerRecord.getRecordKey());
+ boolean isDelete = mergedRecord.isDelete(readerSchema, props);
Review Comment:
This should be `mergeResultSchema`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9715] Custom merger update [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13742:
URL: https://github.com/apache/hudi/pull/13742#discussion_r2302487604
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/BufferedRecordMergerFactory.java:
##########
@@ -465,46 +451,45 @@ public CustomPayloadRecordMerger(
}
@Override
- public Option<BufferedRecord<T>> deltaMergeNonDeleteRecord(BufferedRecord<T> newRecord, BufferedRecord<T> existingRecord) throws IOException {
- Option<Pair<HoodieRecord, Schema>> mergedRecordAndSchema = getMergedRecord(existingRecord, newRecord, false);
- if (mergedRecordAndSchema.isEmpty()) {
- // An empty Option indicates that the output represents a delete.
- return Option.of(new BufferedRecord<>(newRecord.getRecordKey(), OrderingValues.getDefault(), null, null, HoodieOperation.DELETE));
- }
- HoodieRecord mergedRecord = mergedRecordAndSchema.get().getLeft();
- Schema mergeResultSchema = mergedRecordAndSchema.get().getRight();
+ public Option<BufferedRecord<T>> deltaMergeRecords(BufferedRecord<T> newRecord, BufferedRecord<T> existingRecord) throws IOException {
+ Pair<HoodieRecord, Schema> mergedRecordAndSchema = getMergedRecord(existingRecord, newRecord, false);
+ HoodieRecord mergedRecord = mergedRecordAndSchema.getLeft();
+ Schema mergeResultSchema = mergedRecordAndSchema.getRight();
// Special handling for SENTINEL record in Expression Payload. This is returned if the condition does not match.
if (mergedRecord.getData() == HoodieRecord.SENTINEL) {
return Option.empty();
}
- T combinedRecordData = recordContext.convertAvroRecord(mergedRecord.toIndexedRecord(mergeResultSchema, props).get().getData());
+ Option<T> combinedRecordData = (mergedRecord.toIndexedRecord(mergeResultSchema, props).map(indexedRecord -> recordContext.convertAvroRecord(indexedRecord.getData())));
Review Comment:
yes it is possible, it would just require the whole method to be overridden instead of sharing the code
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9715] Custom merger update [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13742:
URL: https://github.com/apache/hudi/pull/13742#issuecomment-3234483882
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7988",
"triggerID" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"triggerType" : "PUSH"
}, {
"hash" : "ecb694295f7a88371f33df6de60370058670439e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7992",
"triggerID" : "ecb694295f7a88371f33df6de60370058670439e",
"triggerType" : "PUSH"
}, {
"hash" : "efd7462406be4f6d90dd5e42f152d66f683ebdf4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "efd7462406be4f6d90dd5e42f152d66f683ebdf4",
"triggerType" : "PUSH"
}, {
"hash" : "3c67160d5ceec883db5d826ff655865a3209f2f3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7994",
"triggerID" : "3c67160d5ceec883db5d826ff655865a3209f2f3",
"triggerType" : "PUSH"
}, {
"hash" : "f4f84d9f6a480923df27cfe5553d8aa65b2bbb7a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7996",
"triggerID" : "f4f84d9f6a480923df27cfe5553d8aa65b2bbb7a",
"triggerType" : "PUSH"
}, {
"hash" : "521b26bc2c60b7921e2f7aa5fffa620ab7f9a096",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "521b26bc2c60b7921e2f7aa5fffa620ab7f9a096",
"triggerType" : "PUSH"
}, {
"hash" : "22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50",
"triggerType" : "PUSH"
}, {
"hash" : "02a319ba884595a4e3b5453f366b660c1c11f767",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8007",
"triggerID" : "02a319ba884595a4e3b5453f366b660c1c11f767",
"triggerType" : "PUSH"
}, {
"hash" : "3e59b6f8b486016911c9ad14c583f6538cc036ac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8009",
"triggerID" : "3e59b6f8b486016911c9ad14c583f6538cc036ac",
"triggerType" : "PUSH"
}, {
"hash" : "003f1c81efedafa0ddd78868ce83357d8803c5e5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "003f1c81efedafa0ddd78868ce83357d8803c5e5",
"triggerType" : "PUSH"
}, {
"hash" : "40c04768d5233a38be1b4c01ce895189a4af4c3a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8013",
"triggerID" : "40c04768d5233a38be1b4c01ce895189a4af4c3a",
"triggerType" : "PUSH"
}, {
"hash" : "4a699a8dcf79e61ea3bb754e67e9cc4a32e86aa1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8084",
"triggerID" : "4a699a8dcf79e61ea3bb754e67e9cc4a32e86aa1",
"triggerType" : "PUSH"
}, {
"hash" : "fe55d8ec4923cb6c64e83ae4f32217c72599a398",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "fe55d8ec4923cb6c64e83ae4f32217c72599a398",
"triggerType" : "PUSH"
}, {
"hash" : "87cd79787dc8126b6877ba0d3407cfe8f343933a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8087",
"triggerID" : "87cd79787dc8126b6877ba0d3407cfe8f343933a",
"triggerType" : "PUSH"
}, {
"hash" : "594874ffdca647cd268b3947fa4680c57c53d679",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8088",
"triggerID" : "594874ffdca647cd268b3947fa4680c57c53d679",
"triggerType" : "PUSH"
}, {
"hash" : "7b95754f0df2c1a087a6a28722a7471ee12cecf3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8090",
"triggerID" : "7b95754f0df2c1a087a6a28722a7471ee12cecf3",
"triggerType" : "PUSH"
}, {
"hash" : "789c2da879ca66f6b9be71d1814be38b5d8cfb58",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8102",
"triggerID" : "789c2da879ca66f6b9be71d1814be38b5d8cfb58",
"triggerType" : "PUSH"
}, {
"hash" : "63f33fae3e22a9f80de51b2a6e1ed0427f631d0b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8103",
"triggerID" : "63f33fae3e22a9f80de51b2a6e1ed0427f631d0b",
"triggerType" : "PUSH"
}, {
"hash" : "e04c466fa81e9bcfaabe36825127098f5d1bb890",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e04c466fa81e9bcfaabe36825127098f5d1bb890",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* efd7462406be4f6d90dd5e42f152d66f683ebdf4 UNKNOWN
* 521b26bc2c60b7921e2f7aa5fffa620ab7f9a096 UNKNOWN
* 22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50 UNKNOWN
* 003f1c81efedafa0ddd78868ce83357d8803c5e5 UNKNOWN
* fe55d8ec4923cb6c64e83ae4f32217c72599a398 UNKNOWN
* 63f33fae3e22a9f80de51b2a6e1ed0427f631d0b Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8103)
* e04c466fa81e9bcfaabe36825127098f5d1bb890 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT] Custom merger update [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13742:
URL: https://github.com/apache/hudi/pull/13742#issuecomment-3207881742
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7988",
"triggerID" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a9251b050ad82bceba70f95140ec4f1c5ac5a594 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7988)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT] Custom merger update [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13742:
URL: https://github.com/apache/hudi/pull/13742#issuecomment-3210855712
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7988",
"triggerID" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"triggerType" : "PUSH"
}, {
"hash" : "ecb694295f7a88371f33df6de60370058670439e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7992",
"triggerID" : "ecb694295f7a88371f33df6de60370058670439e",
"triggerType" : "PUSH"
}, {
"hash" : "efd7462406be4f6d90dd5e42f152d66f683ebdf4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "efd7462406be4f6d90dd5e42f152d66f683ebdf4",
"triggerType" : "PUSH"
}, {
"hash" : "3c67160d5ceec883db5d826ff655865a3209f2f3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7994",
"triggerID" : "3c67160d5ceec883db5d826ff655865a3209f2f3",
"triggerType" : "PUSH"
}, {
"hash" : "f4f84d9f6a480923df27cfe5553d8aa65b2bbb7a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7996",
"triggerID" : "f4f84d9f6a480923df27cfe5553d8aa65b2bbb7a",
"triggerType" : "PUSH"
}, {
"hash" : "521b26bc2c60b7921e2f7aa5fffa620ab7f9a096",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "521b26bc2c60b7921e2f7aa5fffa620ab7f9a096",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* efd7462406be4f6d90dd5e42f152d66f683ebdf4 UNKNOWN
* f4f84d9f6a480923df27cfe5553d8aa65b2bbb7a Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7996)
* 521b26bc2c60b7921e2f7aa5fffa620ab7f9a096 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT] Custom merger update [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13742:
URL: https://github.com/apache/hudi/pull/13742#issuecomment-3208655963
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7988",
"triggerID" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"triggerType" : "PUSH"
}, {
"hash" : "ecb694295f7a88371f33df6de60370058670439e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7992",
"triggerID" : "ecb694295f7a88371f33df6de60370058670439e",
"triggerType" : "PUSH"
}, {
"hash" : "efd7462406be4f6d90dd5e42f152d66f683ebdf4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "efd7462406be4f6d90dd5e42f152d66f683ebdf4",
"triggerType" : "PUSH"
}, {
"hash" : "3c67160d5ceec883db5d826ff655865a3209f2f3",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7994",
"triggerID" : "3c67160d5ceec883db5d826ff655865a3209f2f3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ecb694295f7a88371f33df6de60370058670439e Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7992)
* efd7462406be4f6d90dd5e42f152d66f683ebdf4 UNKNOWN
* 3c67160d5ceec883db5d826ff655865a3209f2f3 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7994)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT] Custom merger update [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13742:
URL: https://github.com/apache/hudi/pull/13742#issuecomment-3210909142
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7988",
"triggerID" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"triggerType" : "PUSH"
}, {
"hash" : "ecb694295f7a88371f33df6de60370058670439e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7992",
"triggerID" : "ecb694295f7a88371f33df6de60370058670439e",
"triggerType" : "PUSH"
}, {
"hash" : "efd7462406be4f6d90dd5e42f152d66f683ebdf4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "efd7462406be4f6d90dd5e42f152d66f683ebdf4",
"triggerType" : "PUSH"
}, {
"hash" : "3c67160d5ceec883db5d826ff655865a3209f2f3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7994",
"triggerID" : "3c67160d5ceec883db5d826ff655865a3209f2f3",
"triggerType" : "PUSH"
}, {
"hash" : "f4f84d9f6a480923df27cfe5553d8aa65b2bbb7a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7996",
"triggerID" : "f4f84d9f6a480923df27cfe5553d8aa65b2bbb7a",
"triggerType" : "PUSH"
}, {
"hash" : "521b26bc2c60b7921e2f7aa5fffa620ab7f9a096",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "521b26bc2c60b7921e2f7aa5fffa620ab7f9a096",
"triggerType" : "PUSH"
}, {
"hash" : "22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* efd7462406be4f6d90dd5e42f152d66f683ebdf4 UNKNOWN
* f4f84d9f6a480923df27cfe5553d8aa65b2bbb7a Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7996)
* 521b26bc2c60b7921e2f7aa5fffa620ab7f9a096 UNKNOWN
* 22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT] Custom merger update [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13742:
URL: https://github.com/apache/hudi/pull/13742#issuecomment-3211517107
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7988",
"triggerID" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"triggerType" : "PUSH"
}, {
"hash" : "ecb694295f7a88371f33df6de60370058670439e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7992",
"triggerID" : "ecb694295f7a88371f33df6de60370058670439e",
"triggerType" : "PUSH"
}, {
"hash" : "efd7462406be4f6d90dd5e42f152d66f683ebdf4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "efd7462406be4f6d90dd5e42f152d66f683ebdf4",
"triggerType" : "PUSH"
}, {
"hash" : "3c67160d5ceec883db5d826ff655865a3209f2f3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7994",
"triggerID" : "3c67160d5ceec883db5d826ff655865a3209f2f3",
"triggerType" : "PUSH"
}, {
"hash" : "f4f84d9f6a480923df27cfe5553d8aa65b2bbb7a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7996",
"triggerID" : "f4f84d9f6a480923df27cfe5553d8aa65b2bbb7a",
"triggerType" : "PUSH"
}, {
"hash" : "521b26bc2c60b7921e2f7aa5fffa620ab7f9a096",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "521b26bc2c60b7921e2f7aa5fffa620ab7f9a096",
"triggerType" : "PUSH"
}, {
"hash" : "22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50",
"triggerType" : "PUSH"
}, {
"hash" : "02a319ba884595a4e3b5453f366b660c1c11f767",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8007",
"triggerID" : "02a319ba884595a4e3b5453f366b660c1c11f767",
"triggerType" : "PUSH"
}, {
"hash" : "3e59b6f8b486016911c9ad14c583f6538cc036ac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8009",
"triggerID" : "3e59b6f8b486016911c9ad14c583f6538cc036ac",
"triggerType" : "PUSH"
}, {
"hash" : "003f1c81efedafa0ddd78868ce83357d8803c5e5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "003f1c81efedafa0ddd78868ce83357d8803c5e5",
"triggerType" : "PUSH"
}, {
"hash" : "40c04768d5233a38be1b4c01ce895189a4af4c3a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8013",
"triggerID" : "40c04768d5233a38be1b4c01ce895189a4af4c3a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* efd7462406be4f6d90dd5e42f152d66f683ebdf4 UNKNOWN
* 521b26bc2c60b7921e2f7aa5fffa620ab7f9a096 UNKNOWN
* 22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50 UNKNOWN
* 003f1c81efedafa0ddd78868ce83357d8803c5e5 UNKNOWN
* 40c04768d5233a38be1b4c01ce895189a4af4c3a Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8013)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9715] Custom merger update [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13742:
URL: https://github.com/apache/hudi/pull/13742#discussion_r2302493064
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/HoodieSparkRecordMerger.java:
##########
@@ -39,33 +41,33 @@ public HoodieRecord.HoodieRecordType getRecordType() {
* Basic handling of deletes that is used by many of the spark mergers
* returns null if merger specific logic should be used
*/
- protected Option<Pair<HoodieRecord, Schema>> handleDeletes(HoodieRecord older, Schema oldSchema, HoodieRecord newer, Schema newSchema, TypedProperties props) {
+ protected Pair<HoodieRecord, Schema> handleDeletes(HoodieRecord older, Schema oldSchema, HoodieRecord newer, Schema newSchema, TypedProperties props) {
ValidationUtils.checkArgument(older.getRecordType() == HoodieRecord.HoodieRecordType.SPARK);
ValidationUtils.checkArgument(newer.getRecordType() == HoodieRecord.HoodieRecordType.SPARK);
if (newer instanceof HoodieSparkRecord) {
HoodieSparkRecord newSparkRecord = (HoodieSparkRecord) newer;
if (newSparkRecord.isDelete(newSchema, props)) {
// Delete record
- return Option.empty();
+ return Pair.of(new HoodieEmptyRecord<>(newer.getKey(), HoodieOperation.DELETE, OrderingValues.getDefault(), HoodieRecord.HoodieRecordType.SPARK), newSchema);
Review Comment:
We can take this on as part of this PR if we want. These older mergers are deprecated now in favor of MergeModes so it will be good to get them aligned once and for all.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9715] Custom merger update [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13742:
URL: https://github.com/apache/hudi/pull/13742#discussion_r2302685967
##########
hudi-common/src/main/java/org/apache/hudi/common/model/HoodieRecordMerger.java:
##########
@@ -63,8 +63,15 @@ public interface HoodieRecordMerger extends Serializable {
* It'd be associative operation: f(a, f(b, c)) = f(f(a, b), c) (which we can translate as having 3 versions A, B, C
* of the single record, both orders of operations applications have to yield the same result)
* This method takes only full records for merging.
+ *
+ * @param older Older record in terms of commit time ordering.
+ * @param oldSchema The schema of the older record.
+ * @param newer Newer record in terms of commit time ordering.
+ * @param newSchema The schema of the newer record.
+ * @param props The additional properties for the merging operation.
+ * @return The merged record and schema. The record is expected to be non-null. If the record represents a deletion, the operation must be set as {@link HoodieOperation#DELETE}.
*/
- Option<Pair<HoodieRecord, Schema>> merge(HoodieRecord older, Schema oldSchema, HoodieRecord newer, Schema newSchema, TypedProperties props) throws IOException;
+ Pair<HoodieRecord, Schema> merge(HoodieRecord older, Schema oldSchema, HoodieRecord newer, Schema newSchema, TypedProperties props) throws IOException;
Review Comment:
> there is no way to currently pass in the payload information to the HoodieAvroRecordMerger
We can, just let the T in BufferedRecord be the payload instance? For HoodieRecord, we did the similiar thing, the T can be eigher engine row or a payload instance. If we assume all the buffered records takes payload instance for payload based merging, it should work. It's just that we can not have a generic type T on the params `BufferedRecord`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9715] Custom merger update [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13742:
URL: https://github.com/apache/hudi/pull/13742#discussion_r2302468926
##########
hudi-common/src/main/java/org/apache/hudi/common/model/HoodieRecordMerger.java:
##########
@@ -63,8 +63,15 @@ public interface HoodieRecordMerger extends Serializable {
* It'd be associative operation: f(a, f(b, c)) = f(f(a, b), c) (which we can translate as having 3 versions A, B, C
* of the single record, both orders of operations applications have to yield the same result)
* This method takes only full records for merging.
+ *
+ * @param older Older record in terms of commit time ordering.
+ * @param oldSchema The schema of the older record.
+ * @param newer Newer record in terms of commit time ordering.
+ * @param newSchema The schema of the newer record.
+ * @param props The additional properties for the merging operation.
+ * @return The merged record and schema. The record is expected to be non-null. If the record represents a deletion, the operation must be set as {@link HoodieOperation#DELETE}.
*/
- Option<Pair<HoodieRecord, Schema>> merge(HoodieRecord older, Schema oldSchema, HoodieRecord newer, Schema newSchema, TypedProperties props) throws IOException;
+ Pair<HoodieRecord, Schema> merge(HoodieRecord older, Schema oldSchema, HoodieRecord newer, Schema newSchema, TypedProperties props) throws IOException;
Review Comment:
If others are ok with a larger API change, I really like the idea of just forcing the user to define their merger in terms of BufferedRecord and leverage the RecordContext. This will allow them to ensure that their merging will work with any record context and be support in any of the engines we support (hive/flink/spark/java).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9715] Custom merger update [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13742:
URL: https://github.com/apache/hudi/pull/13742#discussion_r2299911083
##########
hudi-common/src/main/java/org/apache/hudi/common/model/DefaultHoodieRecordPayload.java:
##########
@@ -74,11 +74,7 @@ public OverwriteWithLatestAvroPayload preCombine(OverwriteWithLatestAvroPayload
@Override
public Option<IndexedRecord> combineAndGetUpdateValue(IndexedRecord currentValue, Schema schema, Properties properties) throws IOException {
- if (recordBytes.length == 0) {
- return Option.empty();
- }
-
- GenericRecord incomingRecord = HoodieAvroUtils.bytesToAvro(recordBytes, schema);
+ Option<IndexedRecord> incomingRecord = recordBytes.length == 0 ? Option.empty() : Option.of(HoodieAvroUtils.bytesToAvro(recordBytes, schema));
Review Comment:
not sure why we need the change, the logic seems the same before/after this patch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9715] Custom merger update [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13742:
URL: https://github.com/apache/hudi/pull/13742#discussion_r2300803147
##########
hudi-common/src/main/java/org/apache/hudi/common/model/HoodieRecordMerger.java:
##########
@@ -63,8 +63,15 @@ public interface HoodieRecordMerger extends Serializable {
* It'd be associative operation: f(a, f(b, c)) = f(f(a, b), c) (which we can translate as having 3 versions A, B, C
* of the single record, both orders of operations applications have to yield the same result)
* This method takes only full records for merging.
+ *
+ * @param older Older record in terms of commit time ordering.
+ * @param oldSchema The schema of the older record.
+ * @param newer Newer record in terms of commit time ordering.
+ * @param newSchema The schema of the newer record.
+ * @param props The additional properties for the merging operation.
+ * @return The merged record and schema. The record is expected to be non-null. If the record represents a deletion, the operation must be set as {@link HoodieOperation#DELETE}.
*/
- Option<Pair<HoodieRecord, Schema>> merge(HoodieRecord older, Schema oldSchema, HoodieRecord newer, Schema newSchema, TypedProperties props) throws IOException;
+ Pair<HoodieRecord, Schema> merge(HoodieRecord older, Schema oldSchema, HoodieRecord newer, Schema newSchema, TypedProperties props) throws IOException;
Review Comment:
Based on the changes in the patch, using `HoodieRecord` brings in a lot of intricate changes in write merging, buffered record mergers and deletes handling in mergers themself, making the bar very high to make the behavior correct and consistent among multiple engine, that I prefer we have a "change record" abstraction here to simply the things, or just use `BufferedRecord` because it is a very near abstaction and it has been widely used in merging now(just need to rename it to better suit as a developer/user API).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT] Custom merger update [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13742:
URL: https://github.com/apache/hudi/pull/13742#issuecomment-3218372868
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7988",
"triggerID" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"triggerType" : "PUSH"
}, {
"hash" : "ecb694295f7a88371f33df6de60370058670439e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7992",
"triggerID" : "ecb694295f7a88371f33df6de60370058670439e",
"triggerType" : "PUSH"
}, {
"hash" : "efd7462406be4f6d90dd5e42f152d66f683ebdf4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "efd7462406be4f6d90dd5e42f152d66f683ebdf4",
"triggerType" : "PUSH"
}, {
"hash" : "3c67160d5ceec883db5d826ff655865a3209f2f3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7994",
"triggerID" : "3c67160d5ceec883db5d826ff655865a3209f2f3",
"triggerType" : "PUSH"
}, {
"hash" : "f4f84d9f6a480923df27cfe5553d8aa65b2bbb7a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7996",
"triggerID" : "f4f84d9f6a480923df27cfe5553d8aa65b2bbb7a",
"triggerType" : "PUSH"
}, {
"hash" : "521b26bc2c60b7921e2f7aa5fffa620ab7f9a096",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "521b26bc2c60b7921e2f7aa5fffa620ab7f9a096",
"triggerType" : "PUSH"
}, {
"hash" : "22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50",
"triggerType" : "PUSH"
}, {
"hash" : "02a319ba884595a4e3b5453f366b660c1c11f767",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8007",
"triggerID" : "02a319ba884595a4e3b5453f366b660c1c11f767",
"triggerType" : "PUSH"
}, {
"hash" : "3e59b6f8b486016911c9ad14c583f6538cc036ac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8009",
"triggerID" : "3e59b6f8b486016911c9ad14c583f6538cc036ac",
"triggerType" : "PUSH"
}, {
"hash" : "003f1c81efedafa0ddd78868ce83357d8803c5e5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "003f1c81efedafa0ddd78868ce83357d8803c5e5",
"triggerType" : "PUSH"
}, {
"hash" : "40c04768d5233a38be1b4c01ce895189a4af4c3a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8013",
"triggerID" : "40c04768d5233a38be1b4c01ce895189a4af4c3a",
"triggerType" : "PUSH"
}, {
"hash" : "4a699a8dcf79e61ea3bb754e67e9cc4a32e86aa1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8084",
"triggerID" : "4a699a8dcf79e61ea3bb754e67e9cc4a32e86aa1",
"triggerType" : "PUSH"
}, {
"hash" : "fe55d8ec4923cb6c64e83ae4f32217c72599a398",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "fe55d8ec4923cb6c64e83ae4f32217c72599a398",
"triggerType" : "PUSH"
}, {
"hash" : "87cd79787dc8126b6877ba0d3407cfe8f343933a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8087",
"triggerID" : "87cd79787dc8126b6877ba0d3407cfe8f343933a",
"triggerType" : "PUSH"
}, {
"hash" : "594874ffdca647cd268b3947fa4680c57c53d679",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8088",
"triggerID" : "594874ffdca647cd268b3947fa4680c57c53d679",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* efd7462406be4f6d90dd5e42f152d66f683ebdf4 UNKNOWN
* 521b26bc2c60b7921e2f7aa5fffa620ab7f9a096 UNKNOWN
* 22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50 UNKNOWN
* 003f1c81efedafa0ddd78868ce83357d8803c5e5 UNKNOWN
* fe55d8ec4923cb6c64e83ae4f32217c72599a398 UNKNOWN
* 87cd79787dc8126b6877ba0d3407cfe8f343933a Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8087)
* 594874ffdca647cd268b3947fa4680c57c53d679 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8088)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9715] Custom merger update [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13742:
URL: https://github.com/apache/hudi/pull/13742#discussion_r2308914341
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/BufferedRecordMergerFactory.java:
##########
@@ -449,95 +383,36 @@ protected Option<Pair<HoodieRecord, Schema>> getMergedRecord(BufferedRecord<T> o
* based on {@code CUSTOM} merge mode and a given record payload class.
*/
private static class CustomPayloadRecordMerger<T> extends BaseCustomMerger<T> {
- private final String[] orderingFieldNames;
protected final String payloadClass;
public CustomPayloadRecordMerger(
RecordContext<T> recordContext,
Option<HoodieRecordMerger> recordMerger,
- List<String> orderingFieldNames,
String payloadClass,
Schema readerSchema,
TypedProperties props) {
super(recordContext, recordMerger, readerSchema, props);
- this.orderingFieldNames = orderingFieldNames.toArray(new String[0]);
this.payloadClass = payloadClass;
+ props.setProperty(HoodieAvroRecordMerger.PAYLOAD_CLASS_PROP, payloadClass);
}
@Override
- public Option<BufferedRecord<T>> deltaMergeNonDeleteRecord(BufferedRecord<T> newRecord, BufferedRecord<T> existingRecord) throws IOException {
- Option<Pair<HoodieRecord, Schema>> mergedRecordAndSchema = getMergedRecord(existingRecord, newRecord, false);
- if (mergedRecordAndSchema.isEmpty()) {
- // An empty Option indicates that the output represents a delete.
- return Option.of(new BufferedRecord<>(newRecord.getRecordKey(), OrderingValues.getDefault(), null, null, HoodieOperation.DELETE));
- }
- HoodieRecord mergedRecord = mergedRecordAndSchema.get().getLeft();
- Schema mergeResultSchema = mergedRecordAndSchema.get().getRight();
- // Special handling for SENTINEL record in Expression Payload. This is returned if the condition does not match.
- if (mergedRecord.getData() == HoodieRecord.SENTINEL) {
- return Option.empty();
- }
- T combinedRecordData = recordContext.convertAvroRecord(mergedRecord.toIndexedRecord(mergeResultSchema, props).get().getData());
+ public Option<BufferedRecord<T>> deltaMergeRecords(BufferedRecord<T> newRecord, BufferedRecord<T> existingRecord) throws IOException {
+ BufferedRecord<T> mergedRecord = getMergedRecord(existingRecord, newRecord, false);
// If pre-combine does not return existing record, update it
- if (combinedRecordData != existingRecord.getRecord()) {
- // For pkless we need to use record key from existing record
- return Option.of(BufferedRecords.fromEngineRecord(combinedRecordData, mergeResultSchema, recordContext, orderingFieldNames,
- existingRecord.getRecordKey(), mergedRecord.isDelete(mergeResultSchema, props)));
+ if (mergedRecord.getRecord() != existingRecord.getRecord()) {
+ return Option.of(mergedRecord);
}
return Option.empty();
}
@Override
- public BufferedRecord<T> mergeNonDeleteRecord(BufferedRecord<T> olderRecord, BufferedRecord<T> newerRecord) throws IOException {
- Option<Pair<HoodieRecord, Schema>> mergedRecordAndSchema = getMergedRecord(olderRecord, newerRecord, true);
- if (mergedRecordAndSchema.isEmpty()) {
- return BufferedRecords.createDelete(newerRecord.getRecordKey());
- }
- HoodieRecord mergedRecord = mergedRecordAndSchema.get().getLeft();
- Schema mergeResultSchema = mergedRecordAndSchema.get().getRight();
- // Special handling for SENTINEL record in Expression Payload
- if (mergedRecord.getData() == HoodieRecord.SENTINEL) {
- return olderRecord;
- }
- if (!mergedRecord.isDelete(mergeResultSchema, props)) {
- IndexedRecord indexedRecord = (IndexedRecord) mergedRecord.getData();
- return BufferedRecords.fromEngineRecord(
- recordContext.convertAvroRecord(indexedRecord), mergeResultSchema, recordContext, orderingFieldNames, newerRecord.getRecordKey(), false);
- }
- return BufferedRecords.createDelete(newerRecord.getRecordKey());
+ public BufferedRecord<T> mergeRecords(BufferedRecord<T> olderRecord, BufferedRecord<T> newerRecord) throws IOException {
+ return getMergedRecord(olderRecord, newerRecord, true);
}
- protected Pair<HoodieRecord, HoodieRecord> getDeltaMergeRecords(BufferedRecord<T> olderRecord, BufferedRecord<T> newerRecord) {
- HoodieRecord oldHoodieRecord = constructHoodieAvroRecord(recordContext, olderRecord, payloadClass);
- HoodieRecord newHoodieRecord = constructHoodieAvroRecord(recordContext, newerRecord, payloadClass);
- return Pair.of(oldHoodieRecord, newHoodieRecord);
- }
-
- protected Pair<HoodieRecord, HoodieRecord> getFinalMergeRecords(BufferedRecord<T> olderRecord, BufferedRecord<T> newerRecord) {
- return getDeltaMergeRecords(olderRecord, newerRecord);
- }
-
- protected Option<Pair<HoodieRecord, Schema>> getMergedRecord(BufferedRecord<T> olderRecord, BufferedRecord<T> newerRecord, boolean isFinalMerge) throws IOException {
- Pair<HoodieRecord, HoodieRecord> records = isFinalMerge ? getFinalMergeRecords(olderRecord, newerRecord) : getDeltaMergeRecords(olderRecord, newerRecord);
- return recordMerger.merge(records.getLeft(), getSchemaForAvroPayloadMerge(olderRecord), records.getRight(), getSchemaForAvroPayloadMerge(newerRecord), props);
- }
-
- protected HoodieRecord constructHoodieAvroRecord(RecordContext<T> recordContext, BufferedRecord<T> bufferedRecord, String payloadClass) {
- GenericRecord record = null;
- if (!bufferedRecord.isDelete()) {
- Schema recordSchema = recordContext.getSchemaFromBufferRecord(bufferedRecord);
- record = recordContext.convertToAvroRecord(bufferedRecord.getRecord(), recordSchema);
- }
- HoodieKey hoodieKey = new HoodieKey(bufferedRecord.getRecordKey(), null);
- return new HoodieAvroRecord<>(hoodieKey,
- HoodieRecordUtils.loadPayload(payloadClass, record, bufferedRecord.getOrderingValue()), null);
- }
-
- protected Schema getSchemaForAvroPayloadMerge(BufferedRecord<T> bufferedRecord) {
- if (bufferedRecord.getSchemaId() == null) {
- return readerSchema;
- }
- return recordContext.getSchemaFromBufferRecord(bufferedRecord);
+ protected BufferedRecord<T> getMergedRecord(BufferedRecord<T> olderRecord, BufferedRecord<T> newerRecord, boolean isFinalMerge) throws IOException {
Review Comment:
if the `isFinalMerge` is useless now, we can just get rid of this method
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9715] Custom merger update [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13742:
URL: https://github.com/apache/hudi/pull/13742#issuecomment-3235630019
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7988",
"triggerID" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"triggerType" : "PUSH"
}, {
"hash" : "ecb694295f7a88371f33df6de60370058670439e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7992",
"triggerID" : "ecb694295f7a88371f33df6de60370058670439e",
"triggerType" : "PUSH"
}, {
"hash" : "efd7462406be4f6d90dd5e42f152d66f683ebdf4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "efd7462406be4f6d90dd5e42f152d66f683ebdf4",
"triggerType" : "PUSH"
}, {
"hash" : "3c67160d5ceec883db5d826ff655865a3209f2f3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7994",
"triggerID" : "3c67160d5ceec883db5d826ff655865a3209f2f3",
"triggerType" : "PUSH"
}, {
"hash" : "f4f84d9f6a480923df27cfe5553d8aa65b2bbb7a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7996",
"triggerID" : "f4f84d9f6a480923df27cfe5553d8aa65b2bbb7a",
"triggerType" : "PUSH"
}, {
"hash" : "521b26bc2c60b7921e2f7aa5fffa620ab7f9a096",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "521b26bc2c60b7921e2f7aa5fffa620ab7f9a096",
"triggerType" : "PUSH"
}, {
"hash" : "22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50",
"triggerType" : "PUSH"
}, {
"hash" : "02a319ba884595a4e3b5453f366b660c1c11f767",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8007",
"triggerID" : "02a319ba884595a4e3b5453f366b660c1c11f767",
"triggerType" : "PUSH"
}, {
"hash" : "3e59b6f8b486016911c9ad14c583f6538cc036ac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8009",
"triggerID" : "3e59b6f8b486016911c9ad14c583f6538cc036ac",
"triggerType" : "PUSH"
}, {
"hash" : "003f1c81efedafa0ddd78868ce83357d8803c5e5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "003f1c81efedafa0ddd78868ce83357d8803c5e5",
"triggerType" : "PUSH"
}, {
"hash" : "40c04768d5233a38be1b4c01ce895189a4af4c3a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8013",
"triggerID" : "40c04768d5233a38be1b4c01ce895189a4af4c3a",
"triggerType" : "PUSH"
}, {
"hash" : "4a699a8dcf79e61ea3bb754e67e9cc4a32e86aa1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8084",
"triggerID" : "4a699a8dcf79e61ea3bb754e67e9cc4a32e86aa1",
"triggerType" : "PUSH"
}, {
"hash" : "fe55d8ec4923cb6c64e83ae4f32217c72599a398",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "fe55d8ec4923cb6c64e83ae4f32217c72599a398",
"triggerType" : "PUSH"
}, {
"hash" : "87cd79787dc8126b6877ba0d3407cfe8f343933a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8087",
"triggerID" : "87cd79787dc8126b6877ba0d3407cfe8f343933a",
"triggerType" : "PUSH"
}, {
"hash" : "594874ffdca647cd268b3947fa4680c57c53d679",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8088",
"triggerID" : "594874ffdca647cd268b3947fa4680c57c53d679",
"triggerType" : "PUSH"
}, {
"hash" : "7b95754f0df2c1a087a6a28722a7471ee12cecf3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8090",
"triggerID" : "7b95754f0df2c1a087a6a28722a7471ee12cecf3",
"triggerType" : "PUSH"
}, {
"hash" : "789c2da879ca66f6b9be71d1814be38b5d8cfb58",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8102",
"triggerID" : "789c2da879ca66f6b9be71d1814be38b5d8cfb58",
"triggerType" : "PUSH"
}, {
"hash" : "63f33fae3e22a9f80de51b2a6e1ed0427f631d0b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8103",
"triggerID" : "63f33fae3e22a9f80de51b2a6e1ed0427f631d0b",
"triggerType" : "PUSH"
}, {
"hash" : "e04c466fa81e9bcfaabe36825127098f5d1bb890",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8224",
"triggerID" : "e04c466fa81e9bcfaabe36825127098f5d1bb890",
"triggerType" : "PUSH"
}, {
"hash" : "cf574f46a1783a39b2074542aad9539ec3c4f36a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8235",
"triggerID" : "cf574f46a1783a39b2074542aad9539ec3c4f36a",
"triggerType" : "PUSH"
}, {
"hash" : "2eb81e923adfa77176f137c442143deaac9e9916",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2eb81e923adfa77176f137c442143deaac9e9916",
"triggerType" : "PUSH"
}, {
"hash" : "12dc3e3eaf746bb1f2bc3f4e93fa14d9adb0f6fd",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8244",
"triggerID" : "12dc3e3eaf746bb1f2bc3f4e93fa14d9adb0f6fd",
"triggerType" : "PUSH"
}, {
"hash" : "af5ab73103f3af882412bbe99c9322165e8941a0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "af5ab73103f3af882412bbe99c9322165e8941a0",
"triggerType" : "PUSH"
}, {
"hash" : "a9c28c4d913644fea23d57a5dc2224f36fd2b63e",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8248",
"triggerID" : "a9c28c4d913644fea23d57a5dc2224f36fd2b63e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* efd7462406be4f6d90dd5e42f152d66f683ebdf4 UNKNOWN
* 521b26bc2c60b7921e2f7aa5fffa620ab7f9a096 UNKNOWN
* 22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50 UNKNOWN
* 003f1c81efedafa0ddd78868ce83357d8803c5e5 UNKNOWN
* fe55d8ec4923cb6c64e83ae4f32217c72599a398 UNKNOWN
* 2eb81e923adfa77176f137c442143deaac9e9916 UNKNOWN
* 12dc3e3eaf746bb1f2bc3f4e93fa14d9adb0f6fd Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8244)
* af5ab73103f3af882412bbe99c9322165e8941a0 UNKNOWN
* a9c28c4d913644fea23d57a5dc2224f36fd2b63e Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8248)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9715] Custom merger update [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13742:
URL: https://github.com/apache/hudi/pull/13742#discussion_r2299937065
##########
hudi-common/src/main/java/org/apache/hudi/common/model/HoodieAvroRecordMerger.java:
##########
@@ -55,10 +82,10 @@ public HoodieRecordType getRecordType() {
private Option<IndexedRecord> combineAndGetUpdateValue(HoodieRecord older, HoodieRecord newer, Schema oldSchema, Schema newSchema, Properties props) throws IOException {
Option<IndexedRecord> previousAvroData = older.toIndexedRecord(oldSchema, props).map(HoodieAvroIndexedRecord::getData);
- if (!previousAvroData.isPresent()) {
- return Option.empty();
+ HoodieRecordPayload payload = ((HoodieAvroRecord) newer).getData();
+ if (previousAvroData.isEmpty()) {
Review Comment:
this is always false now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(merger): [RFC-101] Update HoodieRecordMerger interface and implementations [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 merged PR #13742:
URL: https://github.com/apache/hudi/pull/13742
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT] Custom merger update [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13742:
URL: https://github.com/apache/hudi/pull/13742#issuecomment-3208571553
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7988",
"triggerID" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"triggerType" : "PUSH"
}, {
"hash" : "ecb694295f7a88371f33df6de60370058670439e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7992",
"triggerID" : "ecb694295f7a88371f33df6de60370058670439e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ecb694295f7a88371f33df6de60370058670439e Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7992)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT] Custom merger update [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13742:
URL: https://github.com/apache/hudi/pull/13742#issuecomment-3220697894
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7988",
"triggerID" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"triggerType" : "PUSH"
}, {
"hash" : "ecb694295f7a88371f33df6de60370058670439e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7992",
"triggerID" : "ecb694295f7a88371f33df6de60370058670439e",
"triggerType" : "PUSH"
}, {
"hash" : "efd7462406be4f6d90dd5e42f152d66f683ebdf4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "efd7462406be4f6d90dd5e42f152d66f683ebdf4",
"triggerType" : "PUSH"
}, {
"hash" : "3c67160d5ceec883db5d826ff655865a3209f2f3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7994",
"triggerID" : "3c67160d5ceec883db5d826ff655865a3209f2f3",
"triggerType" : "PUSH"
}, {
"hash" : "f4f84d9f6a480923df27cfe5553d8aa65b2bbb7a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7996",
"triggerID" : "f4f84d9f6a480923df27cfe5553d8aa65b2bbb7a",
"triggerType" : "PUSH"
}, {
"hash" : "521b26bc2c60b7921e2f7aa5fffa620ab7f9a096",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "521b26bc2c60b7921e2f7aa5fffa620ab7f9a096",
"triggerType" : "PUSH"
}, {
"hash" : "22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50",
"triggerType" : "PUSH"
}, {
"hash" : "02a319ba884595a4e3b5453f366b660c1c11f767",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8007",
"triggerID" : "02a319ba884595a4e3b5453f366b660c1c11f767",
"triggerType" : "PUSH"
}, {
"hash" : "3e59b6f8b486016911c9ad14c583f6538cc036ac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8009",
"triggerID" : "3e59b6f8b486016911c9ad14c583f6538cc036ac",
"triggerType" : "PUSH"
}, {
"hash" : "003f1c81efedafa0ddd78868ce83357d8803c5e5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "003f1c81efedafa0ddd78868ce83357d8803c5e5",
"triggerType" : "PUSH"
}, {
"hash" : "40c04768d5233a38be1b4c01ce895189a4af4c3a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8013",
"triggerID" : "40c04768d5233a38be1b4c01ce895189a4af4c3a",
"triggerType" : "PUSH"
}, {
"hash" : "4a699a8dcf79e61ea3bb754e67e9cc4a32e86aa1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8084",
"triggerID" : "4a699a8dcf79e61ea3bb754e67e9cc4a32e86aa1",
"triggerType" : "PUSH"
}, {
"hash" : "fe55d8ec4923cb6c64e83ae4f32217c72599a398",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "fe55d8ec4923cb6c64e83ae4f32217c72599a398",
"triggerType" : "PUSH"
}, {
"hash" : "87cd79787dc8126b6877ba0d3407cfe8f343933a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8087",
"triggerID" : "87cd79787dc8126b6877ba0d3407cfe8f343933a",
"triggerType" : "PUSH"
}, {
"hash" : "594874ffdca647cd268b3947fa4680c57c53d679",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8088",
"triggerID" : "594874ffdca647cd268b3947fa4680c57c53d679",
"triggerType" : "PUSH"
}, {
"hash" : "7b95754f0df2c1a087a6a28722a7471ee12cecf3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8090",
"triggerID" : "7b95754f0df2c1a087a6a28722a7471ee12cecf3",
"triggerType" : "PUSH"
}, {
"hash" : "789c2da879ca66f6b9be71d1814be38b5d8cfb58",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8102",
"triggerID" : "789c2da879ca66f6b9be71d1814be38b5d8cfb58",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* efd7462406be4f6d90dd5e42f152d66f683ebdf4 UNKNOWN
* 521b26bc2c60b7921e2f7aa5fffa620ab7f9a096 UNKNOWN
* 22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50 UNKNOWN
* 003f1c81efedafa0ddd78868ce83357d8803c5e5 UNKNOWN
* fe55d8ec4923cb6c64e83ae4f32217c72599a398 UNKNOWN
* 7b95754f0df2c1a087a6a28722a7471ee12cecf3 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8090)
* 789c2da879ca66f6b9be71d1814be38b5d8cfb58 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8102)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9715] Custom merger update [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13742:
URL: https://github.com/apache/hudi/pull/13742#discussion_r2308901734
##########
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/testutils/HoodieSparkDeleteRecordMerger.java:
##########
@@ -21,11 +21,9 @@
import org.apache.hudi.HoodieSparkRecordMerger;
import org.apache.hudi.common.config.TypedProperties;
-import org.apache.hudi.common.model.HoodieRecord;
-import org.apache.hudi.common.util.Option;
-import org.apache.hudi.common.util.collection.Pair;
-
-import org.apache.avro.Schema;
+import org.apache.hudi.common.engine.RecordContext;
Review Comment:
nit: fix the import sequence.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9715] Custom merger update [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13742:
URL: https://github.com/apache/hudi/pull/13742#discussion_r2308921306
##########
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/testutils/HoodieSparkDeleteRecordMerger.java:
##########
@@ -21,11 +21,9 @@
import org.apache.hudi.HoodieSparkRecordMerger;
import org.apache.hudi.common.config.TypedProperties;
-import org.apache.hudi.common.model.HoodieRecord;
-import org.apache.hudi.common.util.Option;
-import org.apache.hudi.common.util.collection.Pair;
-
-import org.apache.avro.Schema;
+import org.apache.hudi.common.engine.RecordContext;
Review Comment:
`org.apache.hudi.` prefix should be together?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT] Custom merger update [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13742:
URL: https://github.com/apache/hudi/pull/13742#issuecomment-3210942297
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7988",
"triggerID" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"triggerType" : "PUSH"
}, {
"hash" : "ecb694295f7a88371f33df6de60370058670439e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7992",
"triggerID" : "ecb694295f7a88371f33df6de60370058670439e",
"triggerType" : "PUSH"
}, {
"hash" : "efd7462406be4f6d90dd5e42f152d66f683ebdf4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "efd7462406be4f6d90dd5e42f152d66f683ebdf4",
"triggerType" : "PUSH"
}, {
"hash" : "3c67160d5ceec883db5d826ff655865a3209f2f3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7994",
"triggerID" : "3c67160d5ceec883db5d826ff655865a3209f2f3",
"triggerType" : "PUSH"
}, {
"hash" : "f4f84d9f6a480923df27cfe5553d8aa65b2bbb7a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7996",
"triggerID" : "f4f84d9f6a480923df27cfe5553d8aa65b2bbb7a",
"triggerType" : "PUSH"
}, {
"hash" : "521b26bc2c60b7921e2f7aa5fffa620ab7f9a096",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "521b26bc2c60b7921e2f7aa5fffa620ab7f9a096",
"triggerType" : "PUSH"
}, {
"hash" : "22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50",
"triggerType" : "PUSH"
}, {
"hash" : "02a319ba884595a4e3b5453f366b660c1c11f767",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "02a319ba884595a4e3b5453f366b660c1c11f767",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* efd7462406be4f6d90dd5e42f152d66f683ebdf4 UNKNOWN
* f4f84d9f6a480923df27cfe5553d8aa65b2bbb7a Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7996)
* 521b26bc2c60b7921e2f7aa5fffa620ab7f9a096 UNKNOWN
* 22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50 UNKNOWN
* 02a319ba884595a4e3b5453f366b660c1c11f767 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9715] Custom merger update [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13742:
URL: https://github.com/apache/hudi/pull/13742#discussion_r2299961583
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/BufferedRecordMergerFactory.java:
##########
@@ -465,46 +451,45 @@ public CustomPayloadRecordMerger(
}
@Override
- public Option<BufferedRecord<T>> deltaMergeNonDeleteRecord(BufferedRecord<T> newRecord, BufferedRecord<T> existingRecord) throws IOException {
- Option<Pair<HoodieRecord, Schema>> mergedRecordAndSchema = getMergedRecord(existingRecord, newRecord, false);
- if (mergedRecordAndSchema.isEmpty()) {
- // An empty Option indicates that the output represents a delete.
- return Option.of(new BufferedRecord<>(newRecord.getRecordKey(), OrderingValues.getDefault(), null, null, HoodieOperation.DELETE));
- }
- HoodieRecord mergedRecord = mergedRecordAndSchema.get().getLeft();
- Schema mergeResultSchema = mergedRecordAndSchema.get().getRight();
+ public Option<BufferedRecord<T>> deltaMergeRecords(BufferedRecord<T> newRecord, BufferedRecord<T> existingRecord) throws IOException {
+ Pair<HoodieRecord, Schema> mergedRecordAndSchema = getMergedRecord(existingRecord, newRecord, false);
+ HoodieRecord mergedRecord = mergedRecordAndSchema.getLeft();
+ Schema mergeResultSchema = mergedRecordAndSchema.getRight();
// Special handling for SENTINEL record in Expression Payload. This is returned if the condition does not match.
if (mergedRecord.getData() == HoodieRecord.SENTINEL) {
return Option.empty();
}
- T combinedRecordData = recordContext.convertAvroRecord(mergedRecord.toIndexedRecord(mergeResultSchema, props).get().getData());
+ Option<T> combinedRecordData = (mergedRecord.toIndexedRecord(mergeResultSchema, props).map(indexedRecord -> recordContext.convertAvroRecord(indexedRecord.getData())));
// If pre-combine does not return existing record, update it
- if (combinedRecordData != existingRecord.getRecord()) {
+ if (combinedRecordData.map(record -> record != existingRecord.getRecord()).orElse(true)) {
// For pkless we need to use record key from existing record
- return Option.of(BufferedRecords.fromEngineRecord(combinedRecordData, mergeResultSchema, recordContext, orderingFieldNames,
- existingRecord.getRecordKey(), mergedRecord.isDelete(mergeResultSchema, props)));
+ boolean isDelete = mergedRecord.isDelete(readerSchema, props);
+ Comparable orderingValue = mergedRecord.getOrderingValue(mergeResultSchema, props, orderingFieldNames);
+ T mergedEngineRecord = mergedRecord.toIndexedRecord(mergeResultSchema, props)
+ .map(hoodieAvroIndexedRecord -> recordContext.convertAvroRecord(hoodieAvroIndexedRecord.getData()))
+ .orElse(null);
+ return Option.of(BufferedRecords.fromEngineRecord(mergedEngineRecord, readerSchema, recordContext, orderingValue, existingRecord.getRecordKey(), isDelete));
+
}
return Option.empty();
}
@Override
- public BufferedRecord<T> mergeNonDeleteRecord(BufferedRecord<T> olderRecord, BufferedRecord<T> newerRecord) throws IOException {
- Option<Pair<HoodieRecord, Schema>> mergedRecordAndSchema = getMergedRecord(olderRecord, newerRecord, true);
- if (mergedRecordAndSchema.isEmpty()) {
- return BufferedRecords.createDelete(newerRecord.getRecordKey());
- }
- HoodieRecord mergedRecord = mergedRecordAndSchema.get().getLeft();
- Schema mergeResultSchema = mergedRecordAndSchema.get().getRight();
+ public BufferedRecord<T> mergeRecords(BufferedRecord<T> olderRecord, BufferedRecord<T> newerRecord) throws IOException {
+ Pair<HoodieRecord, Schema> mergedRecordAndSchema = getMergedRecord(olderRecord, newerRecord, true);
+
+ HoodieRecord mergedRecord = mergedRecordAndSchema.getLeft();
+ Schema mergeResultSchema = mergedRecordAndSchema.getRight();
// Special handling for SENTINEL record in Expression Payload
if (mergedRecord.getData() == HoodieRecord.SENTINEL) {
return olderRecord;
}
- if (!mergedRecord.isDelete(mergeResultSchema, props)) {
- IndexedRecord indexedRecord = (IndexedRecord) mergedRecord.getData();
- return BufferedRecords.fromEngineRecord(
- recordContext.convertAvroRecord(indexedRecord), mergeResultSchema, recordContext, orderingFieldNames, newerRecord.getRecordKey(), false);
- }
- return BufferedRecords.createDelete(newerRecord.getRecordKey());
+ boolean isDelete = mergedRecord.isDelete(readerSchema, props);
Review Comment:
why use `readerSchema` instead of `mergeResultSchema`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9715] Custom merger update [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13742:
URL: https://github.com/apache/hudi/pull/13742#discussion_r2302489871
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/BufferedRecordMergerFactory.java:
##########
@@ -338,16 +334,15 @@ public Option<DeleteRecord> deltaMerge(DeleteRecord deleteRecord, BufferedRecord
public BufferedRecord<T> finalMerge(BufferedRecord<T> olderRecord, BufferedRecord<T> newerRecord) throws IOException {
// TODO(HUDI-7843): decouple the merging logic from the merger
// and use the record merge mode to control how to merge partial updates
- Option<Pair<HoodieRecord, Schema>> mergedRecord = recordMerger.get().partialMerge(
+ Pair<HoodieRecord, Schema> mergedRecord = recordMerger.get().partialMerge(
Review Comment:
I'm not sure but I was trying to keep the changes to a minimum originally
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9715] Custom merger update [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13742:
URL: https://github.com/apache/hudi/pull/13742#discussion_r2299957029
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/BufferedRecordMergerFactory.java:
##########
@@ -465,46 +451,45 @@ public CustomPayloadRecordMerger(
}
@Override
- public Option<BufferedRecord<T>> deltaMergeNonDeleteRecord(BufferedRecord<T> newRecord, BufferedRecord<T> existingRecord) throws IOException {
- Option<Pair<HoodieRecord, Schema>> mergedRecordAndSchema = getMergedRecord(existingRecord, newRecord, false);
- if (mergedRecordAndSchema.isEmpty()) {
- // An empty Option indicates that the output represents a delete.
- return Option.of(new BufferedRecord<>(newRecord.getRecordKey(), OrderingValues.getDefault(), null, null, HoodieOperation.DELETE));
- }
- HoodieRecord mergedRecord = mergedRecordAndSchema.get().getLeft();
- Schema mergeResultSchema = mergedRecordAndSchema.get().getRight();
+ public Option<BufferedRecord<T>> deltaMergeRecords(BufferedRecord<T> newRecord, BufferedRecord<T> existingRecord) throws IOException {
+ Pair<HoodieRecord, Schema> mergedRecordAndSchema = getMergedRecord(existingRecord, newRecord, false);
+ HoodieRecord mergedRecord = mergedRecordAndSchema.getLeft();
+ Schema mergeResultSchema = mergedRecordAndSchema.getRight();
// Special handling for SENTINEL record in Expression Payload. This is returned if the condition does not match.
if (mergedRecord.getData() == HoodieRecord.SENTINEL) {
return Option.empty();
}
- T combinedRecordData = recordContext.convertAvroRecord(mergedRecord.toIndexedRecord(mergeResultSchema, props).get().getData());
+ Option<T> combinedRecordData = (mergedRecord.toIndexedRecord(mergeResultSchema, props).map(indexedRecord -> recordContext.convertAvroRecord(indexedRecord.getData())));
// If pre-combine does not return existing record, update it
- if (combinedRecordData != existingRecord.getRecord()) {
+ if (combinedRecordData.map(record -> record != existingRecord.getRecord()).orElse(true)) {
// For pkless we need to use record key from existing record
- return Option.of(BufferedRecords.fromEngineRecord(combinedRecordData, mergeResultSchema, recordContext, orderingFieldNames,
- existingRecord.getRecordKey(), mergedRecord.isDelete(mergeResultSchema, props)));
+ boolean isDelete = mergedRecord.isDelete(readerSchema, props);
+ Comparable orderingValue = mergedRecord.getOrderingValue(mergeResultSchema, props, orderingFieldNames);
+ T mergedEngineRecord = mergedRecord.toIndexedRecord(mergeResultSchema, props)
Review Comment:
why convert to avro record then to engine record? `#convertAvroRecord` already returns engine record? `combinedRecordData` is already a T?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9715] Custom merger update [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13742:
URL: https://github.com/apache/hudi/pull/13742#discussion_r2299991113
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/BufferedRecordMergerFactory.java:
##########
@@ -517,20 +502,21 @@ protected Pair<HoodieRecord, HoodieRecord> getFinalMergeRecords(BufferedRecord<T
return getDeltaMergeRecords(olderRecord, newerRecord);
}
- protected Option<Pair<HoodieRecord, Schema>> getMergedRecord(BufferedRecord<T> olderRecord, BufferedRecord<T> newerRecord, boolean isFinalMerge) throws IOException {
+ protected Pair<HoodieRecord, Schema> getMergedRecord(BufferedRecord<T> olderRecord, BufferedRecord<T> newerRecord, boolean isFinalMerge) throws IOException {
Pair<HoodieRecord, HoodieRecord> records = isFinalMerge ? getFinalMergeRecords(olderRecord, newerRecord) : getDeltaMergeRecords(olderRecord, newerRecord);
return recordMerger.merge(records.getLeft(), getSchemaForAvroPayloadMerge(olderRecord), records.getRight(), getSchemaForAvroPayloadMerge(newerRecord), props);
}
protected HoodieRecord constructHoodieAvroRecord(RecordContext<T> recordContext, BufferedRecord<T> bufferedRecord, String payloadClass) {
GenericRecord record = null;
- if (!bufferedRecord.isDelete()) {
+ if (bufferedRecord.getRecord() != null) {
Schema recordSchema = recordContext.getSchemaFromBufferRecord(bufferedRecord);
record = recordContext.convertToAvroRecord(bufferedRecord.getRecord(), recordSchema);
}
HoodieKey hoodieKey = new HoodieKey(bufferedRecord.getRecordKey(), null);
+ Comparable orderingValue = recordContext.convertOrderingValueFromEngineType(bufferedRecord.getOrderingValue());
Review Comment:
wow, this is so hacky.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9715] Custom merger update [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13742:
URL: https://github.com/apache/hudi/pull/13742#discussion_r2310528671
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieWriteMergeHandle.java:
##########
@@ -120,6 +125,8 @@ public HoodieWriteMergeHandle(HoodieWriteConfig config, String instantTime, Hood
super(config, instantTime, hoodieTable, partitionPath, fileId, taskContextSupplier, baseFile, keyGeneratorOpt, false);
populateIncomingRecordsMap(recordItr);
initMarkerFileAndFileWriter(fileId, partitionPath);
+ this.readerContext = hoodieTable.getReaderContextFactoryForWrite().getContext();
Review Comment:
Updated the `FileGroupReaderBasedMergeHandle` to avoid this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9715] Custom merger update [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13742:
URL: https://github.com/apache/hudi/pull/13742#issuecomment-3237455869
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7988",
"triggerID" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"triggerType" : "PUSH"
}, {
"hash" : "ecb694295f7a88371f33df6de60370058670439e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7992",
"triggerID" : "ecb694295f7a88371f33df6de60370058670439e",
"triggerType" : "PUSH"
}, {
"hash" : "efd7462406be4f6d90dd5e42f152d66f683ebdf4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "efd7462406be4f6d90dd5e42f152d66f683ebdf4",
"triggerType" : "PUSH"
}, {
"hash" : "3c67160d5ceec883db5d826ff655865a3209f2f3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7994",
"triggerID" : "3c67160d5ceec883db5d826ff655865a3209f2f3",
"triggerType" : "PUSH"
}, {
"hash" : "f4f84d9f6a480923df27cfe5553d8aa65b2bbb7a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7996",
"triggerID" : "f4f84d9f6a480923df27cfe5553d8aa65b2bbb7a",
"triggerType" : "PUSH"
}, {
"hash" : "521b26bc2c60b7921e2f7aa5fffa620ab7f9a096",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "521b26bc2c60b7921e2f7aa5fffa620ab7f9a096",
"triggerType" : "PUSH"
}, {
"hash" : "22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50",
"triggerType" : "PUSH"
}, {
"hash" : "02a319ba884595a4e3b5453f366b660c1c11f767",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8007",
"triggerID" : "02a319ba884595a4e3b5453f366b660c1c11f767",
"triggerType" : "PUSH"
}, {
"hash" : "3e59b6f8b486016911c9ad14c583f6538cc036ac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8009",
"triggerID" : "3e59b6f8b486016911c9ad14c583f6538cc036ac",
"triggerType" : "PUSH"
}, {
"hash" : "003f1c81efedafa0ddd78868ce83357d8803c5e5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "003f1c81efedafa0ddd78868ce83357d8803c5e5",
"triggerType" : "PUSH"
}, {
"hash" : "40c04768d5233a38be1b4c01ce895189a4af4c3a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8013",
"triggerID" : "40c04768d5233a38be1b4c01ce895189a4af4c3a",
"triggerType" : "PUSH"
}, {
"hash" : "4a699a8dcf79e61ea3bb754e67e9cc4a32e86aa1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8084",
"triggerID" : "4a699a8dcf79e61ea3bb754e67e9cc4a32e86aa1",
"triggerType" : "PUSH"
}, {
"hash" : "fe55d8ec4923cb6c64e83ae4f32217c72599a398",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "fe55d8ec4923cb6c64e83ae4f32217c72599a398",
"triggerType" : "PUSH"
}, {
"hash" : "87cd79787dc8126b6877ba0d3407cfe8f343933a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8087",
"triggerID" : "87cd79787dc8126b6877ba0d3407cfe8f343933a",
"triggerType" : "PUSH"
}, {
"hash" : "594874ffdca647cd268b3947fa4680c57c53d679",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8088",
"triggerID" : "594874ffdca647cd268b3947fa4680c57c53d679",
"triggerType" : "PUSH"
}, {
"hash" : "7b95754f0df2c1a087a6a28722a7471ee12cecf3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8090",
"triggerID" : "7b95754f0df2c1a087a6a28722a7471ee12cecf3",
"triggerType" : "PUSH"
}, {
"hash" : "789c2da879ca66f6b9be71d1814be38b5d8cfb58",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8102",
"triggerID" : "789c2da879ca66f6b9be71d1814be38b5d8cfb58",
"triggerType" : "PUSH"
}, {
"hash" : "63f33fae3e22a9f80de51b2a6e1ed0427f631d0b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8103",
"triggerID" : "63f33fae3e22a9f80de51b2a6e1ed0427f631d0b",
"triggerType" : "PUSH"
}, {
"hash" : "e04c466fa81e9bcfaabe36825127098f5d1bb890",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8224",
"triggerID" : "e04c466fa81e9bcfaabe36825127098f5d1bb890",
"triggerType" : "PUSH"
}, {
"hash" : "cf574f46a1783a39b2074542aad9539ec3c4f36a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8235",
"triggerID" : "cf574f46a1783a39b2074542aad9539ec3c4f36a",
"triggerType" : "PUSH"
}, {
"hash" : "2eb81e923adfa77176f137c442143deaac9e9916",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2eb81e923adfa77176f137c442143deaac9e9916",
"triggerType" : "PUSH"
}, {
"hash" : "12dc3e3eaf746bb1f2bc3f4e93fa14d9adb0f6fd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8244",
"triggerID" : "12dc3e3eaf746bb1f2bc3f4e93fa14d9adb0f6fd",
"triggerType" : "PUSH"
}, {
"hash" : "af5ab73103f3af882412bbe99c9322165e8941a0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "af5ab73103f3af882412bbe99c9322165e8941a0",
"triggerType" : "PUSH"
}, {
"hash" : "a9c28c4d913644fea23d57a5dc2224f36fd2b63e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8248",
"triggerID" : "a9c28c4d913644fea23d57a5dc2224f36fd2b63e",
"triggerType" : "PUSH"
}, {
"hash" : "98381a6bd2bcccf3da08fb992e599fe1165f2ddb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8252",
"triggerID" : "98381a6bd2bcccf3da08fb992e599fe1165f2ddb",
"triggerType" : "PUSH"
}, {
"hash" : "94aa9a669d2fe1fb4919a4735fab2665553b63c0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8258",
"triggerID" : "94aa9a669d2fe1fb4919a4735fab2665553b63c0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* efd7462406be4f6d90dd5e42f152d66f683ebdf4 UNKNOWN
* 521b26bc2c60b7921e2f7aa5fffa620ab7f9a096 UNKNOWN
* 22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50 UNKNOWN
* 003f1c81efedafa0ddd78868ce83357d8803c5e5 UNKNOWN
* fe55d8ec4923cb6c64e83ae4f32217c72599a398 UNKNOWN
* 2eb81e923adfa77176f137c442143deaac9e9916 UNKNOWN
* af5ab73103f3af882412bbe99c9322165e8941a0 UNKNOWN
* 94aa9a669d2fe1fb4919a4735fab2665553b63c0 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8258)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9715] Custom merger update [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13742:
URL: https://github.com/apache/hudi/pull/13742#issuecomment-3237648040
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7988",
"triggerID" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"triggerType" : "PUSH"
}, {
"hash" : "ecb694295f7a88371f33df6de60370058670439e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7992",
"triggerID" : "ecb694295f7a88371f33df6de60370058670439e",
"triggerType" : "PUSH"
}, {
"hash" : "efd7462406be4f6d90dd5e42f152d66f683ebdf4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "efd7462406be4f6d90dd5e42f152d66f683ebdf4",
"triggerType" : "PUSH"
}, {
"hash" : "3c67160d5ceec883db5d826ff655865a3209f2f3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7994",
"triggerID" : "3c67160d5ceec883db5d826ff655865a3209f2f3",
"triggerType" : "PUSH"
}, {
"hash" : "f4f84d9f6a480923df27cfe5553d8aa65b2bbb7a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7996",
"triggerID" : "f4f84d9f6a480923df27cfe5553d8aa65b2bbb7a",
"triggerType" : "PUSH"
}, {
"hash" : "521b26bc2c60b7921e2f7aa5fffa620ab7f9a096",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "521b26bc2c60b7921e2f7aa5fffa620ab7f9a096",
"triggerType" : "PUSH"
}, {
"hash" : "22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50",
"triggerType" : "PUSH"
}, {
"hash" : "02a319ba884595a4e3b5453f366b660c1c11f767",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8007",
"triggerID" : "02a319ba884595a4e3b5453f366b660c1c11f767",
"triggerType" : "PUSH"
}, {
"hash" : "3e59b6f8b486016911c9ad14c583f6538cc036ac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8009",
"triggerID" : "3e59b6f8b486016911c9ad14c583f6538cc036ac",
"triggerType" : "PUSH"
}, {
"hash" : "003f1c81efedafa0ddd78868ce83357d8803c5e5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "003f1c81efedafa0ddd78868ce83357d8803c5e5",
"triggerType" : "PUSH"
}, {
"hash" : "40c04768d5233a38be1b4c01ce895189a4af4c3a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8013",
"triggerID" : "40c04768d5233a38be1b4c01ce895189a4af4c3a",
"triggerType" : "PUSH"
}, {
"hash" : "4a699a8dcf79e61ea3bb754e67e9cc4a32e86aa1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8084",
"triggerID" : "4a699a8dcf79e61ea3bb754e67e9cc4a32e86aa1",
"triggerType" : "PUSH"
}, {
"hash" : "fe55d8ec4923cb6c64e83ae4f32217c72599a398",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "fe55d8ec4923cb6c64e83ae4f32217c72599a398",
"triggerType" : "PUSH"
}, {
"hash" : "87cd79787dc8126b6877ba0d3407cfe8f343933a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8087",
"triggerID" : "87cd79787dc8126b6877ba0d3407cfe8f343933a",
"triggerType" : "PUSH"
}, {
"hash" : "594874ffdca647cd268b3947fa4680c57c53d679",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8088",
"triggerID" : "594874ffdca647cd268b3947fa4680c57c53d679",
"triggerType" : "PUSH"
}, {
"hash" : "7b95754f0df2c1a087a6a28722a7471ee12cecf3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8090",
"triggerID" : "7b95754f0df2c1a087a6a28722a7471ee12cecf3",
"triggerType" : "PUSH"
}, {
"hash" : "789c2da879ca66f6b9be71d1814be38b5d8cfb58",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8102",
"triggerID" : "789c2da879ca66f6b9be71d1814be38b5d8cfb58",
"triggerType" : "PUSH"
}, {
"hash" : "63f33fae3e22a9f80de51b2a6e1ed0427f631d0b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8103",
"triggerID" : "63f33fae3e22a9f80de51b2a6e1ed0427f631d0b",
"triggerType" : "PUSH"
}, {
"hash" : "e04c466fa81e9bcfaabe36825127098f5d1bb890",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8224",
"triggerID" : "e04c466fa81e9bcfaabe36825127098f5d1bb890",
"triggerType" : "PUSH"
}, {
"hash" : "cf574f46a1783a39b2074542aad9539ec3c4f36a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8235",
"triggerID" : "cf574f46a1783a39b2074542aad9539ec3c4f36a",
"triggerType" : "PUSH"
}, {
"hash" : "2eb81e923adfa77176f137c442143deaac9e9916",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2eb81e923adfa77176f137c442143deaac9e9916",
"triggerType" : "PUSH"
}, {
"hash" : "12dc3e3eaf746bb1f2bc3f4e93fa14d9adb0f6fd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8244",
"triggerID" : "12dc3e3eaf746bb1f2bc3f4e93fa14d9adb0f6fd",
"triggerType" : "PUSH"
}, {
"hash" : "af5ab73103f3af882412bbe99c9322165e8941a0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "af5ab73103f3af882412bbe99c9322165e8941a0",
"triggerType" : "PUSH"
}, {
"hash" : "a9c28c4d913644fea23d57a5dc2224f36fd2b63e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8248",
"triggerID" : "a9c28c4d913644fea23d57a5dc2224f36fd2b63e",
"triggerType" : "PUSH"
}, {
"hash" : "98381a6bd2bcccf3da08fb992e599fe1165f2ddb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8252",
"triggerID" : "98381a6bd2bcccf3da08fb992e599fe1165f2ddb",
"triggerType" : "PUSH"
}, {
"hash" : "94aa9a669d2fe1fb4919a4735fab2665553b63c0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8258",
"triggerID" : "94aa9a669d2fe1fb4919a4735fab2665553b63c0",
"triggerType" : "PUSH"
}, {
"hash" : "94aa9a669d2fe1fb4919a4735fab2665553b63c0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3237521433",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* efd7462406be4f6d90dd5e42f152d66f683ebdf4 UNKNOWN
* 521b26bc2c60b7921e2f7aa5fffa620ab7f9a096 UNKNOWN
* 22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50 UNKNOWN
* 003f1c81efedafa0ddd78868ce83357d8803c5e5 UNKNOWN
* fe55d8ec4923cb6c64e83ae4f32217c72599a398 UNKNOWN
* 2eb81e923adfa77176f137c442143deaac9e9916 UNKNOWN
* af5ab73103f3af882412bbe99c9322165e8941a0 UNKNOWN
* 94aa9a669d2fe1fb4919a4735fab2665553b63c0 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8258)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9715] Custom merger update [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13742:
URL: https://github.com/apache/hudi/pull/13742#issuecomment-3300006631
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "94aa9a669d2fe1fb4919a4735fab2665553b63c0",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8258",
"triggerID" : "94aa9a669d2fe1fb4919a4735fab2665553b63c0",
"triggerType" : "PUSH"
}, {
"hash" : "94aa9a669d2fe1fb4919a4735fab2665553b63c0",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8258",
"triggerID" : "3237521433",
"triggerType" : "MANUAL"
}, {
"hash" : "57b338240d2ac6faa9ec0a329f43f13e28019eb8",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8564",
"triggerID" : "57b338240d2ac6faa9ec0a329f43f13e28019eb8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 94aa9a669d2fe1fb4919a4735fab2665553b63c0 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8258)
* 57b338240d2ac6faa9ec0a329f43f13e28019eb8 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8564)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9715] Custom merger update [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13742:
URL: https://github.com/apache/hudi/pull/13742#discussion_r2355803806
##########
hudi-common/src/main/java/org/apache/hudi/common/model/HoodieRecordMerger.java:
##########
@@ -136,6 +136,7 @@ default Option<Pair<HoodieRecord, Schema>> partialMerge(HoodieRecord older, Sche
*
* <p> This interface is experimental and might be evolved in the future.
**/
+ @Deprecated
Review Comment:
Added the clean up in a different PR https://github.com/apache/hudi/pull/13904
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT] Custom merger update [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13742:
URL: https://github.com/apache/hudi/pull/13742#issuecomment-3211139664
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7988",
"triggerID" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"triggerType" : "PUSH"
}, {
"hash" : "ecb694295f7a88371f33df6de60370058670439e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7992",
"triggerID" : "ecb694295f7a88371f33df6de60370058670439e",
"triggerType" : "PUSH"
}, {
"hash" : "efd7462406be4f6d90dd5e42f152d66f683ebdf4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "efd7462406be4f6d90dd5e42f152d66f683ebdf4",
"triggerType" : "PUSH"
}, {
"hash" : "3c67160d5ceec883db5d826ff655865a3209f2f3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7994",
"triggerID" : "3c67160d5ceec883db5d826ff655865a3209f2f3",
"triggerType" : "PUSH"
}, {
"hash" : "f4f84d9f6a480923df27cfe5553d8aa65b2bbb7a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7996",
"triggerID" : "f4f84d9f6a480923df27cfe5553d8aa65b2bbb7a",
"triggerType" : "PUSH"
}, {
"hash" : "521b26bc2c60b7921e2f7aa5fffa620ab7f9a096",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "521b26bc2c60b7921e2f7aa5fffa620ab7f9a096",
"triggerType" : "PUSH"
}, {
"hash" : "22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50",
"triggerType" : "PUSH"
}, {
"hash" : "02a319ba884595a4e3b5453f366b660c1c11f767",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8007",
"triggerID" : "02a319ba884595a4e3b5453f366b660c1c11f767",
"triggerType" : "PUSH"
}, {
"hash" : "3e59b6f8b486016911c9ad14c583f6538cc036ac",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3e59b6f8b486016911c9ad14c583f6538cc036ac",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* efd7462406be4f6d90dd5e42f152d66f683ebdf4 UNKNOWN
* 521b26bc2c60b7921e2f7aa5fffa620ab7f9a096 UNKNOWN
* 22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50 UNKNOWN
* 02a319ba884595a4e3b5453f366b660c1c11f767 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8007)
* 3e59b6f8b486016911c9ad14c583f6538cc036ac UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9715] Custom merger update [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13742:
URL: https://github.com/apache/hudi/pull/13742#issuecomment-3300278962
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "94aa9a669d2fe1fb4919a4735fab2665553b63c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8258",
"triggerID" : "94aa9a669d2fe1fb4919a4735fab2665553b63c0",
"triggerType" : "PUSH"
}, {
"hash" : "94aa9a669d2fe1fb4919a4735fab2665553b63c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8258",
"triggerID" : "3237521433",
"triggerType" : "MANUAL"
}, {
"hash" : "57b338240d2ac6faa9ec0a329f43f13e28019eb8",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8564",
"triggerID" : "57b338240d2ac6faa9ec0a329f43f13e28019eb8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 57b338240d2ac6faa9ec0a329f43f13e28019eb8 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8564)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT] Custom merger update [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13742:
URL: https://github.com/apache/hudi/pull/13742#issuecomment-3220848154
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7988",
"triggerID" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"triggerType" : "PUSH"
}, {
"hash" : "ecb694295f7a88371f33df6de60370058670439e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7992",
"triggerID" : "ecb694295f7a88371f33df6de60370058670439e",
"triggerType" : "PUSH"
}, {
"hash" : "efd7462406be4f6d90dd5e42f152d66f683ebdf4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "efd7462406be4f6d90dd5e42f152d66f683ebdf4",
"triggerType" : "PUSH"
}, {
"hash" : "3c67160d5ceec883db5d826ff655865a3209f2f3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7994",
"triggerID" : "3c67160d5ceec883db5d826ff655865a3209f2f3",
"triggerType" : "PUSH"
}, {
"hash" : "f4f84d9f6a480923df27cfe5553d8aa65b2bbb7a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7996",
"triggerID" : "f4f84d9f6a480923df27cfe5553d8aa65b2bbb7a",
"triggerType" : "PUSH"
}, {
"hash" : "521b26bc2c60b7921e2f7aa5fffa620ab7f9a096",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "521b26bc2c60b7921e2f7aa5fffa620ab7f9a096",
"triggerType" : "PUSH"
}, {
"hash" : "22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50",
"triggerType" : "PUSH"
}, {
"hash" : "02a319ba884595a4e3b5453f366b660c1c11f767",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8007",
"triggerID" : "02a319ba884595a4e3b5453f366b660c1c11f767",
"triggerType" : "PUSH"
}, {
"hash" : "3e59b6f8b486016911c9ad14c583f6538cc036ac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8009",
"triggerID" : "3e59b6f8b486016911c9ad14c583f6538cc036ac",
"triggerType" : "PUSH"
}, {
"hash" : "003f1c81efedafa0ddd78868ce83357d8803c5e5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "003f1c81efedafa0ddd78868ce83357d8803c5e5",
"triggerType" : "PUSH"
}, {
"hash" : "40c04768d5233a38be1b4c01ce895189a4af4c3a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8013",
"triggerID" : "40c04768d5233a38be1b4c01ce895189a4af4c3a",
"triggerType" : "PUSH"
}, {
"hash" : "4a699a8dcf79e61ea3bb754e67e9cc4a32e86aa1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8084",
"triggerID" : "4a699a8dcf79e61ea3bb754e67e9cc4a32e86aa1",
"triggerType" : "PUSH"
}, {
"hash" : "fe55d8ec4923cb6c64e83ae4f32217c72599a398",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "fe55d8ec4923cb6c64e83ae4f32217c72599a398",
"triggerType" : "PUSH"
}, {
"hash" : "87cd79787dc8126b6877ba0d3407cfe8f343933a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8087",
"triggerID" : "87cd79787dc8126b6877ba0d3407cfe8f343933a",
"triggerType" : "PUSH"
}, {
"hash" : "594874ffdca647cd268b3947fa4680c57c53d679",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8088",
"triggerID" : "594874ffdca647cd268b3947fa4680c57c53d679",
"triggerType" : "PUSH"
}, {
"hash" : "7b95754f0df2c1a087a6a28722a7471ee12cecf3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8090",
"triggerID" : "7b95754f0df2c1a087a6a28722a7471ee12cecf3",
"triggerType" : "PUSH"
}, {
"hash" : "789c2da879ca66f6b9be71d1814be38b5d8cfb58",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8102",
"triggerID" : "789c2da879ca66f6b9be71d1814be38b5d8cfb58",
"triggerType" : "PUSH"
}, {
"hash" : "63f33fae3e22a9f80de51b2a6e1ed0427f631d0b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "63f33fae3e22a9f80de51b2a6e1ed0427f631d0b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* efd7462406be4f6d90dd5e42f152d66f683ebdf4 UNKNOWN
* 521b26bc2c60b7921e2f7aa5fffa620ab7f9a096 UNKNOWN
* 22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50 UNKNOWN
* 003f1c81efedafa0ddd78868ce83357d8803c5e5 UNKNOWN
* fe55d8ec4923cb6c64e83ae4f32217c72599a398 UNKNOWN
* 7b95754f0df2c1a087a6a28722a7471ee12cecf3 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8090)
* 789c2da879ca66f6b9be71d1814be38b5d8cfb58 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8102)
* 63f33fae3e22a9f80de51b2a6e1ed0427f631d0b UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT] Custom merger update [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13742:
URL: https://github.com/apache/hudi/pull/13742#issuecomment-3218255972
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7988",
"triggerID" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"triggerType" : "PUSH"
}, {
"hash" : "ecb694295f7a88371f33df6de60370058670439e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7992",
"triggerID" : "ecb694295f7a88371f33df6de60370058670439e",
"triggerType" : "PUSH"
}, {
"hash" : "efd7462406be4f6d90dd5e42f152d66f683ebdf4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "efd7462406be4f6d90dd5e42f152d66f683ebdf4",
"triggerType" : "PUSH"
}, {
"hash" : "3c67160d5ceec883db5d826ff655865a3209f2f3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7994",
"triggerID" : "3c67160d5ceec883db5d826ff655865a3209f2f3",
"triggerType" : "PUSH"
}, {
"hash" : "f4f84d9f6a480923df27cfe5553d8aa65b2bbb7a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7996",
"triggerID" : "f4f84d9f6a480923df27cfe5553d8aa65b2bbb7a",
"triggerType" : "PUSH"
}, {
"hash" : "521b26bc2c60b7921e2f7aa5fffa620ab7f9a096",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "521b26bc2c60b7921e2f7aa5fffa620ab7f9a096",
"triggerType" : "PUSH"
}, {
"hash" : "22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50",
"triggerType" : "PUSH"
}, {
"hash" : "02a319ba884595a4e3b5453f366b660c1c11f767",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8007",
"triggerID" : "02a319ba884595a4e3b5453f366b660c1c11f767",
"triggerType" : "PUSH"
}, {
"hash" : "3e59b6f8b486016911c9ad14c583f6538cc036ac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8009",
"triggerID" : "3e59b6f8b486016911c9ad14c583f6538cc036ac",
"triggerType" : "PUSH"
}, {
"hash" : "003f1c81efedafa0ddd78868ce83357d8803c5e5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "003f1c81efedafa0ddd78868ce83357d8803c5e5",
"triggerType" : "PUSH"
}, {
"hash" : "40c04768d5233a38be1b4c01ce895189a4af4c3a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8013",
"triggerID" : "40c04768d5233a38be1b4c01ce895189a4af4c3a",
"triggerType" : "PUSH"
}, {
"hash" : "4a699a8dcf79e61ea3bb754e67e9cc4a32e86aa1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8084",
"triggerID" : "4a699a8dcf79e61ea3bb754e67e9cc4a32e86aa1",
"triggerType" : "PUSH"
}, {
"hash" : "fe55d8ec4923cb6c64e83ae4f32217c72599a398",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "fe55d8ec4923cb6c64e83ae4f32217c72599a398",
"triggerType" : "PUSH"
}, {
"hash" : "87cd79787dc8126b6877ba0d3407cfe8f343933a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8087",
"triggerID" : "87cd79787dc8126b6877ba0d3407cfe8f343933a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* efd7462406be4f6d90dd5e42f152d66f683ebdf4 UNKNOWN
* 521b26bc2c60b7921e2f7aa5fffa620ab7f9a096 UNKNOWN
* 22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50 UNKNOWN
* 003f1c81efedafa0ddd78868ce83357d8803c5e5 UNKNOWN
* fe55d8ec4923cb6c64e83ae4f32217c72599a398 UNKNOWN
* 87cd79787dc8126b6877ba0d3407cfe8f343933a Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8087)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9715] Custom merger update [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13742:
URL: https://github.com/apache/hudi/pull/13742#discussion_r2300157763
##########
hudi-common/src/main/java/org/apache/hudi/common/model/HoodieRecordMerger.java:
##########
@@ -63,8 +63,15 @@ public interface HoodieRecordMerger extends Serializable {
* It'd be associative operation: f(a, f(b, c)) = f(f(a, b), c) (which we can translate as having 3 versions A, B, C
* of the single record, both orders of operations applications have to yield the same result)
* This method takes only full records for merging.
+ *
+ * @param older Older record in terms of commit time ordering.
+ * @param oldSchema The schema of the older record.
+ * @param newer Newer record in terms of commit time ordering.
+ * @param newSchema The schema of the newer record.
+ * @param props The additional properties for the merging operation.
+ * @return The merged record and schema. The record is expected to be non-null. If the record represents a deletion, the operation must be set as {@link HoodieOperation#DELETE}.
*/
- Option<Pair<HoodieRecord, Schema>> merge(HoodieRecord older, Schema oldSchema, HoodieRecord newer, Schema newSchema, TypedProperties props) throws IOException;
+ Pair<HoodieRecord, Schema> merge(HoodieRecord older, Schema oldSchema, HoodieRecord newer, Schema newSchema, TypedProperties props) throws IOException;
Review Comment:
if we just takes two buffered records, things would be much easier:
```java
BufferedRecord merge(BufferedRecord older, BufferedRecord newer, RecordContext context);
```
the benefits to do this:
- BufferedRecord is the core abstraction for merging, it is more easier to evolve in the future;
- avoid unnecessary hoodie record -> buffered record -> hoodie record conversion;
- more clear impl/semantics for deletes handling in mergers.
We just need to rename the `BufferedRecord` to be a more general name when it been used as a user API.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9715] Custom merger update [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #13742:
URL: https://github.com/apache/hudi/pull/13742#discussion_r2299386989
##########
hudi-common/src/main/java/org/apache/hudi/common/model/HoodieRecordMerger.java:
##########
@@ -63,8 +63,15 @@ public interface HoodieRecordMerger extends Serializable {
* It'd be associative operation: f(a, f(b, c)) = f(f(a, b), c) (which we can translate as having 3 versions A, B, C
* of the single record, both orders of operations applications have to yield the same result)
* This method takes only full records for merging.
+ *
+ * @param older Older record in terms of commit time ordering.
+ * @param oldSchema The schema of the older record.
+ * @param newer Newer record in terms of commit time ordering.
+ * @param newSchema The schema of the newer record.
+ * @param props The additional properties for the merging operation.
+ * @return The merged record and schema. The record is expected to be non-null. If the record represents a deletion, the operation must be set as {@link HoodieOperation#DELETE}.
Review Comment:
thanks for the java docs. on similar lines, can we enhance java docs for BufferedRecordMerger apis as well. esply when is Option.empty expected to be returned.
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieWriteMergeHandle.java:
##########
@@ -236,19 +236,19 @@ public boolean isEmptyNewRecords() {
return keyToNewRecords.isEmpty();
}
- protected boolean writeUpdateRecord(HoodieRecord<T> newRecord, HoodieRecord<T> oldRecord, Option<HoodieRecord> combineRecordOpt, Schema writerSchema) throws IOException {
+ protected boolean writeUpdateRecord(HoodieRecord<T> newRecord, HoodieRecord<T> oldRecord, HoodieRecord combineRecord, Schema writerSchema) throws IOException {
boolean isDelete = false;
- if (combineRecordOpt.isPresent()) {
- if (oldRecord.getData() != combineRecordOpt.get().getData()) {
- // the incoming record is chosen
- isDelete = isDeleteRecord(newRecord);
- } else {
- // the incoming record is dropped
- return false;
+ if (oldRecord.getData() != combineRecord.getData()) {
+ // the incoming record is chosen
+ isDelete = isDeleteRecord(combineRecord);
+ if (!isDelete) {
+ updatedRecordsWritten++;
}
- updatedRecordsWritten++;
+ } else {
+ // the incoming record is dropped
Review Comment:
can you help me understand something in this context.
say we have rk1_v1 and we have new incoming rg1_v2.
and v2 has lower ordering value.
So, L 241 is false (bcoz, old matches the combined record)
and don't we need to write the old record to the new file? how come we are returning right away in L 249?
##########
hudi-common/src/main/java/org/apache/hudi/common/model/HoodieAvroRecordMerger.java:
##########
@@ -43,9 +44,35 @@ public String getMergingStrategy() {
}
@Override
- public Option<Pair<HoodieRecord, Schema>> merge(HoodieRecord older, Schema oldSchema, HoodieRecord newer, Schema newSchema, TypedProperties props) throws IOException {
- return combineAndGetUpdateValue(older, newer, oldSchema, newSchema, props)
- .map(r -> Pair.of(new HoodieAvroIndexedRecord(r), r.getSchema()));
+ public Pair<HoodieRecord, Schema> merge(HoodieRecord older, Schema oldSchema, HoodieRecord newer, Schema newSchema, TypedProperties props) throws IOException {
Review Comment:
can you confirm we added UTs for this change?
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/commit/BaseWriteHelper.java:
##########
@@ -161,8 +160,7 @@ protected static <T> HoodieRecord<T> reduceRecords(TypedProperties props, Buffer
HoodieRecord<T> reducedRecord = merged.map(bufferedRecord -> recordContext.constructHoodieRecord(bufferedRecord, next.getPartitionPath())).orElse(previous);
boolean choosePrevious = merged.isEmpty();
HoodieKey reducedKey = choosePrevious ? previous.getKey() : next.getKey();
- HoodieOperation operation = choosePrevious ? previous.getOperation() : next.getOperation();
- return reducedRecord.newInstance(reducedKey, operation);
+ return reducedRecord.newInstance(reducedKey);
Review Comment:
are we not interested in operation for deduping use-case and hence we are not carrying the operation over to the reducedRecord?
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieWriteMergeHandle.java:
##########
@@ -236,19 +236,19 @@ public boolean isEmptyNewRecords() {
return keyToNewRecords.isEmpty();
}
- protected boolean writeUpdateRecord(HoodieRecord<T> newRecord, HoodieRecord<T> oldRecord, Option<HoodieRecord> combineRecordOpt, Schema writerSchema) throws IOException {
+ protected boolean writeUpdateRecord(HoodieRecord<T> newRecord, HoodieRecord<T> oldRecord, HoodieRecord combineRecord, Schema writerSchema) throws IOException {
boolean isDelete = false;
- if (combineRecordOpt.isPresent()) {
- if (oldRecord.getData() != combineRecordOpt.get().getData()) {
- // the incoming record is chosen
- isDelete = isDeleteRecord(newRecord);
- } else {
- // the incoming record is dropped
- return false;
+ if (oldRecord.getData() != combineRecord.getData()) {
+ // the incoming record is chosen
+ isDelete = isDeleteRecord(combineRecord);
+ if (!isDelete) {
+ updatedRecordsWritten++;
}
- updatedRecordsWritten++;
+ } else {
+ // the incoming record is dropped
Review Comment:
ok, I see the caller is handing the copying of old record.
Is there any different handling we do at the caller end?
why not handle writing both combined or old record here at L 251 only?
may be we have some valid reason, just trying to understand the reasoning
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/BufferedRecordMergerFactory.java:
##########
@@ -338,16 +334,15 @@ public Option<DeleteRecord> deltaMerge(DeleteRecord deleteRecord, BufferedRecord
public BufferedRecord<T> finalMerge(BufferedRecord<T> olderRecord, BufferedRecord<T> newerRecord) throws IOException {
// TODO(HUDI-7843): decouple the merging logic from the merger
// and use the record merge mode to control how to merge partial updates
- Option<Pair<HoodieRecord, Schema>> mergedRecord = recordMerger.get().partialMerge(
+ Pair<HoodieRecord, Schema> mergedRecord = recordMerger.get().partialMerge(
recordContext.constructHoodieRecord(olderRecord), recordContext.getSchemaFromBufferRecord(olderRecord),
recordContext.constructHoodieRecord(newerRecord), recordContext.getSchemaFromBufferRecord(newerRecord),
readerSchema, props);
- if (mergedRecord.isPresent()
- && !mergedRecord.get().getLeft().isDelete(mergedRecord.get().getRight(), props)) {
- HoodieRecord hoodieRecord = mergedRecord.get().getLeft();
- if (!mergedRecord.get().getRight().equals(readerSchema)) {
- hoodieRecord = hoodieRecord.rewriteRecordWithNewSchema(mergedRecord.get().getRight(), null, readerSchema);
+ if (!mergedRecord.getLeft().isDelete(mergedRecord.getRight(), props)) {
+ HoodieRecord hoodieRecord = mergedRecord.getLeft();
+ if (!mergedRecord.getRight().equals(readerSchema)) {
+ hoodieRecord = hoodieRecord.rewriteRecordWithNewSchema(mergedRecord.getRight(), null, readerSchema);
Review Comment:
I remember there was a discussion around retaining ordering values for delete records as well.
I assume we are not fixing them in this patch right?
for eg, L 349 does not carry over the ordering value. so, just wanted to confirm my understanding.
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/BufferedRecordMergerFactory.java:
##########
@@ -465,46 +451,45 @@ public CustomPayloadRecordMerger(
}
@Override
- public Option<BufferedRecord<T>> deltaMergeNonDeleteRecord(BufferedRecord<T> newRecord, BufferedRecord<T> existingRecord) throws IOException {
- Option<Pair<HoodieRecord, Schema>> mergedRecordAndSchema = getMergedRecord(existingRecord, newRecord, false);
- if (mergedRecordAndSchema.isEmpty()) {
- // An empty Option indicates that the output represents a delete.
- return Option.of(new BufferedRecord<>(newRecord.getRecordKey(), OrderingValues.getDefault(), null, null, HoodieOperation.DELETE));
- }
- HoodieRecord mergedRecord = mergedRecordAndSchema.get().getLeft();
- Schema mergeResultSchema = mergedRecordAndSchema.get().getRight();
+ public Option<BufferedRecord<T>> deltaMergeRecords(BufferedRecord<T> newRecord, BufferedRecord<T> existingRecord) throws IOException {
+ Pair<HoodieRecord, Schema> mergedRecordAndSchema = getMergedRecord(existingRecord, newRecord, false);
+ HoodieRecord mergedRecord = mergedRecordAndSchema.getLeft();
+ Schema mergeResultSchema = mergedRecordAndSchema.getRight();
// Special handling for SENTINEL record in Expression Payload. This is returned if the condition does not match.
if (mergedRecord.getData() == HoodieRecord.SENTINEL) {
return Option.empty();
}
- T combinedRecordData = recordContext.convertAvroRecord(mergedRecord.toIndexedRecord(mergeResultSchema, props).get().getData());
+ Option<T> combinedRecordData = (mergedRecord.toIndexedRecord(mergeResultSchema, props).map(indexedRecord -> recordContext.convertAvroRecord(indexedRecord.getData())));
Review Comment:
minor: now that we have a separate merger for ExpressionPayload, do we need L 459 to 461?
##########
hudi-common/src/main/java/org/apache/hudi/common/model/HoodieRecordMerger.java:
##########
@@ -136,6 +136,7 @@ default Option<Pair<HoodieRecord, Schema>> partialMerge(HoodieRecord older, Sche
*
* <p> This interface is experimental and might be evolved in the future.
**/
+ @Deprecated
Review Comment:
I vote to clean it up
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/BufferedRecordMergerFactory.java:
##########
@@ -338,16 +334,15 @@ public Option<DeleteRecord> deltaMerge(DeleteRecord deleteRecord, BufferedRecord
public BufferedRecord<T> finalMerge(BufferedRecord<T> olderRecord, BufferedRecord<T> newerRecord) throws IOException {
// TODO(HUDI-7843): decouple the merging logic from the merger
// and use the record merge mode to control how to merge partial updates
- Option<Pair<HoodieRecord, Schema>> mergedRecord = recordMerger.get().partialMerge(
+ Pair<HoodieRecord, Schema> mergedRecord = recordMerger.get().partialMerge(
Review Comment:
minor: Is it an intentional choice that we name "combinedRecord" above, and "mergedRecord" here.
why not use the same name everywhere.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9715] Custom merger update [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13742:
URL: https://github.com/apache/hudi/pull/13742#discussion_r2299731466
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/commit/BaseWriteHelper.java:
##########
@@ -161,8 +160,7 @@ protected static <T> HoodieRecord<T> reduceRecords(TypedProperties props, Buffer
HoodieRecord<T> reducedRecord = merged.map(bufferedRecord -> recordContext.constructHoodieRecord(bufferedRecord, next.getPartitionPath())).orElse(previous);
boolean choosePrevious = merged.isEmpty();
HoodieKey reducedKey = choosePrevious ? previous.getKey() : next.getKey();
- HoodieOperation operation = choosePrevious ? previous.getOperation() : next.getOperation();
- return reducedRecord.newInstance(reducedKey, operation);
+ return reducedRecord.newInstance(reducedKey);
Review Comment:
The `recordContext.constructHoodieRecord(bufferedRecord, next.getPartitionPath())` keeps the original operation of the hoodie record, the operation is inferred from there which has the correct handling of `isDelete`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9715] Custom merger update [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13742:
URL: https://github.com/apache/hudi/pull/13742#discussion_r2299907678
##########
hudi-common/src/main/java/org/apache/hudi/common/model/DefaultHoodieRecordPayload.java:
##########
@@ -172,10 +168,18 @@ protected boolean needUpdatingPersistedRecord(IndexedRecord currentValue,
Comparable persistedOrderingVal = OrderingValues.create(
orderingFields,
field -> (Comparable) HoodieAvroUtils.getNestedFieldVal((GenericRecord) currentValue, field, true, consistentLogicalTimestampEnabled));
- Comparable incomingOrderingVal = OrderingValues.create(
- orderingFields,
- field -> (Comparable) HoodieAvroUtils.getNestedFieldVal((GenericRecord) incomingRecord, field, true, consistentLogicalTimestampEnabled));
+ Comparable incomingOrderingVal = getIncomingOrderingVal(incomingRecord, orderingFields, consistentLogicalTimestampEnabled);
+ // If the incoming record is a delete record without an ordering value, it is processed as "commit time" ordering.
+ if (incomingRecord.isEmpty() && incomingOrderingVal == OrderingValues.getDefault()) {
Review Comment:
use `OrderingValues.isDefault`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9715] Custom merger update [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13742:
URL: https://github.com/apache/hudi/pull/13742#discussion_r2308930419
##########
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/testutils/HoodieSparkDeleteRecordMerger.java:
##########
@@ -21,11 +21,9 @@
import org.apache.hudi.HoodieSparkRecordMerger;
import org.apache.hudi.common.config.TypedProperties;
-import org.apache.hudi.common.model.HoodieRecord;
-import org.apache.hudi.common.util.Option;
-import org.apache.hudi.common.util.collection.Pair;
-
-import org.apache.avro.Schema;
+import org.apache.hudi.common.engine.RecordContext;
Review Comment:
Yes, that is how it is structured. Package will always be separated from the first import by 1 line in the Hudi style guide
<img width="506" height="195" alt="image" src="https://github.com/user-attachments/assets/26ce5887-5f77-4b93-8546-f43b7ad18200" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9715] Custom merger update [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13742:
URL: https://github.com/apache/hudi/pull/13742#discussion_r2308947419
##########
hudi-common/src/main/java/org/apache/hudi/common/model/HoodieRecordMerger.java:
##########
@@ -228,4 +228,11 @@ static String getCustomRecordMergeStrategyId(String payloadClassName, String rec
return null;
}
}
+
+ static <T> boolean mergingCommitTimeOrderedDelete(BufferedRecord<T> oldRecord, BufferedRecord<T> newRecord) {
Review Comment:
just name it `isCommitTimeOrderingDelete` should be fine.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9715] Custom merger update [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13742:
URL: https://github.com/apache/hudi/pull/13742#issuecomment-3237152157
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7988",
"triggerID" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"triggerType" : "PUSH"
}, {
"hash" : "ecb694295f7a88371f33df6de60370058670439e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7992",
"triggerID" : "ecb694295f7a88371f33df6de60370058670439e",
"triggerType" : "PUSH"
}, {
"hash" : "efd7462406be4f6d90dd5e42f152d66f683ebdf4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "efd7462406be4f6d90dd5e42f152d66f683ebdf4",
"triggerType" : "PUSH"
}, {
"hash" : "3c67160d5ceec883db5d826ff655865a3209f2f3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7994",
"triggerID" : "3c67160d5ceec883db5d826ff655865a3209f2f3",
"triggerType" : "PUSH"
}, {
"hash" : "f4f84d9f6a480923df27cfe5553d8aa65b2bbb7a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7996",
"triggerID" : "f4f84d9f6a480923df27cfe5553d8aa65b2bbb7a",
"triggerType" : "PUSH"
}, {
"hash" : "521b26bc2c60b7921e2f7aa5fffa620ab7f9a096",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "521b26bc2c60b7921e2f7aa5fffa620ab7f9a096",
"triggerType" : "PUSH"
}, {
"hash" : "22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50",
"triggerType" : "PUSH"
}, {
"hash" : "02a319ba884595a4e3b5453f366b660c1c11f767",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8007",
"triggerID" : "02a319ba884595a4e3b5453f366b660c1c11f767",
"triggerType" : "PUSH"
}, {
"hash" : "3e59b6f8b486016911c9ad14c583f6538cc036ac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8009",
"triggerID" : "3e59b6f8b486016911c9ad14c583f6538cc036ac",
"triggerType" : "PUSH"
}, {
"hash" : "003f1c81efedafa0ddd78868ce83357d8803c5e5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "003f1c81efedafa0ddd78868ce83357d8803c5e5",
"triggerType" : "PUSH"
}, {
"hash" : "40c04768d5233a38be1b4c01ce895189a4af4c3a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8013",
"triggerID" : "40c04768d5233a38be1b4c01ce895189a4af4c3a",
"triggerType" : "PUSH"
}, {
"hash" : "4a699a8dcf79e61ea3bb754e67e9cc4a32e86aa1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8084",
"triggerID" : "4a699a8dcf79e61ea3bb754e67e9cc4a32e86aa1",
"triggerType" : "PUSH"
}, {
"hash" : "fe55d8ec4923cb6c64e83ae4f32217c72599a398",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "fe55d8ec4923cb6c64e83ae4f32217c72599a398",
"triggerType" : "PUSH"
}, {
"hash" : "87cd79787dc8126b6877ba0d3407cfe8f343933a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8087",
"triggerID" : "87cd79787dc8126b6877ba0d3407cfe8f343933a",
"triggerType" : "PUSH"
}, {
"hash" : "594874ffdca647cd268b3947fa4680c57c53d679",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8088",
"triggerID" : "594874ffdca647cd268b3947fa4680c57c53d679",
"triggerType" : "PUSH"
}, {
"hash" : "7b95754f0df2c1a087a6a28722a7471ee12cecf3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8090",
"triggerID" : "7b95754f0df2c1a087a6a28722a7471ee12cecf3",
"triggerType" : "PUSH"
}, {
"hash" : "789c2da879ca66f6b9be71d1814be38b5d8cfb58",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8102",
"triggerID" : "789c2da879ca66f6b9be71d1814be38b5d8cfb58",
"triggerType" : "PUSH"
}, {
"hash" : "63f33fae3e22a9f80de51b2a6e1ed0427f631d0b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8103",
"triggerID" : "63f33fae3e22a9f80de51b2a6e1ed0427f631d0b",
"triggerType" : "PUSH"
}, {
"hash" : "e04c466fa81e9bcfaabe36825127098f5d1bb890",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8224",
"triggerID" : "e04c466fa81e9bcfaabe36825127098f5d1bb890",
"triggerType" : "PUSH"
}, {
"hash" : "cf574f46a1783a39b2074542aad9539ec3c4f36a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8235",
"triggerID" : "cf574f46a1783a39b2074542aad9539ec3c4f36a",
"triggerType" : "PUSH"
}, {
"hash" : "2eb81e923adfa77176f137c442143deaac9e9916",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2eb81e923adfa77176f137c442143deaac9e9916",
"triggerType" : "PUSH"
}, {
"hash" : "12dc3e3eaf746bb1f2bc3f4e93fa14d9adb0f6fd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8244",
"triggerID" : "12dc3e3eaf746bb1f2bc3f4e93fa14d9adb0f6fd",
"triggerType" : "PUSH"
}, {
"hash" : "af5ab73103f3af882412bbe99c9322165e8941a0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "af5ab73103f3af882412bbe99c9322165e8941a0",
"triggerType" : "PUSH"
}, {
"hash" : "a9c28c4d913644fea23d57a5dc2224f36fd2b63e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8248",
"triggerID" : "a9c28c4d913644fea23d57a5dc2224f36fd2b63e",
"triggerType" : "PUSH"
}, {
"hash" : "98381a6bd2bcccf3da08fb992e599fe1165f2ddb",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8252",
"triggerID" : "98381a6bd2bcccf3da08fb992e599fe1165f2ddb",
"triggerType" : "PUSH"
}, {
"hash" : "94aa9a669d2fe1fb4919a4735fab2665553b63c0",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8258",
"triggerID" : "94aa9a669d2fe1fb4919a4735fab2665553b63c0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* efd7462406be4f6d90dd5e42f152d66f683ebdf4 UNKNOWN
* 521b26bc2c60b7921e2f7aa5fffa620ab7f9a096 UNKNOWN
* 22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50 UNKNOWN
* 003f1c81efedafa0ddd78868ce83357d8803c5e5 UNKNOWN
* fe55d8ec4923cb6c64e83ae4f32217c72599a398 UNKNOWN
* 2eb81e923adfa77176f137c442143deaac9e9916 UNKNOWN
* af5ab73103f3af882412bbe99c9322165e8941a0 UNKNOWN
* 98381a6bd2bcccf3da08fb992e599fe1165f2ddb Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8252)
* 94aa9a669d2fe1fb4919a4735fab2665553b63c0 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8258)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT] Custom merger update [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13742:
URL: https://github.com/apache/hudi/pull/13742#issuecomment-3211316702
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7988",
"triggerID" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"triggerType" : "PUSH"
}, {
"hash" : "ecb694295f7a88371f33df6de60370058670439e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7992",
"triggerID" : "ecb694295f7a88371f33df6de60370058670439e",
"triggerType" : "PUSH"
}, {
"hash" : "efd7462406be4f6d90dd5e42f152d66f683ebdf4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "efd7462406be4f6d90dd5e42f152d66f683ebdf4",
"triggerType" : "PUSH"
}, {
"hash" : "3c67160d5ceec883db5d826ff655865a3209f2f3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7994",
"triggerID" : "3c67160d5ceec883db5d826ff655865a3209f2f3",
"triggerType" : "PUSH"
}, {
"hash" : "f4f84d9f6a480923df27cfe5553d8aa65b2bbb7a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7996",
"triggerID" : "f4f84d9f6a480923df27cfe5553d8aa65b2bbb7a",
"triggerType" : "PUSH"
}, {
"hash" : "521b26bc2c60b7921e2f7aa5fffa620ab7f9a096",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "521b26bc2c60b7921e2f7aa5fffa620ab7f9a096",
"triggerType" : "PUSH"
}, {
"hash" : "22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50",
"triggerType" : "PUSH"
}, {
"hash" : "02a319ba884595a4e3b5453f366b660c1c11f767",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8007",
"triggerID" : "02a319ba884595a4e3b5453f366b660c1c11f767",
"triggerType" : "PUSH"
}, {
"hash" : "3e59b6f8b486016911c9ad14c583f6538cc036ac",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8009",
"triggerID" : "3e59b6f8b486016911c9ad14c583f6538cc036ac",
"triggerType" : "PUSH"
}, {
"hash" : "003f1c81efedafa0ddd78868ce83357d8803c5e5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "003f1c81efedafa0ddd78868ce83357d8803c5e5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* efd7462406be4f6d90dd5e42f152d66f683ebdf4 UNKNOWN
* 521b26bc2c60b7921e2f7aa5fffa620ab7f9a096 UNKNOWN
* 22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50 UNKNOWN
* 3e59b6f8b486016911c9ad14c583f6538cc036ac Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8009)
* 003f1c81efedafa0ddd78868ce83357d8803c5e5 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(merger): [RFC-101] Update HoodieRecordMerger interface and implementations [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13742:
URL: https://github.com/apache/hudi/pull/13742#issuecomment-3305132428
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "94aa9a669d2fe1fb4919a4735fab2665553b63c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8258",
"triggerID" : "94aa9a669d2fe1fb4919a4735fab2665553b63c0",
"triggerType" : "PUSH"
}, {
"hash" : "94aa9a669d2fe1fb4919a4735fab2665553b63c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8258",
"triggerID" : "3237521433",
"triggerType" : "MANUAL"
}, {
"hash" : "57b338240d2ac6faa9ec0a329f43f13e28019eb8",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8564",
"triggerID" : "57b338240d2ac6faa9ec0a329f43f13e28019eb8",
"triggerType" : "PUSH"
}, {
"hash" : "57b338240d2ac6faa9ec0a329f43f13e28019eb8",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8564",
"triggerID" : "3305128927",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 57b338240d2ac6faa9ec0a329f43f13e28019eb8 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8564)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9715] Custom merger update [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13742:
URL: https://github.com/apache/hudi/pull/13742#discussion_r2300005688
##########
hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/DefaultHiveRecordMerger.java:
##########
@@ -39,33 +41,33 @@ public class DefaultHiveRecordMerger extends HoodieHiveRecordMerger {
private String[] orderingFields;
@Override
- public Option<Pair<HoodieRecord, Schema>> merge(HoodieRecord older, Schema oldSchema, HoodieRecord newer, Schema newSchema, TypedProperties props) throws IOException {
+ public Pair<HoodieRecord, Schema> merge(HoodieRecord older, Schema oldSchema, HoodieRecord newer, Schema newSchema, TypedProperties props) throws IOException {
ValidationUtils.checkArgument(older.getRecordType() == HoodieRecord.HoodieRecordType.HIVE);
ValidationUtils.checkArgument(newer.getRecordType() == HoodieRecord.HoodieRecordType.HIVE);
if (newer instanceof HoodieHiveRecord) {
HoodieHiveRecord newHiveRecord = (HoodieHiveRecord) newer;
if (newHiveRecord.isDelete(newSchema, props)) {
- return Option.empty();
+ return Pair.of(new HoodieEmptyRecord<>(newer.getKey(), HoodieOperation.DELETE, OrderingValues.getDefault(), HoodieRecord.HoodieRecordType.HIVE), newSchema);
Review Comment:
why not takes the ordering value from the original record instead of `OrderingValues.getDefault()`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9715] Custom merger update [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #13742:
URL: https://github.com/apache/hudi/pull/13742#discussion_r2301141732
##########
hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/HiveRecordContext.java:
##########
@@ -126,6 +125,25 @@ public Comparable convertValueToEngineType(Comparable value) {
return value;
}
+ public Comparable convertValueFromEngineType(Comparable value) {
Review Comment:
can you confirm we have added UTs for these.
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/BufferedRecordMergerFactory.java:
##########
@@ -465,46 +451,45 @@ public CustomPayloadRecordMerger(
}
@Override
- public Option<BufferedRecord<T>> deltaMergeNonDeleteRecord(BufferedRecord<T> newRecord, BufferedRecord<T> existingRecord) throws IOException {
- Option<Pair<HoodieRecord, Schema>> mergedRecordAndSchema = getMergedRecord(existingRecord, newRecord, false);
- if (mergedRecordAndSchema.isEmpty()) {
- // An empty Option indicates that the output represents a delete.
- return Option.of(new BufferedRecord<>(newRecord.getRecordKey(), OrderingValues.getDefault(), null, null, HoodieOperation.DELETE));
- }
- HoodieRecord mergedRecord = mergedRecordAndSchema.get().getLeft();
- Schema mergeResultSchema = mergedRecordAndSchema.get().getRight();
+ public Option<BufferedRecord<T>> deltaMergeRecords(BufferedRecord<T> newRecord, BufferedRecord<T> existingRecord) throws IOException {
+ Pair<HoodieRecord, Schema> mergedRecordAndSchema = getMergedRecord(existingRecord, newRecord, false);
+ HoodieRecord mergedRecord = mergedRecordAndSchema.getLeft();
+ Schema mergeResultSchema = mergedRecordAndSchema.getRight();
// Special handling for SENTINEL record in Expression Payload. This is returned if the condition does not match.
if (mergedRecord.getData() == HoodieRecord.SENTINEL) {
return Option.empty();
}
- T combinedRecordData = recordContext.convertAvroRecord(mergedRecord.toIndexedRecord(mergeResultSchema, props).get().getData());
+ Option<T> combinedRecordData = (mergedRecord.toIndexedRecord(mergeResultSchema, props).map(indexedRecord -> recordContext.convertAvroRecord(indexedRecord.getData())));
// If pre-combine does not return existing record, update it
- if (combinedRecordData != existingRecord.getRecord()) {
+ if (combinedRecordData.map(record -> record != existingRecord.getRecord()).orElse(true)) {
// For pkless we need to use record key from existing record
- return Option.of(BufferedRecords.fromEngineRecord(combinedRecordData, mergeResultSchema, recordContext, orderingFieldNames,
- existingRecord.getRecordKey(), mergedRecord.isDelete(mergeResultSchema, props)));
+ boolean isDelete = mergedRecord.isDelete(readerSchema, props);
Review Comment:
readerSchema -> mergeResultSchema
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/HoodieSparkRecordMerger.java:
##########
@@ -39,33 +41,33 @@ public HoodieRecord.HoodieRecordType getRecordType() {
* Basic handling of deletes that is used by many of the spark mergers
* returns null if merger specific logic should be used
*/
- protected Option<Pair<HoodieRecord, Schema>> handleDeletes(HoodieRecord older, Schema oldSchema, HoodieRecord newer, Schema newSchema, TypedProperties props) {
+ protected Pair<HoodieRecord, Schema> handleDeletes(HoodieRecord older, Schema oldSchema, HoodieRecord newer, Schema newSchema, TypedProperties props) {
ValidationUtils.checkArgument(older.getRecordType() == HoodieRecord.HoodieRecordType.SPARK);
ValidationUtils.checkArgument(newer.getRecordType() == HoodieRecord.HoodieRecordType.SPARK);
if (newer instanceof HoodieSparkRecord) {
HoodieSparkRecord newSparkRecord = (HoodieSparkRecord) newer;
if (newSparkRecord.isDelete(newSchema, props)) {
// Delete record
- return Option.empty();
+ return Pair.of(new HoodieEmptyRecord<>(newer.getKey(), HoodieOperation.DELETE, OrderingValues.getDefault(), HoodieRecord.HoodieRecordType.SPARK), newSchema);
Review Comment:
can you track all these inconsistencies in some jira, so that we can revisit at some point.
##########
hudi-common/src/main/java/org/apache/hudi/common/model/DefaultHoodieRecordPayload.java:
##########
@@ -74,11 +74,7 @@ public OverwriteWithLatestAvroPayload preCombine(OverwriteWithLatestAvroPayload
@Override
public Option<IndexedRecord> combineAndGetUpdateValue(IndexedRecord currentValue, Schema schema, Properties properties) throws IOException {
- if (recordBytes.length == 0) {
- return Option.empty();
- }
-
- GenericRecord incomingRecord = HoodieAvroUtils.bytesToAvro(recordBytes, schema);
+ Option<IndexedRecord> incomingRecord = recordBytes.length == 0 ? Option.empty() : Option.of(HoodieAvroUtils.bytesToAvro(recordBytes, schema));
Review Comment:
danny: diff is, previously, we return right away if bytes is empty. but now, we do not. we proceed after line 77. and thats where the fix applies that tim responded above.
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/BufferedRecordMergerFactory.java:
##########
@@ -567,42 +553,36 @@ public Option<BufferedRecord<T>> deltaMerge(BufferedRecord<T> newRecord, Buffere
if (existingRecord == null) {
return Option.of(newRecord);
}
- if (existingRecord.isDelete() || newRecord.isDelete()) {
- if (shouldKeepNewerRecord(existingRecord, newRecord)) {
- // IMPORTANT:
- // this is needed when the fallback HoodieAvroRecordMerger got used, the merger would
- // return Option.empty when the old payload data is empty(a delete) and ignores its ordering value directly.
- return Option.of(newRecord);
- } else {
- return Option.empty();
- }
- }
- return deltaMergeNonDeleteRecord(newRecord, existingRecord);
+ return deltaMergeRecords(newRecord, existingRecord);
}
- public abstract Option<BufferedRecord<T>> deltaMergeNonDeleteRecord(BufferedRecord<T> newRecord, BufferedRecord<T> existingRecord) throws IOException;
+ public abstract Option<BufferedRecord<T>> deltaMergeRecords(BufferedRecord<T> newRecord, BufferedRecord<T> existingRecord) throws IOException;
@Override
public Option<DeleteRecord> deltaMerge(DeleteRecord deleteRecord, BufferedRecord<T> existingRecord) {
- return deltaMergeDeleteRecord(deleteRecord, existingRecord);
+ BufferedRecord<T> deleteBufferedRecord = BufferedRecords.fromDeleteRecord(deleteRecord, recordContext);
+ try {
+ Option<BufferedRecord<T>> merged = deltaMerge(deleteBufferedRecord, existingRecord);
+ // If the delete record is chosen, return an option with the delete record, otherwise return empty.
+ return merged.isPresent() ? Option.of(deleteRecord) : Option.empty();
+ } catch (IOException e) {
+ throw new HoodieIOException("Failed to process delete record", e);
+ }
}
@Override
public BufferedRecord<T> finalMerge(BufferedRecord<T> olderRecord, BufferedRecord<T> newerRecord) throws IOException {
- if (olderRecord.isDelete() || newerRecord.isDelete()) {
- if (shouldKeepNewerRecord(olderRecord, newerRecord)) {
- // IMPORTANT:
- // this is needed when the fallback HoodieAvroRecordMerger got used, the merger would
- // return Option.empty when the new payload data is empty(a delete) and ignores its ordering value directly.
- return newerRecord;
- } else {
- return olderRecord;
- }
+ if (olderRecord == null) {
+ return newerRecord;
+ }
+ // handle special case for deletes that are sent to older partitions in global-index, this delete takes precedence regardless of the previous value
+ if (newerRecord.getHoodieOperation() == HoodieOperation.UPDATE_BEFORE) {
Review Comment:
in case of global index, won't the delete record will have commit time ordering value. So, merging w/ existing records for older partitions should work w/o this special handling is what I thought.
can you help explain why we need this special handling?
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/BufferedRecordMergerFactory.java:
##########
@@ -465,46 +451,45 @@ public CustomPayloadRecordMerger(
}
@Override
- public Option<BufferedRecord<T>> deltaMergeNonDeleteRecord(BufferedRecord<T> newRecord, BufferedRecord<T> existingRecord) throws IOException {
- Option<Pair<HoodieRecord, Schema>> mergedRecordAndSchema = getMergedRecord(existingRecord, newRecord, false);
- if (mergedRecordAndSchema.isEmpty()) {
- // An empty Option indicates that the output represents a delete.
- return Option.of(new BufferedRecord<>(newRecord.getRecordKey(), OrderingValues.getDefault(), null, null, HoodieOperation.DELETE));
- }
- HoodieRecord mergedRecord = mergedRecordAndSchema.get().getLeft();
- Schema mergeResultSchema = mergedRecordAndSchema.get().getRight();
+ public Option<BufferedRecord<T>> deltaMergeRecords(BufferedRecord<T> newRecord, BufferedRecord<T> existingRecord) throws IOException {
+ Pair<HoodieRecord, Schema> mergedRecordAndSchema = getMergedRecord(existingRecord, newRecord, false);
+ HoodieRecord mergedRecord = mergedRecordAndSchema.getLeft();
+ Schema mergeResultSchema = mergedRecordAndSchema.getRight();
// Special handling for SENTINEL record in Expression Payload. This is returned if the condition does not match.
if (mergedRecord.getData() == HoodieRecord.SENTINEL) {
return Option.empty();
}
- T combinedRecordData = recordContext.convertAvroRecord(mergedRecord.toIndexedRecord(mergeResultSchema, props).get().getData());
+ Option<T> combinedRecordData = (mergedRecord.toIndexedRecord(mergeResultSchema, props).map(indexedRecord -> recordContext.convertAvroRecord(indexedRecord.getData())));
Review Comment:
essentially whereever we are handling SENTINEL, can we remove from other mergers?
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/BufferedRecordMergerFactory.java:
##########
@@ -465,46 +451,45 @@ public CustomPayloadRecordMerger(
}
@Override
- public Option<BufferedRecord<T>> deltaMergeNonDeleteRecord(BufferedRecord<T> newRecord, BufferedRecord<T> existingRecord) throws IOException {
- Option<Pair<HoodieRecord, Schema>> mergedRecordAndSchema = getMergedRecord(existingRecord, newRecord, false);
- if (mergedRecordAndSchema.isEmpty()) {
- // An empty Option indicates that the output represents a delete.
- return Option.of(new BufferedRecord<>(newRecord.getRecordKey(), OrderingValues.getDefault(), null, null, HoodieOperation.DELETE));
- }
- HoodieRecord mergedRecord = mergedRecordAndSchema.get().getLeft();
- Schema mergeResultSchema = mergedRecordAndSchema.get().getRight();
+ public Option<BufferedRecord<T>> deltaMergeRecords(BufferedRecord<T> newRecord, BufferedRecord<T> existingRecord) throws IOException {
+ Pair<HoodieRecord, Schema> mergedRecordAndSchema = getMergedRecord(existingRecord, newRecord, false);
+ HoodieRecord mergedRecord = mergedRecordAndSchema.getLeft();
+ Schema mergeResultSchema = mergedRecordAndSchema.getRight();
// Special handling for SENTINEL record in Expression Payload. This is returned if the condition does not match.
if (mergedRecord.getData() == HoodieRecord.SENTINEL) {
return Option.empty();
}
- T combinedRecordData = recordContext.convertAvroRecord(mergedRecord.toIndexedRecord(mergeResultSchema, props).get().getData());
+ Option<T> combinedRecordData = (mergedRecord.toIndexedRecord(mergeResultSchema, props).map(indexedRecord -> recordContext.convertAvroRecord(indexedRecord.getData())));
// If pre-combine does not return existing record, update it
- if (combinedRecordData != existingRecord.getRecord()) {
+ if (combinedRecordData.map(record -> record != existingRecord.getRecord()).orElse(true)) {
// For pkless we need to use record key from existing record
- return Option.of(BufferedRecords.fromEngineRecord(combinedRecordData, mergeResultSchema, recordContext, orderingFieldNames,
- existingRecord.getRecordKey(), mergedRecord.isDelete(mergeResultSchema, props)));
+ boolean isDelete = mergedRecord.isDelete(readerSchema, props);
+ Comparable orderingValue = mergedRecord.getOrderingValue(mergeResultSchema, props, orderingFieldNames);
+ T mergedEngineRecord = mergedRecord.toIndexedRecord(mergeResultSchema, props)
+ .map(hoodieAvroIndexedRecord -> recordContext.convertAvroRecord(hoodieAvroIndexedRecord.getData()))
+ .orElse(null);
+ return Option.of(BufferedRecords.fromEngineRecord(mergedEngineRecord, readerSchema, recordContext, orderingValue, existingRecord.getRecordKey(), isDelete));
+
}
return Option.empty();
}
@Override
- public BufferedRecord<T> mergeNonDeleteRecord(BufferedRecord<T> olderRecord, BufferedRecord<T> newerRecord) throws IOException {
- Option<Pair<HoodieRecord, Schema>> mergedRecordAndSchema = getMergedRecord(olderRecord, newerRecord, true);
- if (mergedRecordAndSchema.isEmpty()) {
- return BufferedRecords.createDelete(newerRecord.getRecordKey());
- }
- HoodieRecord mergedRecord = mergedRecordAndSchema.get().getLeft();
- Schema mergeResultSchema = mergedRecordAndSchema.get().getRight();
+ public BufferedRecord<T> mergeRecords(BufferedRecord<T> olderRecord, BufferedRecord<T> newerRecord) throws IOException {
+ Pair<HoodieRecord, Schema> mergedRecordAndSchema = getMergedRecord(olderRecord, newerRecord, true);
+
+ HoodieRecord mergedRecord = mergedRecordAndSchema.getLeft();
+ Schema mergeResultSchema = mergedRecordAndSchema.getRight();
// Special handling for SENTINEL record in Expression Payload
if (mergedRecord.getData() == HoodieRecord.SENTINEL) {
return olderRecord;
}
- if (!mergedRecord.isDelete(mergeResultSchema, props)) {
- IndexedRecord indexedRecord = (IndexedRecord) mergedRecord.getData();
- return BufferedRecords.fromEngineRecord(
- recordContext.convertAvroRecord(indexedRecord), mergeResultSchema, recordContext, orderingFieldNames, newerRecord.getRecordKey(), false);
- }
- return BufferedRecords.createDelete(newerRecord.getRecordKey());
+ boolean isDelete = mergedRecord.isDelete(readerSchema, props);
+ Comparable orderingValue = mergedRecord.getOrderingValue(mergeResultSchema, props, orderingFieldNames);
+ T mergedEngineRecord = mergedRecord.toIndexedRecord(mergeResultSchema, props)
Review Comment:
I am with you Danny. we should def fix such unnecessary conversions.
one thing I am worried about is the time to get this patch merged and release 1.1.
in general, lets categorize functional and performance, and within performance, whether its regressing from existing behavior or not.
For perf enhancements which on par w/ existing code, I would recommend, we track these in a follow up jira and attend to them post 1.1.
We wanted to get everything done by squeezing every performance issues that we spot, it keeps delaying the release.
I am pretty sure if we introduce the changes you are proposing, we might have to fix non trivial amount of places.
##########
hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/DefaultHiveRecordMerger.java:
##########
@@ -39,33 +41,33 @@ public class DefaultHiveRecordMerger extends HoodieHiveRecordMerger {
private String[] orderingFields;
@Override
- public Option<Pair<HoodieRecord, Schema>> merge(HoodieRecord older, Schema oldSchema, HoodieRecord newer, Schema newSchema, TypedProperties props) throws IOException {
+ public Pair<HoodieRecord, Schema> merge(HoodieRecord older, Schema oldSchema, HoodieRecord newer, Schema newSchema, TypedProperties props) throws IOException {
ValidationUtils.checkArgument(older.getRecordType() == HoodieRecord.HoodieRecordType.HIVE);
ValidationUtils.checkArgument(newer.getRecordType() == HoodieRecord.HoodieRecordType.HIVE);
if (newer instanceof HoodieHiveRecord) {
HoodieHiveRecord newHiveRecord = (HoodieHiveRecord) newer;
if (newHiveRecord.isDelete(newSchema, props)) {
- return Option.empty();
+ return Pair.of(new HoodieEmptyRecord<>(newer.getKey(), HoodieOperation.DELETE, OrderingValues.getDefault(), HoodieRecord.HoodieRecordType.HIVE), newSchema);
Review Comment:
I also noticed this gap some where else.
if this is not breaking existing behavior, can we file a follow up jira and revisit entire deletes + ordering value gaps separately rather than trying to fix it in the same patch.
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/BufferedRecordMergerFactory.java:
##########
@@ -465,46 +451,45 @@ public CustomPayloadRecordMerger(
}
@Override
- public Option<BufferedRecord<T>> deltaMergeNonDeleteRecord(BufferedRecord<T> newRecord, BufferedRecord<T> existingRecord) throws IOException {
- Option<Pair<HoodieRecord, Schema>> mergedRecordAndSchema = getMergedRecord(existingRecord, newRecord, false);
- if (mergedRecordAndSchema.isEmpty()) {
- // An empty Option indicates that the output represents a delete.
- return Option.of(new BufferedRecord<>(newRecord.getRecordKey(), OrderingValues.getDefault(), null, null, HoodieOperation.DELETE));
- }
- HoodieRecord mergedRecord = mergedRecordAndSchema.get().getLeft();
- Schema mergeResultSchema = mergedRecordAndSchema.get().getRight();
+ public Option<BufferedRecord<T>> deltaMergeRecords(BufferedRecord<T> newRecord, BufferedRecord<T> existingRecord) throws IOException {
+ Pair<HoodieRecord, Schema> mergedRecordAndSchema = getMergedRecord(existingRecord, newRecord, false);
+ HoodieRecord mergedRecord = mergedRecordAndSchema.getLeft();
+ Schema mergeResultSchema = mergedRecordAndSchema.getRight();
// Special handling for SENTINEL record in Expression Payload. This is returned if the condition does not match.
if (mergedRecord.getData() == HoodieRecord.SENTINEL) {
return Option.empty();
}
- T combinedRecordData = recordContext.convertAvroRecord(mergedRecord.toIndexedRecord(mergeResultSchema, props).get().getData());
+ Option<T> combinedRecordData = (mergedRecord.toIndexedRecord(mergeResultSchema, props).map(indexedRecord -> recordContext.convertAvroRecord(indexedRecord.getData())));
// If pre-combine does not return existing record, update it
- if (combinedRecordData != existingRecord.getRecord()) {
+ if (combinedRecordData.map(record -> record != existingRecord.getRecord()).orElse(true)) {
// For pkless we need to use record key from existing record
- return Option.of(BufferedRecords.fromEngineRecord(combinedRecordData, mergeResultSchema, recordContext, orderingFieldNames,
- existingRecord.getRecordKey(), mergedRecord.isDelete(mergeResultSchema, props)));
+ boolean isDelete = mergedRecord.isDelete(readerSchema, props);
+ Comparable orderingValue = mergedRecord.getOrderingValue(mergeResultSchema, props, orderingFieldNames);
+ T mergedEngineRecord = mergedRecord.toIndexedRecord(mergeResultSchema, props)
Review Comment:
yes. combinedRecordData at L462 and mergedEngineRecord at L468 looks to be same. we should avoid it
##########
hudi-spark-datasource/hudi-spark/src/test/java/org/apache/hudi/io/CustomPayload.java:
##########
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.io;
+
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordPayload;
+import org.apache.hudi.common.util.Option;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.IndexedRecord;
+
+import java.io.IOException;
+
+public class CustomPayload implements HoodieRecordPayload<CustomPayload> {
+ private final GenericRecord record;
+ private final Comparable orderingValue;
+
+ public CustomPayload(GenericRecord record, Comparable orderingValue) {
+ this.record = record;
+ this.orderingValue = orderingValue;
+ }
+
+ @Override
+ public CustomPayload preCombine(CustomPayload other) {
+ return this; // No-op for this test
+ }
+
+ @Override
+ public Option<IndexedRecord> combineAndGetUpdateValue(IndexedRecord currentValue, Schema schema) {
+ Long olderTimestamp = (Long) ((GenericRecord) currentValue).get("timestamp");
+ Long newerTimestamp = orderingValue == null ? (Long) record.get("timestamp") : (Long) orderingValue;
+ if (olderTimestamp.equals(newerTimestamp)) {
+ // If the timestamps are the same, we do not update
+ return handleDeleteRecord(currentValue);
+ } else if (olderTimestamp < newerTimestamp) {
+ // Custom merger chooses record with lower ordering value
Review Comment:
minor. "Custom merger" -> "Custom payload"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9715] Custom merger update [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13742:
URL: https://github.com/apache/hudi/pull/13742#discussion_r2302697064
##########
hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/DefaultHiveRecordMerger.java:
##########
@@ -39,33 +41,33 @@ public class DefaultHiveRecordMerger extends HoodieHiveRecordMerger {
private String[] orderingFields;
@Override
- public Option<Pair<HoodieRecord, Schema>> merge(HoodieRecord older, Schema oldSchema, HoodieRecord newer, Schema newSchema, TypedProperties props) throws IOException {
+ public Pair<HoodieRecord, Schema> merge(HoodieRecord older, Schema oldSchema, HoodieRecord newer, Schema newSchema, TypedProperties props) throws IOException {
ValidationUtils.checkArgument(older.getRecordType() == HoodieRecord.HoodieRecordType.HIVE);
ValidationUtils.checkArgument(newer.getRecordType() == HoodieRecord.HoodieRecordType.HIVE);
if (newer instanceof HoodieHiveRecord) {
HoodieHiveRecord newHiveRecord = (HoodieHiveRecord) newer;
if (newHiveRecord.isDelete(newSchema, props)) {
- return Option.empty();
+ return Pair.of(new HoodieEmptyRecord<>(newer.getKey(), HoodieOperation.DELETE, OrderingValues.getDefault(), HoodieRecord.HoodieRecordType.HIVE), newSchema);
Review Comment:
not really, this is a regression, before this path, all the engines work with disorder deletes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9715] Custom merger update [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13742:
URL: https://github.com/apache/hudi/pull/13742#discussion_r2302697064
##########
hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/DefaultHiveRecordMerger.java:
##########
@@ -39,33 +41,33 @@ public class DefaultHiveRecordMerger extends HoodieHiveRecordMerger {
private String[] orderingFields;
@Override
- public Option<Pair<HoodieRecord, Schema>> merge(HoodieRecord older, Schema oldSchema, HoodieRecord newer, Schema newSchema, TypedProperties props) throws IOException {
+ public Pair<HoodieRecord, Schema> merge(HoodieRecord older, Schema oldSchema, HoodieRecord newer, Schema newSchema, TypedProperties props) throws IOException {
ValidationUtils.checkArgument(older.getRecordType() == HoodieRecord.HoodieRecordType.HIVE);
ValidationUtils.checkArgument(newer.getRecordType() == HoodieRecord.HoodieRecordType.HIVE);
if (newer instanceof HoodieHiveRecord) {
HoodieHiveRecord newHiveRecord = (HoodieHiveRecord) newer;
if (newHiveRecord.isDelete(newSchema, props)) {
- return Option.empty();
+ return Pair.of(new HoodieEmptyRecord<>(newer.getKey(), HoodieOperation.DELETE, OrderingValues.getDefault(), HoodieRecord.HoodieRecordType.HIVE), newSchema);
Review Comment:
not really, this is a regression, before this path, it works with disorder deletes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9715] Custom merger update [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13742:
URL: https://github.com/apache/hudi/pull/13742#discussion_r2302495296
##########
hudi-common/src/main/java/org/apache/hudi/common/model/HoodieAvroRecordMerger.java:
##########
@@ -43,9 +44,35 @@ public String getMergingStrategy() {
}
@Override
- public Option<Pair<HoodieRecord, Schema>> merge(HoodieRecord older, Schema oldSchema, HoodieRecord newer, Schema newSchema, TypedProperties props) throws IOException {
- return combineAndGetUpdateValue(older, newer, oldSchema, newSchema, props)
- .map(r -> Pair.of(new HoodieAvroIndexedRecord(r), r.getSchema()));
+ public Pair<HoodieRecord, Schema> merge(HoodieRecord older, Schema oldSchema, HoodieRecord newer, Schema newSchema, TypedProperties props) throws IOException {
+ Option<IndexedRecord> previousAvroData = older.toIndexedRecord(oldSchema, props).map(HoodieAvroIndexedRecord::getData);
+ HoodieRecordPayload payload = ((HoodieAvroRecord) newer).getData();
+ Option<IndexedRecord> updatedValue;
+ Comparable orderingVal;
+ if (previousAvroData.isEmpty()) {
+ updatedValue = payload.getInsertValue(newSchema, props);
+ orderingVal = payload.getOrderingValue();
+ } else {
+ updatedValue = ((HoodieAvroRecord) newer).getData().combineAndGetUpdateValue(previousAvroData.get(), newSchema, props);
+ if (updatedValue.map(value -> value == previousAvroData.get()).orElse(false)) {
+ // Ordering value is set during record creation if one is required for this payload, so ordering fields are not required here
+ return Pair.of(older, oldSchema);
+ } else {
+ orderingVal = payload.getOrderingValue();
+ }
+ }
+ if (updatedValue.isEmpty()) {
+ // pass through the record that results in the delete operation. If both newer and older are deletes, we return the newer one.
+ if (newer.isDelete(newSchema, props)) {
+ return Pair.of(newer.newInstance(newer.getKey(), HoodieOperation.DELETE), newSchema);
+ } else if (older.isDelete(oldSchema, props)) {
+ // if older is delete, return it
+ return Pair.of(older.newInstance(newer.getKey(), HoodieOperation.DELETE), oldSchema);
+ }
+ // If neither of the records are a delete, return a delete record with an empty value and default ordering value.
+ return Pair.of(new HoodieEmptyRecord<>(newer.getKey(), HoodieOperation.DELETE, OrderingValues.getDefault(), HoodieRecordType.AVRO), newSchema);
+ }
+ return Pair.of(new HoodieAvroIndexedRecord(newer.getKey(), updatedValue.get(), orderingVal), updatedValue.get().getSchema());
Review Comment:
I was not sure if we can imply that the operation is the same as the `newer` record's value.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT] Custom merger update [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13742:
URL: https://github.com/apache/hudi/pull/13742#issuecomment-3208831366
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7988",
"triggerID" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"triggerType" : "PUSH"
}, {
"hash" : "ecb694295f7a88371f33df6de60370058670439e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7992",
"triggerID" : "ecb694295f7a88371f33df6de60370058670439e",
"triggerType" : "PUSH"
}, {
"hash" : "efd7462406be4f6d90dd5e42f152d66f683ebdf4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "efd7462406be4f6d90dd5e42f152d66f683ebdf4",
"triggerType" : "PUSH"
}, {
"hash" : "3c67160d5ceec883db5d826ff655865a3209f2f3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7994",
"triggerID" : "3c67160d5ceec883db5d826ff655865a3209f2f3",
"triggerType" : "PUSH"
}, {
"hash" : "f4f84d9f6a480923df27cfe5553d8aa65b2bbb7a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7996",
"triggerID" : "f4f84d9f6a480923df27cfe5553d8aa65b2bbb7a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* efd7462406be4f6d90dd5e42f152d66f683ebdf4 UNKNOWN
* f4f84d9f6a480923df27cfe5553d8aa65b2bbb7a Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7996)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT] Custom merger update [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13742:
URL: https://github.com/apache/hudi/pull/13742#issuecomment-3208616772
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7988",
"triggerID" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"triggerType" : "PUSH"
}, {
"hash" : "ecb694295f7a88371f33df6de60370058670439e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7992",
"triggerID" : "ecb694295f7a88371f33df6de60370058670439e",
"triggerType" : "PUSH"
}, {
"hash" : "efd7462406be4f6d90dd5e42f152d66f683ebdf4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "efd7462406be4f6d90dd5e42f152d66f683ebdf4",
"triggerType" : "PUSH"
}, {
"hash" : "3c67160d5ceec883db5d826ff655865a3209f2f3",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3c67160d5ceec883db5d826ff655865a3209f2f3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ecb694295f7a88371f33df6de60370058670439e Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7992)
* efd7462406be4f6d90dd5e42f152d66f683ebdf4 UNKNOWN
* 3c67160d5ceec883db5d826ff655865a3209f2f3 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT] Custom merger update [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13742:
URL: https://github.com/apache/hudi/pull/13742#discussion_r2296724086
##########
hudi-spark-datasource/hudi-spark/src/test/java/org/apache/hudi/io/TestMergeHandle.java:
##########
@@ -673,74 +672,4 @@ public Map<String, HoodieRecord> getValidDeletes() {
return validDeletes;
}
}
-
- public static class CustomMerger implements HoodieRecordMerger {
Review Comment:
Both CustomMerger and CustomPayload are extracted into their own classes for reuse in testing
##########
hudi-spark-datasource/hudi-spark/src/test/java/org/apache/hudi/io/TestMergeHandle.java:
##########
@@ -501,13 +499,14 @@ private InputAndExpectedDataSet prepareInputFor2ndBatch(HoodieWriteConfig config
recordsToDelete.add(deleteRecordSameOrderingValue);
recordsToDelete.add(deleteRecordLowerOrderingValue);
recordsToDelete.add(deleteRecordHigherOrderingValue);
+ // Custom merger chooses record with lower ordering value
Review Comment:
Validates that the custom ordering is respected for deletes
##########
hudi-common/src/test/java/org/apache/hudi/common/model/TestDefaultHoodieRecordPayload.java:
##########
@@ -78,8 +78,8 @@ public void testActiveRecords(String key) throws IOException {
record2.put("ts", 1L);
record2.put("_hoodie_is_deleted", false);
- DefaultHoodieRecordPayload payload1 = new DefaultHoodieRecordPayload(record1, 1);
- DefaultHoodieRecordPayload payload2 = new DefaultHoodieRecordPayload(record2, 2);
+ DefaultHoodieRecordPayload payload1 = new DefaultHoodieRecordPayload(record1, 0L);
Review Comment:
Updated the tests so that the ordering value matches the field value
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT] Custom merger update [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13742:
URL: https://github.com/apache/hudi/pull/13742#issuecomment-3218415208
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7988",
"triggerID" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"triggerType" : "PUSH"
}, {
"hash" : "ecb694295f7a88371f33df6de60370058670439e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7992",
"triggerID" : "ecb694295f7a88371f33df6de60370058670439e",
"triggerType" : "PUSH"
}, {
"hash" : "efd7462406be4f6d90dd5e42f152d66f683ebdf4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "efd7462406be4f6d90dd5e42f152d66f683ebdf4",
"triggerType" : "PUSH"
}, {
"hash" : "3c67160d5ceec883db5d826ff655865a3209f2f3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7994",
"triggerID" : "3c67160d5ceec883db5d826ff655865a3209f2f3",
"triggerType" : "PUSH"
}, {
"hash" : "f4f84d9f6a480923df27cfe5553d8aa65b2bbb7a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7996",
"triggerID" : "f4f84d9f6a480923df27cfe5553d8aa65b2bbb7a",
"triggerType" : "PUSH"
}, {
"hash" : "521b26bc2c60b7921e2f7aa5fffa620ab7f9a096",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "521b26bc2c60b7921e2f7aa5fffa620ab7f9a096",
"triggerType" : "PUSH"
}, {
"hash" : "22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50",
"triggerType" : "PUSH"
}, {
"hash" : "02a319ba884595a4e3b5453f366b660c1c11f767",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8007",
"triggerID" : "02a319ba884595a4e3b5453f366b660c1c11f767",
"triggerType" : "PUSH"
}, {
"hash" : "3e59b6f8b486016911c9ad14c583f6538cc036ac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8009",
"triggerID" : "3e59b6f8b486016911c9ad14c583f6538cc036ac",
"triggerType" : "PUSH"
}, {
"hash" : "003f1c81efedafa0ddd78868ce83357d8803c5e5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "003f1c81efedafa0ddd78868ce83357d8803c5e5",
"triggerType" : "PUSH"
}, {
"hash" : "40c04768d5233a38be1b4c01ce895189a4af4c3a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8013",
"triggerID" : "40c04768d5233a38be1b4c01ce895189a4af4c3a",
"triggerType" : "PUSH"
}, {
"hash" : "4a699a8dcf79e61ea3bb754e67e9cc4a32e86aa1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8084",
"triggerID" : "4a699a8dcf79e61ea3bb754e67e9cc4a32e86aa1",
"triggerType" : "PUSH"
}, {
"hash" : "fe55d8ec4923cb6c64e83ae4f32217c72599a398",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "fe55d8ec4923cb6c64e83ae4f32217c72599a398",
"triggerType" : "PUSH"
}, {
"hash" : "87cd79787dc8126b6877ba0d3407cfe8f343933a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8087",
"triggerID" : "87cd79787dc8126b6877ba0d3407cfe8f343933a",
"triggerType" : "PUSH"
}, {
"hash" : "594874ffdca647cd268b3947fa4680c57c53d679",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8088",
"triggerID" : "594874ffdca647cd268b3947fa4680c57c53d679",
"triggerType" : "PUSH"
}, {
"hash" : "7b95754f0df2c1a087a6a28722a7471ee12cecf3",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7b95754f0df2c1a087a6a28722a7471ee12cecf3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* efd7462406be4f6d90dd5e42f152d66f683ebdf4 UNKNOWN
* 521b26bc2c60b7921e2f7aa5fffa620ab7f9a096 UNKNOWN
* 22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50 UNKNOWN
* 003f1c81efedafa0ddd78868ce83357d8803c5e5 UNKNOWN
* fe55d8ec4923cb6c64e83ae4f32217c72599a398 UNKNOWN
* 594874ffdca647cd268b3947fa4680c57c53d679 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8088)
* 7b95754f0df2c1a087a6a28722a7471ee12cecf3 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9715] Custom merger update [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13742:
URL: https://github.com/apache/hudi/pull/13742#issuecomment-3235141016
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7988",
"triggerID" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"triggerType" : "PUSH"
}, {
"hash" : "ecb694295f7a88371f33df6de60370058670439e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7992",
"triggerID" : "ecb694295f7a88371f33df6de60370058670439e",
"triggerType" : "PUSH"
}, {
"hash" : "efd7462406be4f6d90dd5e42f152d66f683ebdf4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "efd7462406be4f6d90dd5e42f152d66f683ebdf4",
"triggerType" : "PUSH"
}, {
"hash" : "3c67160d5ceec883db5d826ff655865a3209f2f3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7994",
"triggerID" : "3c67160d5ceec883db5d826ff655865a3209f2f3",
"triggerType" : "PUSH"
}, {
"hash" : "f4f84d9f6a480923df27cfe5553d8aa65b2bbb7a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7996",
"triggerID" : "f4f84d9f6a480923df27cfe5553d8aa65b2bbb7a",
"triggerType" : "PUSH"
}, {
"hash" : "521b26bc2c60b7921e2f7aa5fffa620ab7f9a096",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "521b26bc2c60b7921e2f7aa5fffa620ab7f9a096",
"triggerType" : "PUSH"
}, {
"hash" : "22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50",
"triggerType" : "PUSH"
}, {
"hash" : "02a319ba884595a4e3b5453f366b660c1c11f767",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8007",
"triggerID" : "02a319ba884595a4e3b5453f366b660c1c11f767",
"triggerType" : "PUSH"
}, {
"hash" : "3e59b6f8b486016911c9ad14c583f6538cc036ac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8009",
"triggerID" : "3e59b6f8b486016911c9ad14c583f6538cc036ac",
"triggerType" : "PUSH"
}, {
"hash" : "003f1c81efedafa0ddd78868ce83357d8803c5e5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "003f1c81efedafa0ddd78868ce83357d8803c5e5",
"triggerType" : "PUSH"
}, {
"hash" : "40c04768d5233a38be1b4c01ce895189a4af4c3a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8013",
"triggerID" : "40c04768d5233a38be1b4c01ce895189a4af4c3a",
"triggerType" : "PUSH"
}, {
"hash" : "4a699a8dcf79e61ea3bb754e67e9cc4a32e86aa1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8084",
"triggerID" : "4a699a8dcf79e61ea3bb754e67e9cc4a32e86aa1",
"triggerType" : "PUSH"
}, {
"hash" : "fe55d8ec4923cb6c64e83ae4f32217c72599a398",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "fe55d8ec4923cb6c64e83ae4f32217c72599a398",
"triggerType" : "PUSH"
}, {
"hash" : "87cd79787dc8126b6877ba0d3407cfe8f343933a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8087",
"triggerID" : "87cd79787dc8126b6877ba0d3407cfe8f343933a",
"triggerType" : "PUSH"
}, {
"hash" : "594874ffdca647cd268b3947fa4680c57c53d679",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8088",
"triggerID" : "594874ffdca647cd268b3947fa4680c57c53d679",
"triggerType" : "PUSH"
}, {
"hash" : "7b95754f0df2c1a087a6a28722a7471ee12cecf3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8090",
"triggerID" : "7b95754f0df2c1a087a6a28722a7471ee12cecf3",
"triggerType" : "PUSH"
}, {
"hash" : "789c2da879ca66f6b9be71d1814be38b5d8cfb58",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8102",
"triggerID" : "789c2da879ca66f6b9be71d1814be38b5d8cfb58",
"triggerType" : "PUSH"
}, {
"hash" : "63f33fae3e22a9f80de51b2a6e1ed0427f631d0b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8103",
"triggerID" : "63f33fae3e22a9f80de51b2a6e1ed0427f631d0b",
"triggerType" : "PUSH"
}, {
"hash" : "e04c466fa81e9bcfaabe36825127098f5d1bb890",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8224",
"triggerID" : "e04c466fa81e9bcfaabe36825127098f5d1bb890",
"triggerType" : "PUSH"
}, {
"hash" : "cf574f46a1783a39b2074542aad9539ec3c4f36a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8235",
"triggerID" : "cf574f46a1783a39b2074542aad9539ec3c4f36a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* efd7462406be4f6d90dd5e42f152d66f683ebdf4 UNKNOWN
* 521b26bc2c60b7921e2f7aa5fffa620ab7f9a096 UNKNOWN
* 22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50 UNKNOWN
* 003f1c81efedafa0ddd78868ce83357d8803c5e5 UNKNOWN
* fe55d8ec4923cb6c64e83ae4f32217c72599a398 UNKNOWN
* cf574f46a1783a39b2074542aad9539ec3c4f36a Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8235)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9715] Custom merger update [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13742:
URL: https://github.com/apache/hudi/pull/13742#discussion_r2308924819
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/HoodieSparkRecordMerger.java:
##########
@@ -39,33 +41,33 @@ public HoodieRecord.HoodieRecordType getRecordType() {
* Basic handling of deletes that is used by many of the spark mergers
* returns null if merger specific logic should be used
*/
- protected Option<Pair<HoodieRecord, Schema>> handleDeletes(HoodieRecord older, Schema oldSchema, HoodieRecord newer, Schema newSchema, TypedProperties props) {
+ protected Pair<HoodieRecord, Schema> handleDeletes(HoodieRecord older, Schema oldSchema, HoodieRecord newer, Schema newSchema, TypedProperties props) {
Review Comment:
guess this is useless now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT] Custom merger update [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13742:
URL: https://github.com/apache/hudi/pull/13742#discussion_r2298416877
##########
hudi-common/src/main/java/org/apache/hudi/common/model/HoodieRecordMerger.java:
##########
@@ -136,6 +136,7 @@ default Option<Pair<HoodieRecord, Schema>> partialMerge(HoodieRecord older, Sche
*
* <p> This interface is experimental and might be evolved in the future.
**/
+ @Deprecated
Review Comment:
There is also an opportunity to remove this method as part of the 1.1 release.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9715] Custom merger update [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13742:
URL: https://github.com/apache/hudi/pull/13742#issuecomment-3220902212
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7988",
"triggerID" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"triggerType" : "PUSH"
}, {
"hash" : "ecb694295f7a88371f33df6de60370058670439e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7992",
"triggerID" : "ecb694295f7a88371f33df6de60370058670439e",
"triggerType" : "PUSH"
}, {
"hash" : "efd7462406be4f6d90dd5e42f152d66f683ebdf4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "efd7462406be4f6d90dd5e42f152d66f683ebdf4",
"triggerType" : "PUSH"
}, {
"hash" : "3c67160d5ceec883db5d826ff655865a3209f2f3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7994",
"triggerID" : "3c67160d5ceec883db5d826ff655865a3209f2f3",
"triggerType" : "PUSH"
}, {
"hash" : "f4f84d9f6a480923df27cfe5553d8aa65b2bbb7a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7996",
"triggerID" : "f4f84d9f6a480923df27cfe5553d8aa65b2bbb7a",
"triggerType" : "PUSH"
}, {
"hash" : "521b26bc2c60b7921e2f7aa5fffa620ab7f9a096",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "521b26bc2c60b7921e2f7aa5fffa620ab7f9a096",
"triggerType" : "PUSH"
}, {
"hash" : "22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50",
"triggerType" : "PUSH"
}, {
"hash" : "02a319ba884595a4e3b5453f366b660c1c11f767",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8007",
"triggerID" : "02a319ba884595a4e3b5453f366b660c1c11f767",
"triggerType" : "PUSH"
}, {
"hash" : "3e59b6f8b486016911c9ad14c583f6538cc036ac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8009",
"triggerID" : "3e59b6f8b486016911c9ad14c583f6538cc036ac",
"triggerType" : "PUSH"
}, {
"hash" : "003f1c81efedafa0ddd78868ce83357d8803c5e5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "003f1c81efedafa0ddd78868ce83357d8803c5e5",
"triggerType" : "PUSH"
}, {
"hash" : "40c04768d5233a38be1b4c01ce895189a4af4c3a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8013",
"triggerID" : "40c04768d5233a38be1b4c01ce895189a4af4c3a",
"triggerType" : "PUSH"
}, {
"hash" : "4a699a8dcf79e61ea3bb754e67e9cc4a32e86aa1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8084",
"triggerID" : "4a699a8dcf79e61ea3bb754e67e9cc4a32e86aa1",
"triggerType" : "PUSH"
}, {
"hash" : "fe55d8ec4923cb6c64e83ae4f32217c72599a398",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "fe55d8ec4923cb6c64e83ae4f32217c72599a398",
"triggerType" : "PUSH"
}, {
"hash" : "87cd79787dc8126b6877ba0d3407cfe8f343933a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8087",
"triggerID" : "87cd79787dc8126b6877ba0d3407cfe8f343933a",
"triggerType" : "PUSH"
}, {
"hash" : "594874ffdca647cd268b3947fa4680c57c53d679",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8088",
"triggerID" : "594874ffdca647cd268b3947fa4680c57c53d679",
"triggerType" : "PUSH"
}, {
"hash" : "7b95754f0df2c1a087a6a28722a7471ee12cecf3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8090",
"triggerID" : "7b95754f0df2c1a087a6a28722a7471ee12cecf3",
"triggerType" : "PUSH"
}, {
"hash" : "789c2da879ca66f6b9be71d1814be38b5d8cfb58",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8102",
"triggerID" : "789c2da879ca66f6b9be71d1814be38b5d8cfb58",
"triggerType" : "PUSH"
}, {
"hash" : "63f33fae3e22a9f80de51b2a6e1ed0427f631d0b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8103",
"triggerID" : "63f33fae3e22a9f80de51b2a6e1ed0427f631d0b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* efd7462406be4f6d90dd5e42f152d66f683ebdf4 UNKNOWN
* 521b26bc2c60b7921e2f7aa5fffa620ab7f9a096 UNKNOWN
* 22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50 UNKNOWN
* 003f1c81efedafa0ddd78868ce83357d8803c5e5 UNKNOWN
* fe55d8ec4923cb6c64e83ae4f32217c72599a398 UNKNOWN
* 7b95754f0df2c1a087a6a28722a7471ee12cecf3 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8090)
* 789c2da879ca66f6b9be71d1814be38b5d8cfb58 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8102)
* 63f33fae3e22a9f80de51b2a6e1ed0427f631d0b Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8103)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9715] Custom merger update [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13742:
URL: https://github.com/apache/hudi/pull/13742#issuecomment-3220948810
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7988",
"triggerID" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"triggerType" : "PUSH"
}, {
"hash" : "ecb694295f7a88371f33df6de60370058670439e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7992",
"triggerID" : "ecb694295f7a88371f33df6de60370058670439e",
"triggerType" : "PUSH"
}, {
"hash" : "efd7462406be4f6d90dd5e42f152d66f683ebdf4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "efd7462406be4f6d90dd5e42f152d66f683ebdf4",
"triggerType" : "PUSH"
}, {
"hash" : "3c67160d5ceec883db5d826ff655865a3209f2f3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7994",
"triggerID" : "3c67160d5ceec883db5d826ff655865a3209f2f3",
"triggerType" : "PUSH"
}, {
"hash" : "f4f84d9f6a480923df27cfe5553d8aa65b2bbb7a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7996",
"triggerID" : "f4f84d9f6a480923df27cfe5553d8aa65b2bbb7a",
"triggerType" : "PUSH"
}, {
"hash" : "521b26bc2c60b7921e2f7aa5fffa620ab7f9a096",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "521b26bc2c60b7921e2f7aa5fffa620ab7f9a096",
"triggerType" : "PUSH"
}, {
"hash" : "22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50",
"triggerType" : "PUSH"
}, {
"hash" : "02a319ba884595a4e3b5453f366b660c1c11f767",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8007",
"triggerID" : "02a319ba884595a4e3b5453f366b660c1c11f767",
"triggerType" : "PUSH"
}, {
"hash" : "3e59b6f8b486016911c9ad14c583f6538cc036ac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8009",
"triggerID" : "3e59b6f8b486016911c9ad14c583f6538cc036ac",
"triggerType" : "PUSH"
}, {
"hash" : "003f1c81efedafa0ddd78868ce83357d8803c5e5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "003f1c81efedafa0ddd78868ce83357d8803c5e5",
"triggerType" : "PUSH"
}, {
"hash" : "40c04768d5233a38be1b4c01ce895189a4af4c3a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8013",
"triggerID" : "40c04768d5233a38be1b4c01ce895189a4af4c3a",
"triggerType" : "PUSH"
}, {
"hash" : "4a699a8dcf79e61ea3bb754e67e9cc4a32e86aa1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8084",
"triggerID" : "4a699a8dcf79e61ea3bb754e67e9cc4a32e86aa1",
"triggerType" : "PUSH"
}, {
"hash" : "fe55d8ec4923cb6c64e83ae4f32217c72599a398",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "fe55d8ec4923cb6c64e83ae4f32217c72599a398",
"triggerType" : "PUSH"
}, {
"hash" : "87cd79787dc8126b6877ba0d3407cfe8f343933a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8087",
"triggerID" : "87cd79787dc8126b6877ba0d3407cfe8f343933a",
"triggerType" : "PUSH"
}, {
"hash" : "594874ffdca647cd268b3947fa4680c57c53d679",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8088",
"triggerID" : "594874ffdca647cd268b3947fa4680c57c53d679",
"triggerType" : "PUSH"
}, {
"hash" : "7b95754f0df2c1a087a6a28722a7471ee12cecf3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8090",
"triggerID" : "7b95754f0df2c1a087a6a28722a7471ee12cecf3",
"triggerType" : "PUSH"
}, {
"hash" : "789c2da879ca66f6b9be71d1814be38b5d8cfb58",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8102",
"triggerID" : "789c2da879ca66f6b9be71d1814be38b5d8cfb58",
"triggerType" : "PUSH"
}, {
"hash" : "63f33fae3e22a9f80de51b2a6e1ed0427f631d0b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8103",
"triggerID" : "63f33fae3e22a9f80de51b2a6e1ed0427f631d0b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* efd7462406be4f6d90dd5e42f152d66f683ebdf4 UNKNOWN
* 521b26bc2c60b7921e2f7aa5fffa620ab7f9a096 UNKNOWN
* 22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50 UNKNOWN
* 003f1c81efedafa0ddd78868ce83357d8803c5e5 UNKNOWN
* fe55d8ec4923cb6c64e83ae4f32217c72599a398 UNKNOWN
* 789c2da879ca66f6b9be71d1814be38b5d8cfb58 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8102)
* 63f33fae3e22a9f80de51b2a6e1ed0427f631d0b Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8103)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9715] Custom merger update [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13742:
URL: https://github.com/apache/hudi/pull/13742#discussion_r2308990683
##########
hudi-common/src/main/java/org/apache/hudi/common/model/HoodieRecordMerger.java:
##########
@@ -111,17 +119,16 @@ public interface HoodieRecordMerger extends Serializable {
* ts | price | tags
* 16 | 2.8 | fruit,juicy
*
- * @param older Older record.
- * @param oldSchema Schema of the older record.
- * @param newer Newer record.
- * @param newSchema Schema of the newer record.
- * @param readerSchema Reader schema containing all the fields to read. This is used to maintain
- * the ordering of the fields of the merged record.
- * @param props Configuration in {@link TypedProperties}.
- * @return The merged record and schema.
+ * @param older Older record.
+ * @param newer Newer record.
+ * @param readerSchema Reader schema containing all the fields to read. This is used to maintain
+ * the ordering of the fields of the merged record.
+ * @param recordContext the record context for accessing and manipulating the records.
+ * @param props Configuration in {@link TypedProperties}.
+ * @return The merged record and schema. The record is expected to be non-null. If the record represents a deletion, the operation must be set as {@link HoodieOperation#DELETE}.
* @throws IOException upon merging error.
*/
- default Option<Pair<HoodieRecord, Schema>> partialMerge(HoodieRecord older, Schema oldSchema, HoodieRecord newer, Schema newSchema, Schema readerSchema, TypedProperties props) throws IOException {
+ default <T> BufferedRecord<T> partialMerge(BufferedRecord<T> older, BufferedRecord<T> newer, Schema readerSchema, RecordContext<T> recordContext, TypedProperties props) throws IOException {
Review Comment:
we also need to fix the `fullOuterMerge` to use buffered record or just remove it because I didn't see any uses of it in the repo.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9715] Custom merger update [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13742:
URL: https://github.com/apache/hudi/pull/13742#issuecomment-3237027641
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7988",
"triggerID" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"triggerType" : "PUSH"
}, {
"hash" : "ecb694295f7a88371f33df6de60370058670439e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7992",
"triggerID" : "ecb694295f7a88371f33df6de60370058670439e",
"triggerType" : "PUSH"
}, {
"hash" : "efd7462406be4f6d90dd5e42f152d66f683ebdf4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "efd7462406be4f6d90dd5e42f152d66f683ebdf4",
"triggerType" : "PUSH"
}, {
"hash" : "3c67160d5ceec883db5d826ff655865a3209f2f3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7994",
"triggerID" : "3c67160d5ceec883db5d826ff655865a3209f2f3",
"triggerType" : "PUSH"
}, {
"hash" : "f4f84d9f6a480923df27cfe5553d8aa65b2bbb7a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7996",
"triggerID" : "f4f84d9f6a480923df27cfe5553d8aa65b2bbb7a",
"triggerType" : "PUSH"
}, {
"hash" : "521b26bc2c60b7921e2f7aa5fffa620ab7f9a096",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "521b26bc2c60b7921e2f7aa5fffa620ab7f9a096",
"triggerType" : "PUSH"
}, {
"hash" : "22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50",
"triggerType" : "PUSH"
}, {
"hash" : "02a319ba884595a4e3b5453f366b660c1c11f767",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8007",
"triggerID" : "02a319ba884595a4e3b5453f366b660c1c11f767",
"triggerType" : "PUSH"
}, {
"hash" : "3e59b6f8b486016911c9ad14c583f6538cc036ac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8009",
"triggerID" : "3e59b6f8b486016911c9ad14c583f6538cc036ac",
"triggerType" : "PUSH"
}, {
"hash" : "003f1c81efedafa0ddd78868ce83357d8803c5e5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "003f1c81efedafa0ddd78868ce83357d8803c5e5",
"triggerType" : "PUSH"
}, {
"hash" : "40c04768d5233a38be1b4c01ce895189a4af4c3a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8013",
"triggerID" : "40c04768d5233a38be1b4c01ce895189a4af4c3a",
"triggerType" : "PUSH"
}, {
"hash" : "4a699a8dcf79e61ea3bb754e67e9cc4a32e86aa1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8084",
"triggerID" : "4a699a8dcf79e61ea3bb754e67e9cc4a32e86aa1",
"triggerType" : "PUSH"
}, {
"hash" : "fe55d8ec4923cb6c64e83ae4f32217c72599a398",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "fe55d8ec4923cb6c64e83ae4f32217c72599a398",
"triggerType" : "PUSH"
}, {
"hash" : "87cd79787dc8126b6877ba0d3407cfe8f343933a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8087",
"triggerID" : "87cd79787dc8126b6877ba0d3407cfe8f343933a",
"triggerType" : "PUSH"
}, {
"hash" : "594874ffdca647cd268b3947fa4680c57c53d679",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8088",
"triggerID" : "594874ffdca647cd268b3947fa4680c57c53d679",
"triggerType" : "PUSH"
}, {
"hash" : "7b95754f0df2c1a087a6a28722a7471ee12cecf3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8090",
"triggerID" : "7b95754f0df2c1a087a6a28722a7471ee12cecf3",
"triggerType" : "PUSH"
}, {
"hash" : "789c2da879ca66f6b9be71d1814be38b5d8cfb58",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8102",
"triggerID" : "789c2da879ca66f6b9be71d1814be38b5d8cfb58",
"triggerType" : "PUSH"
}, {
"hash" : "63f33fae3e22a9f80de51b2a6e1ed0427f631d0b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8103",
"triggerID" : "63f33fae3e22a9f80de51b2a6e1ed0427f631d0b",
"triggerType" : "PUSH"
}, {
"hash" : "e04c466fa81e9bcfaabe36825127098f5d1bb890",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8224",
"triggerID" : "e04c466fa81e9bcfaabe36825127098f5d1bb890",
"triggerType" : "PUSH"
}, {
"hash" : "cf574f46a1783a39b2074542aad9539ec3c4f36a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8235",
"triggerID" : "cf574f46a1783a39b2074542aad9539ec3c4f36a",
"triggerType" : "PUSH"
}, {
"hash" : "2eb81e923adfa77176f137c442143deaac9e9916",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2eb81e923adfa77176f137c442143deaac9e9916",
"triggerType" : "PUSH"
}, {
"hash" : "12dc3e3eaf746bb1f2bc3f4e93fa14d9adb0f6fd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8244",
"triggerID" : "12dc3e3eaf746bb1f2bc3f4e93fa14d9adb0f6fd",
"triggerType" : "PUSH"
}, {
"hash" : "af5ab73103f3af882412bbe99c9322165e8941a0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "af5ab73103f3af882412bbe99c9322165e8941a0",
"triggerType" : "PUSH"
}, {
"hash" : "a9c28c4d913644fea23d57a5dc2224f36fd2b63e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8248",
"triggerID" : "a9c28c4d913644fea23d57a5dc2224f36fd2b63e",
"triggerType" : "PUSH"
}, {
"hash" : "98381a6bd2bcccf3da08fb992e599fe1165f2ddb",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8252",
"triggerID" : "98381a6bd2bcccf3da08fb992e599fe1165f2ddb",
"triggerType" : "PUSH"
}, {
"hash" : "94aa9a669d2fe1fb4919a4735fab2665553b63c0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "94aa9a669d2fe1fb4919a4735fab2665553b63c0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* efd7462406be4f6d90dd5e42f152d66f683ebdf4 UNKNOWN
* 521b26bc2c60b7921e2f7aa5fffa620ab7f9a096 UNKNOWN
* 22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50 UNKNOWN
* 003f1c81efedafa0ddd78868ce83357d8803c5e5 UNKNOWN
* fe55d8ec4923cb6c64e83ae4f32217c72599a398 UNKNOWN
* 2eb81e923adfa77176f137c442143deaac9e9916 UNKNOWN
* af5ab73103f3af882412bbe99c9322165e8941a0 UNKNOWN
* 98381a6bd2bcccf3da08fb992e599fe1165f2ddb Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8252)
* 94aa9a669d2fe1fb4919a4735fab2665553b63c0 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9715] Custom merger update [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13742:
URL: https://github.com/apache/hudi/pull/13742#discussion_r2308896327
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/DefaultSparkRecordMerger.java:
##########
@@ -37,46 +35,38 @@
*/
public class DefaultSparkRecordMerger extends HoodieSparkRecordMerger {
- private String[] orderingFields;
-
@Override
public String getMergingStrategy() {
return HoodieRecordMerger.EVENT_TIME_BASED_MERGE_STRATEGY_UUID;
}
@Override
- public Option<Pair<HoodieRecord, Schema>> merge(HoodieRecord older, Schema oldSchema, HoodieRecord newer, Schema newSchema, TypedProperties props) throws IOException {
- Option<Pair<HoodieRecord, Schema>> deleteHandlingResult = handleDeletes(older, oldSchema, newer, newSchema, props);
- if (deleteHandlingResult != null) {
- return deleteHandlingResult;
- }
-
- if (orderingFields == null) {
- orderingFields = ConfigUtils.getOrderingFields(props);
- }
- if (older.getOrderingValue(oldSchema, props, orderingFields).compareTo(newer.getOrderingValue(newSchema, props, orderingFields)) > 0) {
- return Option.of(Pair.of(older, oldSchema));
+ public <T> BufferedRecord<T> merge(BufferedRecord<T> older, BufferedRecord<T> newer, RecordContext<T> recordContext, TypedProperties props) throws IOException {
+ if (older.getOrderingValue().compareTo(newer.getOrderingValue()) > 0) {
Review Comment:
we may need to handle the special commit time ordering value `0` so that the `#compareTo` not throws when the ordering value is not a Long, just like what we currently do in the `BufferedRecordMerger`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9715] Custom merger update [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13742:
URL: https://github.com/apache/hudi/pull/13742#discussion_r2308896327
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/DefaultSparkRecordMerger.java:
##########
@@ -37,46 +35,38 @@
*/
public class DefaultSparkRecordMerger extends HoodieSparkRecordMerger {
- private String[] orderingFields;
-
@Override
public String getMergingStrategy() {
return HoodieRecordMerger.EVENT_TIME_BASED_MERGE_STRATEGY_UUID;
}
@Override
- public Option<Pair<HoodieRecord, Schema>> merge(HoodieRecord older, Schema oldSchema, HoodieRecord newer, Schema newSchema, TypedProperties props) throws IOException {
- Option<Pair<HoodieRecord, Schema>> deleteHandlingResult = handleDeletes(older, oldSchema, newer, newSchema, props);
- if (deleteHandlingResult != null) {
- return deleteHandlingResult;
- }
-
- if (orderingFields == null) {
- orderingFields = ConfigUtils.getOrderingFields(props);
- }
- if (older.getOrderingValue(oldSchema, props, orderingFields).compareTo(newer.getOrderingValue(newSchema, props, orderingFields)) > 0) {
- return Option.of(Pair.of(older, oldSchema));
+ public <T> BufferedRecord<T> merge(BufferedRecord<T> older, BufferedRecord<T> newer, RecordContext<T> recordContext, TypedProperties props) throws IOException {
+ if (older.getOrderingValue().compareTo(newer.getOrderingValue()) > 0) {
Review Comment:
we may need to handle the special commit time ordering value `0` so that the `#compareTo` not throws when the ordering value is not a Long.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9715] Custom merger update [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13742:
URL: https://github.com/apache/hudi/pull/13742#issuecomment-3235687490
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7988",
"triggerID" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"triggerType" : "PUSH"
}, {
"hash" : "ecb694295f7a88371f33df6de60370058670439e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7992",
"triggerID" : "ecb694295f7a88371f33df6de60370058670439e",
"triggerType" : "PUSH"
}, {
"hash" : "efd7462406be4f6d90dd5e42f152d66f683ebdf4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "efd7462406be4f6d90dd5e42f152d66f683ebdf4",
"triggerType" : "PUSH"
}, {
"hash" : "3c67160d5ceec883db5d826ff655865a3209f2f3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7994",
"triggerID" : "3c67160d5ceec883db5d826ff655865a3209f2f3",
"triggerType" : "PUSH"
}, {
"hash" : "f4f84d9f6a480923df27cfe5553d8aa65b2bbb7a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7996",
"triggerID" : "f4f84d9f6a480923df27cfe5553d8aa65b2bbb7a",
"triggerType" : "PUSH"
}, {
"hash" : "521b26bc2c60b7921e2f7aa5fffa620ab7f9a096",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "521b26bc2c60b7921e2f7aa5fffa620ab7f9a096",
"triggerType" : "PUSH"
}, {
"hash" : "22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50",
"triggerType" : "PUSH"
}, {
"hash" : "02a319ba884595a4e3b5453f366b660c1c11f767",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8007",
"triggerID" : "02a319ba884595a4e3b5453f366b660c1c11f767",
"triggerType" : "PUSH"
}, {
"hash" : "3e59b6f8b486016911c9ad14c583f6538cc036ac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8009",
"triggerID" : "3e59b6f8b486016911c9ad14c583f6538cc036ac",
"triggerType" : "PUSH"
}, {
"hash" : "003f1c81efedafa0ddd78868ce83357d8803c5e5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "003f1c81efedafa0ddd78868ce83357d8803c5e5",
"triggerType" : "PUSH"
}, {
"hash" : "40c04768d5233a38be1b4c01ce895189a4af4c3a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8013",
"triggerID" : "40c04768d5233a38be1b4c01ce895189a4af4c3a",
"triggerType" : "PUSH"
}, {
"hash" : "4a699a8dcf79e61ea3bb754e67e9cc4a32e86aa1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8084",
"triggerID" : "4a699a8dcf79e61ea3bb754e67e9cc4a32e86aa1",
"triggerType" : "PUSH"
}, {
"hash" : "fe55d8ec4923cb6c64e83ae4f32217c72599a398",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "fe55d8ec4923cb6c64e83ae4f32217c72599a398",
"triggerType" : "PUSH"
}, {
"hash" : "87cd79787dc8126b6877ba0d3407cfe8f343933a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8087",
"triggerID" : "87cd79787dc8126b6877ba0d3407cfe8f343933a",
"triggerType" : "PUSH"
}, {
"hash" : "594874ffdca647cd268b3947fa4680c57c53d679",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8088",
"triggerID" : "594874ffdca647cd268b3947fa4680c57c53d679",
"triggerType" : "PUSH"
}, {
"hash" : "7b95754f0df2c1a087a6a28722a7471ee12cecf3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8090",
"triggerID" : "7b95754f0df2c1a087a6a28722a7471ee12cecf3",
"triggerType" : "PUSH"
}, {
"hash" : "789c2da879ca66f6b9be71d1814be38b5d8cfb58",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8102",
"triggerID" : "789c2da879ca66f6b9be71d1814be38b5d8cfb58",
"triggerType" : "PUSH"
}, {
"hash" : "63f33fae3e22a9f80de51b2a6e1ed0427f631d0b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8103",
"triggerID" : "63f33fae3e22a9f80de51b2a6e1ed0427f631d0b",
"triggerType" : "PUSH"
}, {
"hash" : "e04c466fa81e9bcfaabe36825127098f5d1bb890",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8224",
"triggerID" : "e04c466fa81e9bcfaabe36825127098f5d1bb890",
"triggerType" : "PUSH"
}, {
"hash" : "cf574f46a1783a39b2074542aad9539ec3c4f36a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8235",
"triggerID" : "cf574f46a1783a39b2074542aad9539ec3c4f36a",
"triggerType" : "PUSH"
}, {
"hash" : "2eb81e923adfa77176f137c442143deaac9e9916",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2eb81e923adfa77176f137c442143deaac9e9916",
"triggerType" : "PUSH"
}, {
"hash" : "12dc3e3eaf746bb1f2bc3f4e93fa14d9adb0f6fd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8244",
"triggerID" : "12dc3e3eaf746bb1f2bc3f4e93fa14d9adb0f6fd",
"triggerType" : "PUSH"
}, {
"hash" : "af5ab73103f3af882412bbe99c9322165e8941a0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "af5ab73103f3af882412bbe99c9322165e8941a0",
"triggerType" : "PUSH"
}, {
"hash" : "a9c28c4d913644fea23d57a5dc2224f36fd2b63e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8248",
"triggerID" : "a9c28c4d913644fea23d57a5dc2224f36fd2b63e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* efd7462406be4f6d90dd5e42f152d66f683ebdf4 UNKNOWN
* 521b26bc2c60b7921e2f7aa5fffa620ab7f9a096 UNKNOWN
* 22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50 UNKNOWN
* 003f1c81efedafa0ddd78868ce83357d8803c5e5 UNKNOWN
* fe55d8ec4923cb6c64e83ae4f32217c72599a398 UNKNOWN
* 2eb81e923adfa77176f137c442143deaac9e9916 UNKNOWN
* af5ab73103f3af882412bbe99c9322165e8941a0 UNKNOWN
* a9c28c4d913644fea23d57a5dc2224f36fd2b63e Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8248)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT] Custom merger update [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13742:
URL: https://github.com/apache/hudi/pull/13742#issuecomment-3207864908
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7988",
"triggerID" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a9251b050ad82bceba70f95140ec4f1c5ac5a594 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7988)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT] Custom merger update [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13742:
URL: https://github.com/apache/hudi/pull/13742#issuecomment-3207857264
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a9251b050ad82bceba70f95140ec4f1c5ac5a594 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9715] Custom merger update [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13742:
URL: https://github.com/apache/hudi/pull/13742#discussion_r2299977999
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/BufferedRecordMergerFactory.java:
##########
@@ -465,46 +451,45 @@ public CustomPayloadRecordMerger(
}
@Override
- public Option<BufferedRecord<T>> deltaMergeNonDeleteRecord(BufferedRecord<T> newRecord, BufferedRecord<T> existingRecord) throws IOException {
- Option<Pair<HoodieRecord, Schema>> mergedRecordAndSchema = getMergedRecord(existingRecord, newRecord, false);
- if (mergedRecordAndSchema.isEmpty()) {
- // An empty Option indicates that the output represents a delete.
- return Option.of(new BufferedRecord<>(newRecord.getRecordKey(), OrderingValues.getDefault(), null, null, HoodieOperation.DELETE));
- }
- HoodieRecord mergedRecord = mergedRecordAndSchema.get().getLeft();
- Schema mergeResultSchema = mergedRecordAndSchema.get().getRight();
+ public Option<BufferedRecord<T>> deltaMergeRecords(BufferedRecord<T> newRecord, BufferedRecord<T> existingRecord) throws IOException {
+ Pair<HoodieRecord, Schema> mergedRecordAndSchema = getMergedRecord(existingRecord, newRecord, false);
+ HoodieRecord mergedRecord = mergedRecordAndSchema.getLeft();
+ Schema mergeResultSchema = mergedRecordAndSchema.getRight();
// Special handling for SENTINEL record in Expression Payload. This is returned if the condition does not match.
if (mergedRecord.getData() == HoodieRecord.SENTINEL) {
return Option.empty();
}
- T combinedRecordData = recordContext.convertAvroRecord(mergedRecord.toIndexedRecord(mergeResultSchema, props).get().getData());
+ Option<T> combinedRecordData = (mergedRecord.toIndexedRecord(mergeResultSchema, props).map(indexedRecord -> recordContext.convertAvroRecord(indexedRecord.getData())));
// If pre-combine does not return existing record, update it
- if (combinedRecordData != existingRecord.getRecord()) {
+ if (combinedRecordData.map(record -> record != existingRecord.getRecord()).orElse(true)) {
// For pkless we need to use record key from existing record
- return Option.of(BufferedRecords.fromEngineRecord(combinedRecordData, mergeResultSchema, recordContext, orderingFieldNames,
- existingRecord.getRecordKey(), mergedRecord.isDelete(mergeResultSchema, props)));
+ boolean isDelete = mergedRecord.isDelete(readerSchema, props);
+ Comparable orderingValue = mergedRecord.getOrderingValue(mergeResultSchema, props, orderingFieldNames);
+ T mergedEngineRecord = mergedRecord.toIndexedRecord(mergeResultSchema, props)
+ .map(hoodieAvroIndexedRecord -> recordContext.convertAvroRecord(hoodieAvroIndexedRecord.getData()))
+ .orElse(null);
+ return Option.of(BufferedRecords.fromEngineRecord(mergedEngineRecord, readerSchema, recordContext, orderingValue, existingRecord.getRecordKey(), isDelete));
+
}
return Option.empty();
}
@Override
- public BufferedRecord<T> mergeNonDeleteRecord(BufferedRecord<T> olderRecord, BufferedRecord<T> newerRecord) throws IOException {
- Option<Pair<HoodieRecord, Schema>> mergedRecordAndSchema = getMergedRecord(olderRecord, newerRecord, true);
- if (mergedRecordAndSchema.isEmpty()) {
- return BufferedRecords.createDelete(newerRecord.getRecordKey());
- }
- HoodieRecord mergedRecord = mergedRecordAndSchema.get().getLeft();
- Schema mergeResultSchema = mergedRecordAndSchema.get().getRight();
+ public BufferedRecord<T> mergeRecords(BufferedRecord<T> olderRecord, BufferedRecord<T> newerRecord) throws IOException {
+ Pair<HoodieRecord, Schema> mergedRecordAndSchema = getMergedRecord(olderRecord, newerRecord, true);
+
+ HoodieRecord mergedRecord = mergedRecordAndSchema.getLeft();
+ Schema mergeResultSchema = mergedRecordAndSchema.getRight();
// Special handling for SENTINEL record in Expression Payload
if (mergedRecord.getData() == HoodieRecord.SENTINEL) {
return olderRecord;
}
- if (!mergedRecord.isDelete(mergeResultSchema, props)) {
- IndexedRecord indexedRecord = (IndexedRecord) mergedRecord.getData();
- return BufferedRecords.fromEngineRecord(
- recordContext.convertAvroRecord(indexedRecord), mergeResultSchema, recordContext, orderingFieldNames, newerRecord.getRecordKey(), false);
- }
- return BufferedRecords.createDelete(newerRecord.getRecordKey());
+ boolean isDelete = mergedRecord.isDelete(readerSchema, props);
+ Comparable orderingValue = mergedRecord.getOrderingValue(mergeResultSchema, props, orderingFieldNames);
+ T mergedEngineRecord = mergedRecord.toIndexedRecord(mergeResultSchema, props)
Review Comment:
The hoodie record -> toIndexedRecord -> `recordContext.convertAvroRecord` works but it would induce unnecessary avro conversion for engine hoodie record, maybe we introudce a new API on HoodieRecord named `HoodieRecord.toEngineRecord` or `HoodieRecord.getEngineRow` for payload based hoodie record?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9715] Custom merger update [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13742:
URL: https://github.com/apache/hudi/pull/13742#discussion_r2300000482
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/BufferedRecordMergerFactory.java:
##########
@@ -567,42 +553,36 @@ public Option<BufferedRecord<T>> deltaMerge(BufferedRecord<T> newRecord, Buffere
if (existingRecord == null) {
return Option.of(newRecord);
}
- if (existingRecord.isDelete() || newRecord.isDelete()) {
- if (shouldKeepNewerRecord(existingRecord, newRecord)) {
- // IMPORTANT:
- // this is needed when the fallback HoodieAvroRecordMerger got used, the merger would
- // return Option.empty when the old payload data is empty(a delete) and ignores its ordering value directly.
- return Option.of(newRecord);
- } else {
- return Option.empty();
- }
- }
- return deltaMergeNonDeleteRecord(newRecord, existingRecord);
+ return deltaMergeRecords(newRecord, existingRecord);
}
- public abstract Option<BufferedRecord<T>> deltaMergeNonDeleteRecord(BufferedRecord<T> newRecord, BufferedRecord<T> existingRecord) throws IOException;
+ public abstract Option<BufferedRecord<T>> deltaMergeRecords(BufferedRecord<T> newRecord, BufferedRecord<T> existingRecord) throws IOException;
@Override
public Option<DeleteRecord> deltaMerge(DeleteRecord deleteRecord, BufferedRecord<T> existingRecord) {
- return deltaMergeDeleteRecord(deleteRecord, existingRecord);
+ BufferedRecord<T> deleteBufferedRecord = BufferedRecords.fromDeleteRecord(deleteRecord, recordContext);
+ try {
+ Option<BufferedRecord<T>> merged = deltaMerge(deleteBufferedRecord, existingRecord);
+ // If the delete record is chosen, return an option with the delete record, otherwise return empty.
+ return merged.isPresent() ? Option.of(deleteRecord) : Option.empty();
+ } catch (IOException e) {
+ throw new HoodieIOException("Failed to process delete record", e);
+ }
}
@Override
public BufferedRecord<T> finalMerge(BufferedRecord<T> olderRecord, BufferedRecord<T> newerRecord) throws IOException {
- if (olderRecord.isDelete() || newerRecord.isDelete()) {
- if (shouldKeepNewerRecord(olderRecord, newerRecord)) {
- // IMPORTANT:
- // this is needed when the fallback HoodieAvroRecordMerger got used, the merger would
- // return Option.empty when the new payload data is empty(a delete) and ignores its ordering value directly.
- return newerRecord;
- } else {
- return olderRecord;
- }
+ if (olderRecord == null) {
+ return newerRecord;
+ }
+ // handle special case for deletes that are sent to older partitions in global-index, this delete takes precedence regardless of the previous value
+ if (newerRecord.getHoodieOperation() == HoodieOperation.UPDATE_BEFORE) {
Review Comment:
we better not have this special handing because for cdc inputs of Flink, each record could takea operation, the `#mergeRecords` should handle it correctly because the ordering value is commit time `0`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT] Custom merger update [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13742:
URL: https://github.com/apache/hudi/pull/13742#issuecomment-3218237082
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7988",
"triggerID" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"triggerType" : "PUSH"
}, {
"hash" : "ecb694295f7a88371f33df6de60370058670439e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7992",
"triggerID" : "ecb694295f7a88371f33df6de60370058670439e",
"triggerType" : "PUSH"
}, {
"hash" : "efd7462406be4f6d90dd5e42f152d66f683ebdf4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "efd7462406be4f6d90dd5e42f152d66f683ebdf4",
"triggerType" : "PUSH"
}, {
"hash" : "3c67160d5ceec883db5d826ff655865a3209f2f3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7994",
"triggerID" : "3c67160d5ceec883db5d826ff655865a3209f2f3",
"triggerType" : "PUSH"
}, {
"hash" : "f4f84d9f6a480923df27cfe5553d8aa65b2bbb7a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7996",
"triggerID" : "f4f84d9f6a480923df27cfe5553d8aa65b2bbb7a",
"triggerType" : "PUSH"
}, {
"hash" : "521b26bc2c60b7921e2f7aa5fffa620ab7f9a096",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "521b26bc2c60b7921e2f7aa5fffa620ab7f9a096",
"triggerType" : "PUSH"
}, {
"hash" : "22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50",
"triggerType" : "PUSH"
}, {
"hash" : "02a319ba884595a4e3b5453f366b660c1c11f767",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8007",
"triggerID" : "02a319ba884595a4e3b5453f366b660c1c11f767",
"triggerType" : "PUSH"
}, {
"hash" : "3e59b6f8b486016911c9ad14c583f6538cc036ac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8009",
"triggerID" : "3e59b6f8b486016911c9ad14c583f6538cc036ac",
"triggerType" : "PUSH"
}, {
"hash" : "003f1c81efedafa0ddd78868ce83357d8803c5e5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "003f1c81efedafa0ddd78868ce83357d8803c5e5",
"triggerType" : "PUSH"
}, {
"hash" : "40c04768d5233a38be1b4c01ce895189a4af4c3a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8013",
"triggerID" : "40c04768d5233a38be1b4c01ce895189a4af4c3a",
"triggerType" : "PUSH"
}, {
"hash" : "4a699a8dcf79e61ea3bb754e67e9cc4a32e86aa1",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8084",
"triggerID" : "4a699a8dcf79e61ea3bb754e67e9cc4a32e86aa1",
"triggerType" : "PUSH"
}, {
"hash" : "fe55d8ec4923cb6c64e83ae4f32217c72599a398",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "fe55d8ec4923cb6c64e83ae4f32217c72599a398",
"triggerType" : "PUSH"
}, {
"hash" : "87cd79787dc8126b6877ba0d3407cfe8f343933a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8087",
"triggerID" : "87cd79787dc8126b6877ba0d3407cfe8f343933a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* efd7462406be4f6d90dd5e42f152d66f683ebdf4 UNKNOWN
* 521b26bc2c60b7921e2f7aa5fffa620ab7f9a096 UNKNOWN
* 22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50 UNKNOWN
* 003f1c81efedafa0ddd78868ce83357d8803c5e5 UNKNOWN
* 4a699a8dcf79e61ea3bb754e67e9cc4a32e86aa1 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8084)
* fe55d8ec4923cb6c64e83ae4f32217c72599a398 UNKNOWN
* 87cd79787dc8126b6877ba0d3407cfe8f343933a Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8087)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9715] Custom merger update [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #13742:
URL: https://github.com/apache/hudi/pull/13742#discussion_r2302012267
##########
hudi-common/src/main/java/org/apache/hudi/common/model/HoodieAvroRecordMerger.java:
##########
@@ -43,9 +44,35 @@ public String getMergingStrategy() {
}
@Override
- public Option<Pair<HoodieRecord, Schema>> merge(HoodieRecord older, Schema oldSchema, HoodieRecord newer, Schema newSchema, TypedProperties props) throws IOException {
- return combineAndGetUpdateValue(older, newer, oldSchema, newSchema, props)
- .map(r -> Pair.of(new HoodieAvroIndexedRecord(r), r.getSchema()));
+ public Pair<HoodieRecord, Schema> merge(HoodieRecord older, Schema oldSchema, HoodieRecord newer, Schema newSchema, TypedProperties props) throws IOException {
+ Option<IndexedRecord> previousAvroData = older.toIndexedRecord(oldSchema, props).map(HoodieAvroIndexedRecord::getData);
+ HoodieRecordPayload payload = ((HoodieAvroRecord) newer).getData();
+ Option<IndexedRecord> updatedValue;
+ Comparable orderingVal;
+ if (previousAvroData.isEmpty()) {
+ updatedValue = payload.getInsertValue(newSchema, props);
+ orderingVal = payload.getOrderingValue();
+ } else {
+ updatedValue = ((HoodieAvroRecord) newer).getData().combineAndGetUpdateValue(previousAvroData.get(), newSchema, props);
Review Comment:
Is it possible the newer record is null, like when we merge base file with records from log files?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9715] Custom merger update [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13742:
URL: https://github.com/apache/hudi/pull/13742#discussion_r2302484262
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/BufferedRecordMergerFactory.java:
##########
@@ -465,46 +451,45 @@ public CustomPayloadRecordMerger(
}
@Override
- public Option<BufferedRecord<T>> deltaMergeNonDeleteRecord(BufferedRecord<T> newRecord, BufferedRecord<T> existingRecord) throws IOException {
- Option<Pair<HoodieRecord, Schema>> mergedRecordAndSchema = getMergedRecord(existingRecord, newRecord, false);
- if (mergedRecordAndSchema.isEmpty()) {
- // An empty Option indicates that the output represents a delete.
- return Option.of(new BufferedRecord<>(newRecord.getRecordKey(), OrderingValues.getDefault(), null, null, HoodieOperation.DELETE));
- }
- HoodieRecord mergedRecord = mergedRecordAndSchema.get().getLeft();
- Schema mergeResultSchema = mergedRecordAndSchema.get().getRight();
+ public Option<BufferedRecord<T>> deltaMergeRecords(BufferedRecord<T> newRecord, BufferedRecord<T> existingRecord) throws IOException {
+ Pair<HoodieRecord, Schema> mergedRecordAndSchema = getMergedRecord(existingRecord, newRecord, false);
+ HoodieRecord mergedRecord = mergedRecordAndSchema.getLeft();
+ Schema mergeResultSchema = mergedRecordAndSchema.getRight();
// Special handling for SENTINEL record in Expression Payload. This is returned if the condition does not match.
if (mergedRecord.getData() == HoodieRecord.SENTINEL) {
return Option.empty();
}
- T combinedRecordData = recordContext.convertAvroRecord(mergedRecord.toIndexedRecord(mergeResultSchema, props).get().getData());
+ Option<T> combinedRecordData = (mergedRecord.toIndexedRecord(mergeResultSchema, props).map(indexedRecord -> recordContext.convertAvroRecord(indexedRecord.getData())));
// If pre-combine does not return existing record, update it
- if (combinedRecordData != existingRecord.getRecord()) {
+ if (combinedRecordData.map(record -> record != existingRecord.getRecord()).orElse(true)) {
// For pkless we need to use record key from existing record
- return Option.of(BufferedRecords.fromEngineRecord(combinedRecordData, mergeResultSchema, recordContext, orderingFieldNames,
- existingRecord.getRecordKey(), mergedRecord.isDelete(mergeResultSchema, props)));
+ boolean isDelete = mergedRecord.isDelete(readerSchema, props);
+ Comparable orderingValue = mergedRecord.getOrderingValue(mergeResultSchema, props, orderingFieldNames);
+ T mergedEngineRecord = mergedRecord.toIndexedRecord(mergeResultSchema, props)
+ .map(hoodieAvroIndexedRecord -> recordContext.convertAvroRecord(hoodieAvroIndexedRecord.getData()))
+ .orElse(null);
+ return Option.of(BufferedRecords.fromEngineRecord(mergedEngineRecord, readerSchema, recordContext, orderingValue, existingRecord.getRecordKey(), isDelete));
Review Comment:
`mergeResultSchema`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9715] Custom merger update [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13742:
URL: https://github.com/apache/hudi/pull/13742#discussion_r2300803147
##########
hudi-common/src/main/java/org/apache/hudi/common/model/HoodieRecordMerger.java:
##########
@@ -63,8 +63,15 @@ public interface HoodieRecordMerger extends Serializable {
* It'd be associative operation: f(a, f(b, c)) = f(f(a, b), c) (which we can translate as having 3 versions A, B, C
* of the single record, both orders of operations applications have to yield the same result)
* This method takes only full records for merging.
+ *
+ * @param older Older record in terms of commit time ordering.
+ * @param oldSchema The schema of the older record.
+ * @param newer Newer record in terms of commit time ordering.
+ * @param newSchema The schema of the newer record.
+ * @param props The additional properties for the merging operation.
+ * @return The merged record and schema. The record is expected to be non-null. If the record represents a deletion, the operation must be set as {@link HoodieOperation#DELETE}.
*/
- Option<Pair<HoodieRecord, Schema>> merge(HoodieRecord older, Schema oldSchema, HoodieRecord newer, Schema newSchema, TypedProperties props) throws IOException;
+ Pair<HoodieRecord, Schema> merge(HoodieRecord older, Schema oldSchema, HoodieRecord newer, Schema newSchema, TypedProperties props) throws IOException;
Review Comment:
Based on the changes in the patch, using `HoodieRecord` brings in a lot of intricate changes in write merging, buffered record mergers and deletes handling in mergers themself, that I prefer we have a "change record" abstraction here to simply the things, or just use `BufferedRecord` because it is a very near abstaction and it has been widely used in merging now(just to name it better as a developer/user API).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9715] Custom merger update [hudi]
Posted by "cshuo (via GitHub)" <gi...@apache.org>.
cshuo commented on code in PR #13742:
URL: https://github.com/apache/hudi/pull/13742#discussion_r2300957205
##########
hudi-spark-datasource/hudi-spark/src/test/java/org/apache/hudi/io/CustomMerger.java:
##########
@@ -0,0 +1,63 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.io;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordMerger;
+import org.apache.hudi.common.util.collection.Pair;
+
+import org.apache.avro.Schema;
+
+import java.io.IOException;
+import java.util.UUID;
+
+public class CustomMerger implements HoodieRecordMerger {
+ private static final String STRATEGY_ID = UUID.randomUUID().toString();
+
+ public static String getStrategyId() {
+ return STRATEGY_ID;
+ }
+
+ @Override
+ public Pair<HoodieRecord, Schema> merge(HoodieRecord older, Schema oldSchema, HoodieRecord newer, Schema newSchema, TypedProperties props) throws IOException {
+ Long olderTimestamp = (Long) older.getOrderingValue(oldSchema, props, new String[0]);
+ Long newerTimestamp = (Long) newer.getOrderingValue(oldSchema, props, new String[0]);
Review Comment:
`oldSchema` -> `newSchema`?
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieWriteMergeHandle.java:
##########
@@ -149,6 +158,7 @@ public HoodieWriteMergeHandle(HoodieWriteConfig config, String instantTime, Hood
public HoodieWriteMergeHandle(HoodieWriteConfig config, String instantTime, String partitionPath,
String fileId, HoodieTable<T, I, K, O> hoodieTable, TaskContextSupplier taskContextSupplier) {
super(config, instantTime, partitionPath, fileId, hoodieTable, taskContextSupplier, true);
+ this.orderingFields = ConfigUtils.getOrderingFields(config.getProps());
Review Comment:
can we also initialize `readerContext` in this constructor and make the `readerContext` final? This constructor is for fg based handle, seems we can pass the `readerContext` in parameters.
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieWriteMergeHandle.java:
##########
@@ -120,6 +125,8 @@ public HoodieWriteMergeHandle(HoodieWriteConfig config, String instantTime, Hood
super(config, instantTime, hoodieTable, partitionPath, fileId, taskContextSupplier, baseFile, keyGeneratorOpt, false);
populateIncomingRecordsMap(recordItr);
initMarkerFileAndFileWriter(fileId, partitionPath);
+ this.readerContext = hoodieTable.getReaderContextFactoryForWrite().getContext();
Review Comment:
There is duplicate re-initialization for `readerContext` after this change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9715] Custom merger update [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13742:
URL: https://github.com/apache/hudi/pull/13742#issuecomment-3235585984
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7988",
"triggerID" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"triggerType" : "PUSH"
}, {
"hash" : "ecb694295f7a88371f33df6de60370058670439e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7992",
"triggerID" : "ecb694295f7a88371f33df6de60370058670439e",
"triggerType" : "PUSH"
}, {
"hash" : "efd7462406be4f6d90dd5e42f152d66f683ebdf4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "efd7462406be4f6d90dd5e42f152d66f683ebdf4",
"triggerType" : "PUSH"
}, {
"hash" : "3c67160d5ceec883db5d826ff655865a3209f2f3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7994",
"triggerID" : "3c67160d5ceec883db5d826ff655865a3209f2f3",
"triggerType" : "PUSH"
}, {
"hash" : "f4f84d9f6a480923df27cfe5553d8aa65b2bbb7a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7996",
"triggerID" : "f4f84d9f6a480923df27cfe5553d8aa65b2bbb7a",
"triggerType" : "PUSH"
}, {
"hash" : "521b26bc2c60b7921e2f7aa5fffa620ab7f9a096",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "521b26bc2c60b7921e2f7aa5fffa620ab7f9a096",
"triggerType" : "PUSH"
}, {
"hash" : "22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50",
"triggerType" : "PUSH"
}, {
"hash" : "02a319ba884595a4e3b5453f366b660c1c11f767",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8007",
"triggerID" : "02a319ba884595a4e3b5453f366b660c1c11f767",
"triggerType" : "PUSH"
}, {
"hash" : "3e59b6f8b486016911c9ad14c583f6538cc036ac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8009",
"triggerID" : "3e59b6f8b486016911c9ad14c583f6538cc036ac",
"triggerType" : "PUSH"
}, {
"hash" : "003f1c81efedafa0ddd78868ce83357d8803c5e5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "003f1c81efedafa0ddd78868ce83357d8803c5e5",
"triggerType" : "PUSH"
}, {
"hash" : "40c04768d5233a38be1b4c01ce895189a4af4c3a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8013",
"triggerID" : "40c04768d5233a38be1b4c01ce895189a4af4c3a",
"triggerType" : "PUSH"
}, {
"hash" : "4a699a8dcf79e61ea3bb754e67e9cc4a32e86aa1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8084",
"triggerID" : "4a699a8dcf79e61ea3bb754e67e9cc4a32e86aa1",
"triggerType" : "PUSH"
}, {
"hash" : "fe55d8ec4923cb6c64e83ae4f32217c72599a398",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "fe55d8ec4923cb6c64e83ae4f32217c72599a398",
"triggerType" : "PUSH"
}, {
"hash" : "87cd79787dc8126b6877ba0d3407cfe8f343933a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8087",
"triggerID" : "87cd79787dc8126b6877ba0d3407cfe8f343933a",
"triggerType" : "PUSH"
}, {
"hash" : "594874ffdca647cd268b3947fa4680c57c53d679",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8088",
"triggerID" : "594874ffdca647cd268b3947fa4680c57c53d679",
"triggerType" : "PUSH"
}, {
"hash" : "7b95754f0df2c1a087a6a28722a7471ee12cecf3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8090",
"triggerID" : "7b95754f0df2c1a087a6a28722a7471ee12cecf3",
"triggerType" : "PUSH"
}, {
"hash" : "789c2da879ca66f6b9be71d1814be38b5d8cfb58",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8102",
"triggerID" : "789c2da879ca66f6b9be71d1814be38b5d8cfb58",
"triggerType" : "PUSH"
}, {
"hash" : "63f33fae3e22a9f80de51b2a6e1ed0427f631d0b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8103",
"triggerID" : "63f33fae3e22a9f80de51b2a6e1ed0427f631d0b",
"triggerType" : "PUSH"
}, {
"hash" : "e04c466fa81e9bcfaabe36825127098f5d1bb890",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8224",
"triggerID" : "e04c466fa81e9bcfaabe36825127098f5d1bb890",
"triggerType" : "PUSH"
}, {
"hash" : "cf574f46a1783a39b2074542aad9539ec3c4f36a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8235",
"triggerID" : "cf574f46a1783a39b2074542aad9539ec3c4f36a",
"triggerType" : "PUSH"
}, {
"hash" : "2eb81e923adfa77176f137c442143deaac9e9916",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2eb81e923adfa77176f137c442143deaac9e9916",
"triggerType" : "PUSH"
}, {
"hash" : "12dc3e3eaf746bb1f2bc3f4e93fa14d9adb0f6fd",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8244",
"triggerID" : "12dc3e3eaf746bb1f2bc3f4e93fa14d9adb0f6fd",
"triggerType" : "PUSH"
}, {
"hash" : "af5ab73103f3af882412bbe99c9322165e8941a0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "af5ab73103f3af882412bbe99c9322165e8941a0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* efd7462406be4f6d90dd5e42f152d66f683ebdf4 UNKNOWN
* 521b26bc2c60b7921e2f7aa5fffa620ab7f9a096 UNKNOWN
* 22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50 UNKNOWN
* 003f1c81efedafa0ddd78868ce83357d8803c5e5 UNKNOWN
* fe55d8ec4923cb6c64e83ae4f32217c72599a398 UNKNOWN
* 2eb81e923adfa77176f137c442143deaac9e9916 UNKNOWN
* 12dc3e3eaf746bb1f2bc3f4e93fa14d9adb0f6fd Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8244)
* af5ab73103f3af882412bbe99c9322165e8941a0 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9715] Custom merger update [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13742:
URL: https://github.com/apache/hudi/pull/13742#issuecomment-3236561180
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7988",
"triggerID" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"triggerType" : "PUSH"
}, {
"hash" : "ecb694295f7a88371f33df6de60370058670439e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7992",
"triggerID" : "ecb694295f7a88371f33df6de60370058670439e",
"triggerType" : "PUSH"
}, {
"hash" : "efd7462406be4f6d90dd5e42f152d66f683ebdf4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "efd7462406be4f6d90dd5e42f152d66f683ebdf4",
"triggerType" : "PUSH"
}, {
"hash" : "3c67160d5ceec883db5d826ff655865a3209f2f3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7994",
"triggerID" : "3c67160d5ceec883db5d826ff655865a3209f2f3",
"triggerType" : "PUSH"
}, {
"hash" : "f4f84d9f6a480923df27cfe5553d8aa65b2bbb7a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7996",
"triggerID" : "f4f84d9f6a480923df27cfe5553d8aa65b2bbb7a",
"triggerType" : "PUSH"
}, {
"hash" : "521b26bc2c60b7921e2f7aa5fffa620ab7f9a096",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "521b26bc2c60b7921e2f7aa5fffa620ab7f9a096",
"triggerType" : "PUSH"
}, {
"hash" : "22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50",
"triggerType" : "PUSH"
}, {
"hash" : "02a319ba884595a4e3b5453f366b660c1c11f767",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8007",
"triggerID" : "02a319ba884595a4e3b5453f366b660c1c11f767",
"triggerType" : "PUSH"
}, {
"hash" : "3e59b6f8b486016911c9ad14c583f6538cc036ac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8009",
"triggerID" : "3e59b6f8b486016911c9ad14c583f6538cc036ac",
"triggerType" : "PUSH"
}, {
"hash" : "003f1c81efedafa0ddd78868ce83357d8803c5e5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "003f1c81efedafa0ddd78868ce83357d8803c5e5",
"triggerType" : "PUSH"
}, {
"hash" : "40c04768d5233a38be1b4c01ce895189a4af4c3a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8013",
"triggerID" : "40c04768d5233a38be1b4c01ce895189a4af4c3a",
"triggerType" : "PUSH"
}, {
"hash" : "4a699a8dcf79e61ea3bb754e67e9cc4a32e86aa1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8084",
"triggerID" : "4a699a8dcf79e61ea3bb754e67e9cc4a32e86aa1",
"triggerType" : "PUSH"
}, {
"hash" : "fe55d8ec4923cb6c64e83ae4f32217c72599a398",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "fe55d8ec4923cb6c64e83ae4f32217c72599a398",
"triggerType" : "PUSH"
}, {
"hash" : "87cd79787dc8126b6877ba0d3407cfe8f343933a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8087",
"triggerID" : "87cd79787dc8126b6877ba0d3407cfe8f343933a",
"triggerType" : "PUSH"
}, {
"hash" : "594874ffdca647cd268b3947fa4680c57c53d679",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8088",
"triggerID" : "594874ffdca647cd268b3947fa4680c57c53d679",
"triggerType" : "PUSH"
}, {
"hash" : "7b95754f0df2c1a087a6a28722a7471ee12cecf3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8090",
"triggerID" : "7b95754f0df2c1a087a6a28722a7471ee12cecf3",
"triggerType" : "PUSH"
}, {
"hash" : "789c2da879ca66f6b9be71d1814be38b5d8cfb58",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8102",
"triggerID" : "789c2da879ca66f6b9be71d1814be38b5d8cfb58",
"triggerType" : "PUSH"
}, {
"hash" : "63f33fae3e22a9f80de51b2a6e1ed0427f631d0b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8103",
"triggerID" : "63f33fae3e22a9f80de51b2a6e1ed0427f631d0b",
"triggerType" : "PUSH"
}, {
"hash" : "e04c466fa81e9bcfaabe36825127098f5d1bb890",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8224",
"triggerID" : "e04c466fa81e9bcfaabe36825127098f5d1bb890",
"triggerType" : "PUSH"
}, {
"hash" : "cf574f46a1783a39b2074542aad9539ec3c4f36a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8235",
"triggerID" : "cf574f46a1783a39b2074542aad9539ec3c4f36a",
"triggerType" : "PUSH"
}, {
"hash" : "2eb81e923adfa77176f137c442143deaac9e9916",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2eb81e923adfa77176f137c442143deaac9e9916",
"triggerType" : "PUSH"
}, {
"hash" : "12dc3e3eaf746bb1f2bc3f4e93fa14d9adb0f6fd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8244",
"triggerID" : "12dc3e3eaf746bb1f2bc3f4e93fa14d9adb0f6fd",
"triggerType" : "PUSH"
}, {
"hash" : "af5ab73103f3af882412bbe99c9322165e8941a0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "af5ab73103f3af882412bbe99c9322165e8941a0",
"triggerType" : "PUSH"
}, {
"hash" : "a9c28c4d913644fea23d57a5dc2224f36fd2b63e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8248",
"triggerID" : "a9c28c4d913644fea23d57a5dc2224f36fd2b63e",
"triggerType" : "PUSH"
}, {
"hash" : "98381a6bd2bcccf3da08fb992e599fe1165f2ddb",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "98381a6bd2bcccf3da08fb992e599fe1165f2ddb",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* efd7462406be4f6d90dd5e42f152d66f683ebdf4 UNKNOWN
* 521b26bc2c60b7921e2f7aa5fffa620ab7f9a096 UNKNOWN
* 22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50 UNKNOWN
* 003f1c81efedafa0ddd78868ce83357d8803c5e5 UNKNOWN
* fe55d8ec4923cb6c64e83ae4f32217c72599a398 UNKNOWN
* 2eb81e923adfa77176f137c442143deaac9e9916 UNKNOWN
* af5ab73103f3af882412bbe99c9322165e8941a0 UNKNOWN
* a9c28c4d913644fea23d57a5dc2224f36fd2b63e Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8248)
* 98381a6bd2bcccf3da08fb992e599fe1165f2ddb UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT] Custom merger update [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13742:
URL: https://github.com/apache/hudi/pull/13742#issuecomment-3218433339
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7988",
"triggerID" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"triggerType" : "PUSH"
}, {
"hash" : "ecb694295f7a88371f33df6de60370058670439e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7992",
"triggerID" : "ecb694295f7a88371f33df6de60370058670439e",
"triggerType" : "PUSH"
}, {
"hash" : "efd7462406be4f6d90dd5e42f152d66f683ebdf4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "efd7462406be4f6d90dd5e42f152d66f683ebdf4",
"triggerType" : "PUSH"
}, {
"hash" : "3c67160d5ceec883db5d826ff655865a3209f2f3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7994",
"triggerID" : "3c67160d5ceec883db5d826ff655865a3209f2f3",
"triggerType" : "PUSH"
}, {
"hash" : "f4f84d9f6a480923df27cfe5553d8aa65b2bbb7a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7996",
"triggerID" : "f4f84d9f6a480923df27cfe5553d8aa65b2bbb7a",
"triggerType" : "PUSH"
}, {
"hash" : "521b26bc2c60b7921e2f7aa5fffa620ab7f9a096",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "521b26bc2c60b7921e2f7aa5fffa620ab7f9a096",
"triggerType" : "PUSH"
}, {
"hash" : "22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50",
"triggerType" : "PUSH"
}, {
"hash" : "02a319ba884595a4e3b5453f366b660c1c11f767",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8007",
"triggerID" : "02a319ba884595a4e3b5453f366b660c1c11f767",
"triggerType" : "PUSH"
}, {
"hash" : "3e59b6f8b486016911c9ad14c583f6538cc036ac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8009",
"triggerID" : "3e59b6f8b486016911c9ad14c583f6538cc036ac",
"triggerType" : "PUSH"
}, {
"hash" : "003f1c81efedafa0ddd78868ce83357d8803c5e5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "003f1c81efedafa0ddd78868ce83357d8803c5e5",
"triggerType" : "PUSH"
}, {
"hash" : "40c04768d5233a38be1b4c01ce895189a4af4c3a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8013",
"triggerID" : "40c04768d5233a38be1b4c01ce895189a4af4c3a",
"triggerType" : "PUSH"
}, {
"hash" : "4a699a8dcf79e61ea3bb754e67e9cc4a32e86aa1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8084",
"triggerID" : "4a699a8dcf79e61ea3bb754e67e9cc4a32e86aa1",
"triggerType" : "PUSH"
}, {
"hash" : "fe55d8ec4923cb6c64e83ae4f32217c72599a398",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "fe55d8ec4923cb6c64e83ae4f32217c72599a398",
"triggerType" : "PUSH"
}, {
"hash" : "87cd79787dc8126b6877ba0d3407cfe8f343933a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8087",
"triggerID" : "87cd79787dc8126b6877ba0d3407cfe8f343933a",
"triggerType" : "PUSH"
}, {
"hash" : "594874ffdca647cd268b3947fa4680c57c53d679",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8088",
"triggerID" : "594874ffdca647cd268b3947fa4680c57c53d679",
"triggerType" : "PUSH"
}, {
"hash" : "7b95754f0df2c1a087a6a28722a7471ee12cecf3",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8090",
"triggerID" : "7b95754f0df2c1a087a6a28722a7471ee12cecf3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* efd7462406be4f6d90dd5e42f152d66f683ebdf4 UNKNOWN
* 521b26bc2c60b7921e2f7aa5fffa620ab7f9a096 UNKNOWN
* 22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50 UNKNOWN
* 003f1c81efedafa0ddd78868ce83357d8803c5e5 UNKNOWN
* fe55d8ec4923cb6c64e83ae4f32217c72599a398 UNKNOWN
* 594874ffdca647cd268b3947fa4680c57c53d679 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8088)
* 7b95754f0df2c1a087a6a28722a7471ee12cecf3 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8090)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9715] Custom merger update [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13742:
URL: https://github.com/apache/hudi/pull/13742#issuecomment-3300002074
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "94aa9a669d2fe1fb4919a4735fab2665553b63c0",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8258",
"triggerID" : "94aa9a669d2fe1fb4919a4735fab2665553b63c0",
"triggerType" : "PUSH"
}, {
"hash" : "94aa9a669d2fe1fb4919a4735fab2665553b63c0",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8258",
"triggerID" : "3237521433",
"triggerType" : "MANUAL"
}, {
"hash" : "57b338240d2ac6faa9ec0a329f43f13e28019eb8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "57b338240d2ac6faa9ec0a329f43f13e28019eb8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 94aa9a669d2fe1fb4919a4735fab2665553b63c0 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8258)
* 57b338240d2ac6faa9ec0a329f43f13e28019eb8 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9715] Custom merger update [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13742:
URL: https://github.com/apache/hudi/pull/13742#issuecomment-3234570784
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7988",
"triggerID" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"triggerType" : "PUSH"
}, {
"hash" : "ecb694295f7a88371f33df6de60370058670439e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7992",
"triggerID" : "ecb694295f7a88371f33df6de60370058670439e",
"triggerType" : "PUSH"
}, {
"hash" : "efd7462406be4f6d90dd5e42f152d66f683ebdf4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "efd7462406be4f6d90dd5e42f152d66f683ebdf4",
"triggerType" : "PUSH"
}, {
"hash" : "3c67160d5ceec883db5d826ff655865a3209f2f3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7994",
"triggerID" : "3c67160d5ceec883db5d826ff655865a3209f2f3",
"triggerType" : "PUSH"
}, {
"hash" : "f4f84d9f6a480923df27cfe5553d8aa65b2bbb7a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7996",
"triggerID" : "f4f84d9f6a480923df27cfe5553d8aa65b2bbb7a",
"triggerType" : "PUSH"
}, {
"hash" : "521b26bc2c60b7921e2f7aa5fffa620ab7f9a096",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "521b26bc2c60b7921e2f7aa5fffa620ab7f9a096",
"triggerType" : "PUSH"
}, {
"hash" : "22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50",
"triggerType" : "PUSH"
}, {
"hash" : "02a319ba884595a4e3b5453f366b660c1c11f767",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8007",
"triggerID" : "02a319ba884595a4e3b5453f366b660c1c11f767",
"triggerType" : "PUSH"
}, {
"hash" : "3e59b6f8b486016911c9ad14c583f6538cc036ac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8009",
"triggerID" : "3e59b6f8b486016911c9ad14c583f6538cc036ac",
"triggerType" : "PUSH"
}, {
"hash" : "003f1c81efedafa0ddd78868ce83357d8803c5e5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "003f1c81efedafa0ddd78868ce83357d8803c5e5",
"triggerType" : "PUSH"
}, {
"hash" : "40c04768d5233a38be1b4c01ce895189a4af4c3a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8013",
"triggerID" : "40c04768d5233a38be1b4c01ce895189a4af4c3a",
"triggerType" : "PUSH"
}, {
"hash" : "4a699a8dcf79e61ea3bb754e67e9cc4a32e86aa1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8084",
"triggerID" : "4a699a8dcf79e61ea3bb754e67e9cc4a32e86aa1",
"triggerType" : "PUSH"
}, {
"hash" : "fe55d8ec4923cb6c64e83ae4f32217c72599a398",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "fe55d8ec4923cb6c64e83ae4f32217c72599a398",
"triggerType" : "PUSH"
}, {
"hash" : "87cd79787dc8126b6877ba0d3407cfe8f343933a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8087",
"triggerID" : "87cd79787dc8126b6877ba0d3407cfe8f343933a",
"triggerType" : "PUSH"
}, {
"hash" : "594874ffdca647cd268b3947fa4680c57c53d679",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8088",
"triggerID" : "594874ffdca647cd268b3947fa4680c57c53d679",
"triggerType" : "PUSH"
}, {
"hash" : "7b95754f0df2c1a087a6a28722a7471ee12cecf3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8090",
"triggerID" : "7b95754f0df2c1a087a6a28722a7471ee12cecf3",
"triggerType" : "PUSH"
}, {
"hash" : "789c2da879ca66f6b9be71d1814be38b5d8cfb58",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8102",
"triggerID" : "789c2da879ca66f6b9be71d1814be38b5d8cfb58",
"triggerType" : "PUSH"
}, {
"hash" : "63f33fae3e22a9f80de51b2a6e1ed0427f631d0b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8103",
"triggerID" : "63f33fae3e22a9f80de51b2a6e1ed0427f631d0b",
"triggerType" : "PUSH"
}, {
"hash" : "e04c466fa81e9bcfaabe36825127098f5d1bb890",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8224",
"triggerID" : "e04c466fa81e9bcfaabe36825127098f5d1bb890",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* efd7462406be4f6d90dd5e42f152d66f683ebdf4 UNKNOWN
* 521b26bc2c60b7921e2f7aa5fffa620ab7f9a096 UNKNOWN
* 22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50 UNKNOWN
* 003f1c81efedafa0ddd78868ce83357d8803c5e5 UNKNOWN
* fe55d8ec4923cb6c64e83ae4f32217c72599a398 UNKNOWN
* 63f33fae3e22a9f80de51b2a6e1ed0427f631d0b Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8103)
* e04c466fa81e9bcfaabe36825127098f5d1bb890 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8224)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9715] Custom merger update [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13742:
URL: https://github.com/apache/hudi/pull/13742#issuecomment-3234686914
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7988",
"triggerID" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"triggerType" : "PUSH"
}, {
"hash" : "ecb694295f7a88371f33df6de60370058670439e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7992",
"triggerID" : "ecb694295f7a88371f33df6de60370058670439e",
"triggerType" : "PUSH"
}, {
"hash" : "efd7462406be4f6d90dd5e42f152d66f683ebdf4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "efd7462406be4f6d90dd5e42f152d66f683ebdf4",
"triggerType" : "PUSH"
}, {
"hash" : "3c67160d5ceec883db5d826ff655865a3209f2f3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7994",
"triggerID" : "3c67160d5ceec883db5d826ff655865a3209f2f3",
"triggerType" : "PUSH"
}, {
"hash" : "f4f84d9f6a480923df27cfe5553d8aa65b2bbb7a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7996",
"triggerID" : "f4f84d9f6a480923df27cfe5553d8aa65b2bbb7a",
"triggerType" : "PUSH"
}, {
"hash" : "521b26bc2c60b7921e2f7aa5fffa620ab7f9a096",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "521b26bc2c60b7921e2f7aa5fffa620ab7f9a096",
"triggerType" : "PUSH"
}, {
"hash" : "22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50",
"triggerType" : "PUSH"
}, {
"hash" : "02a319ba884595a4e3b5453f366b660c1c11f767",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8007",
"triggerID" : "02a319ba884595a4e3b5453f366b660c1c11f767",
"triggerType" : "PUSH"
}, {
"hash" : "3e59b6f8b486016911c9ad14c583f6538cc036ac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8009",
"triggerID" : "3e59b6f8b486016911c9ad14c583f6538cc036ac",
"triggerType" : "PUSH"
}, {
"hash" : "003f1c81efedafa0ddd78868ce83357d8803c5e5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "003f1c81efedafa0ddd78868ce83357d8803c5e5",
"triggerType" : "PUSH"
}, {
"hash" : "40c04768d5233a38be1b4c01ce895189a4af4c3a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8013",
"triggerID" : "40c04768d5233a38be1b4c01ce895189a4af4c3a",
"triggerType" : "PUSH"
}, {
"hash" : "4a699a8dcf79e61ea3bb754e67e9cc4a32e86aa1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8084",
"triggerID" : "4a699a8dcf79e61ea3bb754e67e9cc4a32e86aa1",
"triggerType" : "PUSH"
}, {
"hash" : "fe55d8ec4923cb6c64e83ae4f32217c72599a398",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "fe55d8ec4923cb6c64e83ae4f32217c72599a398",
"triggerType" : "PUSH"
}, {
"hash" : "87cd79787dc8126b6877ba0d3407cfe8f343933a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8087",
"triggerID" : "87cd79787dc8126b6877ba0d3407cfe8f343933a",
"triggerType" : "PUSH"
}, {
"hash" : "594874ffdca647cd268b3947fa4680c57c53d679",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8088",
"triggerID" : "594874ffdca647cd268b3947fa4680c57c53d679",
"triggerType" : "PUSH"
}, {
"hash" : "7b95754f0df2c1a087a6a28722a7471ee12cecf3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8090",
"triggerID" : "7b95754f0df2c1a087a6a28722a7471ee12cecf3",
"triggerType" : "PUSH"
}, {
"hash" : "789c2da879ca66f6b9be71d1814be38b5d8cfb58",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8102",
"triggerID" : "789c2da879ca66f6b9be71d1814be38b5d8cfb58",
"triggerType" : "PUSH"
}, {
"hash" : "63f33fae3e22a9f80de51b2a6e1ed0427f631d0b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8103",
"triggerID" : "63f33fae3e22a9f80de51b2a6e1ed0427f631d0b",
"triggerType" : "PUSH"
}, {
"hash" : "e04c466fa81e9bcfaabe36825127098f5d1bb890",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8224",
"triggerID" : "e04c466fa81e9bcfaabe36825127098f5d1bb890",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* efd7462406be4f6d90dd5e42f152d66f683ebdf4 UNKNOWN
* 521b26bc2c60b7921e2f7aa5fffa620ab7f9a096 UNKNOWN
* 22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50 UNKNOWN
* 003f1c81efedafa0ddd78868ce83357d8803c5e5 UNKNOWN
* fe55d8ec4923cb6c64e83ae4f32217c72599a398 UNKNOWN
* e04c466fa81e9bcfaabe36825127098f5d1bb890 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8224)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9715] Custom merger update [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13742:
URL: https://github.com/apache/hudi/pull/13742#discussion_r2308933595
##########
hudi-common/src/main/java/org/apache/hudi/common/model/HoodieAvroRecordMerger.java:
##########
@@ -19,51 +19,101 @@
package org.apache.hudi.common.model;
import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.engine.RecordContext;
import org.apache.hudi.common.model.HoodieRecord.HoodieRecordType;
+import org.apache.hudi.common.table.read.BufferedRecord;
+import org.apache.hudi.common.table.read.BufferedRecords;
+import org.apache.hudi.common.util.ConfigUtils;
+import org.apache.hudi.common.util.HoodieRecordUtils;
import org.apache.hudi.common.util.Option;
-import org.apache.hudi.common.util.collection.Pair;
import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
import org.apache.avro.generic.IndexedRecord;
import java.io.IOException;
-import java.util.Properties;
/**
* Record merger for Hoodie avro record.
*
* <p>It should only be used for base record from disk to merge with incoming record.
*/
public class HoodieAvroRecordMerger implements HoodieRecordMerger, OperationModeAwareness {
- public static final HoodieAvroRecordMerger INSTANCE = new HoodieAvroRecordMerger();
+ public static final String PAYLOAD_CLASS_PROP = "_hoodie.merger.payload.class";
Review Comment:
Yes, since it will differ from the table's payload class. Not sure what the best way is to ensure the merger has the right payload class set during the merge into operations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9715] Custom merger update [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13742:
URL: https://github.com/apache/hudi/pull/13742#issuecomment-3235540142
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7988",
"triggerID" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"triggerType" : "PUSH"
}, {
"hash" : "ecb694295f7a88371f33df6de60370058670439e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7992",
"triggerID" : "ecb694295f7a88371f33df6de60370058670439e",
"triggerType" : "PUSH"
}, {
"hash" : "efd7462406be4f6d90dd5e42f152d66f683ebdf4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "efd7462406be4f6d90dd5e42f152d66f683ebdf4",
"triggerType" : "PUSH"
}, {
"hash" : "3c67160d5ceec883db5d826ff655865a3209f2f3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7994",
"triggerID" : "3c67160d5ceec883db5d826ff655865a3209f2f3",
"triggerType" : "PUSH"
}, {
"hash" : "f4f84d9f6a480923df27cfe5553d8aa65b2bbb7a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7996",
"triggerID" : "f4f84d9f6a480923df27cfe5553d8aa65b2bbb7a",
"triggerType" : "PUSH"
}, {
"hash" : "521b26bc2c60b7921e2f7aa5fffa620ab7f9a096",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "521b26bc2c60b7921e2f7aa5fffa620ab7f9a096",
"triggerType" : "PUSH"
}, {
"hash" : "22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50",
"triggerType" : "PUSH"
}, {
"hash" : "02a319ba884595a4e3b5453f366b660c1c11f767",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8007",
"triggerID" : "02a319ba884595a4e3b5453f366b660c1c11f767",
"triggerType" : "PUSH"
}, {
"hash" : "3e59b6f8b486016911c9ad14c583f6538cc036ac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8009",
"triggerID" : "3e59b6f8b486016911c9ad14c583f6538cc036ac",
"triggerType" : "PUSH"
}, {
"hash" : "003f1c81efedafa0ddd78868ce83357d8803c5e5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "003f1c81efedafa0ddd78868ce83357d8803c5e5",
"triggerType" : "PUSH"
}, {
"hash" : "40c04768d5233a38be1b4c01ce895189a4af4c3a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8013",
"triggerID" : "40c04768d5233a38be1b4c01ce895189a4af4c3a",
"triggerType" : "PUSH"
}, {
"hash" : "4a699a8dcf79e61ea3bb754e67e9cc4a32e86aa1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8084",
"triggerID" : "4a699a8dcf79e61ea3bb754e67e9cc4a32e86aa1",
"triggerType" : "PUSH"
}, {
"hash" : "fe55d8ec4923cb6c64e83ae4f32217c72599a398",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "fe55d8ec4923cb6c64e83ae4f32217c72599a398",
"triggerType" : "PUSH"
}, {
"hash" : "87cd79787dc8126b6877ba0d3407cfe8f343933a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8087",
"triggerID" : "87cd79787dc8126b6877ba0d3407cfe8f343933a",
"triggerType" : "PUSH"
}, {
"hash" : "594874ffdca647cd268b3947fa4680c57c53d679",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8088",
"triggerID" : "594874ffdca647cd268b3947fa4680c57c53d679",
"triggerType" : "PUSH"
}, {
"hash" : "7b95754f0df2c1a087a6a28722a7471ee12cecf3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8090",
"triggerID" : "7b95754f0df2c1a087a6a28722a7471ee12cecf3",
"triggerType" : "PUSH"
}, {
"hash" : "789c2da879ca66f6b9be71d1814be38b5d8cfb58",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8102",
"triggerID" : "789c2da879ca66f6b9be71d1814be38b5d8cfb58",
"triggerType" : "PUSH"
}, {
"hash" : "63f33fae3e22a9f80de51b2a6e1ed0427f631d0b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8103",
"triggerID" : "63f33fae3e22a9f80de51b2a6e1ed0427f631d0b",
"triggerType" : "PUSH"
}, {
"hash" : "e04c466fa81e9bcfaabe36825127098f5d1bb890",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8224",
"triggerID" : "e04c466fa81e9bcfaabe36825127098f5d1bb890",
"triggerType" : "PUSH"
}, {
"hash" : "cf574f46a1783a39b2074542aad9539ec3c4f36a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8235",
"triggerID" : "cf574f46a1783a39b2074542aad9539ec3c4f36a",
"triggerType" : "PUSH"
}, {
"hash" : "2eb81e923adfa77176f137c442143deaac9e9916",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2eb81e923adfa77176f137c442143deaac9e9916",
"triggerType" : "PUSH"
}, {
"hash" : "12dc3e3eaf746bb1f2bc3f4e93fa14d9adb0f6fd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "12dc3e3eaf746bb1f2bc3f4e93fa14d9adb0f6fd",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* efd7462406be4f6d90dd5e42f152d66f683ebdf4 UNKNOWN
* 521b26bc2c60b7921e2f7aa5fffa620ab7f9a096 UNKNOWN
* 22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50 UNKNOWN
* 003f1c81efedafa0ddd78868ce83357d8803c5e5 UNKNOWN
* fe55d8ec4923cb6c64e83ae4f32217c72599a398 UNKNOWN
* cf574f46a1783a39b2074542aad9539ec3c4f36a Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8235)
* 2eb81e923adfa77176f137c442143deaac9e9916 UNKNOWN
* 12dc3e3eaf746bb1f2bc3f4e93fa14d9adb0f6fd UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9715] Custom merger update [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13742:
URL: https://github.com/apache/hudi/pull/13742#issuecomment-3235558444
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7988",
"triggerID" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"triggerType" : "PUSH"
}, {
"hash" : "ecb694295f7a88371f33df6de60370058670439e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7992",
"triggerID" : "ecb694295f7a88371f33df6de60370058670439e",
"triggerType" : "PUSH"
}, {
"hash" : "efd7462406be4f6d90dd5e42f152d66f683ebdf4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "efd7462406be4f6d90dd5e42f152d66f683ebdf4",
"triggerType" : "PUSH"
}, {
"hash" : "3c67160d5ceec883db5d826ff655865a3209f2f3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7994",
"triggerID" : "3c67160d5ceec883db5d826ff655865a3209f2f3",
"triggerType" : "PUSH"
}, {
"hash" : "f4f84d9f6a480923df27cfe5553d8aa65b2bbb7a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7996",
"triggerID" : "f4f84d9f6a480923df27cfe5553d8aa65b2bbb7a",
"triggerType" : "PUSH"
}, {
"hash" : "521b26bc2c60b7921e2f7aa5fffa620ab7f9a096",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "521b26bc2c60b7921e2f7aa5fffa620ab7f9a096",
"triggerType" : "PUSH"
}, {
"hash" : "22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50",
"triggerType" : "PUSH"
}, {
"hash" : "02a319ba884595a4e3b5453f366b660c1c11f767",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8007",
"triggerID" : "02a319ba884595a4e3b5453f366b660c1c11f767",
"triggerType" : "PUSH"
}, {
"hash" : "3e59b6f8b486016911c9ad14c583f6538cc036ac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8009",
"triggerID" : "3e59b6f8b486016911c9ad14c583f6538cc036ac",
"triggerType" : "PUSH"
}, {
"hash" : "003f1c81efedafa0ddd78868ce83357d8803c5e5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "003f1c81efedafa0ddd78868ce83357d8803c5e5",
"triggerType" : "PUSH"
}, {
"hash" : "40c04768d5233a38be1b4c01ce895189a4af4c3a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8013",
"triggerID" : "40c04768d5233a38be1b4c01ce895189a4af4c3a",
"triggerType" : "PUSH"
}, {
"hash" : "4a699a8dcf79e61ea3bb754e67e9cc4a32e86aa1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8084",
"triggerID" : "4a699a8dcf79e61ea3bb754e67e9cc4a32e86aa1",
"triggerType" : "PUSH"
}, {
"hash" : "fe55d8ec4923cb6c64e83ae4f32217c72599a398",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "fe55d8ec4923cb6c64e83ae4f32217c72599a398",
"triggerType" : "PUSH"
}, {
"hash" : "87cd79787dc8126b6877ba0d3407cfe8f343933a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8087",
"triggerID" : "87cd79787dc8126b6877ba0d3407cfe8f343933a",
"triggerType" : "PUSH"
}, {
"hash" : "594874ffdca647cd268b3947fa4680c57c53d679",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8088",
"triggerID" : "594874ffdca647cd268b3947fa4680c57c53d679",
"triggerType" : "PUSH"
}, {
"hash" : "7b95754f0df2c1a087a6a28722a7471ee12cecf3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8090",
"triggerID" : "7b95754f0df2c1a087a6a28722a7471ee12cecf3",
"triggerType" : "PUSH"
}, {
"hash" : "789c2da879ca66f6b9be71d1814be38b5d8cfb58",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8102",
"triggerID" : "789c2da879ca66f6b9be71d1814be38b5d8cfb58",
"triggerType" : "PUSH"
}, {
"hash" : "63f33fae3e22a9f80de51b2a6e1ed0427f631d0b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8103",
"triggerID" : "63f33fae3e22a9f80de51b2a6e1ed0427f631d0b",
"triggerType" : "PUSH"
}, {
"hash" : "e04c466fa81e9bcfaabe36825127098f5d1bb890",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8224",
"triggerID" : "e04c466fa81e9bcfaabe36825127098f5d1bb890",
"triggerType" : "PUSH"
}, {
"hash" : "cf574f46a1783a39b2074542aad9539ec3c4f36a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8235",
"triggerID" : "cf574f46a1783a39b2074542aad9539ec3c4f36a",
"triggerType" : "PUSH"
}, {
"hash" : "2eb81e923adfa77176f137c442143deaac9e9916",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2eb81e923adfa77176f137c442143deaac9e9916",
"triggerType" : "PUSH"
}, {
"hash" : "12dc3e3eaf746bb1f2bc3f4e93fa14d9adb0f6fd",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8244",
"triggerID" : "12dc3e3eaf746bb1f2bc3f4e93fa14d9adb0f6fd",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* efd7462406be4f6d90dd5e42f152d66f683ebdf4 UNKNOWN
* 521b26bc2c60b7921e2f7aa5fffa620ab7f9a096 UNKNOWN
* 22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50 UNKNOWN
* 003f1c81efedafa0ddd78868ce83357d8803c5e5 UNKNOWN
* fe55d8ec4923cb6c64e83ae4f32217c72599a398 UNKNOWN
* 2eb81e923adfa77176f137c442143deaac9e9916 UNKNOWN
* 12dc3e3eaf746bb1f2bc3f4e93fa14d9adb0f6fd Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8244)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9715] Custom merger update [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13742:
URL: https://github.com/apache/hudi/pull/13742#issuecomment-3235595046
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7988",
"triggerID" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"triggerType" : "PUSH"
}, {
"hash" : "ecb694295f7a88371f33df6de60370058670439e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7992",
"triggerID" : "ecb694295f7a88371f33df6de60370058670439e",
"triggerType" : "PUSH"
}, {
"hash" : "efd7462406be4f6d90dd5e42f152d66f683ebdf4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "efd7462406be4f6d90dd5e42f152d66f683ebdf4",
"triggerType" : "PUSH"
}, {
"hash" : "3c67160d5ceec883db5d826ff655865a3209f2f3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7994",
"triggerID" : "3c67160d5ceec883db5d826ff655865a3209f2f3",
"triggerType" : "PUSH"
}, {
"hash" : "f4f84d9f6a480923df27cfe5553d8aa65b2bbb7a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7996",
"triggerID" : "f4f84d9f6a480923df27cfe5553d8aa65b2bbb7a",
"triggerType" : "PUSH"
}, {
"hash" : "521b26bc2c60b7921e2f7aa5fffa620ab7f9a096",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "521b26bc2c60b7921e2f7aa5fffa620ab7f9a096",
"triggerType" : "PUSH"
}, {
"hash" : "22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50",
"triggerType" : "PUSH"
}, {
"hash" : "02a319ba884595a4e3b5453f366b660c1c11f767",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8007",
"triggerID" : "02a319ba884595a4e3b5453f366b660c1c11f767",
"triggerType" : "PUSH"
}, {
"hash" : "3e59b6f8b486016911c9ad14c583f6538cc036ac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8009",
"triggerID" : "3e59b6f8b486016911c9ad14c583f6538cc036ac",
"triggerType" : "PUSH"
}, {
"hash" : "003f1c81efedafa0ddd78868ce83357d8803c5e5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "003f1c81efedafa0ddd78868ce83357d8803c5e5",
"triggerType" : "PUSH"
}, {
"hash" : "40c04768d5233a38be1b4c01ce895189a4af4c3a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8013",
"triggerID" : "40c04768d5233a38be1b4c01ce895189a4af4c3a",
"triggerType" : "PUSH"
}, {
"hash" : "4a699a8dcf79e61ea3bb754e67e9cc4a32e86aa1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8084",
"triggerID" : "4a699a8dcf79e61ea3bb754e67e9cc4a32e86aa1",
"triggerType" : "PUSH"
}, {
"hash" : "fe55d8ec4923cb6c64e83ae4f32217c72599a398",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "fe55d8ec4923cb6c64e83ae4f32217c72599a398",
"triggerType" : "PUSH"
}, {
"hash" : "87cd79787dc8126b6877ba0d3407cfe8f343933a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8087",
"triggerID" : "87cd79787dc8126b6877ba0d3407cfe8f343933a",
"triggerType" : "PUSH"
}, {
"hash" : "594874ffdca647cd268b3947fa4680c57c53d679",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8088",
"triggerID" : "594874ffdca647cd268b3947fa4680c57c53d679",
"triggerType" : "PUSH"
}, {
"hash" : "7b95754f0df2c1a087a6a28722a7471ee12cecf3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8090",
"triggerID" : "7b95754f0df2c1a087a6a28722a7471ee12cecf3",
"triggerType" : "PUSH"
}, {
"hash" : "789c2da879ca66f6b9be71d1814be38b5d8cfb58",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8102",
"triggerID" : "789c2da879ca66f6b9be71d1814be38b5d8cfb58",
"triggerType" : "PUSH"
}, {
"hash" : "63f33fae3e22a9f80de51b2a6e1ed0427f631d0b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8103",
"triggerID" : "63f33fae3e22a9f80de51b2a6e1ed0427f631d0b",
"triggerType" : "PUSH"
}, {
"hash" : "e04c466fa81e9bcfaabe36825127098f5d1bb890",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8224",
"triggerID" : "e04c466fa81e9bcfaabe36825127098f5d1bb890",
"triggerType" : "PUSH"
}, {
"hash" : "cf574f46a1783a39b2074542aad9539ec3c4f36a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8235",
"triggerID" : "cf574f46a1783a39b2074542aad9539ec3c4f36a",
"triggerType" : "PUSH"
}, {
"hash" : "2eb81e923adfa77176f137c442143deaac9e9916",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2eb81e923adfa77176f137c442143deaac9e9916",
"triggerType" : "PUSH"
}, {
"hash" : "12dc3e3eaf746bb1f2bc3f4e93fa14d9adb0f6fd",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8244",
"triggerID" : "12dc3e3eaf746bb1f2bc3f4e93fa14d9adb0f6fd",
"triggerType" : "PUSH"
}, {
"hash" : "af5ab73103f3af882412bbe99c9322165e8941a0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "af5ab73103f3af882412bbe99c9322165e8941a0",
"triggerType" : "PUSH"
}, {
"hash" : "a9c28c4d913644fea23d57a5dc2224f36fd2b63e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a9c28c4d913644fea23d57a5dc2224f36fd2b63e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* efd7462406be4f6d90dd5e42f152d66f683ebdf4 UNKNOWN
* 521b26bc2c60b7921e2f7aa5fffa620ab7f9a096 UNKNOWN
* 22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50 UNKNOWN
* 003f1c81efedafa0ddd78868ce83357d8803c5e5 UNKNOWN
* fe55d8ec4923cb6c64e83ae4f32217c72599a398 UNKNOWN
* 2eb81e923adfa77176f137c442143deaac9e9916 UNKNOWN
* 12dc3e3eaf746bb1f2bc3f4e93fa14d9adb0f6fd Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8244)
* af5ab73103f3af882412bbe99c9322165e8941a0 UNKNOWN
* a9c28c4d913644fea23d57a5dc2224f36fd2b63e UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT] Custom merger update [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13742:
URL: https://github.com/apache/hudi/pull/13742#issuecomment-3211024112
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7988",
"triggerID" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"triggerType" : "PUSH"
}, {
"hash" : "ecb694295f7a88371f33df6de60370058670439e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7992",
"triggerID" : "ecb694295f7a88371f33df6de60370058670439e",
"triggerType" : "PUSH"
}, {
"hash" : "efd7462406be4f6d90dd5e42f152d66f683ebdf4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "efd7462406be4f6d90dd5e42f152d66f683ebdf4",
"triggerType" : "PUSH"
}, {
"hash" : "3c67160d5ceec883db5d826ff655865a3209f2f3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7994",
"triggerID" : "3c67160d5ceec883db5d826ff655865a3209f2f3",
"triggerType" : "PUSH"
}, {
"hash" : "f4f84d9f6a480923df27cfe5553d8aa65b2bbb7a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7996",
"triggerID" : "f4f84d9f6a480923df27cfe5553d8aa65b2bbb7a",
"triggerType" : "PUSH"
}, {
"hash" : "521b26bc2c60b7921e2f7aa5fffa620ab7f9a096",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "521b26bc2c60b7921e2f7aa5fffa620ab7f9a096",
"triggerType" : "PUSH"
}, {
"hash" : "22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50",
"triggerType" : "PUSH"
}, {
"hash" : "02a319ba884595a4e3b5453f366b660c1c11f767",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8007",
"triggerID" : "02a319ba884595a4e3b5453f366b660c1c11f767",
"triggerType" : "PUSH"
}, {
"hash" : "3e59b6f8b486016911c9ad14c583f6538cc036ac",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3e59b6f8b486016911c9ad14c583f6538cc036ac",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* efd7462406be4f6d90dd5e42f152d66f683ebdf4 UNKNOWN
* f4f84d9f6a480923df27cfe5553d8aa65b2bbb7a Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7996)
* 521b26bc2c60b7921e2f7aa5fffa620ab7f9a096 UNKNOWN
* 22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50 UNKNOWN
* 02a319ba884595a4e3b5453f366b660c1c11f767 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8007)
* 3e59b6f8b486016911c9ad14c583f6538cc036ac UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT] Custom merger update [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13742:
URL: https://github.com/apache/hudi/pull/13742#issuecomment-3211423146
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7988",
"triggerID" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"triggerType" : "PUSH"
}, {
"hash" : "ecb694295f7a88371f33df6de60370058670439e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7992",
"triggerID" : "ecb694295f7a88371f33df6de60370058670439e",
"triggerType" : "PUSH"
}, {
"hash" : "efd7462406be4f6d90dd5e42f152d66f683ebdf4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "efd7462406be4f6d90dd5e42f152d66f683ebdf4",
"triggerType" : "PUSH"
}, {
"hash" : "3c67160d5ceec883db5d826ff655865a3209f2f3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7994",
"triggerID" : "3c67160d5ceec883db5d826ff655865a3209f2f3",
"triggerType" : "PUSH"
}, {
"hash" : "f4f84d9f6a480923df27cfe5553d8aa65b2bbb7a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7996",
"triggerID" : "f4f84d9f6a480923df27cfe5553d8aa65b2bbb7a",
"triggerType" : "PUSH"
}, {
"hash" : "521b26bc2c60b7921e2f7aa5fffa620ab7f9a096",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "521b26bc2c60b7921e2f7aa5fffa620ab7f9a096",
"triggerType" : "PUSH"
}, {
"hash" : "22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50",
"triggerType" : "PUSH"
}, {
"hash" : "02a319ba884595a4e3b5453f366b660c1c11f767",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8007",
"triggerID" : "02a319ba884595a4e3b5453f366b660c1c11f767",
"triggerType" : "PUSH"
}, {
"hash" : "3e59b6f8b486016911c9ad14c583f6538cc036ac",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8009",
"triggerID" : "3e59b6f8b486016911c9ad14c583f6538cc036ac",
"triggerType" : "PUSH"
}, {
"hash" : "003f1c81efedafa0ddd78868ce83357d8803c5e5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "003f1c81efedafa0ddd78868ce83357d8803c5e5",
"triggerType" : "PUSH"
}, {
"hash" : "40c04768d5233a38be1b4c01ce895189a4af4c3a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8013",
"triggerID" : "40c04768d5233a38be1b4c01ce895189a4af4c3a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* efd7462406be4f6d90dd5e42f152d66f683ebdf4 UNKNOWN
* 521b26bc2c60b7921e2f7aa5fffa620ab7f9a096 UNKNOWN
* 22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50 UNKNOWN
* 3e59b6f8b486016911c9ad14c583f6538cc036ac Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8009)
* 003f1c81efedafa0ddd78868ce83357d8803c5e5 UNKNOWN
* 40c04768d5233a38be1b4c01ce895189a4af4c3a Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8013)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT] Custom merger update [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13742:
URL: https://github.com/apache/hudi/pull/13742#issuecomment-3208770548
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7988",
"triggerID" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"triggerType" : "PUSH"
}, {
"hash" : "ecb694295f7a88371f33df6de60370058670439e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7992",
"triggerID" : "ecb694295f7a88371f33df6de60370058670439e",
"triggerType" : "PUSH"
}, {
"hash" : "efd7462406be4f6d90dd5e42f152d66f683ebdf4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "efd7462406be4f6d90dd5e42f152d66f683ebdf4",
"triggerType" : "PUSH"
}, {
"hash" : "3c67160d5ceec883db5d826ff655865a3209f2f3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7994",
"triggerID" : "3c67160d5ceec883db5d826ff655865a3209f2f3",
"triggerType" : "PUSH"
}, {
"hash" : "f4f84d9f6a480923df27cfe5553d8aa65b2bbb7a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f4f84d9f6a480923df27cfe5553d8aa65b2bbb7a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* efd7462406be4f6d90dd5e42f152d66f683ebdf4 UNKNOWN
* 3c67160d5ceec883db5d826ff655865a3209f2f3 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7994)
* f4f84d9f6a480923df27cfe5553d8aa65b2bbb7a UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9715] Custom merger update [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13742:
URL: https://github.com/apache/hudi/pull/13742#discussion_r2299739927
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/HoodieSparkRecordMerger.java:
##########
@@ -39,33 +41,33 @@ public HoodieRecord.HoodieRecordType getRecordType() {
* Basic handling of deletes that is used by many of the spark mergers
* returns null if merger specific logic should be used
*/
- protected Option<Pair<HoodieRecord, Schema>> handleDeletes(HoodieRecord older, Schema oldSchema, HoodieRecord newer, Schema newSchema, TypedProperties props) {
+ protected Pair<HoodieRecord, Schema> handleDeletes(HoodieRecord older, Schema oldSchema, HoodieRecord newer, Schema newSchema, TypedProperties props) {
ValidationUtils.checkArgument(older.getRecordType() == HoodieRecord.HoodieRecordType.SPARK);
ValidationUtils.checkArgument(newer.getRecordType() == HoodieRecord.HoodieRecordType.SPARK);
if (newer instanceof HoodieSparkRecord) {
HoodieSparkRecord newSparkRecord = (HoodieSparkRecord) newer;
if (newSparkRecord.isDelete(newSchema, props)) {
// Delete record
- return Option.empty();
+ return Pair.of(new HoodieEmptyRecord<>(newer.getKey(), HoodieOperation.DELETE, OrderingValues.getDefault(), HoodieRecord.HoodieRecordType.SPARK), newSchema);
Review Comment:
need to investigate why not just compare the ordering val for older/newer deletes here? Does it mean spark does not support disorder deletes as of now?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9715] Custom merger update [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13742:
URL: https://github.com/apache/hudi/pull/13742#discussion_r2302303611
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/BufferedRecordMergerFactory.java:
##########
@@ -465,46 +451,45 @@ public CustomPayloadRecordMerger(
}
@Override
- public Option<BufferedRecord<T>> deltaMergeNonDeleteRecord(BufferedRecord<T> newRecord, BufferedRecord<T> existingRecord) throws IOException {
- Option<Pair<HoodieRecord, Schema>> mergedRecordAndSchema = getMergedRecord(existingRecord, newRecord, false);
- if (mergedRecordAndSchema.isEmpty()) {
- // An empty Option indicates that the output represents a delete.
- return Option.of(new BufferedRecord<>(newRecord.getRecordKey(), OrderingValues.getDefault(), null, null, HoodieOperation.DELETE));
- }
- HoodieRecord mergedRecord = mergedRecordAndSchema.get().getLeft();
- Schema mergeResultSchema = mergedRecordAndSchema.get().getRight();
+ public Option<BufferedRecord<T>> deltaMergeRecords(BufferedRecord<T> newRecord, BufferedRecord<T> existingRecord) throws IOException {
+ Pair<HoodieRecord, Schema> mergedRecordAndSchema = getMergedRecord(existingRecord, newRecord, false);
+ HoodieRecord mergedRecord = mergedRecordAndSchema.getLeft();
+ Schema mergeResultSchema = mergedRecordAndSchema.getRight();
// Special handling for SENTINEL record in Expression Payload. This is returned if the condition does not match.
if (mergedRecord.getData() == HoodieRecord.SENTINEL) {
return Option.empty();
}
- T combinedRecordData = recordContext.convertAvroRecord(mergedRecord.toIndexedRecord(mergeResultSchema, props).get().getData());
+ Option<T> combinedRecordData = (mergedRecord.toIndexedRecord(mergeResultSchema, props).map(indexedRecord -> recordContext.convertAvroRecord(indexedRecord.getData())));
// If pre-combine does not return existing record, update it
- if (combinedRecordData != existingRecord.getRecord()) {
+ if (combinedRecordData.map(record -> record != existingRecord.getRecord()).orElse(true)) {
// For pkless we need to use record key from existing record
- return Option.of(BufferedRecords.fromEngineRecord(combinedRecordData, mergeResultSchema, recordContext, orderingFieldNames,
- existingRecord.getRecordKey(), mergedRecord.isDelete(mergeResultSchema, props)));
+ boolean isDelete = mergedRecord.isDelete(readerSchema, props);
+ Comparable orderingValue = mergedRecord.getOrderingValue(mergeResultSchema, props, orderingFieldNames);
+ T mergedEngineRecord = mergedRecord.toIndexedRecord(mergeResultSchema, props)
+ .map(hoodieAvroIndexedRecord -> recordContext.convertAvroRecord(hoodieAvroIndexedRecord.getData()))
+ .orElse(null);
+ return Option.of(BufferedRecords.fromEngineRecord(mergedEngineRecord, readerSchema, recordContext, orderingValue, existingRecord.getRecordKey(), isDelete));
+
}
return Option.empty();
}
@Override
- public BufferedRecord<T> mergeNonDeleteRecord(BufferedRecord<T> olderRecord, BufferedRecord<T> newerRecord) throws IOException {
- Option<Pair<HoodieRecord, Schema>> mergedRecordAndSchema = getMergedRecord(olderRecord, newerRecord, true);
- if (mergedRecordAndSchema.isEmpty()) {
- return BufferedRecords.createDelete(newerRecord.getRecordKey());
- }
- HoodieRecord mergedRecord = mergedRecordAndSchema.get().getLeft();
- Schema mergeResultSchema = mergedRecordAndSchema.get().getRight();
+ public BufferedRecord<T> mergeRecords(BufferedRecord<T> olderRecord, BufferedRecord<T> newerRecord) throws IOException {
+ Pair<HoodieRecord, Schema> mergedRecordAndSchema = getMergedRecord(olderRecord, newerRecord, true);
+
+ HoodieRecord mergedRecord = mergedRecordAndSchema.getLeft();
+ Schema mergeResultSchema = mergedRecordAndSchema.getRight();
// Special handling for SENTINEL record in Expression Payload
if (mergedRecord.getData() == HoodieRecord.SENTINEL) {
return olderRecord;
}
- if (!mergedRecord.isDelete(mergeResultSchema, props)) {
- IndexedRecord indexedRecord = (IndexedRecord) mergedRecord.getData();
- return BufferedRecords.fromEngineRecord(
- recordContext.convertAvroRecord(indexedRecord), mergeResultSchema, recordContext, orderingFieldNames, newerRecord.getRecordKey(), false);
- }
- return BufferedRecords.createDelete(newerRecord.getRecordKey());
+ boolean isDelete = mergedRecord.isDelete(readerSchema, props);
+ Comparable orderingValue = mergedRecord.getOrderingValue(mergeResultSchema, props, orderingFieldNames);
+ T mergedEngineRecord = mergedRecord.toIndexedRecord(mergeResultSchema, props)
Review Comment:
If we update the merger API to work with BufferedRecord, this will also be simplified so that is another pro for making that change
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9715] Custom merger update [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13742:
URL: https://github.com/apache/hudi/pull/13742#discussion_r2302445172
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieWriteMergeHandle.java:
##########
@@ -236,19 +236,19 @@ public boolean isEmptyNewRecords() {
return keyToNewRecords.isEmpty();
}
- protected boolean writeUpdateRecord(HoodieRecord<T> newRecord, HoodieRecord<T> oldRecord, Option<HoodieRecord> combineRecordOpt, Schema writerSchema) throws IOException {
+ protected boolean writeUpdateRecord(HoodieRecord<T> newRecord, HoodieRecord<T> oldRecord, HoodieRecord combineRecord, Schema writerSchema) throws IOException {
boolean isDelete = false;
- if (combineRecordOpt.isPresent()) {
- if (oldRecord.getData() != combineRecordOpt.get().getData()) {
- // the incoming record is chosen
- isDelete = isDeleteRecord(newRecord);
- } else {
- // the incoming record is dropped
- return false;
+ if (oldRecord.getData() != combineRecord.getData()) {
+ // the incoming record is chosen
+ isDelete = isDeleteRecord(combineRecord);
+ if (!isDelete) {
+ updatedRecordsWritten++;
}
- updatedRecordsWritten++;
+ } else {
+ // the incoming record is dropped
Review Comment:
This is just copying the old functionality, I don't have the context on the original design
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9715] Custom merger update [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13742:
URL: https://github.com/apache/hudi/pull/13742#discussion_r2302521697
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/BufferedRecordMergerFactory.java:
##########
@@ -567,42 +553,36 @@ public Option<BufferedRecord<T>> deltaMerge(BufferedRecord<T> newRecord, Buffere
if (existingRecord == null) {
return Option.of(newRecord);
}
- if (existingRecord.isDelete() || newRecord.isDelete()) {
- if (shouldKeepNewerRecord(existingRecord, newRecord)) {
- // IMPORTANT:
- // this is needed when the fallback HoodieAvroRecordMerger got used, the merger would
- // return Option.empty when the old payload data is empty(a delete) and ignores its ordering value directly.
- return Option.of(newRecord);
- } else {
- return Option.empty();
- }
- }
- return deltaMergeNonDeleteRecord(newRecord, existingRecord);
+ return deltaMergeRecords(newRecord, existingRecord);
}
- public abstract Option<BufferedRecord<T>> deltaMergeNonDeleteRecord(BufferedRecord<T> newRecord, BufferedRecord<T> existingRecord) throws IOException;
+ public abstract Option<BufferedRecord<T>> deltaMergeRecords(BufferedRecord<T> newRecord, BufferedRecord<T> existingRecord) throws IOException;
@Override
public Option<DeleteRecord> deltaMerge(DeleteRecord deleteRecord, BufferedRecord<T> existingRecord) {
- return deltaMergeDeleteRecord(deleteRecord, existingRecord);
+ BufferedRecord<T> deleteBufferedRecord = BufferedRecords.fromDeleteRecord(deleteRecord, recordContext);
+ try {
+ Option<BufferedRecord<T>> merged = deltaMerge(deleteBufferedRecord, existingRecord);
+ // If the delete record is chosen, return an option with the delete record, otherwise return empty.
+ return merged.isPresent() ? Option.of(deleteRecord) : Option.empty();
+ } catch (IOException e) {
+ throw new HoodieIOException("Failed to process delete record", e);
+ }
}
@Override
public BufferedRecord<T> finalMerge(BufferedRecord<T> olderRecord, BufferedRecord<T> newerRecord) throws IOException {
- if (olderRecord.isDelete() || newerRecord.isDelete()) {
- if (shouldKeepNewerRecord(olderRecord, newerRecord)) {
- // IMPORTANT:
- // this is needed when the fallback HoodieAvroRecordMerger got used, the merger would
- // return Option.empty when the new payload data is empty(a delete) and ignores its ordering value directly.
- return newerRecord;
- } else {
- return olderRecord;
- }
+ if (olderRecord == null) {
+ return newerRecord;
+ }
+ // handle special case for deletes that are sent to older partitions in global-index, this delete takes precedence regardless of the previous value
+ if (newerRecord.getHoodieOperation() == HoodieOperation.UPDATE_BEFORE) {
Review Comment:
Right now the global index with Merge-Into tests will fail without this. It will require some more investigation into why that is the case
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9715] Custom merger update [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13742:
URL: https://github.com/apache/hudi/pull/13742#issuecomment-3236645845
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7988",
"triggerID" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"triggerType" : "PUSH"
}, {
"hash" : "ecb694295f7a88371f33df6de60370058670439e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7992",
"triggerID" : "ecb694295f7a88371f33df6de60370058670439e",
"triggerType" : "PUSH"
}, {
"hash" : "efd7462406be4f6d90dd5e42f152d66f683ebdf4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "efd7462406be4f6d90dd5e42f152d66f683ebdf4",
"triggerType" : "PUSH"
}, {
"hash" : "3c67160d5ceec883db5d826ff655865a3209f2f3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7994",
"triggerID" : "3c67160d5ceec883db5d826ff655865a3209f2f3",
"triggerType" : "PUSH"
}, {
"hash" : "f4f84d9f6a480923df27cfe5553d8aa65b2bbb7a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7996",
"triggerID" : "f4f84d9f6a480923df27cfe5553d8aa65b2bbb7a",
"triggerType" : "PUSH"
}, {
"hash" : "521b26bc2c60b7921e2f7aa5fffa620ab7f9a096",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "521b26bc2c60b7921e2f7aa5fffa620ab7f9a096",
"triggerType" : "PUSH"
}, {
"hash" : "22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50",
"triggerType" : "PUSH"
}, {
"hash" : "02a319ba884595a4e3b5453f366b660c1c11f767",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8007",
"triggerID" : "02a319ba884595a4e3b5453f366b660c1c11f767",
"triggerType" : "PUSH"
}, {
"hash" : "3e59b6f8b486016911c9ad14c583f6538cc036ac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8009",
"triggerID" : "3e59b6f8b486016911c9ad14c583f6538cc036ac",
"triggerType" : "PUSH"
}, {
"hash" : "003f1c81efedafa0ddd78868ce83357d8803c5e5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "003f1c81efedafa0ddd78868ce83357d8803c5e5",
"triggerType" : "PUSH"
}, {
"hash" : "40c04768d5233a38be1b4c01ce895189a4af4c3a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8013",
"triggerID" : "40c04768d5233a38be1b4c01ce895189a4af4c3a",
"triggerType" : "PUSH"
}, {
"hash" : "4a699a8dcf79e61ea3bb754e67e9cc4a32e86aa1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8084",
"triggerID" : "4a699a8dcf79e61ea3bb754e67e9cc4a32e86aa1",
"triggerType" : "PUSH"
}, {
"hash" : "fe55d8ec4923cb6c64e83ae4f32217c72599a398",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "fe55d8ec4923cb6c64e83ae4f32217c72599a398",
"triggerType" : "PUSH"
}, {
"hash" : "87cd79787dc8126b6877ba0d3407cfe8f343933a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8087",
"triggerID" : "87cd79787dc8126b6877ba0d3407cfe8f343933a",
"triggerType" : "PUSH"
}, {
"hash" : "594874ffdca647cd268b3947fa4680c57c53d679",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8088",
"triggerID" : "594874ffdca647cd268b3947fa4680c57c53d679",
"triggerType" : "PUSH"
}, {
"hash" : "7b95754f0df2c1a087a6a28722a7471ee12cecf3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8090",
"triggerID" : "7b95754f0df2c1a087a6a28722a7471ee12cecf3",
"triggerType" : "PUSH"
}, {
"hash" : "789c2da879ca66f6b9be71d1814be38b5d8cfb58",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8102",
"triggerID" : "789c2da879ca66f6b9be71d1814be38b5d8cfb58",
"triggerType" : "PUSH"
}, {
"hash" : "63f33fae3e22a9f80de51b2a6e1ed0427f631d0b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8103",
"triggerID" : "63f33fae3e22a9f80de51b2a6e1ed0427f631d0b",
"triggerType" : "PUSH"
}, {
"hash" : "e04c466fa81e9bcfaabe36825127098f5d1bb890",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8224",
"triggerID" : "e04c466fa81e9bcfaabe36825127098f5d1bb890",
"triggerType" : "PUSH"
}, {
"hash" : "cf574f46a1783a39b2074542aad9539ec3c4f36a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8235",
"triggerID" : "cf574f46a1783a39b2074542aad9539ec3c4f36a",
"triggerType" : "PUSH"
}, {
"hash" : "2eb81e923adfa77176f137c442143deaac9e9916",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2eb81e923adfa77176f137c442143deaac9e9916",
"triggerType" : "PUSH"
}, {
"hash" : "12dc3e3eaf746bb1f2bc3f4e93fa14d9adb0f6fd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8244",
"triggerID" : "12dc3e3eaf746bb1f2bc3f4e93fa14d9adb0f6fd",
"triggerType" : "PUSH"
}, {
"hash" : "af5ab73103f3af882412bbe99c9322165e8941a0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "af5ab73103f3af882412bbe99c9322165e8941a0",
"triggerType" : "PUSH"
}, {
"hash" : "a9c28c4d913644fea23d57a5dc2224f36fd2b63e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8248",
"triggerID" : "a9c28c4d913644fea23d57a5dc2224f36fd2b63e",
"triggerType" : "PUSH"
}, {
"hash" : "98381a6bd2bcccf3da08fb992e599fe1165f2ddb",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8252",
"triggerID" : "98381a6bd2bcccf3da08fb992e599fe1165f2ddb",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* efd7462406be4f6d90dd5e42f152d66f683ebdf4 UNKNOWN
* 521b26bc2c60b7921e2f7aa5fffa620ab7f9a096 UNKNOWN
* 22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50 UNKNOWN
* 003f1c81efedafa0ddd78868ce83357d8803c5e5 UNKNOWN
* fe55d8ec4923cb6c64e83ae4f32217c72599a398 UNKNOWN
* 2eb81e923adfa77176f137c442143deaac9e9916 UNKNOWN
* af5ab73103f3af882412bbe99c9322165e8941a0 UNKNOWN
* a9c28c4d913644fea23d57a5dc2224f36fd2b63e Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8248)
* 98381a6bd2bcccf3da08fb992e599fe1165f2ddb Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8252)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9715] Custom merger update [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13742:
URL: https://github.com/apache/hudi/pull/13742#discussion_r2303855644
##########
hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/DefaultHiveRecordMerger.java:
##########
@@ -39,33 +41,33 @@ public class DefaultHiveRecordMerger extends HoodieHiveRecordMerger {
private String[] orderingFields;
@Override
- public Option<Pair<HoodieRecord, Schema>> merge(HoodieRecord older, Schema oldSchema, HoodieRecord newer, Schema newSchema, TypedProperties props) throws IOException {
+ public Pair<HoodieRecord, Schema> merge(HoodieRecord older, Schema oldSchema, HoodieRecord newer, Schema newSchema, TypedProperties props) throws IOException {
ValidationUtils.checkArgument(older.getRecordType() == HoodieRecord.HoodieRecordType.HIVE);
ValidationUtils.checkArgument(newer.getRecordType() == HoodieRecord.HoodieRecordType.HIVE);
if (newer instanceof HoodieHiveRecord) {
HoodieHiveRecord newHiveRecord = (HoodieHiveRecord) newer;
if (newHiveRecord.isDelete(newSchema, props)) {
- return Option.empty();
+ return Pair.of(new HoodieEmptyRecord<>(newer.getKey(), HoodieOperation.DELETE, OrderingValues.getDefault(), HoodieRecord.HoodieRecordType.HIVE), newSchema);
Review Comment:
Previously this would return an empty option so it does not have any ability to track the ordering value after this based on my understanding. What am I missing @danny0405?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9715] Custom merger update [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13742:
URL: https://github.com/apache/hudi/pull/13742#discussion_r2302685967
##########
hudi-common/src/main/java/org/apache/hudi/common/model/HoodieRecordMerger.java:
##########
@@ -63,8 +63,15 @@ public interface HoodieRecordMerger extends Serializable {
* It'd be associative operation: f(a, f(b, c)) = f(f(a, b), c) (which we can translate as having 3 versions A, B, C
* of the single record, both orders of operations applications have to yield the same result)
* This method takes only full records for merging.
+ *
+ * @param older Older record in terms of commit time ordering.
+ * @param oldSchema The schema of the older record.
+ * @param newer Newer record in terms of commit time ordering.
+ * @param newSchema The schema of the newer record.
+ * @param props The additional properties for the merging operation.
+ * @return The merged record and schema. The record is expected to be non-null. If the record represents a deletion, the operation must be set as {@link HoodieOperation#DELETE}.
*/
- Option<Pair<HoodieRecord, Schema>> merge(HoodieRecord older, Schema oldSchema, HoodieRecord newer, Schema newSchema, TypedProperties props) throws IOException;
+ Pair<HoodieRecord, Schema> merge(HoodieRecord older, Schema oldSchema, HoodieRecord newer, Schema newSchema, TypedProperties props) throws IOException;
Review Comment:
> there is no way to currently pass in the payload information to the HoodieAvroRecordMerger
We can, just let the T in BufferedRecord be the payload instance?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9715] Custom merger update [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13742:
URL: https://github.com/apache/hudi/pull/13742#discussion_r2299957785
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/BufferedRecordMergerFactory.java:
##########
@@ -465,46 +451,45 @@ public CustomPayloadRecordMerger(
}
@Override
- public Option<BufferedRecord<T>> deltaMergeNonDeleteRecord(BufferedRecord<T> newRecord, BufferedRecord<T> existingRecord) throws IOException {
- Option<Pair<HoodieRecord, Schema>> mergedRecordAndSchema = getMergedRecord(existingRecord, newRecord, false);
- if (mergedRecordAndSchema.isEmpty()) {
- // An empty Option indicates that the output represents a delete.
- return Option.of(new BufferedRecord<>(newRecord.getRecordKey(), OrderingValues.getDefault(), null, null, HoodieOperation.DELETE));
- }
- HoodieRecord mergedRecord = mergedRecordAndSchema.get().getLeft();
- Schema mergeResultSchema = mergedRecordAndSchema.get().getRight();
+ public Option<BufferedRecord<T>> deltaMergeRecords(BufferedRecord<T> newRecord, BufferedRecord<T> existingRecord) throws IOException {
+ Pair<HoodieRecord, Schema> mergedRecordAndSchema = getMergedRecord(existingRecord, newRecord, false);
+ HoodieRecord mergedRecord = mergedRecordAndSchema.getLeft();
+ Schema mergeResultSchema = mergedRecordAndSchema.getRight();
// Special handling for SENTINEL record in Expression Payload. This is returned if the condition does not match.
if (mergedRecord.getData() == HoodieRecord.SENTINEL) {
return Option.empty();
}
- T combinedRecordData = recordContext.convertAvroRecord(mergedRecord.toIndexedRecord(mergeResultSchema, props).get().getData());
+ Option<T> combinedRecordData = (mergedRecord.toIndexedRecord(mergeResultSchema, props).map(indexedRecord -> recordContext.convertAvroRecord(indexedRecord.getData())));
// If pre-combine does not return existing record, update it
- if (combinedRecordData != existingRecord.getRecord()) {
+ if (combinedRecordData.map(record -> record != existingRecord.getRecord()).orElse(true)) {
// For pkless we need to use record key from existing record
- return Option.of(BufferedRecords.fromEngineRecord(combinedRecordData, mergeResultSchema, recordContext, orderingFieldNames,
- existingRecord.getRecordKey(), mergedRecord.isDelete(mergeResultSchema, props)));
+ boolean isDelete = mergedRecord.isDelete(readerSchema, props);
+ Comparable orderingValue = mergedRecord.getOrderingValue(mergeResultSchema, props, orderingFieldNames);
+ T mergedEngineRecord = mergedRecord.toIndexedRecord(mergeResultSchema, props)
+ .map(hoodieAvroIndexedRecord -> recordContext.convertAvroRecord(hoodieAvroIndexedRecord.getData()))
+ .orElse(null);
+ return Option.of(BufferedRecords.fromEngineRecord(mergedEngineRecord, readerSchema, recordContext, orderingValue, existingRecord.getRecordKey(), isDelete));
Review Comment:
readerSchema or mergeResultSchema?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9715] Custom merger update [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13742:
URL: https://github.com/apache/hudi/pull/13742#discussion_r2299933233
##########
hudi-common/src/main/java/org/apache/hudi/common/model/HoodieAvroRecordMerger.java:
##########
@@ -43,9 +44,35 @@ public String getMergingStrategy() {
}
@Override
- public Option<Pair<HoodieRecord, Schema>> merge(HoodieRecord older, Schema oldSchema, HoodieRecord newer, Schema newSchema, TypedProperties props) throws IOException {
- return combineAndGetUpdateValue(older, newer, oldSchema, newSchema, props)
- .map(r -> Pair.of(new HoodieAvroIndexedRecord(r), r.getSchema()));
+ public Pair<HoodieRecord, Schema> merge(HoodieRecord older, Schema oldSchema, HoodieRecord newer, Schema newSchema, TypedProperties props) throws IOException {
+ Option<IndexedRecord> previousAvroData = older.toIndexedRecord(oldSchema, props).map(HoodieAvroIndexedRecord::getData);
+ HoodieRecordPayload payload = ((HoodieAvroRecord) newer).getData();
+ Option<IndexedRecord> updatedValue;
+ Comparable orderingVal;
+ if (previousAvroData.isEmpty()) {
+ updatedValue = payload.getInsertValue(newSchema, props);
+ orderingVal = payload.getOrderingValue();
+ } else {
+ updatedValue = ((HoodieAvroRecord) newer).getData().combineAndGetUpdateValue(previousAvroData.get(), newSchema, props);
+ if (updatedValue.map(value -> value == previousAvroData.get()).orElse(false)) {
+ // Ordering value is set during record creation if one is required for this payload, so ordering fields are not required here
+ return Pair.of(older, oldSchema);
+ } else {
+ orderingVal = payload.getOrderingValue();
+ }
+ }
+ if (updatedValue.isEmpty()) {
+ // pass through the record that results in the delete operation. If both newer and older are deletes, we return the newer one.
+ if (newer.isDelete(newSchema, props)) {
+ return Pair.of(newer.newInstance(newer.getKey(), HoodieOperation.DELETE), newSchema);
+ } else if (older.isDelete(oldSchema, props)) {
+ // if older is delete, return it
+ return Pair.of(older.newInstance(newer.getKey(), HoodieOperation.DELETE), oldSchema);
+ }
+ // If neither of the records are a delete, return a delete record with an empty value and default ordering value.
+ return Pair.of(new HoodieEmptyRecord<>(newer.getKey(), HoodieOperation.DELETE, OrderingValues.getDefault(), HoodieRecordType.AVRO), newSchema);
+ }
+ return Pair.of(new HoodieAvroIndexedRecord(newer.getKey(), updatedValue.get(), orderingVal), updatedValue.get().getSchema());
Review Comment:
not sure if we need to keep the hoodie operation too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9715] Custom merger update [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #13742:
URL: https://github.com/apache/hudi/pull/13742#discussion_r2301970989
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/common/model/HoodieSparkRecord.java:
##########
@@ -257,7 +258,11 @@ public HoodieRecord truncateRecordKey(Schema recordSchema, Properties props, Str
@Override
protected boolean checkIsDelete(Schema recordSchema, Properties props) {
- if (null == data) {
+ if (data == null) {
+ return true;
+ }
+
+ if (HoodieOperation.isDelete(getOperation())) {
return true;
}
Review Comment:
Should we just combine into one `if` statement?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9715] Custom merger update [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13742:
URL: https://github.com/apache/hudi/pull/13742#discussion_r2302480201
##########
hudi-common/src/main/java/org/apache/hudi/common/model/HoodieRecordMerger.java:
##########
@@ -63,8 +63,15 @@ public interface HoodieRecordMerger extends Serializable {
* It'd be associative operation: f(a, f(b, c)) = f(f(a, b), c) (which we can translate as having 3 versions A, B, C
* of the single record, both orders of operations applications have to yield the same result)
* This method takes only full records for merging.
+ *
+ * @param older Older record in terms of commit time ordering.
+ * @param oldSchema The schema of the older record.
+ * @param newer Newer record in terms of commit time ordering.
+ * @param newSchema The schema of the newer record.
+ * @param props The additional properties for the merging operation.
+ * @return The merged record and schema. The record is expected to be non-null. If the record represents a deletion, the operation must be set as {@link HoodieOperation#DELETE}.
*/
- Option<Pair<HoodieRecord, Schema>> merge(HoodieRecord older, Schema oldSchema, HoodieRecord newer, Schema newSchema, TypedProperties props) throws IOException;
+ Pair<HoodieRecord, Schema> merge(HoodieRecord older, Schema oldSchema, HoodieRecord newer, Schema newSchema, TypedProperties props) throws IOException;
Review Comment:
There may be some practical limitations that will prevent this from happening though. For example, there is no way to currently pass in the payload information to the `HoodieAvroRecordMerger` so we would not know how to construct the payloads properly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9715] Custom merger update [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13742:
URL: https://github.com/apache/hudi/pull/13742#discussion_r2302481535
##########
hudi-common/src/main/java/org/apache/hudi/common/model/HoodieAvroRecordMerger.java:
##########
@@ -43,9 +44,35 @@ public String getMergingStrategy() {
}
@Override
- public Option<Pair<HoodieRecord, Schema>> merge(HoodieRecord older, Schema oldSchema, HoodieRecord newer, Schema newSchema, TypedProperties props) throws IOException {
- return combineAndGetUpdateValue(older, newer, oldSchema, newSchema, props)
- .map(r -> Pair.of(new HoodieAvroIndexedRecord(r), r.getSchema()));
+ public Pair<HoodieRecord, Schema> merge(HoodieRecord older, Schema oldSchema, HoodieRecord newer, Schema newSchema, TypedProperties props) throws IOException {
+ Option<IndexedRecord> previousAvroData = older.toIndexedRecord(oldSchema, props).map(HoodieAvroIndexedRecord::getData);
+ HoodieRecordPayload payload = ((HoodieAvroRecord) newer).getData();
+ Option<IndexedRecord> updatedValue;
+ Comparable orderingVal;
+ if (previousAvroData.isEmpty()) {
+ updatedValue = payload.getInsertValue(newSchema, props);
+ orderingVal = payload.getOrderingValue();
+ } else {
+ updatedValue = ((HoodieAvroRecord) newer).getData().combineAndGetUpdateValue(previousAvroData.get(), newSchema, props);
Review Comment:
The newer record should not be a null object. It is possible that the underlying data field is null if it is a delete.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9715] Custom merger update [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13742:
URL: https://github.com/apache/hudi/pull/13742#discussion_r2300856120
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/BufferedRecordMergerFactory.java:
##########
@@ -517,20 +502,21 @@ protected Pair<HoodieRecord, HoodieRecord> getFinalMergeRecords(BufferedRecord<T
return getDeltaMergeRecords(olderRecord, newerRecord);
}
- protected Option<Pair<HoodieRecord, Schema>> getMergedRecord(BufferedRecord<T> olderRecord, BufferedRecord<T> newerRecord, boolean isFinalMerge) throws IOException {
+ protected Pair<HoodieRecord, Schema> getMergedRecord(BufferedRecord<T> olderRecord, BufferedRecord<T> newerRecord, boolean isFinalMerge) throws IOException {
Pair<HoodieRecord, HoodieRecord> records = isFinalMerge ? getFinalMergeRecords(olderRecord, newerRecord) : getDeltaMergeRecords(olderRecord, newerRecord);
return recordMerger.merge(records.getLeft(), getSchemaForAvroPayloadMerge(olderRecord), records.getRight(), getSchemaForAvroPayloadMerge(newerRecord), props);
}
protected HoodieRecord constructHoodieAvroRecord(RecordContext<T> recordContext, BufferedRecord<T> bufferedRecord, String payloadClass) {
GenericRecord record = null;
- if (!bufferedRecord.isDelete()) {
+ if (bufferedRecord.getRecord() != null) {
Schema recordSchema = recordContext.getSchemaFromBufferRecord(bufferedRecord);
record = recordContext.convertToAvroRecord(bufferedRecord.getRecord(), recordSchema);
}
HoodieKey hoodieKey = new HoodieKey(bufferedRecord.getRecordKey(), null);
+ Comparable orderingValue = recordContext.convertOrderingValueFromEngineType(bufferedRecord.getOrderingValue());
Review Comment:
Any suggestions? The payloads do not use the engine specific types leading to type cast issues.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(merger): [RFC-101] Update HoodieRecordMerger interface and implementations [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on PR #13742:
URL: https://github.com/apache/hudi/pull/13742#issuecomment-3305100269
> @the-other-tim-brown are we planning to change the param from `TypedProperties` to `MergeConfig` in this patch? Maybe we land this one first and address it in a separate PR which ease the review a lot.
@danny0405 if possible that would be great. I think it will be easier to do that in a separate, smaller PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT] Custom merger update [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13742:
URL: https://github.com/apache/hudi/pull/13742#issuecomment-3217431836
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7988",
"triggerID" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"triggerType" : "PUSH"
}, {
"hash" : "ecb694295f7a88371f33df6de60370058670439e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7992",
"triggerID" : "ecb694295f7a88371f33df6de60370058670439e",
"triggerType" : "PUSH"
}, {
"hash" : "efd7462406be4f6d90dd5e42f152d66f683ebdf4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "efd7462406be4f6d90dd5e42f152d66f683ebdf4",
"triggerType" : "PUSH"
}, {
"hash" : "3c67160d5ceec883db5d826ff655865a3209f2f3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7994",
"triggerID" : "3c67160d5ceec883db5d826ff655865a3209f2f3",
"triggerType" : "PUSH"
}, {
"hash" : "f4f84d9f6a480923df27cfe5553d8aa65b2bbb7a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7996",
"triggerID" : "f4f84d9f6a480923df27cfe5553d8aa65b2bbb7a",
"triggerType" : "PUSH"
}, {
"hash" : "521b26bc2c60b7921e2f7aa5fffa620ab7f9a096",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "521b26bc2c60b7921e2f7aa5fffa620ab7f9a096",
"triggerType" : "PUSH"
}, {
"hash" : "22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50",
"triggerType" : "PUSH"
}, {
"hash" : "02a319ba884595a4e3b5453f366b660c1c11f767",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8007",
"triggerID" : "02a319ba884595a4e3b5453f366b660c1c11f767",
"triggerType" : "PUSH"
}, {
"hash" : "3e59b6f8b486016911c9ad14c583f6538cc036ac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8009",
"triggerID" : "3e59b6f8b486016911c9ad14c583f6538cc036ac",
"triggerType" : "PUSH"
}, {
"hash" : "003f1c81efedafa0ddd78868ce83357d8803c5e5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "003f1c81efedafa0ddd78868ce83357d8803c5e5",
"triggerType" : "PUSH"
}, {
"hash" : "40c04768d5233a38be1b4c01ce895189a4af4c3a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8013",
"triggerID" : "40c04768d5233a38be1b4c01ce895189a4af4c3a",
"triggerType" : "PUSH"
}, {
"hash" : "4a699a8dcf79e61ea3bb754e67e9cc4a32e86aa1",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8084",
"triggerID" : "4a699a8dcf79e61ea3bb754e67e9cc4a32e86aa1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* efd7462406be4f6d90dd5e42f152d66f683ebdf4 UNKNOWN
* 521b26bc2c60b7921e2f7aa5fffa620ab7f9a096 UNKNOWN
* 22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50 UNKNOWN
* 003f1c81efedafa0ddd78868ce83357d8803c5e5 UNKNOWN
* 40c04768d5233a38be1b4c01ce895189a4af4c3a Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8013)
* 4a699a8dcf79e61ea3bb754e67e9cc4a32e86aa1 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8084)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT] Custom merger update [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13742:
URL: https://github.com/apache/hudi/pull/13742#issuecomment-3217447107
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7988",
"triggerID" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"triggerType" : "PUSH"
}, {
"hash" : "ecb694295f7a88371f33df6de60370058670439e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7992",
"triggerID" : "ecb694295f7a88371f33df6de60370058670439e",
"triggerType" : "PUSH"
}, {
"hash" : "efd7462406be4f6d90dd5e42f152d66f683ebdf4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "efd7462406be4f6d90dd5e42f152d66f683ebdf4",
"triggerType" : "PUSH"
}, {
"hash" : "3c67160d5ceec883db5d826ff655865a3209f2f3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7994",
"triggerID" : "3c67160d5ceec883db5d826ff655865a3209f2f3",
"triggerType" : "PUSH"
}, {
"hash" : "f4f84d9f6a480923df27cfe5553d8aa65b2bbb7a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7996",
"triggerID" : "f4f84d9f6a480923df27cfe5553d8aa65b2bbb7a",
"triggerType" : "PUSH"
}, {
"hash" : "521b26bc2c60b7921e2f7aa5fffa620ab7f9a096",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "521b26bc2c60b7921e2f7aa5fffa620ab7f9a096",
"triggerType" : "PUSH"
}, {
"hash" : "22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50",
"triggerType" : "PUSH"
}, {
"hash" : "02a319ba884595a4e3b5453f366b660c1c11f767",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8007",
"triggerID" : "02a319ba884595a4e3b5453f366b660c1c11f767",
"triggerType" : "PUSH"
}, {
"hash" : "3e59b6f8b486016911c9ad14c583f6538cc036ac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8009",
"triggerID" : "3e59b6f8b486016911c9ad14c583f6538cc036ac",
"triggerType" : "PUSH"
}, {
"hash" : "003f1c81efedafa0ddd78868ce83357d8803c5e5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "003f1c81efedafa0ddd78868ce83357d8803c5e5",
"triggerType" : "PUSH"
}, {
"hash" : "40c04768d5233a38be1b4c01ce895189a4af4c3a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8013",
"triggerID" : "40c04768d5233a38be1b4c01ce895189a4af4c3a",
"triggerType" : "PUSH"
}, {
"hash" : "4a699a8dcf79e61ea3bb754e67e9cc4a32e86aa1",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8084",
"triggerID" : "4a699a8dcf79e61ea3bb754e67e9cc4a32e86aa1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* efd7462406be4f6d90dd5e42f152d66f683ebdf4 UNKNOWN
* 521b26bc2c60b7921e2f7aa5fffa620ab7f9a096 UNKNOWN
* 22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50 UNKNOWN
* 003f1c81efedafa0ddd78868ce83357d8803c5e5 UNKNOWN
* 4a699a8dcf79e61ea3bb754e67e9cc4a32e86aa1 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8084)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT] Custom merger update [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13742:
URL: https://github.com/apache/hudi/pull/13742#issuecomment-3218387833
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7988",
"triggerID" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"triggerType" : "PUSH"
}, {
"hash" : "ecb694295f7a88371f33df6de60370058670439e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7992",
"triggerID" : "ecb694295f7a88371f33df6de60370058670439e",
"triggerType" : "PUSH"
}, {
"hash" : "efd7462406be4f6d90dd5e42f152d66f683ebdf4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "efd7462406be4f6d90dd5e42f152d66f683ebdf4",
"triggerType" : "PUSH"
}, {
"hash" : "3c67160d5ceec883db5d826ff655865a3209f2f3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7994",
"triggerID" : "3c67160d5ceec883db5d826ff655865a3209f2f3",
"triggerType" : "PUSH"
}, {
"hash" : "f4f84d9f6a480923df27cfe5553d8aa65b2bbb7a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7996",
"triggerID" : "f4f84d9f6a480923df27cfe5553d8aa65b2bbb7a",
"triggerType" : "PUSH"
}, {
"hash" : "521b26bc2c60b7921e2f7aa5fffa620ab7f9a096",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "521b26bc2c60b7921e2f7aa5fffa620ab7f9a096",
"triggerType" : "PUSH"
}, {
"hash" : "22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50",
"triggerType" : "PUSH"
}, {
"hash" : "02a319ba884595a4e3b5453f366b660c1c11f767",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8007",
"triggerID" : "02a319ba884595a4e3b5453f366b660c1c11f767",
"triggerType" : "PUSH"
}, {
"hash" : "3e59b6f8b486016911c9ad14c583f6538cc036ac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8009",
"triggerID" : "3e59b6f8b486016911c9ad14c583f6538cc036ac",
"triggerType" : "PUSH"
}, {
"hash" : "003f1c81efedafa0ddd78868ce83357d8803c5e5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "003f1c81efedafa0ddd78868ce83357d8803c5e5",
"triggerType" : "PUSH"
}, {
"hash" : "40c04768d5233a38be1b4c01ce895189a4af4c3a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8013",
"triggerID" : "40c04768d5233a38be1b4c01ce895189a4af4c3a",
"triggerType" : "PUSH"
}, {
"hash" : "4a699a8dcf79e61ea3bb754e67e9cc4a32e86aa1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8084",
"triggerID" : "4a699a8dcf79e61ea3bb754e67e9cc4a32e86aa1",
"triggerType" : "PUSH"
}, {
"hash" : "fe55d8ec4923cb6c64e83ae4f32217c72599a398",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "fe55d8ec4923cb6c64e83ae4f32217c72599a398",
"triggerType" : "PUSH"
}, {
"hash" : "87cd79787dc8126b6877ba0d3407cfe8f343933a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8087",
"triggerID" : "87cd79787dc8126b6877ba0d3407cfe8f343933a",
"triggerType" : "PUSH"
}, {
"hash" : "594874ffdca647cd268b3947fa4680c57c53d679",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8088",
"triggerID" : "594874ffdca647cd268b3947fa4680c57c53d679",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* efd7462406be4f6d90dd5e42f152d66f683ebdf4 UNKNOWN
* 521b26bc2c60b7921e2f7aa5fffa620ab7f9a096 UNKNOWN
* 22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50 UNKNOWN
* 003f1c81efedafa0ddd78868ce83357d8803c5e5 UNKNOWN
* fe55d8ec4923cb6c64e83ae4f32217c72599a398 UNKNOWN
* 594874ffdca647cd268b3947fa4680c57c53d679 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8088)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT] Custom merger update [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13742:
URL: https://github.com/apache/hudi/pull/13742#issuecomment-3218344515
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7988",
"triggerID" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"triggerType" : "PUSH"
}, {
"hash" : "ecb694295f7a88371f33df6de60370058670439e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7992",
"triggerID" : "ecb694295f7a88371f33df6de60370058670439e",
"triggerType" : "PUSH"
}, {
"hash" : "efd7462406be4f6d90dd5e42f152d66f683ebdf4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "efd7462406be4f6d90dd5e42f152d66f683ebdf4",
"triggerType" : "PUSH"
}, {
"hash" : "3c67160d5ceec883db5d826ff655865a3209f2f3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7994",
"triggerID" : "3c67160d5ceec883db5d826ff655865a3209f2f3",
"triggerType" : "PUSH"
}, {
"hash" : "f4f84d9f6a480923df27cfe5553d8aa65b2bbb7a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7996",
"triggerID" : "f4f84d9f6a480923df27cfe5553d8aa65b2bbb7a",
"triggerType" : "PUSH"
}, {
"hash" : "521b26bc2c60b7921e2f7aa5fffa620ab7f9a096",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "521b26bc2c60b7921e2f7aa5fffa620ab7f9a096",
"triggerType" : "PUSH"
}, {
"hash" : "22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50",
"triggerType" : "PUSH"
}, {
"hash" : "02a319ba884595a4e3b5453f366b660c1c11f767",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8007",
"triggerID" : "02a319ba884595a4e3b5453f366b660c1c11f767",
"triggerType" : "PUSH"
}, {
"hash" : "3e59b6f8b486016911c9ad14c583f6538cc036ac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8009",
"triggerID" : "3e59b6f8b486016911c9ad14c583f6538cc036ac",
"triggerType" : "PUSH"
}, {
"hash" : "003f1c81efedafa0ddd78868ce83357d8803c5e5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "003f1c81efedafa0ddd78868ce83357d8803c5e5",
"triggerType" : "PUSH"
}, {
"hash" : "40c04768d5233a38be1b4c01ce895189a4af4c3a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8013",
"triggerID" : "40c04768d5233a38be1b4c01ce895189a4af4c3a",
"triggerType" : "PUSH"
}, {
"hash" : "4a699a8dcf79e61ea3bb754e67e9cc4a32e86aa1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8084",
"triggerID" : "4a699a8dcf79e61ea3bb754e67e9cc4a32e86aa1",
"triggerType" : "PUSH"
}, {
"hash" : "fe55d8ec4923cb6c64e83ae4f32217c72599a398",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "fe55d8ec4923cb6c64e83ae4f32217c72599a398",
"triggerType" : "PUSH"
}, {
"hash" : "87cd79787dc8126b6877ba0d3407cfe8f343933a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8087",
"triggerID" : "87cd79787dc8126b6877ba0d3407cfe8f343933a",
"triggerType" : "PUSH"
}, {
"hash" : "594874ffdca647cd268b3947fa4680c57c53d679",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "594874ffdca647cd268b3947fa4680c57c53d679",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* efd7462406be4f6d90dd5e42f152d66f683ebdf4 UNKNOWN
* 521b26bc2c60b7921e2f7aa5fffa620ab7f9a096 UNKNOWN
* 22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50 UNKNOWN
* 003f1c81efedafa0ddd78868ce83357d8803c5e5 UNKNOWN
* fe55d8ec4923cb6c64e83ae4f32217c72599a398 UNKNOWN
* 87cd79787dc8126b6877ba0d3407cfe8f343933a Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8087)
* 594874ffdca647cd268b3947fa4680c57c53d679 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT] Custom merger update [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13742:
URL: https://github.com/apache/hudi/pull/13742#issuecomment-3218207885
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7988",
"triggerID" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"triggerType" : "PUSH"
}, {
"hash" : "ecb694295f7a88371f33df6de60370058670439e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7992",
"triggerID" : "ecb694295f7a88371f33df6de60370058670439e",
"triggerType" : "PUSH"
}, {
"hash" : "efd7462406be4f6d90dd5e42f152d66f683ebdf4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "efd7462406be4f6d90dd5e42f152d66f683ebdf4",
"triggerType" : "PUSH"
}, {
"hash" : "3c67160d5ceec883db5d826ff655865a3209f2f3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7994",
"triggerID" : "3c67160d5ceec883db5d826ff655865a3209f2f3",
"triggerType" : "PUSH"
}, {
"hash" : "f4f84d9f6a480923df27cfe5553d8aa65b2bbb7a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7996",
"triggerID" : "f4f84d9f6a480923df27cfe5553d8aa65b2bbb7a",
"triggerType" : "PUSH"
}, {
"hash" : "521b26bc2c60b7921e2f7aa5fffa620ab7f9a096",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "521b26bc2c60b7921e2f7aa5fffa620ab7f9a096",
"triggerType" : "PUSH"
}, {
"hash" : "22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50",
"triggerType" : "PUSH"
}, {
"hash" : "02a319ba884595a4e3b5453f366b660c1c11f767",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8007",
"triggerID" : "02a319ba884595a4e3b5453f366b660c1c11f767",
"triggerType" : "PUSH"
}, {
"hash" : "3e59b6f8b486016911c9ad14c583f6538cc036ac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8009",
"triggerID" : "3e59b6f8b486016911c9ad14c583f6538cc036ac",
"triggerType" : "PUSH"
}, {
"hash" : "003f1c81efedafa0ddd78868ce83357d8803c5e5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "003f1c81efedafa0ddd78868ce83357d8803c5e5",
"triggerType" : "PUSH"
}, {
"hash" : "40c04768d5233a38be1b4c01ce895189a4af4c3a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8013",
"triggerID" : "40c04768d5233a38be1b4c01ce895189a4af4c3a",
"triggerType" : "PUSH"
}, {
"hash" : "4a699a8dcf79e61ea3bb754e67e9cc4a32e86aa1",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8084",
"triggerID" : "4a699a8dcf79e61ea3bb754e67e9cc4a32e86aa1",
"triggerType" : "PUSH"
}, {
"hash" : "fe55d8ec4923cb6c64e83ae4f32217c72599a398",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "fe55d8ec4923cb6c64e83ae4f32217c72599a398",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* efd7462406be4f6d90dd5e42f152d66f683ebdf4 UNKNOWN
* 521b26bc2c60b7921e2f7aa5fffa620ab7f9a096 UNKNOWN
* 22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50 UNKNOWN
* 003f1c81efedafa0ddd78868ce83357d8803c5e5 UNKNOWN
* 4a699a8dcf79e61ea3bb754e67e9cc4a32e86aa1 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8084)
* fe55d8ec4923cb6c64e83ae4f32217c72599a398 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9715] Custom merger update [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13742:
URL: https://github.com/apache/hudi/pull/13742#discussion_r2308934239
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieWriteMergeHandle.java:
##########
@@ -149,6 +158,7 @@ public HoodieWriteMergeHandle(HoodieWriteConfig config, String instantTime, Hood
public HoodieWriteMergeHandle(HoodieWriteConfig config, String instantTime, String partitionPath,
String fileId, HoodieTable<T, I, K, O> hoodieTable, TaskContextSupplier taskContextSupplier) {
super(config, instantTime, partitionPath, fileId, hoodieTable, taskContextSupplier, true);
+ this.orderingFields = ConfigUtils.getOrderingFields(config.getProps());
Review Comment:
Unfortunately no it cannot be final, there is a setter method for the reader context
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9715] Custom merger update [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13742:
URL: https://github.com/apache/hudi/pull/13742#issuecomment-3259466248
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "94aa9a669d2fe1fb4919a4735fab2665553b63c0",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8258",
"triggerID" : "94aa9a669d2fe1fb4919a4735fab2665553b63c0",
"triggerType" : "PUSH"
}, {
"hash" : "94aa9a669d2fe1fb4919a4735fab2665553b63c0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3237521433",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 94aa9a669d2fe1fb4919a4735fab2665553b63c0 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8258)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9715] Custom merger update [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13742:
URL: https://github.com/apache/hudi/pull/13742#issuecomment-3235018635
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7988",
"triggerID" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"triggerType" : "PUSH"
}, {
"hash" : "ecb694295f7a88371f33df6de60370058670439e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7992",
"triggerID" : "ecb694295f7a88371f33df6de60370058670439e",
"triggerType" : "PUSH"
}, {
"hash" : "efd7462406be4f6d90dd5e42f152d66f683ebdf4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "efd7462406be4f6d90dd5e42f152d66f683ebdf4",
"triggerType" : "PUSH"
}, {
"hash" : "3c67160d5ceec883db5d826ff655865a3209f2f3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7994",
"triggerID" : "3c67160d5ceec883db5d826ff655865a3209f2f3",
"triggerType" : "PUSH"
}, {
"hash" : "f4f84d9f6a480923df27cfe5553d8aa65b2bbb7a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7996",
"triggerID" : "f4f84d9f6a480923df27cfe5553d8aa65b2bbb7a",
"triggerType" : "PUSH"
}, {
"hash" : "521b26bc2c60b7921e2f7aa5fffa620ab7f9a096",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "521b26bc2c60b7921e2f7aa5fffa620ab7f9a096",
"triggerType" : "PUSH"
}, {
"hash" : "22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50",
"triggerType" : "PUSH"
}, {
"hash" : "02a319ba884595a4e3b5453f366b660c1c11f767",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8007",
"triggerID" : "02a319ba884595a4e3b5453f366b660c1c11f767",
"triggerType" : "PUSH"
}, {
"hash" : "3e59b6f8b486016911c9ad14c583f6538cc036ac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8009",
"triggerID" : "3e59b6f8b486016911c9ad14c583f6538cc036ac",
"triggerType" : "PUSH"
}, {
"hash" : "003f1c81efedafa0ddd78868ce83357d8803c5e5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "003f1c81efedafa0ddd78868ce83357d8803c5e5",
"triggerType" : "PUSH"
}, {
"hash" : "40c04768d5233a38be1b4c01ce895189a4af4c3a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8013",
"triggerID" : "40c04768d5233a38be1b4c01ce895189a4af4c3a",
"triggerType" : "PUSH"
}, {
"hash" : "4a699a8dcf79e61ea3bb754e67e9cc4a32e86aa1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8084",
"triggerID" : "4a699a8dcf79e61ea3bb754e67e9cc4a32e86aa1",
"triggerType" : "PUSH"
}, {
"hash" : "fe55d8ec4923cb6c64e83ae4f32217c72599a398",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "fe55d8ec4923cb6c64e83ae4f32217c72599a398",
"triggerType" : "PUSH"
}, {
"hash" : "87cd79787dc8126b6877ba0d3407cfe8f343933a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8087",
"triggerID" : "87cd79787dc8126b6877ba0d3407cfe8f343933a",
"triggerType" : "PUSH"
}, {
"hash" : "594874ffdca647cd268b3947fa4680c57c53d679",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8088",
"triggerID" : "594874ffdca647cd268b3947fa4680c57c53d679",
"triggerType" : "PUSH"
}, {
"hash" : "7b95754f0df2c1a087a6a28722a7471ee12cecf3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8090",
"triggerID" : "7b95754f0df2c1a087a6a28722a7471ee12cecf3",
"triggerType" : "PUSH"
}, {
"hash" : "789c2da879ca66f6b9be71d1814be38b5d8cfb58",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8102",
"triggerID" : "789c2da879ca66f6b9be71d1814be38b5d8cfb58",
"triggerType" : "PUSH"
}, {
"hash" : "63f33fae3e22a9f80de51b2a6e1ed0427f631d0b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8103",
"triggerID" : "63f33fae3e22a9f80de51b2a6e1ed0427f631d0b",
"triggerType" : "PUSH"
}, {
"hash" : "e04c466fa81e9bcfaabe36825127098f5d1bb890",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8224",
"triggerID" : "e04c466fa81e9bcfaabe36825127098f5d1bb890",
"triggerType" : "PUSH"
}, {
"hash" : "cf574f46a1783a39b2074542aad9539ec3c4f36a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "cf574f46a1783a39b2074542aad9539ec3c4f36a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* efd7462406be4f6d90dd5e42f152d66f683ebdf4 UNKNOWN
* 521b26bc2c60b7921e2f7aa5fffa620ab7f9a096 UNKNOWN
* 22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50 UNKNOWN
* 003f1c81efedafa0ddd78868ce83357d8803c5e5 UNKNOWN
* fe55d8ec4923cb6c64e83ae4f32217c72599a398 UNKNOWN
* e04c466fa81e9bcfaabe36825127098f5d1bb890 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8224)
* cf574f46a1783a39b2074542aad9539ec3c4f36a UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9715] Custom merger update [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13742:
URL: https://github.com/apache/hudi/pull/13742#discussion_r2308911678
##########
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/testutils/HoodieSparkDeleteRecordMerger.java:
##########
@@ -21,11 +21,9 @@
import org.apache.hudi.HoodieSparkRecordMerger;
import org.apache.hudi.common.config.TypedProperties;
-import org.apache.hudi.common.model.HoodieRecord;
-import org.apache.hudi.common.util.Option;
-import org.apache.hudi.common.util.collection.Pair;
-
-import org.apache.avro.Schema;
+import org.apache.hudi.common.engine.RecordContext;
Review Comment:
What would be the correct sequence? Currently it is alphabetic sorting
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT] Custom merger update [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13742:
URL: https://github.com/apache/hudi/pull/13742#issuecomment-3208816532
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7988",
"triggerID" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"triggerType" : "PUSH"
}, {
"hash" : "ecb694295f7a88371f33df6de60370058670439e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7992",
"triggerID" : "ecb694295f7a88371f33df6de60370058670439e",
"triggerType" : "PUSH"
}, {
"hash" : "efd7462406be4f6d90dd5e42f152d66f683ebdf4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "efd7462406be4f6d90dd5e42f152d66f683ebdf4",
"triggerType" : "PUSH"
}, {
"hash" : "3c67160d5ceec883db5d826ff655865a3209f2f3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7994",
"triggerID" : "3c67160d5ceec883db5d826ff655865a3209f2f3",
"triggerType" : "PUSH"
}, {
"hash" : "f4f84d9f6a480923df27cfe5553d8aa65b2bbb7a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7996",
"triggerID" : "f4f84d9f6a480923df27cfe5553d8aa65b2bbb7a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* efd7462406be4f6d90dd5e42f152d66f683ebdf4 UNKNOWN
* 3c67160d5ceec883db5d826ff655865a3209f2f3 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7994)
* f4f84d9f6a480923df27cfe5553d8aa65b2bbb7a Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7996)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT] Custom merger update [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13742:
URL: https://github.com/apache/hudi/pull/13742#issuecomment-3211218348
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7988",
"triggerID" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"triggerType" : "PUSH"
}, {
"hash" : "ecb694295f7a88371f33df6de60370058670439e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7992",
"triggerID" : "ecb694295f7a88371f33df6de60370058670439e",
"triggerType" : "PUSH"
}, {
"hash" : "efd7462406be4f6d90dd5e42f152d66f683ebdf4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "efd7462406be4f6d90dd5e42f152d66f683ebdf4",
"triggerType" : "PUSH"
}, {
"hash" : "3c67160d5ceec883db5d826ff655865a3209f2f3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7994",
"triggerID" : "3c67160d5ceec883db5d826ff655865a3209f2f3",
"triggerType" : "PUSH"
}, {
"hash" : "f4f84d9f6a480923df27cfe5553d8aa65b2bbb7a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7996",
"triggerID" : "f4f84d9f6a480923df27cfe5553d8aa65b2bbb7a",
"triggerType" : "PUSH"
}, {
"hash" : "521b26bc2c60b7921e2f7aa5fffa620ab7f9a096",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "521b26bc2c60b7921e2f7aa5fffa620ab7f9a096",
"triggerType" : "PUSH"
}, {
"hash" : "22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50",
"triggerType" : "PUSH"
}, {
"hash" : "02a319ba884595a4e3b5453f366b660c1c11f767",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8007",
"triggerID" : "02a319ba884595a4e3b5453f366b660c1c11f767",
"triggerType" : "PUSH"
}, {
"hash" : "3e59b6f8b486016911c9ad14c583f6538cc036ac",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8009",
"triggerID" : "3e59b6f8b486016911c9ad14c583f6538cc036ac",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* efd7462406be4f6d90dd5e42f152d66f683ebdf4 UNKNOWN
* 521b26bc2c60b7921e2f7aa5fffa620ab7f9a096 UNKNOWN
* 22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50 UNKNOWN
* 02a319ba884595a4e3b5453f366b660c1c11f767 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8007)
* 3e59b6f8b486016911c9ad14c583f6538cc036ac Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8009)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT] Custom merger update [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13742:
URL: https://github.com/apache/hudi/pull/13742#issuecomment-3208607643
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7988",
"triggerID" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"triggerType" : "PUSH"
}, {
"hash" : "ecb694295f7a88371f33df6de60370058670439e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7992",
"triggerID" : "ecb694295f7a88371f33df6de60370058670439e",
"triggerType" : "PUSH"
}, {
"hash" : "efd7462406be4f6d90dd5e42f152d66f683ebdf4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "efd7462406be4f6d90dd5e42f152d66f683ebdf4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ecb694295f7a88371f33df6de60370058670439e Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7992)
* efd7462406be4f6d90dd5e42f152d66f683ebdf4 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT] Custom merger update [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13742:
URL: https://github.com/apache/hudi/pull/13742#issuecomment-3210975153
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7988",
"triggerID" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"triggerType" : "PUSH"
}, {
"hash" : "ecb694295f7a88371f33df6de60370058670439e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7992",
"triggerID" : "ecb694295f7a88371f33df6de60370058670439e",
"triggerType" : "PUSH"
}, {
"hash" : "efd7462406be4f6d90dd5e42f152d66f683ebdf4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "efd7462406be4f6d90dd5e42f152d66f683ebdf4",
"triggerType" : "PUSH"
}, {
"hash" : "3c67160d5ceec883db5d826ff655865a3209f2f3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7994",
"triggerID" : "3c67160d5ceec883db5d826ff655865a3209f2f3",
"triggerType" : "PUSH"
}, {
"hash" : "f4f84d9f6a480923df27cfe5553d8aa65b2bbb7a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7996",
"triggerID" : "f4f84d9f6a480923df27cfe5553d8aa65b2bbb7a",
"triggerType" : "PUSH"
}, {
"hash" : "521b26bc2c60b7921e2f7aa5fffa620ab7f9a096",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "521b26bc2c60b7921e2f7aa5fffa620ab7f9a096",
"triggerType" : "PUSH"
}, {
"hash" : "22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50",
"triggerType" : "PUSH"
}, {
"hash" : "02a319ba884595a4e3b5453f366b660c1c11f767",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8007",
"triggerID" : "02a319ba884595a4e3b5453f366b660c1c11f767",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* efd7462406be4f6d90dd5e42f152d66f683ebdf4 UNKNOWN
* f4f84d9f6a480923df27cfe5553d8aa65b2bbb7a Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7996)
* 521b26bc2c60b7921e2f7aa5fffa620ab7f9a096 UNKNOWN
* 22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50 UNKNOWN
* 02a319ba884595a4e3b5453f366b660c1c11f767 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8007)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT] Custom merger update [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13742:
URL: https://github.com/apache/hudi/pull/13742#issuecomment-3208702166
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7988",
"triggerID" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"triggerType" : "PUSH"
}, {
"hash" : "ecb694295f7a88371f33df6de60370058670439e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7992",
"triggerID" : "ecb694295f7a88371f33df6de60370058670439e",
"triggerType" : "PUSH"
}, {
"hash" : "efd7462406be4f6d90dd5e42f152d66f683ebdf4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "efd7462406be4f6d90dd5e42f152d66f683ebdf4",
"triggerType" : "PUSH"
}, {
"hash" : "3c67160d5ceec883db5d826ff655865a3209f2f3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7994",
"triggerID" : "3c67160d5ceec883db5d826ff655865a3209f2f3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* efd7462406be4f6d90dd5e42f152d66f683ebdf4 UNKNOWN
* 3c67160d5ceec883db5d826ff655865a3209f2f3 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7994)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT] Custom merger update [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13742:
URL: https://github.com/apache/hudi/pull/13742#issuecomment-3211232072
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7988",
"triggerID" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"triggerType" : "PUSH"
}, {
"hash" : "ecb694295f7a88371f33df6de60370058670439e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7992",
"triggerID" : "ecb694295f7a88371f33df6de60370058670439e",
"triggerType" : "PUSH"
}, {
"hash" : "efd7462406be4f6d90dd5e42f152d66f683ebdf4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "efd7462406be4f6d90dd5e42f152d66f683ebdf4",
"triggerType" : "PUSH"
}, {
"hash" : "3c67160d5ceec883db5d826ff655865a3209f2f3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7994",
"triggerID" : "3c67160d5ceec883db5d826ff655865a3209f2f3",
"triggerType" : "PUSH"
}, {
"hash" : "f4f84d9f6a480923df27cfe5553d8aa65b2bbb7a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7996",
"triggerID" : "f4f84d9f6a480923df27cfe5553d8aa65b2bbb7a",
"triggerType" : "PUSH"
}, {
"hash" : "521b26bc2c60b7921e2f7aa5fffa620ab7f9a096",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "521b26bc2c60b7921e2f7aa5fffa620ab7f9a096",
"triggerType" : "PUSH"
}, {
"hash" : "22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50",
"triggerType" : "PUSH"
}, {
"hash" : "02a319ba884595a4e3b5453f366b660c1c11f767",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8007",
"triggerID" : "02a319ba884595a4e3b5453f366b660c1c11f767",
"triggerType" : "PUSH"
}, {
"hash" : "3e59b6f8b486016911c9ad14c583f6538cc036ac",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8009",
"triggerID" : "3e59b6f8b486016911c9ad14c583f6538cc036ac",
"triggerType" : "PUSH"
}, {
"hash" : "003f1c81efedafa0ddd78868ce83357d8803c5e5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "003f1c81efedafa0ddd78868ce83357d8803c5e5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* efd7462406be4f6d90dd5e42f152d66f683ebdf4 UNKNOWN
* 521b26bc2c60b7921e2f7aa5fffa620ab7f9a096 UNKNOWN
* 22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50 UNKNOWN
* 02a319ba884595a4e3b5453f366b660c1c11f767 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8007)
* 3e59b6f8b486016911c9ad14c583f6538cc036ac Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8009)
* 003f1c81efedafa0ddd78868ce83357d8803c5e5 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT] Custom merger update [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13742:
URL: https://github.com/apache/hudi/pull/13742#issuecomment-3211339841
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7988",
"triggerID" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"triggerType" : "PUSH"
}, {
"hash" : "ecb694295f7a88371f33df6de60370058670439e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7992",
"triggerID" : "ecb694295f7a88371f33df6de60370058670439e",
"triggerType" : "PUSH"
}, {
"hash" : "efd7462406be4f6d90dd5e42f152d66f683ebdf4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "efd7462406be4f6d90dd5e42f152d66f683ebdf4",
"triggerType" : "PUSH"
}, {
"hash" : "3c67160d5ceec883db5d826ff655865a3209f2f3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7994",
"triggerID" : "3c67160d5ceec883db5d826ff655865a3209f2f3",
"triggerType" : "PUSH"
}, {
"hash" : "f4f84d9f6a480923df27cfe5553d8aa65b2bbb7a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7996",
"triggerID" : "f4f84d9f6a480923df27cfe5553d8aa65b2bbb7a",
"triggerType" : "PUSH"
}, {
"hash" : "521b26bc2c60b7921e2f7aa5fffa620ab7f9a096",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "521b26bc2c60b7921e2f7aa5fffa620ab7f9a096",
"triggerType" : "PUSH"
}, {
"hash" : "22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50",
"triggerType" : "PUSH"
}, {
"hash" : "02a319ba884595a4e3b5453f366b660c1c11f767",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8007",
"triggerID" : "02a319ba884595a4e3b5453f366b660c1c11f767",
"triggerType" : "PUSH"
}, {
"hash" : "3e59b6f8b486016911c9ad14c583f6538cc036ac",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8009",
"triggerID" : "3e59b6f8b486016911c9ad14c583f6538cc036ac",
"triggerType" : "PUSH"
}, {
"hash" : "003f1c81efedafa0ddd78868ce83357d8803c5e5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "003f1c81efedafa0ddd78868ce83357d8803c5e5",
"triggerType" : "PUSH"
}, {
"hash" : "40c04768d5233a38be1b4c01ce895189a4af4c3a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "40c04768d5233a38be1b4c01ce895189a4af4c3a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* efd7462406be4f6d90dd5e42f152d66f683ebdf4 UNKNOWN
* 521b26bc2c60b7921e2f7aa5fffa620ab7f9a096 UNKNOWN
* 22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50 UNKNOWN
* 3e59b6f8b486016911c9ad14c583f6538cc036ac Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8009)
* 003f1c81efedafa0ddd78868ce83357d8803c5e5 UNKNOWN
* 40c04768d5233a38be1b4c01ce895189a4af4c3a UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT] Custom merger update [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13742:
URL: https://github.com/apache/hudi/pull/13742#issuecomment-3218450698
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7988",
"triggerID" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"triggerType" : "PUSH"
}, {
"hash" : "ecb694295f7a88371f33df6de60370058670439e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7992",
"triggerID" : "ecb694295f7a88371f33df6de60370058670439e",
"triggerType" : "PUSH"
}, {
"hash" : "efd7462406be4f6d90dd5e42f152d66f683ebdf4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "efd7462406be4f6d90dd5e42f152d66f683ebdf4",
"triggerType" : "PUSH"
}, {
"hash" : "3c67160d5ceec883db5d826ff655865a3209f2f3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7994",
"triggerID" : "3c67160d5ceec883db5d826ff655865a3209f2f3",
"triggerType" : "PUSH"
}, {
"hash" : "f4f84d9f6a480923df27cfe5553d8aa65b2bbb7a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7996",
"triggerID" : "f4f84d9f6a480923df27cfe5553d8aa65b2bbb7a",
"triggerType" : "PUSH"
}, {
"hash" : "521b26bc2c60b7921e2f7aa5fffa620ab7f9a096",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "521b26bc2c60b7921e2f7aa5fffa620ab7f9a096",
"triggerType" : "PUSH"
}, {
"hash" : "22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50",
"triggerType" : "PUSH"
}, {
"hash" : "02a319ba884595a4e3b5453f366b660c1c11f767",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8007",
"triggerID" : "02a319ba884595a4e3b5453f366b660c1c11f767",
"triggerType" : "PUSH"
}, {
"hash" : "3e59b6f8b486016911c9ad14c583f6538cc036ac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8009",
"triggerID" : "3e59b6f8b486016911c9ad14c583f6538cc036ac",
"triggerType" : "PUSH"
}, {
"hash" : "003f1c81efedafa0ddd78868ce83357d8803c5e5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "003f1c81efedafa0ddd78868ce83357d8803c5e5",
"triggerType" : "PUSH"
}, {
"hash" : "40c04768d5233a38be1b4c01ce895189a4af4c3a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8013",
"triggerID" : "40c04768d5233a38be1b4c01ce895189a4af4c3a",
"triggerType" : "PUSH"
}, {
"hash" : "4a699a8dcf79e61ea3bb754e67e9cc4a32e86aa1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8084",
"triggerID" : "4a699a8dcf79e61ea3bb754e67e9cc4a32e86aa1",
"triggerType" : "PUSH"
}, {
"hash" : "fe55d8ec4923cb6c64e83ae4f32217c72599a398",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "fe55d8ec4923cb6c64e83ae4f32217c72599a398",
"triggerType" : "PUSH"
}, {
"hash" : "87cd79787dc8126b6877ba0d3407cfe8f343933a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8087",
"triggerID" : "87cd79787dc8126b6877ba0d3407cfe8f343933a",
"triggerType" : "PUSH"
}, {
"hash" : "594874ffdca647cd268b3947fa4680c57c53d679",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8088",
"triggerID" : "594874ffdca647cd268b3947fa4680c57c53d679",
"triggerType" : "PUSH"
}, {
"hash" : "7b95754f0df2c1a087a6a28722a7471ee12cecf3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8090",
"triggerID" : "7b95754f0df2c1a087a6a28722a7471ee12cecf3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* efd7462406be4f6d90dd5e42f152d66f683ebdf4 UNKNOWN
* 521b26bc2c60b7921e2f7aa5fffa620ab7f9a096 UNKNOWN
* 22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50 UNKNOWN
* 003f1c81efedafa0ddd78868ce83357d8803c5e5 UNKNOWN
* fe55d8ec4923cb6c64e83ae4f32217c72599a398 UNKNOWN
* 7b95754f0df2c1a087a6a28722a7471ee12cecf3 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8090)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9715] Custom merger update [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13742:
URL: https://github.com/apache/hudi/pull/13742#issuecomment-3259476412
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "94aa9a669d2fe1fb4919a4735fab2665553b63c0",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8258",
"triggerID" : "94aa9a669d2fe1fb4919a4735fab2665553b63c0",
"triggerType" : "PUSH"
}, {
"hash" : "94aa9a669d2fe1fb4919a4735fab2665553b63c0",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8258",
"triggerID" : "3237521433",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 94aa9a669d2fe1fb4919a4735fab2665553b63c0 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8258)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9715] Custom merger update [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13742:
URL: https://github.com/apache/hudi/pull/13742#issuecomment-3259462729
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "94aa9a669d2fe1fb4919a4735fab2665553b63c0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "94aa9a669d2fe1fb4919a4735fab2665553b63c0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 94aa9a669d2fe1fb4919a4735fab2665553b63c0 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(merger): [RFC-101] Update HoodieRecordMerger interface and implementations [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on PR #13742:
URL: https://github.com/apache/hudi/pull/13742#issuecomment-3305128927
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9715] Custom merger update [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13742:
URL: https://github.com/apache/hudi/pull/13742#discussion_r2300860077
##########
hudi-common/src/main/java/org/apache/hudi/common/model/DefaultHoodieRecordPayload.java:
##########
@@ -74,11 +74,7 @@ public OverwriteWithLatestAvroPayload preCombine(OverwriteWithLatestAvroPayload
@Override
public Option<IndexedRecord> combineAndGetUpdateValue(IndexedRecord currentValue, Schema schema, Properties properties) throws IOException {
- if (recordBytes.length == 0) {
- return Option.empty();
- }
-
- GenericRecord incomingRecord = HoodieAvroUtils.bytesToAvro(recordBytes, schema);
+ Option<IndexedRecord> incomingRecord = recordBytes.length == 0 ? Option.empty() : Option.of(HoodieAvroUtils.bytesToAvro(recordBytes, schema));
Review Comment:
It is to allow event time ordering with deletes. Before if we have an empty byte array for the record, it is always returned as a delete and the event time is not considered.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9715] Custom merger update [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #13742:
URL: https://github.com/apache/hudi/pull/13742#discussion_r2302038255
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/BufferedRecordMergerFactory.java:
##########
@@ -567,42 +553,36 @@ public Option<BufferedRecord<T>> deltaMerge(BufferedRecord<T> newRecord, Buffere
if (existingRecord == null) {
return Option.of(newRecord);
}
- if (existingRecord.isDelete() || newRecord.isDelete()) {
- if (shouldKeepNewerRecord(existingRecord, newRecord)) {
- // IMPORTANT:
- // this is needed when the fallback HoodieAvroRecordMerger got used, the merger would
- // return Option.empty when the old payload data is empty(a delete) and ignores its ordering value directly.
- return Option.of(newRecord);
- } else {
- return Option.empty();
- }
- }
- return deltaMergeNonDeleteRecord(newRecord, existingRecord);
+ return deltaMergeRecords(newRecord, existingRecord);
}
- public abstract Option<BufferedRecord<T>> deltaMergeNonDeleteRecord(BufferedRecord<T> newRecord, BufferedRecord<T> existingRecord) throws IOException;
+ public abstract Option<BufferedRecord<T>> deltaMergeRecords(BufferedRecord<T> newRecord, BufferedRecord<T> existingRecord) throws IOException;
@Override
public Option<DeleteRecord> deltaMerge(DeleteRecord deleteRecord, BufferedRecord<T> existingRecord) {
- return deltaMergeDeleteRecord(deleteRecord, existingRecord);
+ BufferedRecord<T> deleteBufferedRecord = BufferedRecords.fromDeleteRecord(deleteRecord, recordContext);
+ try {
+ Option<BufferedRecord<T>> merged = deltaMerge(deleteBufferedRecord, existingRecord);
+ // If the delete record is chosen, return an option with the delete record, otherwise return empty.
+ return merged.isPresent() ? Option.of(deleteRecord) : Option.empty();
+ } catch (IOException e) {
+ throw new HoodieIOException("Failed to process delete record", e);
+ }
}
@Override
public BufferedRecord<T> finalMerge(BufferedRecord<T> olderRecord, BufferedRecord<T> newerRecord) throws IOException {
- if (olderRecord.isDelete() || newerRecord.isDelete()) {
- if (shouldKeepNewerRecord(olderRecord, newerRecord)) {
- // IMPORTANT:
- // this is needed when the fallback HoodieAvroRecordMerger got used, the merger would
- // return Option.empty when the new payload data is empty(a delete) and ignores its ordering value directly.
- return newerRecord;
- } else {
- return olderRecord;
- }
+ if (olderRecord == null) {
Review Comment:
Based on the current implements, both older and newer record arguments here can not be not null.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9715] Custom merger update [hudi]
Posted by "cshuo (via GitHub)" <gi...@apache.org>.
cshuo commented on code in PR #13742:
URL: https://github.com/apache/hudi/pull/13742#discussion_r2308906508
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieWriteMergeHandle.java:
##########
@@ -120,6 +125,8 @@ public HoodieWriteMergeHandle(HoodieWriteConfig config, String instantTime, Hood
super(config, instantTime, hoodieTable, partitionPath, fileId, taskContextSupplier, baseFile, keyGeneratorOpt, false);
populateIncomingRecordsMap(recordItr);
initMarkerFileAndFileWriter(fileId, partitionPath);
+ this.readerContext = hoodieTable.getReaderContextFactoryForWrite().getContext();
Review Comment:
There is duplicate re-initialization for `readerContext` in `FileGroupReaderBasedMergeHandle` after this change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9715] Custom merger update [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13742:
URL: https://github.com/apache/hudi/pull/13742#issuecomment-3236726659
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7988",
"triggerID" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"triggerType" : "PUSH"
}, {
"hash" : "ecb694295f7a88371f33df6de60370058670439e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7992",
"triggerID" : "ecb694295f7a88371f33df6de60370058670439e",
"triggerType" : "PUSH"
}, {
"hash" : "efd7462406be4f6d90dd5e42f152d66f683ebdf4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "efd7462406be4f6d90dd5e42f152d66f683ebdf4",
"triggerType" : "PUSH"
}, {
"hash" : "3c67160d5ceec883db5d826ff655865a3209f2f3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7994",
"triggerID" : "3c67160d5ceec883db5d826ff655865a3209f2f3",
"triggerType" : "PUSH"
}, {
"hash" : "f4f84d9f6a480923df27cfe5553d8aa65b2bbb7a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7996",
"triggerID" : "f4f84d9f6a480923df27cfe5553d8aa65b2bbb7a",
"triggerType" : "PUSH"
}, {
"hash" : "521b26bc2c60b7921e2f7aa5fffa620ab7f9a096",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "521b26bc2c60b7921e2f7aa5fffa620ab7f9a096",
"triggerType" : "PUSH"
}, {
"hash" : "22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50",
"triggerType" : "PUSH"
}, {
"hash" : "02a319ba884595a4e3b5453f366b660c1c11f767",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8007",
"triggerID" : "02a319ba884595a4e3b5453f366b660c1c11f767",
"triggerType" : "PUSH"
}, {
"hash" : "3e59b6f8b486016911c9ad14c583f6538cc036ac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8009",
"triggerID" : "3e59b6f8b486016911c9ad14c583f6538cc036ac",
"triggerType" : "PUSH"
}, {
"hash" : "003f1c81efedafa0ddd78868ce83357d8803c5e5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "003f1c81efedafa0ddd78868ce83357d8803c5e5",
"triggerType" : "PUSH"
}, {
"hash" : "40c04768d5233a38be1b4c01ce895189a4af4c3a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8013",
"triggerID" : "40c04768d5233a38be1b4c01ce895189a4af4c3a",
"triggerType" : "PUSH"
}, {
"hash" : "4a699a8dcf79e61ea3bb754e67e9cc4a32e86aa1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8084",
"triggerID" : "4a699a8dcf79e61ea3bb754e67e9cc4a32e86aa1",
"triggerType" : "PUSH"
}, {
"hash" : "fe55d8ec4923cb6c64e83ae4f32217c72599a398",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "fe55d8ec4923cb6c64e83ae4f32217c72599a398",
"triggerType" : "PUSH"
}, {
"hash" : "87cd79787dc8126b6877ba0d3407cfe8f343933a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8087",
"triggerID" : "87cd79787dc8126b6877ba0d3407cfe8f343933a",
"triggerType" : "PUSH"
}, {
"hash" : "594874ffdca647cd268b3947fa4680c57c53d679",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8088",
"triggerID" : "594874ffdca647cd268b3947fa4680c57c53d679",
"triggerType" : "PUSH"
}, {
"hash" : "7b95754f0df2c1a087a6a28722a7471ee12cecf3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8090",
"triggerID" : "7b95754f0df2c1a087a6a28722a7471ee12cecf3",
"triggerType" : "PUSH"
}, {
"hash" : "789c2da879ca66f6b9be71d1814be38b5d8cfb58",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8102",
"triggerID" : "789c2da879ca66f6b9be71d1814be38b5d8cfb58",
"triggerType" : "PUSH"
}, {
"hash" : "63f33fae3e22a9f80de51b2a6e1ed0427f631d0b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8103",
"triggerID" : "63f33fae3e22a9f80de51b2a6e1ed0427f631d0b",
"triggerType" : "PUSH"
}, {
"hash" : "e04c466fa81e9bcfaabe36825127098f5d1bb890",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8224",
"triggerID" : "e04c466fa81e9bcfaabe36825127098f5d1bb890",
"triggerType" : "PUSH"
}, {
"hash" : "cf574f46a1783a39b2074542aad9539ec3c4f36a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8235",
"triggerID" : "cf574f46a1783a39b2074542aad9539ec3c4f36a",
"triggerType" : "PUSH"
}, {
"hash" : "2eb81e923adfa77176f137c442143deaac9e9916",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2eb81e923adfa77176f137c442143deaac9e9916",
"triggerType" : "PUSH"
}, {
"hash" : "12dc3e3eaf746bb1f2bc3f4e93fa14d9adb0f6fd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8244",
"triggerID" : "12dc3e3eaf746bb1f2bc3f4e93fa14d9adb0f6fd",
"triggerType" : "PUSH"
}, {
"hash" : "af5ab73103f3af882412bbe99c9322165e8941a0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "af5ab73103f3af882412bbe99c9322165e8941a0",
"triggerType" : "PUSH"
}, {
"hash" : "a9c28c4d913644fea23d57a5dc2224f36fd2b63e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8248",
"triggerID" : "a9c28c4d913644fea23d57a5dc2224f36fd2b63e",
"triggerType" : "PUSH"
}, {
"hash" : "98381a6bd2bcccf3da08fb992e599fe1165f2ddb",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8252",
"triggerID" : "98381a6bd2bcccf3da08fb992e599fe1165f2ddb",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* efd7462406be4f6d90dd5e42f152d66f683ebdf4 UNKNOWN
* 521b26bc2c60b7921e2f7aa5fffa620ab7f9a096 UNKNOWN
* 22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50 UNKNOWN
* 003f1c81efedafa0ddd78868ce83357d8803c5e5 UNKNOWN
* fe55d8ec4923cb6c64e83ae4f32217c72599a398 UNKNOWN
* 2eb81e923adfa77176f137c442143deaac9e9916 UNKNOWN
* af5ab73103f3af882412bbe99c9322165e8941a0 UNKNOWN
* 98381a6bd2bcccf3da08fb992e599fe1165f2ddb Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8252)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9715] Custom merger update [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13742:
URL: https://github.com/apache/hudi/pull/13742#discussion_r2308926474
##########
hudi-common/src/main/java/org/apache/hudi/common/model/HoodieAvroRecordMerger.java:
##########
@@ -19,51 +19,101 @@
package org.apache.hudi.common.model;
import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.engine.RecordContext;
import org.apache.hudi.common.model.HoodieRecord.HoodieRecordType;
+import org.apache.hudi.common.table.read.BufferedRecord;
+import org.apache.hudi.common.table.read.BufferedRecords;
+import org.apache.hudi.common.util.ConfigUtils;
+import org.apache.hudi.common.util.HoodieRecordUtils;
import org.apache.hudi.common.util.Option;
-import org.apache.hudi.common.util.collection.Pair;
import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
import org.apache.avro.generic.IndexedRecord;
import java.io.IOException;
-import java.util.Properties;
/**
* Record merger for Hoodie avro record.
*
* <p>It should only be used for base record from disk to merge with incoming record.
*/
public class HoodieAvroRecordMerger implements HoodieRecordMerger, OperationModeAwareness {
- public static final HoodieAvroRecordMerger INSTANCE = new HoodieAvroRecordMerger();
+ public static final String PAYLOAD_CLASS_PROP = "_hoodie.merger.payload.class";
Review Comment:
this is specific for expression payload?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9715] Custom merger update [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13742:
URL: https://github.com/apache/hudi/pull/13742#discussion_r2300803147
##########
hudi-common/src/main/java/org/apache/hudi/common/model/HoodieRecordMerger.java:
##########
@@ -63,8 +63,15 @@ public interface HoodieRecordMerger extends Serializable {
* It'd be associative operation: f(a, f(b, c)) = f(f(a, b), c) (which we can translate as having 3 versions A, B, C
* of the single record, both orders of operations applications have to yield the same result)
* This method takes only full records for merging.
+ *
+ * @param older Older record in terms of commit time ordering.
+ * @param oldSchema The schema of the older record.
+ * @param newer Newer record in terms of commit time ordering.
+ * @param newSchema The schema of the newer record.
+ * @param props The additional properties for the merging operation.
+ * @return The merged record and schema. The record is expected to be non-null. If the record represents a deletion, the operation must be set as {@link HoodieOperation#DELETE}.
*/
- Option<Pair<HoodieRecord, Schema>> merge(HoodieRecord older, Schema oldSchema, HoodieRecord newer, Schema newSchema, TypedProperties props) throws IOException;
+ Pair<HoodieRecord, Schema> merge(HoodieRecord older, Schema oldSchema, HoodieRecord newer, Schema newSchema, TypedProperties props) throws IOException;
Review Comment:
Based on the changes in the patch, using `HoodieRecord` brings in a lot of intricate changes in write merging, buffered record mergers and deletes handling in mergers themself, making the bar very high to make the behavior correct and consistent among multiple engine, that I prefer we have a "change record" abstraction here to simply the things, or just use `BufferedRecord` because it is a very near abstaction and it has been widely used in merging now(just to name it better as a developer/user API).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9715] Custom merger update [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13742:
URL: https://github.com/apache/hudi/pull/13742#discussion_r2308943041
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/DefaultSparkRecordMerger.java:
##########
@@ -37,46 +35,38 @@
*/
public class DefaultSparkRecordMerger extends HoodieSparkRecordMerger {
- private String[] orderingFields;
-
@Override
public String getMergingStrategy() {
return HoodieRecordMerger.EVENT_TIME_BASED_MERGE_STRATEGY_UUID;
}
@Override
- public Option<Pair<HoodieRecord, Schema>> merge(HoodieRecord older, Schema oldSchema, HoodieRecord newer, Schema newSchema, TypedProperties props) throws IOException {
- Option<Pair<HoodieRecord, Schema>> deleteHandlingResult = handleDeletes(older, oldSchema, newer, newSchema, props);
- if (deleteHandlingResult != null) {
- return deleteHandlingResult;
- }
-
- if (orderingFields == null) {
- orderingFields = ConfigUtils.getOrderingFields(props);
- }
- if (older.getOrderingValue(oldSchema, props, orderingFields).compareTo(newer.getOrderingValue(newSchema, props, orderingFields)) > 0) {
- return Option.of(Pair.of(older, oldSchema));
+ public <T> BufferedRecord<T> merge(BufferedRecord<T> older, BufferedRecord<T> newer, RecordContext<T> recordContext, TypedProperties props) throws IOException {
+ if (older.getOrderingValue().compareTo(newer.getOrderingValue()) > 0) {
Review Comment:
Updated the mergers with a helper method and some inline comments attempting to explain the reason for the special case
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9715] Custom merger update [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13742:
URL: https://github.com/apache/hudi/pull/13742#discussion_r2305884287
##########
hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/DefaultHiveRecordMerger.java:
##########
@@ -39,33 +41,33 @@ public class DefaultHiveRecordMerger extends HoodieHiveRecordMerger {
private String[] orderingFields;
@Override
- public Option<Pair<HoodieRecord, Schema>> merge(HoodieRecord older, Schema oldSchema, HoodieRecord newer, Schema newSchema, TypedProperties props) throws IOException {
+ public Pair<HoodieRecord, Schema> merge(HoodieRecord older, Schema oldSchema, HoodieRecord newer, Schema newSchema, TypedProperties props) throws IOException {
ValidationUtils.checkArgument(older.getRecordType() == HoodieRecord.HoodieRecordType.HIVE);
ValidationUtils.checkArgument(newer.getRecordType() == HoodieRecord.HoodieRecordType.HIVE);
if (newer instanceof HoodieHiveRecord) {
HoodieHiveRecord newHiveRecord = (HoodieHiveRecord) newer;
if (newHiveRecord.isDelete(newSchema, props)) {
- return Option.empty();
+ return Pair.of(new HoodieEmptyRecord<>(newer.getKey(), HoodieOperation.DELETE, OrderingValues.getDefault(), HoodieRecord.HoodieRecordType.HIVE), newSchema);
Review Comment:
That is because the deletes are pre handled by the buffered record merger itself. But in this patch, the pre-handling is removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9715] Custom merger update [hudi]
Posted by "vinothchandar (via GitHub)" <gi...@apache.org>.
vinothchandar commented on code in PR #13742:
URL: https://github.com/apache/hudi/pull/13742#discussion_r2300690723
##########
hudi-common/src/main/java/org/apache/hudi/common/model/HoodieRecordMerger.java:
##########
@@ -63,8 +63,15 @@ public interface HoodieRecordMerger extends Serializable {
* It'd be associative operation: f(a, f(b, c)) = f(f(a, b), c) (which we can translate as having 3 versions A, B, C
* of the single record, both orders of operations applications have to yield the same result)
* This method takes only full records for merging.
+ *
+ * @param older Older record in terms of commit time ordering.
+ * @param oldSchema The schema of the older record.
+ * @param newer Newer record in terms of commit time ordering.
+ * @param newSchema The schema of the newer record.
+ * @param props The additional properties for the merging operation.
+ * @return The merged record and schema. The record is expected to be non-null. If the record represents a deletion, the operation must be set as {@link HoodieOperation#DELETE}.
*/
- Option<Pair<HoodieRecord, Schema>> merge(HoodieRecord older, Schema oldSchema, HoodieRecord newer, Schema newSchema, TypedProperties props) throws IOException;
+ Pair<HoodieRecord, Schema> merge(HoodieRecord older, Schema oldSchema, HoodieRecord newer, Schema newSchema, TypedProperties props) throws IOException;
Review Comment:
On the same topic of, whether we need a new Pojo here or is `HoodieRecord` the right choice. HoodieRecord is used here as a change record i.e we merge multiple change records in commit time order, based on the custom logic.. Is this use of the pojo consistent with other places ..
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9715] Custom merger update [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13742:
URL: https://github.com/apache/hudi/pull/13742#issuecomment-3235043452
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7988",
"triggerID" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"triggerType" : "PUSH"
}, {
"hash" : "ecb694295f7a88371f33df6de60370058670439e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7992",
"triggerID" : "ecb694295f7a88371f33df6de60370058670439e",
"triggerType" : "PUSH"
}, {
"hash" : "efd7462406be4f6d90dd5e42f152d66f683ebdf4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "efd7462406be4f6d90dd5e42f152d66f683ebdf4",
"triggerType" : "PUSH"
}, {
"hash" : "3c67160d5ceec883db5d826ff655865a3209f2f3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7994",
"triggerID" : "3c67160d5ceec883db5d826ff655865a3209f2f3",
"triggerType" : "PUSH"
}, {
"hash" : "f4f84d9f6a480923df27cfe5553d8aa65b2bbb7a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7996",
"triggerID" : "f4f84d9f6a480923df27cfe5553d8aa65b2bbb7a",
"triggerType" : "PUSH"
}, {
"hash" : "521b26bc2c60b7921e2f7aa5fffa620ab7f9a096",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "521b26bc2c60b7921e2f7aa5fffa620ab7f9a096",
"triggerType" : "PUSH"
}, {
"hash" : "22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50",
"triggerType" : "PUSH"
}, {
"hash" : "02a319ba884595a4e3b5453f366b660c1c11f767",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8007",
"triggerID" : "02a319ba884595a4e3b5453f366b660c1c11f767",
"triggerType" : "PUSH"
}, {
"hash" : "3e59b6f8b486016911c9ad14c583f6538cc036ac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8009",
"triggerID" : "3e59b6f8b486016911c9ad14c583f6538cc036ac",
"triggerType" : "PUSH"
}, {
"hash" : "003f1c81efedafa0ddd78868ce83357d8803c5e5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "003f1c81efedafa0ddd78868ce83357d8803c5e5",
"triggerType" : "PUSH"
}, {
"hash" : "40c04768d5233a38be1b4c01ce895189a4af4c3a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8013",
"triggerID" : "40c04768d5233a38be1b4c01ce895189a4af4c3a",
"triggerType" : "PUSH"
}, {
"hash" : "4a699a8dcf79e61ea3bb754e67e9cc4a32e86aa1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8084",
"triggerID" : "4a699a8dcf79e61ea3bb754e67e9cc4a32e86aa1",
"triggerType" : "PUSH"
}, {
"hash" : "fe55d8ec4923cb6c64e83ae4f32217c72599a398",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "fe55d8ec4923cb6c64e83ae4f32217c72599a398",
"triggerType" : "PUSH"
}, {
"hash" : "87cd79787dc8126b6877ba0d3407cfe8f343933a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8087",
"triggerID" : "87cd79787dc8126b6877ba0d3407cfe8f343933a",
"triggerType" : "PUSH"
}, {
"hash" : "594874ffdca647cd268b3947fa4680c57c53d679",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8088",
"triggerID" : "594874ffdca647cd268b3947fa4680c57c53d679",
"triggerType" : "PUSH"
}, {
"hash" : "7b95754f0df2c1a087a6a28722a7471ee12cecf3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8090",
"triggerID" : "7b95754f0df2c1a087a6a28722a7471ee12cecf3",
"triggerType" : "PUSH"
}, {
"hash" : "789c2da879ca66f6b9be71d1814be38b5d8cfb58",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8102",
"triggerID" : "789c2da879ca66f6b9be71d1814be38b5d8cfb58",
"triggerType" : "PUSH"
}, {
"hash" : "63f33fae3e22a9f80de51b2a6e1ed0427f631d0b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8103",
"triggerID" : "63f33fae3e22a9f80de51b2a6e1ed0427f631d0b",
"triggerType" : "PUSH"
}, {
"hash" : "e04c466fa81e9bcfaabe36825127098f5d1bb890",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8224",
"triggerID" : "e04c466fa81e9bcfaabe36825127098f5d1bb890",
"triggerType" : "PUSH"
}, {
"hash" : "cf574f46a1783a39b2074542aad9539ec3c4f36a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8235",
"triggerID" : "cf574f46a1783a39b2074542aad9539ec3c4f36a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* efd7462406be4f6d90dd5e42f152d66f683ebdf4 UNKNOWN
* 521b26bc2c60b7921e2f7aa5fffa620ab7f9a096 UNKNOWN
* 22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50 UNKNOWN
* 003f1c81efedafa0ddd78868ce83357d8803c5e5 UNKNOWN
* fe55d8ec4923cb6c64e83ae4f32217c72599a398 UNKNOWN
* e04c466fa81e9bcfaabe36825127098f5d1bb890 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8224)
* cf574f46a1783a39b2074542aad9539ec3c4f36a Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8235)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9715] Custom merger update [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13742:
URL: https://github.com/apache/hudi/pull/13742#discussion_r2308940260
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/BufferedRecordMergerFactory.java:
##########
@@ -449,95 +383,36 @@ protected Option<Pair<HoodieRecord, Schema>> getMergedRecord(BufferedRecord<T> o
* based on {@code CUSTOM} merge mode and a given record payload class.
*/
private static class CustomPayloadRecordMerger<T> extends BaseCustomMerger<T> {
- private final String[] orderingFieldNames;
protected final String payloadClass;
public CustomPayloadRecordMerger(
RecordContext<T> recordContext,
Option<HoodieRecordMerger> recordMerger,
- List<String> orderingFieldNames,
String payloadClass,
Schema readerSchema,
TypedProperties props) {
super(recordContext, recordMerger, readerSchema, props);
- this.orderingFieldNames = orderingFieldNames.toArray(new String[0]);
this.payloadClass = payloadClass;
+ props.setProperty(HoodieAvroRecordMerger.PAYLOAD_CLASS_PROP, payloadClass);
}
@Override
- public Option<BufferedRecord<T>> deltaMergeNonDeleteRecord(BufferedRecord<T> newRecord, BufferedRecord<T> existingRecord) throws IOException {
- Option<Pair<HoodieRecord, Schema>> mergedRecordAndSchema = getMergedRecord(existingRecord, newRecord, false);
- if (mergedRecordAndSchema.isEmpty()) {
- // An empty Option indicates that the output represents a delete.
- return Option.of(new BufferedRecord<>(newRecord.getRecordKey(), OrderingValues.getDefault(), null, null, HoodieOperation.DELETE));
- }
- HoodieRecord mergedRecord = mergedRecordAndSchema.get().getLeft();
- Schema mergeResultSchema = mergedRecordAndSchema.get().getRight();
- // Special handling for SENTINEL record in Expression Payload. This is returned if the condition does not match.
- if (mergedRecord.getData() == HoodieRecord.SENTINEL) {
- return Option.empty();
- }
- T combinedRecordData = recordContext.convertAvroRecord(mergedRecord.toIndexedRecord(mergeResultSchema, props).get().getData());
+ public Option<BufferedRecord<T>> deltaMergeRecords(BufferedRecord<T> newRecord, BufferedRecord<T> existingRecord) throws IOException {
+ BufferedRecord<T> mergedRecord = getMergedRecord(existingRecord, newRecord, false);
// If pre-combine does not return existing record, update it
- if (combinedRecordData != existingRecord.getRecord()) {
- // For pkless we need to use record key from existing record
- return Option.of(BufferedRecords.fromEngineRecord(combinedRecordData, mergeResultSchema, recordContext, orderingFieldNames,
- existingRecord.getRecordKey(), mergedRecord.isDelete(mergeResultSchema, props)));
+ if (mergedRecord.getRecord() != existingRecord.getRecord()) {
+ return Option.of(mergedRecord);
}
return Option.empty();
}
@Override
- public BufferedRecord<T> mergeNonDeleteRecord(BufferedRecord<T> olderRecord, BufferedRecord<T> newerRecord) throws IOException {
- Option<Pair<HoodieRecord, Schema>> mergedRecordAndSchema = getMergedRecord(olderRecord, newerRecord, true);
- if (mergedRecordAndSchema.isEmpty()) {
- return BufferedRecords.createDelete(newerRecord.getRecordKey());
- }
- HoodieRecord mergedRecord = mergedRecordAndSchema.get().getLeft();
- Schema mergeResultSchema = mergedRecordAndSchema.get().getRight();
- // Special handling for SENTINEL record in Expression Payload
- if (mergedRecord.getData() == HoodieRecord.SENTINEL) {
- return olderRecord;
- }
- if (!mergedRecord.isDelete(mergeResultSchema, props)) {
- IndexedRecord indexedRecord = (IndexedRecord) mergedRecord.getData();
- return BufferedRecords.fromEngineRecord(
- recordContext.convertAvroRecord(indexedRecord), mergeResultSchema, recordContext, orderingFieldNames, newerRecord.getRecordKey(), false);
- }
- return BufferedRecords.createDelete(newerRecord.getRecordKey());
+ public BufferedRecord<T> mergeRecords(BufferedRecord<T> olderRecord, BufferedRecord<T> newerRecord) throws IOException {
+ return getMergedRecord(olderRecord, newerRecord, true);
}
- protected Pair<HoodieRecord, HoodieRecord> getDeltaMergeRecords(BufferedRecord<T> olderRecord, BufferedRecord<T> newerRecord) {
- HoodieRecord oldHoodieRecord = constructHoodieAvroRecord(recordContext, olderRecord, payloadClass);
- HoodieRecord newHoodieRecord = constructHoodieAvroRecord(recordContext, newerRecord, payloadClass);
- return Pair.of(oldHoodieRecord, newHoodieRecord);
- }
-
- protected Pair<HoodieRecord, HoodieRecord> getFinalMergeRecords(BufferedRecord<T> olderRecord, BufferedRecord<T> newerRecord) {
- return getDeltaMergeRecords(olderRecord, newerRecord);
- }
-
- protected Option<Pair<HoodieRecord, Schema>> getMergedRecord(BufferedRecord<T> olderRecord, BufferedRecord<T> newerRecord, boolean isFinalMerge) throws IOException {
- Pair<HoodieRecord, HoodieRecord> records = isFinalMerge ? getFinalMergeRecords(olderRecord, newerRecord) : getDeltaMergeRecords(olderRecord, newerRecord);
- return recordMerger.merge(records.getLeft(), getSchemaForAvroPayloadMerge(olderRecord), records.getRight(), getSchemaForAvroPayloadMerge(newerRecord), props);
- }
-
- protected HoodieRecord constructHoodieAvroRecord(RecordContext<T> recordContext, BufferedRecord<T> bufferedRecord, String payloadClass) {
- GenericRecord record = null;
- if (!bufferedRecord.isDelete()) {
- Schema recordSchema = recordContext.getSchemaFromBufferRecord(bufferedRecord);
- record = recordContext.convertToAvroRecord(bufferedRecord.getRecord(), recordSchema);
- }
- HoodieKey hoodieKey = new HoodieKey(bufferedRecord.getRecordKey(), null);
- return new HoodieAvroRecord<>(hoodieKey,
- HoodieRecordUtils.loadPayload(payloadClass, record, bufferedRecord.getOrderingValue()), null);
- }
-
- protected Schema getSchemaForAvroPayloadMerge(BufferedRecord<T> bufferedRecord) {
- if (bufferedRecord.getSchemaId() == null) {
- return readerSchema;
- }
- return recordContext.getSchemaFromBufferRecord(bufferedRecord);
+ protected BufferedRecord<T> getMergedRecord(BufferedRecord<T> olderRecord, BufferedRecord<T> newerRecord, boolean isFinalMerge) throws IOException {
Review Comment:
Cleaned this up along with some more code now that the payload/custom merger paths are more similar
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/HoodieSparkRecordMerger.java:
##########
@@ -39,33 +41,33 @@ public HoodieRecord.HoodieRecordType getRecordType() {
* Basic handling of deletes that is used by many of the spark mergers
* returns null if merger specific logic should be used
*/
- protected Option<Pair<HoodieRecord, Schema>> handleDeletes(HoodieRecord older, Schema oldSchema, HoodieRecord newer, Schema newSchema, TypedProperties props) {
+ protected Pair<HoodieRecord, Schema> handleDeletes(HoodieRecord older, Schema oldSchema, HoodieRecord newer, Schema newSchema, TypedProperties props) {
Review Comment:
Removed this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9715] Custom merger update [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13742:
URL: https://github.com/apache/hudi/pull/13742#discussion_r2299994184
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/BufferedRecordMergerFactory.java:
##########
@@ -567,42 +553,36 @@ public Option<BufferedRecord<T>> deltaMerge(BufferedRecord<T> newRecord, Buffere
if (existingRecord == null) {
return Option.of(newRecord);
}
- if (existingRecord.isDelete() || newRecord.isDelete()) {
- if (shouldKeepNewerRecord(existingRecord, newRecord)) {
- // IMPORTANT:
- // this is needed when the fallback HoodieAvroRecordMerger got used, the merger would
- // return Option.empty when the old payload data is empty(a delete) and ignores its ordering value directly.
- return Option.of(newRecord);
- } else {
- return Option.empty();
- }
- }
- return deltaMergeNonDeleteRecord(newRecord, existingRecord);
+ return deltaMergeRecords(newRecord, existingRecord);
}
- public abstract Option<BufferedRecord<T>> deltaMergeNonDeleteRecord(BufferedRecord<T> newRecord, BufferedRecord<T> existingRecord) throws IOException;
+ public abstract Option<BufferedRecord<T>> deltaMergeRecords(BufferedRecord<T> newRecord, BufferedRecord<T> existingRecord) throws IOException;
@Override
public Option<DeleteRecord> deltaMerge(DeleteRecord deleteRecord, BufferedRecord<T> existingRecord) {
- return deltaMergeDeleteRecord(deleteRecord, existingRecord);
+ BufferedRecord<T> deleteBufferedRecord = BufferedRecords.fromDeleteRecord(deleteRecord, recordContext);
+ try {
+ Option<BufferedRecord<T>> merged = deltaMerge(deleteBufferedRecord, existingRecord);
+ // If the delete record is chosen, return an option with the delete record, otherwise return empty.
+ return merged.isPresent() ? Option.of(deleteRecord) : Option.empty();
+ } catch (IOException e) {
+ throw new HoodieIOException("Failed to process delete record", e);
+ }
}
@Override
public BufferedRecord<T> finalMerge(BufferedRecord<T> olderRecord, BufferedRecord<T> newerRecord) throws IOException {
- if (olderRecord.isDelete() || newerRecord.isDelete()) {
- if (shouldKeepNewerRecord(olderRecord, newerRecord)) {
- // IMPORTANT:
- // this is needed when the fallback HoodieAvroRecordMerger got used, the merger would
- // return Option.empty when the new payload data is empty(a delete) and ignores its ordering value directly.
- return newerRecord;
- } else {
- return olderRecord;
- }
+ if (olderRecord == null) {
Review Comment:
how could the oldRecord be null?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT] Custom merger update [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13742:
URL: https://github.com/apache/hudi/pull/13742#issuecomment-3220630079
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7988",
"triggerID" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"triggerType" : "PUSH"
}, {
"hash" : "ecb694295f7a88371f33df6de60370058670439e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7992",
"triggerID" : "ecb694295f7a88371f33df6de60370058670439e",
"triggerType" : "PUSH"
}, {
"hash" : "efd7462406be4f6d90dd5e42f152d66f683ebdf4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "efd7462406be4f6d90dd5e42f152d66f683ebdf4",
"triggerType" : "PUSH"
}, {
"hash" : "3c67160d5ceec883db5d826ff655865a3209f2f3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7994",
"triggerID" : "3c67160d5ceec883db5d826ff655865a3209f2f3",
"triggerType" : "PUSH"
}, {
"hash" : "f4f84d9f6a480923df27cfe5553d8aa65b2bbb7a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7996",
"triggerID" : "f4f84d9f6a480923df27cfe5553d8aa65b2bbb7a",
"triggerType" : "PUSH"
}, {
"hash" : "521b26bc2c60b7921e2f7aa5fffa620ab7f9a096",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "521b26bc2c60b7921e2f7aa5fffa620ab7f9a096",
"triggerType" : "PUSH"
}, {
"hash" : "22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50",
"triggerType" : "PUSH"
}, {
"hash" : "02a319ba884595a4e3b5453f366b660c1c11f767",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8007",
"triggerID" : "02a319ba884595a4e3b5453f366b660c1c11f767",
"triggerType" : "PUSH"
}, {
"hash" : "3e59b6f8b486016911c9ad14c583f6538cc036ac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8009",
"triggerID" : "3e59b6f8b486016911c9ad14c583f6538cc036ac",
"triggerType" : "PUSH"
}, {
"hash" : "003f1c81efedafa0ddd78868ce83357d8803c5e5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "003f1c81efedafa0ddd78868ce83357d8803c5e5",
"triggerType" : "PUSH"
}, {
"hash" : "40c04768d5233a38be1b4c01ce895189a4af4c3a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8013",
"triggerID" : "40c04768d5233a38be1b4c01ce895189a4af4c3a",
"triggerType" : "PUSH"
}, {
"hash" : "4a699a8dcf79e61ea3bb754e67e9cc4a32e86aa1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8084",
"triggerID" : "4a699a8dcf79e61ea3bb754e67e9cc4a32e86aa1",
"triggerType" : "PUSH"
}, {
"hash" : "fe55d8ec4923cb6c64e83ae4f32217c72599a398",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "fe55d8ec4923cb6c64e83ae4f32217c72599a398",
"triggerType" : "PUSH"
}, {
"hash" : "87cd79787dc8126b6877ba0d3407cfe8f343933a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8087",
"triggerID" : "87cd79787dc8126b6877ba0d3407cfe8f343933a",
"triggerType" : "PUSH"
}, {
"hash" : "594874ffdca647cd268b3947fa4680c57c53d679",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8088",
"triggerID" : "594874ffdca647cd268b3947fa4680c57c53d679",
"triggerType" : "PUSH"
}, {
"hash" : "7b95754f0df2c1a087a6a28722a7471ee12cecf3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8090",
"triggerID" : "7b95754f0df2c1a087a6a28722a7471ee12cecf3",
"triggerType" : "PUSH"
}, {
"hash" : "789c2da879ca66f6b9be71d1814be38b5d8cfb58",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "789c2da879ca66f6b9be71d1814be38b5d8cfb58",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* efd7462406be4f6d90dd5e42f152d66f683ebdf4 UNKNOWN
* 521b26bc2c60b7921e2f7aa5fffa620ab7f9a096 UNKNOWN
* 22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50 UNKNOWN
* 003f1c81efedafa0ddd78868ce83357d8803c5e5 UNKNOWN
* fe55d8ec4923cb6c64e83ae4f32217c72599a398 UNKNOWN
* 7b95754f0df2c1a087a6a28722a7471ee12cecf3 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8090)
* 789c2da879ca66f6b9be71d1814be38b5d8cfb58 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9715] Custom merger update [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13742:
URL: https://github.com/apache/hudi/pull/13742#discussion_r2300863078
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/HoodieSparkRecordMerger.java:
##########
@@ -39,33 +41,33 @@ public HoodieRecord.HoodieRecordType getRecordType() {
* Basic handling of deletes that is used by many of the spark mergers
* returns null if merger specific logic should be used
*/
- protected Option<Pair<HoodieRecord, Schema>> handleDeletes(HoodieRecord older, Schema oldSchema, HoodieRecord newer, Schema newSchema, TypedProperties props) {
+ protected Pair<HoodieRecord, Schema> handleDeletes(HoodieRecord older, Schema oldSchema, HoodieRecord newer, Schema newSchema, TypedProperties props) {
ValidationUtils.checkArgument(older.getRecordType() == HoodieRecord.HoodieRecordType.SPARK);
ValidationUtils.checkArgument(newer.getRecordType() == HoodieRecord.HoodieRecordType.SPARK);
if (newer instanceof HoodieSparkRecord) {
HoodieSparkRecord newSparkRecord = (HoodieSparkRecord) newer;
if (newSparkRecord.isDelete(newSchema, props)) {
// Delete record
- return Option.empty();
+ return Pair.of(new HoodieEmptyRecord<>(newer.getKey(), HoodieOperation.DELETE, OrderingValues.getDefault(), HoodieRecord.HoodieRecordType.SPARK), newSchema);
Review Comment:
There are definitely gaps with respect to event time ordering. Even the payload flow was not allowing ordering value on some deletes so it is quite inconsistent.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9715] Custom merger update [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13742:
URL: https://github.com/apache/hudi/pull/13742#discussion_r2302489488
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/BufferedRecordMergerFactory.java:
##########
@@ -338,16 +334,15 @@ public Option<DeleteRecord> deltaMerge(DeleteRecord deleteRecord, BufferedRecord
public BufferedRecord<T> finalMerge(BufferedRecord<T> olderRecord, BufferedRecord<T> newerRecord) throws IOException {
// TODO(HUDI-7843): decouple the merging logic from the merger
// and use the record merge mode to control how to merge partial updates
- Option<Pair<HoodieRecord, Schema>> mergedRecord = recordMerger.get().partialMerge(
+ Pair<HoodieRecord, Schema> mergedRecord = recordMerger.get().partialMerge(
recordContext.constructHoodieRecord(olderRecord), recordContext.getSchemaFromBufferRecord(olderRecord),
recordContext.constructHoodieRecord(newerRecord), recordContext.getSchemaFromBufferRecord(newerRecord),
readerSchema, props);
- if (mergedRecord.isPresent()
- && !mergedRecord.get().getLeft().isDelete(mergedRecord.get().getRight(), props)) {
- HoodieRecord hoodieRecord = mergedRecord.get().getLeft();
- if (!mergedRecord.get().getRight().equals(readerSchema)) {
- hoodieRecord = hoodieRecord.rewriteRecordWithNewSchema(mergedRecord.get().getRight(), null, readerSchema);
+ if (!mergedRecord.getLeft().isDelete(mergedRecord.getRight(), props)) {
+ HoodieRecord hoodieRecord = mergedRecord.getLeft();
+ if (!mergedRecord.getRight().equals(readerSchema)) {
+ hoodieRecord = hoodieRecord.rewriteRecordWithNewSchema(mergedRecord.getRight(), null, readerSchema);
Review Comment:
The lower lines will return a BufferedRecord and that will extract the ordering field from the record
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9715] Custom merger update [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13742:
URL: https://github.com/apache/hudi/pull/13742#issuecomment-3221041773
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7988",
"triggerID" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"triggerType" : "PUSH"
}, {
"hash" : "ecb694295f7a88371f33df6de60370058670439e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7992",
"triggerID" : "ecb694295f7a88371f33df6de60370058670439e",
"triggerType" : "PUSH"
}, {
"hash" : "efd7462406be4f6d90dd5e42f152d66f683ebdf4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "efd7462406be4f6d90dd5e42f152d66f683ebdf4",
"triggerType" : "PUSH"
}, {
"hash" : "3c67160d5ceec883db5d826ff655865a3209f2f3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7994",
"triggerID" : "3c67160d5ceec883db5d826ff655865a3209f2f3",
"triggerType" : "PUSH"
}, {
"hash" : "f4f84d9f6a480923df27cfe5553d8aa65b2bbb7a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7996",
"triggerID" : "f4f84d9f6a480923df27cfe5553d8aa65b2bbb7a",
"triggerType" : "PUSH"
}, {
"hash" : "521b26bc2c60b7921e2f7aa5fffa620ab7f9a096",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "521b26bc2c60b7921e2f7aa5fffa620ab7f9a096",
"triggerType" : "PUSH"
}, {
"hash" : "22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50",
"triggerType" : "PUSH"
}, {
"hash" : "02a319ba884595a4e3b5453f366b660c1c11f767",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8007",
"triggerID" : "02a319ba884595a4e3b5453f366b660c1c11f767",
"triggerType" : "PUSH"
}, {
"hash" : "3e59b6f8b486016911c9ad14c583f6538cc036ac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8009",
"triggerID" : "3e59b6f8b486016911c9ad14c583f6538cc036ac",
"triggerType" : "PUSH"
}, {
"hash" : "003f1c81efedafa0ddd78868ce83357d8803c5e5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "003f1c81efedafa0ddd78868ce83357d8803c5e5",
"triggerType" : "PUSH"
}, {
"hash" : "40c04768d5233a38be1b4c01ce895189a4af4c3a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8013",
"triggerID" : "40c04768d5233a38be1b4c01ce895189a4af4c3a",
"triggerType" : "PUSH"
}, {
"hash" : "4a699a8dcf79e61ea3bb754e67e9cc4a32e86aa1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8084",
"triggerID" : "4a699a8dcf79e61ea3bb754e67e9cc4a32e86aa1",
"triggerType" : "PUSH"
}, {
"hash" : "fe55d8ec4923cb6c64e83ae4f32217c72599a398",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "fe55d8ec4923cb6c64e83ae4f32217c72599a398",
"triggerType" : "PUSH"
}, {
"hash" : "87cd79787dc8126b6877ba0d3407cfe8f343933a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8087",
"triggerID" : "87cd79787dc8126b6877ba0d3407cfe8f343933a",
"triggerType" : "PUSH"
}, {
"hash" : "594874ffdca647cd268b3947fa4680c57c53d679",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8088",
"triggerID" : "594874ffdca647cd268b3947fa4680c57c53d679",
"triggerType" : "PUSH"
}, {
"hash" : "7b95754f0df2c1a087a6a28722a7471ee12cecf3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8090",
"triggerID" : "7b95754f0df2c1a087a6a28722a7471ee12cecf3",
"triggerType" : "PUSH"
}, {
"hash" : "789c2da879ca66f6b9be71d1814be38b5d8cfb58",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8102",
"triggerID" : "789c2da879ca66f6b9be71d1814be38b5d8cfb58",
"triggerType" : "PUSH"
}, {
"hash" : "63f33fae3e22a9f80de51b2a6e1ed0427f631d0b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8103",
"triggerID" : "63f33fae3e22a9f80de51b2a6e1ed0427f631d0b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* efd7462406be4f6d90dd5e42f152d66f683ebdf4 UNKNOWN
* 521b26bc2c60b7921e2f7aa5fffa620ab7f9a096 UNKNOWN
* 22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50 UNKNOWN
* 003f1c81efedafa0ddd78868ce83357d8803c5e5 UNKNOWN
* fe55d8ec4923cb6c64e83ae4f32217c72599a398 UNKNOWN
* 63f33fae3e22a9f80de51b2a6e1ed0427f631d0b Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8103)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9715] Custom merger update [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13742:
URL: https://github.com/apache/hudi/pull/13742#discussion_r2302730327
##########
hudi-common/src/main/java/org/apache/hudi/common/model/HoodieRecordMerger.java:
##########
@@ -63,8 +63,15 @@ public interface HoodieRecordMerger extends Serializable {
* It'd be associative operation: f(a, f(b, c)) = f(f(a, b), c) (which we can translate as having 3 versions A, B, C
* of the single record, both orders of operations applications have to yield the same result)
* This method takes only full records for merging.
+ *
+ * @param older Older record in terms of commit time ordering.
+ * @param oldSchema The schema of the older record.
+ * @param newer Newer record in terms of commit time ordering.
+ * @param newSchema The schema of the newer record.
+ * @param props The additional properties for the merging operation.
+ * @return The merged record and schema. The record is expected to be non-null. If the record represents a deletion, the operation must be set as {@link HoodieOperation#DELETE}.
*/
- Option<Pair<HoodieRecord, Schema>> merge(HoodieRecord older, Schema oldSchema, HoodieRecord newer, Schema newSchema, TypedProperties props) throws IOException;
+ Pair<HoodieRecord, Schema> merge(HoodieRecord older, Schema oldSchema, HoodieRecord newer, Schema newSchema, TypedProperties props) throws IOException;
Review Comment:
Discussed offline and for payload based merging, the T should be avro `IndexdRecord`, either think though a way to invoke the deprecated payload API or just ignore it and impl event time based merging on it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9715] Custom merger update [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13742:
URL: https://github.com/apache/hudi/pull/13742#discussion_r2308923080
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/DefaultSparkRecordMerger.java:
##########
@@ -37,46 +35,38 @@
*/
public class DefaultSparkRecordMerger extends HoodieSparkRecordMerger {
- private String[] orderingFields;
-
@Override
public String getMergingStrategy() {
return HoodieRecordMerger.EVENT_TIME_BASED_MERGE_STRATEGY_UUID;
}
@Override
- public Option<Pair<HoodieRecord, Schema>> merge(HoodieRecord older, Schema oldSchema, HoodieRecord newer, Schema newSchema, TypedProperties props) throws IOException {
- Option<Pair<HoodieRecord, Schema>> deleteHandlingResult = handleDeletes(older, oldSchema, newer, newSchema, props);
- if (deleteHandlingResult != null) {
- return deleteHandlingResult;
- }
-
- if (orderingFields == null) {
- orderingFields = ConfigUtils.getOrderingFields(props);
- }
- if (older.getOrderingValue(oldSchema, props, orderingFields).compareTo(newer.getOrderingValue(newSchema, props, orderingFields)) > 0) {
- return Option.of(Pair.of(older, oldSchema));
+ public <T> BufferedRecord<T> merge(BufferedRecord<T> older, BufferedRecord<T> newer, RecordContext<T> recordContext, TypedProperties props) throws IOException {
+ if (older.getOrderingValue().compareTo(newer.getOrderingValue()) > 0) {
Review Comment:
yeah, take a feference of the current `BufferedRecordMerger#shouldKeepNewerRecord`, might need to check all the `HoodieRecordMerger` impls.
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/DefaultSparkRecordMerger.java:
##########
@@ -37,46 +35,38 @@
*/
public class DefaultSparkRecordMerger extends HoodieSparkRecordMerger {
- private String[] orderingFields;
-
@Override
public String getMergingStrategy() {
return HoodieRecordMerger.EVENT_TIME_BASED_MERGE_STRATEGY_UUID;
}
@Override
- public Option<Pair<HoodieRecord, Schema>> merge(HoodieRecord older, Schema oldSchema, HoodieRecord newer, Schema newSchema, TypedProperties props) throws IOException {
- Option<Pair<HoodieRecord, Schema>> deleteHandlingResult = handleDeletes(older, oldSchema, newer, newSchema, props);
- if (deleteHandlingResult != null) {
- return deleteHandlingResult;
- }
-
- if (orderingFields == null) {
- orderingFields = ConfigUtils.getOrderingFields(props);
- }
- if (older.getOrderingValue(oldSchema, props, orderingFields).compareTo(newer.getOrderingValue(newSchema, props, orderingFields)) > 0) {
- return Option.of(Pair.of(older, oldSchema));
+ public <T> BufferedRecord<T> merge(BufferedRecord<T> older, BufferedRecord<T> newer, RecordContext<T> recordContext, TypedProperties props) throws IOException {
+ if (older.getOrderingValue().compareTo(newer.getOrderingValue()) > 0) {
Review Comment:
yeah, take a reference of the current `BufferedRecordMerger#shouldKeepNewerRecord`, might need to check all the `HoodieRecordMerger` impls.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9715] Custom merger update [hudi]
Posted by "cshuo (via GitHub)" <gi...@apache.org>.
cshuo commented on code in PR #13742:
URL: https://github.com/apache/hudi/pull/13742#discussion_r2300957205
##########
hudi-spark-datasource/hudi-spark/src/test/java/org/apache/hudi/io/CustomMerger.java:
##########
@@ -0,0 +1,63 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.io;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordMerger;
+import org.apache.hudi.common.util.collection.Pair;
+
+import org.apache.avro.Schema;
+
+import java.io.IOException;
+import java.util.UUID;
+
+public class CustomMerger implements HoodieRecordMerger {
+ private static final String STRATEGY_ID = UUID.randomUUID().toString();
+
+ public static String getStrategyId() {
+ return STRATEGY_ID;
+ }
+
+ @Override
+ public Pair<HoodieRecord, Schema> merge(HoodieRecord older, Schema oldSchema, HoodieRecord newer, Schema newSchema, TypedProperties props) throws IOException {
+ Long olderTimestamp = (Long) older.getOrderingValue(oldSchema, props, new String[0]);
+ Long newerTimestamp = (Long) newer.getOrderingValue(oldSchema, props, new String[0]);
Review Comment:
`oldSchema` -> `newSchema`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(merger): [RFC-101] Update HoodieRecordMerger interface and implementations [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13742:
URL: https://github.com/apache/hudi/pull/13742#issuecomment-3305129053
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "94aa9a669d2fe1fb4919a4735fab2665553b63c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8258",
"triggerID" : "94aa9a669d2fe1fb4919a4735fab2665553b63c0",
"triggerType" : "PUSH"
}, {
"hash" : "94aa9a669d2fe1fb4919a4735fab2665553b63c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8258",
"triggerID" : "3237521433",
"triggerType" : "MANUAL"
}, {
"hash" : "57b338240d2ac6faa9ec0a329f43f13e28019eb8",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8564",
"triggerID" : "57b338240d2ac6faa9ec0a329f43f13e28019eb8",
"triggerType" : "PUSH"
}, {
"hash" : "57b338240d2ac6faa9ec0a329f43f13e28019eb8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3305128927",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 57b338240d2ac6faa9ec0a329f43f13e28019eb8 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8564)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT] Custom merger update [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13742:
URL: https://github.com/apache/hudi/pull/13742#issuecomment-3217422381
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7988",
"triggerID" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"triggerType" : "PUSH"
}, {
"hash" : "ecb694295f7a88371f33df6de60370058670439e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7992",
"triggerID" : "ecb694295f7a88371f33df6de60370058670439e",
"triggerType" : "PUSH"
}, {
"hash" : "efd7462406be4f6d90dd5e42f152d66f683ebdf4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "efd7462406be4f6d90dd5e42f152d66f683ebdf4",
"triggerType" : "PUSH"
}, {
"hash" : "3c67160d5ceec883db5d826ff655865a3209f2f3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7994",
"triggerID" : "3c67160d5ceec883db5d826ff655865a3209f2f3",
"triggerType" : "PUSH"
}, {
"hash" : "f4f84d9f6a480923df27cfe5553d8aa65b2bbb7a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7996",
"triggerID" : "f4f84d9f6a480923df27cfe5553d8aa65b2bbb7a",
"triggerType" : "PUSH"
}, {
"hash" : "521b26bc2c60b7921e2f7aa5fffa620ab7f9a096",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "521b26bc2c60b7921e2f7aa5fffa620ab7f9a096",
"triggerType" : "PUSH"
}, {
"hash" : "22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50",
"triggerType" : "PUSH"
}, {
"hash" : "02a319ba884595a4e3b5453f366b660c1c11f767",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8007",
"triggerID" : "02a319ba884595a4e3b5453f366b660c1c11f767",
"triggerType" : "PUSH"
}, {
"hash" : "3e59b6f8b486016911c9ad14c583f6538cc036ac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8009",
"triggerID" : "3e59b6f8b486016911c9ad14c583f6538cc036ac",
"triggerType" : "PUSH"
}, {
"hash" : "003f1c81efedafa0ddd78868ce83357d8803c5e5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "003f1c81efedafa0ddd78868ce83357d8803c5e5",
"triggerType" : "PUSH"
}, {
"hash" : "40c04768d5233a38be1b4c01ce895189a4af4c3a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8013",
"triggerID" : "40c04768d5233a38be1b4c01ce895189a4af4c3a",
"triggerType" : "PUSH"
}, {
"hash" : "4a699a8dcf79e61ea3bb754e67e9cc4a32e86aa1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4a699a8dcf79e61ea3bb754e67e9cc4a32e86aa1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* efd7462406be4f6d90dd5e42f152d66f683ebdf4 UNKNOWN
* 521b26bc2c60b7921e2f7aa5fffa620ab7f9a096 UNKNOWN
* 22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50 UNKNOWN
* 003f1c81efedafa0ddd78868ce83357d8803c5e5 UNKNOWN
* 40c04768d5233a38be1b4c01ce895189a4af4c3a Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8013)
* 4a699a8dcf79e61ea3bb754e67e9cc4a32e86aa1 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9715] Custom merger update [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13742:
URL: https://github.com/apache/hudi/pull/13742#discussion_r2308910073
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/DefaultSparkRecordMerger.java:
##########
@@ -37,46 +35,38 @@
*/
public class DefaultSparkRecordMerger extends HoodieSparkRecordMerger {
- private String[] orderingFields;
-
@Override
public String getMergingStrategy() {
return HoodieRecordMerger.EVENT_TIME_BASED_MERGE_STRATEGY_UUID;
}
@Override
- public Option<Pair<HoodieRecord, Schema>> merge(HoodieRecord older, Schema oldSchema, HoodieRecord newer, Schema newSchema, TypedProperties props) throws IOException {
- Option<Pair<HoodieRecord, Schema>> deleteHandlingResult = handleDeletes(older, oldSchema, newer, newSchema, props);
- if (deleteHandlingResult != null) {
- return deleteHandlingResult;
- }
-
- if (orderingFields == null) {
- orderingFields = ConfigUtils.getOrderingFields(props);
- }
- if (older.getOrderingValue(oldSchema, props, orderingFields).compareTo(newer.getOrderingValue(newSchema, props, orderingFields)) > 0) {
- return Option.of(Pair.of(older, oldSchema));
+ public <T> BufferedRecord<T> merge(BufferedRecord<T> older, BufferedRecord<T> newer, RecordContext<T> recordContext, TypedProperties props) throws IOException {
+ if (older.getOrderingValue().compareTo(newer.getOrderingValue()) > 0) {
Review Comment:
Something like check if the records are commit time ordered deletes before this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9715] Custom merger update [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on PR #13742:
URL: https://github.com/apache/hudi/pull/13742#issuecomment-3237521433
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9715] Custom merger update [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13742:
URL: https://github.com/apache/hudi/pull/13742#issuecomment-3237652726
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7988",
"triggerID" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"triggerType" : "PUSH"
}, {
"hash" : "ecb694295f7a88371f33df6de60370058670439e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7992",
"triggerID" : "ecb694295f7a88371f33df6de60370058670439e",
"triggerType" : "PUSH"
}, {
"hash" : "efd7462406be4f6d90dd5e42f152d66f683ebdf4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "efd7462406be4f6d90dd5e42f152d66f683ebdf4",
"triggerType" : "PUSH"
}, {
"hash" : "3c67160d5ceec883db5d826ff655865a3209f2f3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7994",
"triggerID" : "3c67160d5ceec883db5d826ff655865a3209f2f3",
"triggerType" : "PUSH"
}, {
"hash" : "f4f84d9f6a480923df27cfe5553d8aa65b2bbb7a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7996",
"triggerID" : "f4f84d9f6a480923df27cfe5553d8aa65b2bbb7a",
"triggerType" : "PUSH"
}, {
"hash" : "521b26bc2c60b7921e2f7aa5fffa620ab7f9a096",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "521b26bc2c60b7921e2f7aa5fffa620ab7f9a096",
"triggerType" : "PUSH"
}, {
"hash" : "22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50",
"triggerType" : "PUSH"
}, {
"hash" : "02a319ba884595a4e3b5453f366b660c1c11f767",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8007",
"triggerID" : "02a319ba884595a4e3b5453f366b660c1c11f767",
"triggerType" : "PUSH"
}, {
"hash" : "3e59b6f8b486016911c9ad14c583f6538cc036ac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8009",
"triggerID" : "3e59b6f8b486016911c9ad14c583f6538cc036ac",
"triggerType" : "PUSH"
}, {
"hash" : "003f1c81efedafa0ddd78868ce83357d8803c5e5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "003f1c81efedafa0ddd78868ce83357d8803c5e5",
"triggerType" : "PUSH"
}, {
"hash" : "40c04768d5233a38be1b4c01ce895189a4af4c3a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8013",
"triggerID" : "40c04768d5233a38be1b4c01ce895189a4af4c3a",
"triggerType" : "PUSH"
}, {
"hash" : "4a699a8dcf79e61ea3bb754e67e9cc4a32e86aa1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8084",
"triggerID" : "4a699a8dcf79e61ea3bb754e67e9cc4a32e86aa1",
"triggerType" : "PUSH"
}, {
"hash" : "fe55d8ec4923cb6c64e83ae4f32217c72599a398",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "fe55d8ec4923cb6c64e83ae4f32217c72599a398",
"triggerType" : "PUSH"
}, {
"hash" : "87cd79787dc8126b6877ba0d3407cfe8f343933a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8087",
"triggerID" : "87cd79787dc8126b6877ba0d3407cfe8f343933a",
"triggerType" : "PUSH"
}, {
"hash" : "594874ffdca647cd268b3947fa4680c57c53d679",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8088",
"triggerID" : "594874ffdca647cd268b3947fa4680c57c53d679",
"triggerType" : "PUSH"
}, {
"hash" : "7b95754f0df2c1a087a6a28722a7471ee12cecf3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8090",
"triggerID" : "7b95754f0df2c1a087a6a28722a7471ee12cecf3",
"triggerType" : "PUSH"
}, {
"hash" : "789c2da879ca66f6b9be71d1814be38b5d8cfb58",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8102",
"triggerID" : "789c2da879ca66f6b9be71d1814be38b5d8cfb58",
"triggerType" : "PUSH"
}, {
"hash" : "63f33fae3e22a9f80de51b2a6e1ed0427f631d0b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8103",
"triggerID" : "63f33fae3e22a9f80de51b2a6e1ed0427f631d0b",
"triggerType" : "PUSH"
}, {
"hash" : "e04c466fa81e9bcfaabe36825127098f5d1bb890",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8224",
"triggerID" : "e04c466fa81e9bcfaabe36825127098f5d1bb890",
"triggerType" : "PUSH"
}, {
"hash" : "cf574f46a1783a39b2074542aad9539ec3c4f36a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8235",
"triggerID" : "cf574f46a1783a39b2074542aad9539ec3c4f36a",
"triggerType" : "PUSH"
}, {
"hash" : "2eb81e923adfa77176f137c442143deaac9e9916",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2eb81e923adfa77176f137c442143deaac9e9916",
"triggerType" : "PUSH"
}, {
"hash" : "12dc3e3eaf746bb1f2bc3f4e93fa14d9adb0f6fd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8244",
"triggerID" : "12dc3e3eaf746bb1f2bc3f4e93fa14d9adb0f6fd",
"triggerType" : "PUSH"
}, {
"hash" : "af5ab73103f3af882412bbe99c9322165e8941a0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "af5ab73103f3af882412bbe99c9322165e8941a0",
"triggerType" : "PUSH"
}, {
"hash" : "a9c28c4d913644fea23d57a5dc2224f36fd2b63e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8248",
"triggerID" : "a9c28c4d913644fea23d57a5dc2224f36fd2b63e",
"triggerType" : "PUSH"
}, {
"hash" : "98381a6bd2bcccf3da08fb992e599fe1165f2ddb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8252",
"triggerID" : "98381a6bd2bcccf3da08fb992e599fe1165f2ddb",
"triggerType" : "PUSH"
}, {
"hash" : "94aa9a669d2fe1fb4919a4735fab2665553b63c0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8258",
"triggerID" : "94aa9a669d2fe1fb4919a4735fab2665553b63c0",
"triggerType" : "PUSH"
}, {
"hash" : "94aa9a669d2fe1fb4919a4735fab2665553b63c0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8258",
"triggerID" : "3237521433",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* efd7462406be4f6d90dd5e42f152d66f683ebdf4 UNKNOWN
* 521b26bc2c60b7921e2f7aa5fffa620ab7f9a096 UNKNOWN
* 22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50 UNKNOWN
* 003f1c81efedafa0ddd78868ce83357d8803c5e5 UNKNOWN
* fe55d8ec4923cb6c64e83ae4f32217c72599a398 UNKNOWN
* 2eb81e923adfa77176f137c442143deaac9e9916 UNKNOWN
* af5ab73103f3af882412bbe99c9322165e8941a0 UNKNOWN
* 94aa9a669d2fe1fb4919a4735fab2665553b63c0 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8258)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT] Custom merger update [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13742:
URL: https://github.com/apache/hudi/pull/13742#issuecomment-3208565841
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7988",
"triggerID" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"triggerType" : "PUSH"
}, {
"hash" : "ecb694295f7a88371f33df6de60370058670439e",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7992",
"triggerID" : "ecb694295f7a88371f33df6de60370058670439e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a9251b050ad82bceba70f95140ec4f1c5ac5a594 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7988)
* ecb694295f7a88371f33df6de60370058670439e Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7992)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT] Custom merger update [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13742:
URL: https://github.com/apache/hudi/pull/13742#issuecomment-3208563207
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7988",
"triggerID" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"triggerType" : "PUSH"
}, {
"hash" : "ecb694295f7a88371f33df6de60370058670439e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ecb694295f7a88371f33df6de60370058670439e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a9251b050ad82bceba70f95140ec4f1c5ac5a594 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7988)
* ecb694295f7a88371f33df6de60370058670439e UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT] Custom merger update [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13742:
URL: https://github.com/apache/hudi/pull/13742#issuecomment-3218234023
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7988",
"triggerID" : "a9251b050ad82bceba70f95140ec4f1c5ac5a594",
"triggerType" : "PUSH"
}, {
"hash" : "ecb694295f7a88371f33df6de60370058670439e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7992",
"triggerID" : "ecb694295f7a88371f33df6de60370058670439e",
"triggerType" : "PUSH"
}, {
"hash" : "efd7462406be4f6d90dd5e42f152d66f683ebdf4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "efd7462406be4f6d90dd5e42f152d66f683ebdf4",
"triggerType" : "PUSH"
}, {
"hash" : "3c67160d5ceec883db5d826ff655865a3209f2f3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7994",
"triggerID" : "3c67160d5ceec883db5d826ff655865a3209f2f3",
"triggerType" : "PUSH"
}, {
"hash" : "f4f84d9f6a480923df27cfe5553d8aa65b2bbb7a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7996",
"triggerID" : "f4f84d9f6a480923df27cfe5553d8aa65b2bbb7a",
"triggerType" : "PUSH"
}, {
"hash" : "521b26bc2c60b7921e2f7aa5fffa620ab7f9a096",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "521b26bc2c60b7921e2f7aa5fffa620ab7f9a096",
"triggerType" : "PUSH"
}, {
"hash" : "22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50",
"triggerType" : "PUSH"
}, {
"hash" : "02a319ba884595a4e3b5453f366b660c1c11f767",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8007",
"triggerID" : "02a319ba884595a4e3b5453f366b660c1c11f767",
"triggerType" : "PUSH"
}, {
"hash" : "3e59b6f8b486016911c9ad14c583f6538cc036ac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8009",
"triggerID" : "3e59b6f8b486016911c9ad14c583f6538cc036ac",
"triggerType" : "PUSH"
}, {
"hash" : "003f1c81efedafa0ddd78868ce83357d8803c5e5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "003f1c81efedafa0ddd78868ce83357d8803c5e5",
"triggerType" : "PUSH"
}, {
"hash" : "40c04768d5233a38be1b4c01ce895189a4af4c3a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8013",
"triggerID" : "40c04768d5233a38be1b4c01ce895189a4af4c3a",
"triggerType" : "PUSH"
}, {
"hash" : "4a699a8dcf79e61ea3bb754e67e9cc4a32e86aa1",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8084",
"triggerID" : "4a699a8dcf79e61ea3bb754e67e9cc4a32e86aa1",
"triggerType" : "PUSH"
}, {
"hash" : "fe55d8ec4923cb6c64e83ae4f32217c72599a398",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "fe55d8ec4923cb6c64e83ae4f32217c72599a398",
"triggerType" : "PUSH"
}, {
"hash" : "87cd79787dc8126b6877ba0d3407cfe8f343933a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "87cd79787dc8126b6877ba0d3407cfe8f343933a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* efd7462406be4f6d90dd5e42f152d66f683ebdf4 UNKNOWN
* 521b26bc2c60b7921e2f7aa5fffa620ab7f9a096 UNKNOWN
* 22fcec855b4e8c7e4aff80dfa8f26c4613ac0e50 UNKNOWN
* 003f1c81efedafa0ddd78868ce83357d8803c5e5 UNKNOWN
* 4a699a8dcf79e61ea3bb754e67e9cc4a32e86aa1 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8084)
* fe55d8ec4923cb6c64e83ae4f32217c72599a398 UNKNOWN
* 87cd79787dc8126b6877ba0d3407cfe8f343933a UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org