You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "the-other-tim-brown (via GitHub)" <gi...@apache.org> on 2025/08/09 01:25:01 UTC
[PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
the-other-tim-brown opened a new pull request, #13699:
URL: https://github.com/apache/hudi/pull/13699
### Change Logs
_Describe context and summary for this change. Highlight if any code was copied._
### Impact
_Describe any public API or user-facing feature change or any performance impact._
### Risk level (write none, low medium or high below)
_If medium or high, explain what verification was done to mitigate the risks._
### Documentation Update
_Describe any necessary documentation update if there is any new feature, config, or user-facing change. If not, put "none"._
- _The config description must be updated if new configs are added or the default value of the configs are changed_
- _Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
ticket number here and follow the [instruction](https://hudi.apache.org/contribute/developer-setup#website) to make
changes to the website._
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9622] Add implementation of MergeHandle backed by the HoodieFileGroupReader [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13699:
URL: https://github.com/apache/hudi/pull/13699#discussion_r2278052272
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/UpdateProcessor.java:
##########
@@ -80,17 +101,56 @@ public BufferedRecord<T> processUpdate(String recordKey, BufferedRecord<T> previ
}
return null;
} else {
- T prevRow = previousRecord != null ? previousRecord.getRecord() : null;
- T mergedRow = mergedRecord.getRecord();
- if (prevRow != null && prevRow != mergedRow) {
- mergedRecord.setHoodieOperation(HoodieOperation.UPDATE_AFTER);
- readStats.incrementNumUpdates();
- } else if (prevRow == null) {
- mergedRecord.setHoodieOperation(HoodieOperation.INSERT);
- readStats.incrementNumInserts();
+ return handleNonDeletes(previousRecord, mergedRecord);
+ }
+ }
+
+ protected BufferedRecord<T> handleNonDeletes(BufferedRecord<T> previousRecord, BufferedRecord<T> mergedRecord) {
Review Comment:
Just added some UTs here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9622] Add implementation of MergeHandle backed by the HoodieFileGroupReader [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13699:
URL: https://github.com/apache/hudi/pull/13699#discussion_r2277906103
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/FileGroupReaderBasedMergeHandle.java:
##########
@@ -198,8 +270,10 @@ public void doMerge() {
}
// Writes the record
try {
- writeToFile(record.getKey(), record, writeSchemaWithMetaFields,
- config.getPayloadConfig().getProps(), preserveMetadata);
+ // if the record is not being updated and is not a new insert for the file group, we must preserve the existing record metadata.
+ boolean shouldPreserveRecordMetadata = preserveMetadata || record.getOperation() == null;
Review Comment:
The record operation is set in the UpdateProcessor.
You are right, the case A is the only one where there is no operation set. For case B, records being updated implies that it is not the same as the previous record so in that case you will want to update the meta-fields. If it is not updated, it will not have an update operation associated with it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9622] Add implementation of MergeHandle backed by the HoodieFileGroupReader [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13699:
URL: https://github.com/apache/hudi/pull/13699#discussion_r2277924704
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/WriteStatus.java:
##########
@@ -109,6 +109,17 @@ public void markSuccess(HoodieRecord record, Option<Map<String, String>> optiona
updateStatsForSuccess(optionalRecordMetadata);
}
+ /**
+ * Allows the writer to manually add record delegates to the index stats.
+ */
+ public void manuallyTrackSuccess() {
Review Comment:
I'm not sure of how we could do this just for the CoW merge case, any ideas for how it can be done?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9622] Add implementation of MergeHandle backed by the HoodieFileGroupReader [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #13699:
URL: https://github.com/apache/hudi/pull/13699#discussion_r2279136874
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/BufferedRecordMergerFactory.java:
##########
@@ -426,6 +428,16 @@ protected Pair<HoodieRecord, HoodieRecord> getFinalMergeRecords(BufferedRecord<T
HoodieRecord newHoodieRecord = constructHoodieAvroRecord(recordContext, newerRecord, payloadClass);
return Pair.of(oldHoodieRecord, newHoodieRecord);
}
+
+ @Override
+ protected Option<Pair<HoodieRecord, Schema>> getMergedRecord(BufferedRecord<T> olderRecord, BufferedRecord<T> newerRecord, boolean isFinalMerge) throws IOException {
Review Comment:
can we file a follow up jira on this please. we don't need to block this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9622] Add implementation of MergeHandle backed by the HoodieFileGroupReader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13699:
URL: https://github.com/apache/hudi/pull/13699#issuecomment-3188930934
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7661",
"triggerID" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"triggerType" : "PUSH"
}, {
"hash" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7666",
"triggerID" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"triggerType" : "PUSH"
}, {
"hash" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7667",
"triggerID" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"triggerType" : "PUSH"
}, {
"hash" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7672",
"triggerID" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"triggerType" : "PUSH"
}, {
"hash" : "965136302f1188e5123de37734c927dc55801b24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7691",
"triggerID" : "965136302f1188e5123de37734c927dc55801b24",
"triggerType" : "PUSH"
}, {
"hash" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7693",
"triggerID" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"triggerType" : "PUSH"
}, {
"hash" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7717",
"triggerID" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"triggerType" : "PUSH"
}, {
"hash" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"triggerType" : "PUSH"
}, {
"hash" : "47f8303d6e61504275ecfddf553e356427e6c822",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7718",
"triggerID" : "47f8303d6e61504275ecfddf553e356427e6c822",
"triggerType" : "PUSH"
}, {
"hash" : "fb191ecf48d13cd81412535f7f8c9ed553d86517",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7725",
"triggerID" : "fb191ecf48d13cd81412535f7f8c9ed553d86517",
"triggerType" : "PUSH"
}, {
"hash" : "064912d3a54714935f2ee77d55cc66816166300f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7727",
"triggerID" : "064912d3a54714935f2ee77d55cc66816166300f",
"triggerType" : "PUSH"
}, {
"hash" : "77a2b45fe13dafc51f7bebb2ba859d40291ccc10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7736",
"triggerID" : "77a2b45fe13dafc51f7bebb2ba859d40291ccc10",
"triggerType" : "PUSH"
}, {
"hash" : "e4eda0e51fee0e8de5d57ea7b38dc642f53aa95d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7758",
"triggerID" : "e4eda0e51fee0e8de5d57ea7b38dc642f53aa95d",
"triggerType" : "PUSH"
}, {
"hash" : "6cc840c51fcecde95d2187fb6fd217b50fa82806",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6cc840c51fcecde95d2187fb6fd217b50fa82806",
"triggerType" : "PUSH"
}, {
"hash" : "31dab8ce9abbe4fc80e6cd81a0e433a039ca3e1d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7760",
"triggerID" : "31dab8ce9abbe4fc80e6cd81a0e433a039ca3e1d",
"triggerType" : "PUSH"
}, {
"hash" : "c04fd95dcc94146a9146c347f4f204656f3d8308",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7763",
"triggerID" : "c04fd95dcc94146a9146c347f4f204656f3d8308",
"triggerType" : "PUSH"
}, {
"hash" : "324147913d47ba8f2555e1991ce59905e5dc0b3f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7767",
"triggerID" : "324147913d47ba8f2555e1991ce59905e5dc0b3f",
"triggerType" : "PUSH"
}, {
"hash" : "3f177d42b831b30f4bd590b057dcab6ded78b634",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7774",
"triggerID" : "3f177d42b831b30f4bd590b057dcab6ded78b634",
"triggerType" : "PUSH"
}, {
"hash" : "8d6d2c06ec82d1072251ac12ddf21a890ddd5bda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7776",
"triggerID" : "8d6d2c06ec82d1072251ac12ddf21a890ddd5bda",
"triggerType" : "PUSH"
}, {
"hash" : "77589738d73b044385f5aa6f7ef87b421f06043b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "77589738d73b044385f5aa6f7ef87b421f06043b",
"triggerType" : "PUSH"
}, {
"hash" : "ad0d7ef0fbc525f880d3baadef5a22196ab97b32",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7781",
"triggerID" : "ad0d7ef0fbc525f880d3baadef5a22196ab97b32",
"triggerType" : "PUSH"
}, {
"hash" : "0db72caa3a6c7d0f0899f243ea4b8975de79e02c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7787",
"triggerID" : "0db72caa3a6c7d0f0899f243ea4b8975de79e02c",
"triggerType" : "PUSH"
}, {
"hash" : "a01ce748dc2d4cd1fd4858d1ecac8a486989e880",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7797",
"triggerID" : "a01ce748dc2d4cd1fd4858d1ecac8a486989e880",
"triggerType" : "PUSH"
}, {
"hash" : "3f35151e28bfa1dd50b31c6e5379d9769627afd9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7799",
"triggerID" : "3f35151e28bfa1dd50b31c6e5379d9769627afd9",
"triggerType" : "PUSH"
}, {
"hash" : "b0054c8fd329489d9a1e9535d9866edffc52fdd0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b0054c8fd329489d9a1e9535d9866edffc52fdd0",
"triggerType" : "PUSH"
}, {
"hash" : "363e54341420cc217bf5af62de1e09ff370f75d2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "363e54341420cc217bf5af62de1e09ff370f75d2",
"triggerType" : "PUSH"
}, {
"hash" : "a893da375eb9c65edabbd22965296eac9b3714f4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a893da375eb9c65edabbd22965296eac9b3714f4",
"triggerType" : "PUSH"
}, {
"hash" : "a7c2756e6a293cbba99bd4e821881e6b961c138f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7804",
"triggerID" : "a7c2756e6a293cbba99bd4e821881e6b961c138f",
"triggerType" : "PUSH"
}, {
"hash" : "4a140fffa35893a13f2e7ac8365d60d51a2d0870",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7813",
"triggerID" : "4a140fffa35893a13f2e7ac8365d60d51a2d0870",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 50022dcc45b0f511dd69706aeaf6464a51be49e2 UNKNOWN
* 6cc840c51fcecde95d2187fb6fd217b50fa82806 UNKNOWN
* 77589738d73b044385f5aa6f7ef87b421f06043b UNKNOWN
* b0054c8fd329489d9a1e9535d9866edffc52fdd0 UNKNOWN
* 363e54341420cc217bf5af62de1e09ff370f75d2 UNKNOWN
* a893da375eb9c65edabbd22965296eac9b3714f4 UNKNOWN
* 4a140fffa35893a13f2e7ac8365d60d51a2d0870 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7813)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9622] Add implementation of MergeHandle backed by the HoodieFileGroupReader [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on PR #13699:
URL: https://github.com/apache/hudi/pull/13699#issuecomment-3192751981
<img width="1492" height="532" alt="image" src="https://github.com/user-attachments/assets/20d71790-d005-4323-bc70-a51ba501a78a" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9622] Add implementation of MergeHandle backed by the HoodieFileGroupReader [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #13699:
URL: https://github.com/apache/hudi/pull/13699#discussion_r2279080060
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/FileGroupReaderBasedMergeHandle.java:
##########
@@ -198,8 +270,10 @@ public void doMerge() {
}
// Writes the record
try {
- writeToFile(record.getKey(), record, writeSchemaWithMetaFields,
- config.getPayloadConfig().getProps(), preserveMetadata);
+ // if the record is not being updated and is not a new insert for the file group, we must preserve the existing record metadata.
+ boolean shouldPreserveRecordMetadata = preserveMetadata || record.getOperation() == null;
Review Comment:
thanks
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/index/HoodieIndexUtils.java:
##########
@@ -526,14 +524,14 @@ public static <R> HoodieData<HoodieRecord<R>> mergeForPartitionUpdatesIfNeeded(
.distinct(updatedConfig.getGlobalIndexReconcileParallelism());
// define the buffered record merger.
ReaderContextFactory<R> readerContextFactory = (ReaderContextFactory<R>) hoodieTable.getContext()
- .getReaderContextFactoryForWrite(hoodieTable.getMetaClient(), config.getRecordMerger().getRecordType(), config.getProps());
+ .getReaderContextFactoryForWrite(hoodieTable.getMetaClient(), config.getRecordMerger().getRecordType(), config.getProps(), true);
Review Comment:
sg
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/UpdateProcessor.java:
##########
@@ -80,17 +101,56 @@ public BufferedRecord<T> processUpdate(String recordKey, BufferedRecord<T> previ
}
return null;
} else {
- T prevRow = previousRecord != null ? previousRecord.getRecord() : null;
- T mergedRow = mergedRecord.getRecord();
- if (prevRow != null && prevRow != mergedRow) {
- mergedRecord.setHoodieOperation(HoodieOperation.UPDATE_AFTER);
- readStats.incrementNumUpdates();
- } else if (prevRow == null) {
- mergedRecord.setHoodieOperation(HoodieOperation.INSERT);
- readStats.incrementNumInserts();
+ return handleNonDeletes(previousRecord, mergedRecord);
+ }
+ }
+
+ protected BufferedRecord<T> handleNonDeletes(BufferedRecord<T> previousRecord, BufferedRecord<T> mergedRecord) {
+ T prevRow = previousRecord != null ? previousRecord.getRecord() : null;
+ T mergedRow = mergedRecord.getRecord();
+ if (prevRow != null && prevRow != mergedRow) {
+ mergedRecord.setHoodieOperation(HoodieOperation.UPDATE_AFTER);
+ readStats.incrementNumUpdates();
+ } else if (prevRow == null) {
+ mergedRecord.setHoodieOperation(HoodieOperation.INSERT);
+ readStats.incrementNumInserts();
+ }
+ return mergedRecord.seal(readerContext.getRecordContext());
+ }
+ }
+
+ class PayloadUpdateProcessor<T> extends StandardUpdateProcessor<T> {
Review Comment:
In the interest of time to land the patch, we can go ahead if CI is green. But would recommend filing one follow up jira and adding all these minor things. we can take it up during bug fixes period as well.
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/FileGroupReaderBasedMergeHandle.java:
##########
@@ -251,6 +344,31 @@ public List<WriteStatus> close() {
}
}
+ private Option<BaseFileUpdateCallback<T>> createCallback() {
+ List<BaseFileUpdateCallback<T>> callbacks = new ArrayList<>();
+ // Handle CDC workflow.
+ if (cdcLogger.isPresent()) {
+ callbacks.add(new CDCCallback<>(cdcLogger.get(), readerContext));
+ }
+ // Indexes are not updated during compaction
+ if (operation.isEmpty()) {
+ // record index callback
+ if (this.writeStatus.isTrackingSuccessfulWrites()) {
+ writeStatus.manuallyTrackSuccess();
+ callbacks.add(new RecordLevelIndexCallback<>(writeStatus, newRecordLocation, partitionPath));
+ }
+ // Stream secondary index stats.
+ if (isSecondaryIndexStatsStreamingWritesEnabled) {
+ callbacks.add(new SecondaryIndexCallback<>(
Review Comment:
I see. but it should be possible to have just 1 callback for both RLI and SI right. I see its simpler and maintainable w/ two diff callbacks for now. So, not really a blocking comment. we can revisit this in future if this causes any perf issues.
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/table/action/commit/BaseSparkCommitActionExecutor.java:
##########
@@ -397,6 +401,9 @@ protected HoodieMergeHandle getUpdateHandle(String partitionPath, String fileId,
mergeHandle.setPartitionFields(partitionFields);
mergeHandle.setPartitionValues(partitionValues);
}
+ if (readerContextFactory != null && mergeHandle instanceof FileGroupReaderBasedMergeHandle) {
Review Comment:
gotcha. thanks for the context. Spark specific requirement is what makes this tricky.
I am good then.
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/buffer/FileGroupRecordBuffer.java:
##########
@@ -76,7 +72,7 @@ abstract class FileGroupRecordBuffer<T> implements HoodieFileGroupRecordBuffer<T
protected final RecordMergeMode recordMergeMode;
protected final PartialUpdateMode partialUpdateMode;
protected final Option<HoodieRecordMerger> recordMerger;
- protected final Option<String> payloadClass;
+ protected final Option<Pair<String, String>> payloadClasses;
Review Comment:
thanks.
trying to think through how our path forward will look like where we are looking to completely move away from payloads. until we retire expression payloads from Merge Into, we have to assume all records will have some payload associated with it right? curious to know how are you tackling this in your next patch. Essentially, what changes we intend to do with ExpressionPayloadRecordMerger.
For eg, if someone creates a new table in v9 today w/o setting any payload class (after your next patch), but just setting the merge mode. and if they issue MIT writes, what will happen w/ ExpressionPayloadRecordMerger. Currently, it converts both base record and incoming record to HoodieAvroRecord which is in payload format before it can actually merge them.
may be you thought through this already or have fixed this in your draft patch. was just curious to understand. we can jam f2f to align if need be.
note: this comment is not a blocking comment to land this patch.
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/FileGroupReaderBasedMergeHandle.java:
##########
@@ -156,40 +215,53 @@ private void init(CompactionOperation operation, String partitionPath) {
fileWriter = HoodieFileWriterFactory.getFileWriter(instantTime, newFilePath, hoodieTable.getStorage(),
config, writeSchemaWithMetaFields, taskContextSupplier, recordType);
} catch (IOException io) {
- LOG.error("Error in update task at commit {}", instantTime, io);
writeStatus.setGlobalError(io);
throw new HoodieUpsertException("Failed to initialize HoodieUpdateHandle for FileId: " + fileId + " on commit "
+ instantTime + " on path " + hoodieTable.getMetaClient().getBasePath(), io);
}
}
+ @Override
+ protected void populateIncomingRecordsMap(Iterator<HoodieRecord<T>> newRecordsItr) {
+ // no op.
+ }
+
+ /**
+ * This is only for spark, the engine context fetched from a serialized hoodie table is always local,
+ * overrides it to spark specific reader context.
+ */
+ public void setReaderContext(HoodieReaderContext<T> readerContext) {
+ this.readerContext = readerContext;
+ }
+
/**
* Reads the file slice of a compaction operation using a file group reader,
* by getting an iterator of the records; then writes the records to a new base file.
*/
@Override
public void doMerge() {
+ // For non-compaction operations, the merger needs to be initialized with the writer properties to handle cases like Merge-Into commands
+ if (operation.isEmpty()) {
+ this.readerContext.initRecordMergerForIngestion(config.getProps());
+ }
boolean usePosition = config.getBooleanOrDefault(MERGE_USE_RECORD_POSITIONS);
- Option<InternalSchema> internalSchemaOption = SerDeHelper.fromJson(config.getInternalSchema());
- TypedProperties props = TypedProperties.copy(config.getProps());
+ Option<InternalSchema> internalSchemaOption = SerDeHelper.fromJson(config.getInternalSchema())
+ .map(internalSchema -> AvroSchemaEvolutionUtils.reconcileSchema(writeSchemaWithMetaFields, internalSchema,
+ config.getBooleanOrDefault(HoodieCommonConfig.SET_NULL_FOR_MISSING_COLUMNS)));
long maxMemoryPerCompaction = IOUtils.getMaxMemoryPerCompaction(taskContextSupplier, config);
props.put(HoodieMemoryConfig.MAX_MEMORY_FOR_MERGE.key(), String.valueOf(maxMemoryPerCompaction));
- Stream<HoodieLogFile> logFiles = operation.getDeltaFileNames().stream().map(logFileName ->
+ Option<Stream<HoodieLogFile>> logFilesStreamOpt = operation.map(op -> op.getDeltaFileNames().stream().map(logFileName ->
new HoodieLogFile(new StoragePath(FSUtils.constructAbsolutePath(
- config.getBasePath(), operation.getPartitionPath()), logFileName)));
+ config.getBasePath(), op.getPartitionPath()), logFileName))));
// Initializes file group reader
- try (HoodieFileGroupReader<T> fileGroupReader = HoodieFileGroupReader.<T>newBuilder().withReaderContext(readerContext).withHoodieTableMetaClient(hoodieTable.getMetaClient())
- .withLatestCommitTime(maxInstantTime).withPartitionPath(partitionPath).withBaseFileOption(Option.ofNullable(baseFileToMerge)).withLogFiles(logFiles)
- .withDataSchema(writeSchemaWithMetaFields).withRequestedSchema(writeSchemaWithMetaFields).withInternalSchema(internalSchemaOption).withProps(props)
- .withShouldUseRecordPosition(usePosition).withSortOutput(hoodieTable.requireSortedRecords())
- .withFileGroupUpdateCallback(cdcLogger.map(logger -> new CDCCallback(logger, readerContext))).withEnableOptimizedLogBlockScan(config.enableOptimizedLogBlocksScan()).build()) {
+ try (HoodieFileGroupReader<T> fileGroupReader = getFileGroupReader(usePosition, internalSchemaOption, props, logFilesStreamOpt, incomingRecordsItr)) {
// Reads the records from the file slice
try (ClosableIterator<HoodieRecord<T>> recordIterator = fileGroupReader.getClosableHoodieRecordIterator()) {
while (recordIterator.hasNext()) {
HoodieRecord<T> record = recordIterator.next();
+ Option<Map<String, String>> recordMetadata = getRecordMetadata(record, writeSchema, props);
Review Comment:
oh, we should avoid the event time metadata tracking for compaction commits right. How are we ensuring that?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13699:
URL: https://github.com/apache/hudi/pull/13699#discussion_r2271943944
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/UpdateProcessor.java:
##########
@@ -80,16 +106,79 @@ public BufferedRecord<T> processUpdate(String recordKey, BufferedRecord<T> previ
}
return null;
} else {
- T prevRow = previousRecord != null ? previousRecord.getRecord() : null;
- T mergedRow = mergedRecord.getRecord();
- if (prevRow != null && prevRow != mergedRow) {
- mergedRecord.setHoodieOperation(HoodieOperation.UPDATE_AFTER);
- readStats.incrementNumUpdates();
- } else if (prevRow == null) {
- mergedRecord.setHoodieOperation(HoodieOperation.INSERT);
- readStats.incrementNumInserts();
+ return handleNonDeletes(previousRecord, mergedRecord);
+ }
+ }
+
+ protected BufferedRecord<T> handleNonDeletes(BufferedRecord<T> previousRecord, BufferedRecord<T> mergedRecord) {
+ T prevRow = previousRecord != null ? previousRecord.getRecord() : null;
+ T mergedRow = mergedRecord.getRecord();
+ if (prevRow != null && prevRow != mergedRow) {
+ mergedRecord.setHoodieOperation(HoodieOperation.UPDATE_AFTER);
+ readStats.incrementNumUpdates();
+ } else if (prevRow == null) {
+ mergedRecord.setHoodieOperation(HoodieOperation.INSERT);
+ readStats.incrementNumInserts();
+ }
+ return mergedRecord.seal(readerContext);
+ }
+ }
+
+ class PayloadUpdateProcessor<T> extends StandardUpdateProcessor<T> {
+ private final String payloadClass;
+ private final Properties properties;
+
+ public PayloadUpdateProcessor(HoodieReadStats readStats, HoodieReaderContext<T> readerContext, boolean emitDeletes,
+ Properties properties, String payloadClass) {
+ super(readStats, readerContext, emitDeletes);
+ this.payloadClass = payloadClass;
+ this.properties = properties;
+ }
+
+ @Override
+ protected BufferedRecord<T> handleNonDeletes(BufferedRecord<T> previousRecord, BufferedRecord<T> mergedRecord) {
+ if (previousRecord == null) {
+ // special case for payloads when there is no previous record
+ Schema recordSchema = readerContext.getRecordContext().decodeAvroSchema(mergedRecord.getSchemaId());
+ GenericRecord record = readerContext.getRecordContext().convertToAvroRecord(mergedRecord.getRecord(), recordSchema);
+ HoodieAvroRecord hoodieRecord = new HoodieAvroRecord<>(null, HoodieRecordUtils.loadPayload(payloadClass, record, mergedRecord.getOrderingValue()));
+ try {
+ if (hoodieRecord.shouldIgnore(recordSchema, properties)) {
+ return null;
+ } else {
+ Schema readerSchema = readerContext.getSchemaHandler().getRequestedSchema();
+ // If the record schema is different from the reader schema, rewrite the record using the payload methods to ensure consistency with legacy writer paths
+ hoodieRecord.rewriteRecordWithNewSchema(recordSchema, properties, readerSchema).toIndexedRecord(readerSchema, properties)
+ .ifPresent(rewrittenRecord -> mergedRecord.replaceRecord(readerContext.getRecordContext().convertAvroRecord(rewrittenRecord.getData())));
+ }
+ } catch (IOException e) {
+ throw new HoodieIOException("Error processing record with payload class: " + payloadClass, e);
+ }
+ }
+ return super.handleNonDeletes(previousRecord, mergedRecord);
+ }
+ }
+
+ class CustomMergerUpdateProcessor<T> extends StandardUpdateProcessor<T> {
+ private final HoodieRecordMerger merger;
+ private final TypedProperties properties;
+
+ CustomMergerUpdateProcessor(HoodieReadStats readStats, HoodieReaderContext<T> readerContext, boolean emitDeletes,
+ TypedProperties properties) {
+ super(readStats, readerContext, emitDeletes);
+ this.merger = readerContext.getRecordMerger().get();
+ this.properties = properties;
+ }
+
+ @Override
+ protected BufferedRecord<T> handleNonDeletes(BufferedRecord<T> previousRecord, BufferedRecord<T> mergedRecord) {
+ try {
+ if (merger.shouldFlush(readerContext.getRecordContext().constructHoodieRecord(mergedRecord), readerContext.getRecordContext().getSchemaFromBufferRecord(mergedRecord), properties)) {
Review Comment:
we have discussed to drop the `shouldFlush` functionality as of now, it is not in good design and the original fix does not cover MOR merging scenarios as mentioned here: https://github.com/apache/hudi/pull/9809#issuecomment-1751669474, let's just drop this support first and fire a JIRA to trace it instead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13699:
URL: https://github.com/apache/hudi/pull/13699#discussion_r2273475148
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/HoodieTable.java:
##########
@@ -150,6 +151,7 @@ public abstract class HoodieTable<T, I, K, O> implements Serializable {
private transient FileSystemViewManager viewManager;
protected final transient HoodieEngineContext context;
+ private final ReaderContextFactory<T> readerContextFactoryForWrite;
Review Comment:
@danny0405 I don't think this is the right way to approach this. Now every usage of HoodieTable will incur the cost of generating the factory, which requires broadcasting on Spark. Why can't we just pass this in when it is required like I had before?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13699:
URL: https://github.com/apache/hudi/pull/13699#discussion_r2275104323
##########
hudi-common/src/main/java/org/apache/hudi/internal/schema/utils/InternalSchemaUtils.java:
##########
@@ -57,15 +62,16 @@ public static InternalSchema pruneInternalSchema(InternalSchema schema, List<Str
List<Integer> prunedIds = names.stream().map(name -> {
int id = schema.findIdByName(name);
if (id == -1) {
- throw new IllegalArgumentException(String.format("cannot prune col: %s which does not exist in hudi table", name));
+ LOG.debug("Cannot find field name {} in schema {}, skipping.", name, schema);
+ return null;
}
return id;
- }).collect(Collectors.toList());
+ }).filter(Objects::nonNull).collect(Collectors.toList());
Review Comment:
It looks like we get around this in the current flows by using `AvroSchemaEvolutionUtils.reconcileSchema` before setting the `InternalSchema`. I will test this out with the schema evolution cases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13699:
URL: https://github.com/apache/hudi/pull/13699#issuecomment-3171264328
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7661",
"triggerID" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"triggerType" : "PUSH"
}, {
"hash" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7666",
"triggerID" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"triggerType" : "PUSH"
}, {
"hash" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ca08e20d8bacc2355f15ae37fab3c1ef61a052af Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7661)
* 7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7666)
* af88f9e4a548191ff0cba41db6ad3e60ba9ea43b UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13699:
URL: https://github.com/apache/hudi/pull/13699#issuecomment-3171895542
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7661",
"triggerID" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"triggerType" : "PUSH"
}, {
"hash" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7666",
"triggerID" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"triggerType" : "PUSH"
}, {
"hash" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7667",
"triggerID" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* af88f9e4a548191ff0cba41db6ad3e60ba9ea43b Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7667)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13699:
URL: https://github.com/apache/hudi/pull/13699#issuecomment-3172264918
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7661",
"triggerID" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"triggerType" : "PUSH"
}, {
"hash" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7666",
"triggerID" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"triggerType" : "PUSH"
}, {
"hash" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7667",
"triggerID" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"triggerType" : "PUSH"
}, {
"hash" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7672",
"triggerID" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5507dfb0f3e741a539dbed0669d24dac9f1fc2b3 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7672)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13699:
URL: https://github.com/apache/hudi/pull/13699#discussion_r2265588763
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/index/HoodieIndexUtils.java:
##########
@@ -430,16 +430,11 @@ private static <R> Option<HoodieRecord<R>> mergeIncomingWithExistingRecordWithEx
//the record was deleted
return Option.empty();
}
- if (mergeResult.getRecord() == null) {
- // SENTINEL case: the record did not match and merge case and should not be modified
+ if (mergeResult.getRecord() == null || mergeResult == existingBufferedRecord) {
Review Comment:
if this is only for merge into and merge into only been utilized with EVENT_TIME merge mode, equals with buffered record is valid, otherwise, better use record equals here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13699:
URL: https://github.com/apache/hudi/pull/13699#discussion_r2265594143
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/FileGroupReaderBasedMergeHandle.java:
##########
@@ -90,7 +147,15 @@ public FileGroupReaderBasedMergeHandle(HoodieWriteConfig config, String instantT
this.maxInstantTime = maxInstantTime;
this.keyToNewRecords = Collections.emptyMap();
this.readerContext = readerContext;
- this.operation = operation;
+ this.operation = Option.of(operation);
+ initRecordTypeAndCdcLogger(enginRecordType);
+ init(operation, this.partitionPath);
+ this.props = TypedProperties.copy(config.getProps());
+ this.isCompaction = true;
Review Comment:
we already have flag `preserveMetadata` to distinguish table service and regular write, can we continue to use that? some functions like SI tracing already relies on the flag `preserveMetadata`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13699:
URL: https://github.com/apache/hudi/pull/13699#discussion_r2265608632
##########
hudi-common/src/main/java/org/apache/hudi/common/engine/HoodieEngineContext.java:
##########
@@ -139,15 +139,16 @@ public abstract <I, K, V> List<V> reduceByKey(
/**
* Returns reader context factory for write operations in the table.
*
- * @param metaClient Table meta client
- * @param recordType Record type
- * @param properties Typed properties
+ * @param metaClient Table meta client
+ * @param recordType Record type
+ * @param properties Typed properties
+ * @param outputsCustomPayloads Whether the reader context factory should output custom payloads. Final merging of records before writes does not require custom payloads.
*/
public ReaderContextFactory<?> getReaderContextFactoryForWrite(HoodieTableMetaClient metaClient, HoodieRecord.HoodieRecordType recordType,
- TypedProperties properties) {
+ TypedProperties properties, boolean outputsCustomPayloads) {
Review Comment:
This flag is only meaningful for avro reader context, is there anyway we can constraint it just to `AvroReaderContextFactory`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13699:
URL: https://github.com/apache/hudi/pull/13699#discussion_r2268392616
##########
hudi-common/src/main/java/org/apache/hudi/common/engine/HoodieEngineContext.java:
##########
@@ -139,15 +139,16 @@ public abstract <I, K, V> List<V> reduceByKey(
/**
* Returns reader context factory for write operations in the table.
*
- * @param metaClient Table meta client
- * @param recordType Record type
- * @param properties Typed properties
+ * @param metaClient Table meta client
+ * @param recordType Record type
+ * @param properties Typed properties
+ * @param outputsCustomPayloads Whether the reader context factory should output custom payloads. Final merging of records before writes does not require custom payloads.
*/
public ReaderContextFactory<?> getReaderContextFactoryForWrite(HoodieTableMetaClient metaClient, HoodieRecord.HoodieRecordType recordType,
- TypedProperties properties) {
+ TypedProperties properties, boolean outputsCustomPayloads) {
Review Comment:
we have a plan to abadon the payload based records in writer path right? So this should be just a temporary solution?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13699:
URL: https://github.com/apache/hudi/pull/13699#issuecomment-3185659881
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7661",
"triggerID" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"triggerType" : "PUSH"
}, {
"hash" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7666",
"triggerID" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"triggerType" : "PUSH"
}, {
"hash" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7667",
"triggerID" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"triggerType" : "PUSH"
}, {
"hash" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7672",
"triggerID" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"triggerType" : "PUSH"
}, {
"hash" : "965136302f1188e5123de37734c927dc55801b24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7691",
"triggerID" : "965136302f1188e5123de37734c927dc55801b24",
"triggerType" : "PUSH"
}, {
"hash" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7693",
"triggerID" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"triggerType" : "PUSH"
}, {
"hash" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7717",
"triggerID" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"triggerType" : "PUSH"
}, {
"hash" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"triggerType" : "PUSH"
}, {
"hash" : "47f8303d6e61504275ecfddf553e356427e6c822",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7718",
"triggerID" : "47f8303d6e61504275ecfddf553e356427e6c822",
"triggerType" : "PUSH"
}, {
"hash" : "fb191ecf48d13cd81412535f7f8c9ed553d86517",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7725",
"triggerID" : "fb191ecf48d13cd81412535f7f8c9ed553d86517",
"triggerType" : "PUSH"
}, {
"hash" : "064912d3a54714935f2ee77d55cc66816166300f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7727",
"triggerID" : "064912d3a54714935f2ee77d55cc66816166300f",
"triggerType" : "PUSH"
}, {
"hash" : "77a2b45fe13dafc51f7bebb2ba859d40291ccc10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7736",
"triggerID" : "77a2b45fe13dafc51f7bebb2ba859d40291ccc10",
"triggerType" : "PUSH"
}, {
"hash" : "e4eda0e51fee0e8de5d57ea7b38dc642f53aa95d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7758",
"triggerID" : "e4eda0e51fee0e8de5d57ea7b38dc642f53aa95d",
"triggerType" : "PUSH"
}, {
"hash" : "6cc840c51fcecde95d2187fb6fd217b50fa82806",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6cc840c51fcecde95d2187fb6fd217b50fa82806",
"triggerType" : "PUSH"
}, {
"hash" : "31dab8ce9abbe4fc80e6cd81a0e433a039ca3e1d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7760",
"triggerID" : "31dab8ce9abbe4fc80e6cd81a0e433a039ca3e1d",
"triggerType" : "PUSH"
}, {
"hash" : "c04fd95dcc94146a9146c347f4f204656f3d8308",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7763",
"triggerID" : "c04fd95dcc94146a9146c347f4f204656f3d8308",
"triggerType" : "PUSH"
}, {
"hash" : "324147913d47ba8f2555e1991ce59905e5dc0b3f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7767",
"triggerID" : "324147913d47ba8f2555e1991ce59905e5dc0b3f",
"triggerType" : "PUSH"
}, {
"hash" : "3f177d42b831b30f4bd590b057dcab6ded78b634",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7774",
"triggerID" : "3f177d42b831b30f4bd590b057dcab6ded78b634",
"triggerType" : "PUSH"
}, {
"hash" : "8d6d2c06ec82d1072251ac12ddf21a890ddd5bda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7776",
"triggerID" : "8d6d2c06ec82d1072251ac12ddf21a890ddd5bda",
"triggerType" : "PUSH"
}, {
"hash" : "77589738d73b044385f5aa6f7ef87b421f06043b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "77589738d73b044385f5aa6f7ef87b421f06043b",
"triggerType" : "PUSH"
}, {
"hash" : "ad0d7ef0fbc525f880d3baadef5a22196ab97b32",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7781",
"triggerID" : "ad0d7ef0fbc525f880d3baadef5a22196ab97b32",
"triggerType" : "PUSH"
}, {
"hash" : "0db72caa3a6c7d0f0899f243ea4b8975de79e02c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7787",
"triggerID" : "0db72caa3a6c7d0f0899f243ea4b8975de79e02c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 50022dcc45b0f511dd69706aeaf6464a51be49e2 UNKNOWN
* 6cc840c51fcecde95d2187fb6fd217b50fa82806 UNKNOWN
* 77589738d73b044385f5aa6f7ef87b421f06043b UNKNOWN
* 0db72caa3a6c7d0f0899f243ea4b8975de79e02c Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7787)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13699:
URL: https://github.com/apache/hudi/pull/13699#issuecomment-3185241941
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7661",
"triggerID" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"triggerType" : "PUSH"
}, {
"hash" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7666",
"triggerID" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"triggerType" : "PUSH"
}, {
"hash" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7667",
"triggerID" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"triggerType" : "PUSH"
}, {
"hash" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7672",
"triggerID" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"triggerType" : "PUSH"
}, {
"hash" : "965136302f1188e5123de37734c927dc55801b24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7691",
"triggerID" : "965136302f1188e5123de37734c927dc55801b24",
"triggerType" : "PUSH"
}, {
"hash" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7693",
"triggerID" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"triggerType" : "PUSH"
}, {
"hash" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7717",
"triggerID" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"triggerType" : "PUSH"
}, {
"hash" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"triggerType" : "PUSH"
}, {
"hash" : "47f8303d6e61504275ecfddf553e356427e6c822",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7718",
"triggerID" : "47f8303d6e61504275ecfddf553e356427e6c822",
"triggerType" : "PUSH"
}, {
"hash" : "fb191ecf48d13cd81412535f7f8c9ed553d86517",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7725",
"triggerID" : "fb191ecf48d13cd81412535f7f8c9ed553d86517",
"triggerType" : "PUSH"
}, {
"hash" : "064912d3a54714935f2ee77d55cc66816166300f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7727",
"triggerID" : "064912d3a54714935f2ee77d55cc66816166300f",
"triggerType" : "PUSH"
}, {
"hash" : "77a2b45fe13dafc51f7bebb2ba859d40291ccc10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7736",
"triggerID" : "77a2b45fe13dafc51f7bebb2ba859d40291ccc10",
"triggerType" : "PUSH"
}, {
"hash" : "e4eda0e51fee0e8de5d57ea7b38dc642f53aa95d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7758",
"triggerID" : "e4eda0e51fee0e8de5d57ea7b38dc642f53aa95d",
"triggerType" : "PUSH"
}, {
"hash" : "6cc840c51fcecde95d2187fb6fd217b50fa82806",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6cc840c51fcecde95d2187fb6fd217b50fa82806",
"triggerType" : "PUSH"
}, {
"hash" : "31dab8ce9abbe4fc80e6cd81a0e433a039ca3e1d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7760",
"triggerID" : "31dab8ce9abbe4fc80e6cd81a0e433a039ca3e1d",
"triggerType" : "PUSH"
}, {
"hash" : "c04fd95dcc94146a9146c347f4f204656f3d8308",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7763",
"triggerID" : "c04fd95dcc94146a9146c347f4f204656f3d8308",
"triggerType" : "PUSH"
}, {
"hash" : "324147913d47ba8f2555e1991ce59905e5dc0b3f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7767",
"triggerID" : "324147913d47ba8f2555e1991ce59905e5dc0b3f",
"triggerType" : "PUSH"
}, {
"hash" : "3f177d42b831b30f4bd590b057dcab6ded78b634",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7774",
"triggerID" : "3f177d42b831b30f4bd590b057dcab6ded78b634",
"triggerType" : "PUSH"
}, {
"hash" : "8d6d2c06ec82d1072251ac12ddf21a890ddd5bda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7776",
"triggerID" : "8d6d2c06ec82d1072251ac12ddf21a890ddd5bda",
"triggerType" : "PUSH"
}, {
"hash" : "77589738d73b044385f5aa6f7ef87b421f06043b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "77589738d73b044385f5aa6f7ef87b421f06043b",
"triggerType" : "PUSH"
}, {
"hash" : "ad0d7ef0fbc525f880d3baadef5a22196ab97b32",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7781",
"triggerID" : "ad0d7ef0fbc525f880d3baadef5a22196ab97b32",
"triggerType" : "PUSH"
}, {
"hash" : "0db72caa3a6c7d0f0899f243ea4b8975de79e02c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0db72caa3a6c7d0f0899f243ea4b8975de79e02c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 50022dcc45b0f511dd69706aeaf6464a51be49e2 UNKNOWN
* 6cc840c51fcecde95d2187fb6fd217b50fa82806 UNKNOWN
* 77589738d73b044385f5aa6f7ef87b421f06043b UNKNOWN
* ad0d7ef0fbc525f880d3baadef5a22196ab97b32 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7781)
* 0db72caa3a6c7d0f0899f243ea4b8975de79e02c UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13699:
URL: https://github.com/apache/hudi/pull/13699#discussion_r2268412640
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/FileGroupReaderBasedMergeHandle.java:
##########
@@ -75,13 +85,60 @@
public class FileGroupReaderBasedMergeHandle<T, I, K, O> extends HoodieWriteMergeHandle<T, I, K, O> {
private static final Logger LOG = LoggerFactory.getLogger(FileGroupReaderBasedMergeHandle.class);
- private final HoodieReaderContext<T> readerContext;
- private final CompactionOperation operation;
+ private final Option<CompactionOperation> operation;
private final String maxInstantTime;
+ private HoodieReaderContext<T> readerContext;
private HoodieReadStats readStats;
- private final HoodieRecord.HoodieRecordType recordType;
- private final Option<HoodieCDCLogger> cdcLogger;
+ private HoodieRecord.HoodieRecordType recordType;
+ private Option<HoodieCDCLogger> cdcLogger;
+ private Option<RecordLevelIndexCallback> recordIndexCallbackOpt;
+ private Option<SecondaryIndexCallback> secondaryIndexCallbackOpt;
+ private final boolean isCompaction;
+ private final TypedProperties props;
+ private Iterator<HoodieRecord> incomingRecordsItr;
+ /**
+ * Constructor for Copy-On-Write (COW) merge path.
+ * Takes in a base path and an iterator of records to be merged with that file.
+ * @param config instance of {@link HoodieWriteConfig} to use.
+ * @param instantTime instant time of the current commit.
+ * @param hoodieTable instance of {@link HoodieTable} being updated.
+ * @param recordItr iterator of records to be merged with the file.
+ * @param partitionPath partition path of the base file.
+ * @param fileId file ID of the base file.
+ * @param taskContextSupplier instance of {@link TaskContextSupplier} to use.
+ * @param keyGeneratorOpt optional instance of {@link BaseKeyGenerator} to use for extracting keys from records.
+ * @param readerContext instance of {@link HoodieReaderContext} to use while merging for accessing fields and transforming records.
+ */
+ public FileGroupReaderBasedMergeHandle(HoodieWriteConfig config, String instantTime, HoodieTable<T, I, K, O> hoodieTable,
+ Iterator<HoodieRecord<T>> recordItr, String partitionPath, String fileId,
+ TaskContextSupplier taskContextSupplier, Option<BaseKeyGenerator> keyGeneratorOpt, HoodieReaderContext<T> readerContext) {
Review Comment:
is `getContextForWrite` returning an EngineContext here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13699:
URL: https://github.com/apache/hudi/pull/13699#discussion_r2268409012
##########
hudi-common/src/main/java/org/apache/hudi/common/engine/HoodieEngineContext.java:
##########
@@ -139,15 +139,16 @@ public abstract <I, K, V> List<V> reduceByKey(
/**
* Returns reader context factory for write operations in the table.
*
- * @param metaClient Table meta client
- * @param recordType Record type
- * @param properties Typed properties
+ * @param metaClient Table meta client
+ * @param recordType Record type
+ * @param properties Typed properties
+ * @param outputsCustomPayloads Whether the reader context factory should output custom payloads. Final merging of records before writes does not require custom payloads.
*/
public ReaderContextFactory<?> getReaderContextFactoryForWrite(HoodieTableMetaClient metaClient, HoodieRecord.HoodieRecordType recordType,
- TypedProperties properties) {
+ TypedProperties properties, boolean outputsCustomPayloads) {
Review Comment:
We'll still need it for ExpressionPayload and for any user provided payload so it is not temporary but these restrictions may allow us to clean things up further.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13699:
URL: https://github.com/apache/hudi/pull/13699#issuecomment-3177534082
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7661",
"triggerID" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"triggerType" : "PUSH"
}, {
"hash" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7666",
"triggerID" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"triggerType" : "PUSH"
}, {
"hash" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7667",
"triggerID" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"triggerType" : "PUSH"
}, {
"hash" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7672",
"triggerID" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"triggerType" : "PUSH"
}, {
"hash" : "965136302f1188e5123de37734c927dc55801b24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7691",
"triggerID" : "965136302f1188e5123de37734c927dc55801b24",
"triggerType" : "PUSH"
}, {
"hash" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7693",
"triggerID" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"triggerType" : "PUSH"
}, {
"hash" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7717",
"triggerID" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"triggerType" : "PUSH"
}, {
"hash" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"triggerType" : "PUSH"
}, {
"hash" : "47f8303d6e61504275ecfddf553e356427e6c822",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7718",
"triggerID" : "47f8303d6e61504275ecfddf553e356427e6c822",
"triggerType" : "PUSH"
}, {
"hash" : "fb191ecf48d13cd81412535f7f8c9ed553d86517",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7725",
"triggerID" : "fb191ecf48d13cd81412535f7f8c9ed553d86517",
"triggerType" : "PUSH"
}, {
"hash" : "064912d3a54714935f2ee77d55cc66816166300f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7727",
"triggerID" : "064912d3a54714935f2ee77d55cc66816166300f",
"triggerType" : "PUSH"
}, {
"hash" : "77a2b45fe13dafc51f7bebb2ba859d40291ccc10",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7736",
"triggerID" : "77a2b45fe13dafc51f7bebb2ba859d40291ccc10",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 50022dcc45b0f511dd69706aeaf6464a51be49e2 UNKNOWN
* 77a2b45fe13dafc51f7bebb2ba859d40291ccc10 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7736)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13699:
URL: https://github.com/apache/hudi/pull/13699#discussion_r2273304993
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieMergeHandleFactory.java:
##########
@@ -60,21 +60,35 @@ public static <T, I, K, O> HoodieMergeHandle<T, I, K, O> create(
String partitionPath,
String fileId,
TaskContextSupplier taskContextSupplier,
- Option<BaseKeyGenerator> keyGeneratorOpt) {
+ Option<BaseKeyGenerator> keyGeneratorOpt,
+ HoodieReaderContext<T> readerContext) {
boolean isFallbackEnabled = writeConfig.isMergeHandleFallbackEnabled();
Pair<String, String> mergeHandleClasses = getMergeHandleClassesWrite(operationType, writeConfig, table);
String logContext = String.format("for fileId %s and partition path %s at commit %s", fileId, partitionPath, instantTime);
LOG.info("Create HoodieMergeHandle implementation {} {}", mergeHandleClasses.getLeft(), logContext);
Class<?>[] constructorParamTypes = new Class<?>[] {
+ HoodieWriteConfig.class, String.class, HoodieTable.class, Iterator.class,
+ String.class, String.class, TaskContextSupplier.class, Option.class, HoodieReaderContext.class
+ };
+
+ Class<?>[] legacyConstructorParamTypes = new Class<?>[] {
HoodieWriteConfig.class, String.class, HoodieTable.class, Iterator.class,
String.class, String.class, TaskContextSupplier.class, Option.class
};
- return instantiateMergeHandle(
- isFallbackEnabled, mergeHandleClasses.getLeft(), mergeHandleClasses.getRight(), logContext, constructorParamTypes,
- writeConfig, instantTime, table, recordItr, partitionPath, fileId, taskContextSupplier, keyGeneratorOpt);
+ try {
+ return instantiateMergeHandle(
+ isFallbackEnabled, mergeHandleClasses.getLeft(), mergeHandleClasses.getRight(), logContext, constructorParamTypes,
+ writeConfig, instantTime, table, recordItr, partitionPath, fileId, taskContextSupplier, keyGeneratorOpt, readerContext);
+ } catch (Exception e) {
+ // Fallback to legacy constructor if the new one fails
+ LOG.warn("Failed to instantiate HoodieMergeHandle with new constructor, falling back to legacy constructor: {}", e.getMessage());
Review Comment:
Another option I was considering is checking if the constructor exists first. Then users can still provide a custom merge handle with the newer or older set of constructor args
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13699:
URL: https://github.com/apache/hudi/pull/13699#issuecomment-3181597124
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7661",
"triggerID" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"triggerType" : "PUSH"
}, {
"hash" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7666",
"triggerID" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"triggerType" : "PUSH"
}, {
"hash" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7667",
"triggerID" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"triggerType" : "PUSH"
}, {
"hash" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7672",
"triggerID" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"triggerType" : "PUSH"
}, {
"hash" : "965136302f1188e5123de37734c927dc55801b24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7691",
"triggerID" : "965136302f1188e5123de37734c927dc55801b24",
"triggerType" : "PUSH"
}, {
"hash" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7693",
"triggerID" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"triggerType" : "PUSH"
}, {
"hash" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7717",
"triggerID" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"triggerType" : "PUSH"
}, {
"hash" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"triggerType" : "PUSH"
}, {
"hash" : "47f8303d6e61504275ecfddf553e356427e6c822",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7718",
"triggerID" : "47f8303d6e61504275ecfddf553e356427e6c822",
"triggerType" : "PUSH"
}, {
"hash" : "fb191ecf48d13cd81412535f7f8c9ed553d86517",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7725",
"triggerID" : "fb191ecf48d13cd81412535f7f8c9ed553d86517",
"triggerType" : "PUSH"
}, {
"hash" : "064912d3a54714935f2ee77d55cc66816166300f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7727",
"triggerID" : "064912d3a54714935f2ee77d55cc66816166300f",
"triggerType" : "PUSH"
}, {
"hash" : "77a2b45fe13dafc51f7bebb2ba859d40291ccc10",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7736",
"triggerID" : "77a2b45fe13dafc51f7bebb2ba859d40291ccc10",
"triggerType" : "PUSH"
}, {
"hash" : "e4eda0e51fee0e8de5d57ea7b38dc642f53aa95d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e4eda0e51fee0e8de5d57ea7b38dc642f53aa95d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 50022dcc45b0f511dd69706aeaf6464a51be49e2 UNKNOWN
* 77a2b45fe13dafc51f7bebb2ba859d40291ccc10 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7736)
* e4eda0e51fee0e8de5d57ea7b38dc642f53aa95d UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13699:
URL: https://github.com/apache/hudi/pull/13699#issuecomment-3182404687
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7661",
"triggerID" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"triggerType" : "PUSH"
}, {
"hash" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7666",
"triggerID" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"triggerType" : "PUSH"
}, {
"hash" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7667",
"triggerID" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"triggerType" : "PUSH"
}, {
"hash" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7672",
"triggerID" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"triggerType" : "PUSH"
}, {
"hash" : "965136302f1188e5123de37734c927dc55801b24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7691",
"triggerID" : "965136302f1188e5123de37734c927dc55801b24",
"triggerType" : "PUSH"
}, {
"hash" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7693",
"triggerID" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"triggerType" : "PUSH"
}, {
"hash" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7717",
"triggerID" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"triggerType" : "PUSH"
}, {
"hash" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"triggerType" : "PUSH"
}, {
"hash" : "47f8303d6e61504275ecfddf553e356427e6c822",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7718",
"triggerID" : "47f8303d6e61504275ecfddf553e356427e6c822",
"triggerType" : "PUSH"
}, {
"hash" : "fb191ecf48d13cd81412535f7f8c9ed553d86517",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7725",
"triggerID" : "fb191ecf48d13cd81412535f7f8c9ed553d86517",
"triggerType" : "PUSH"
}, {
"hash" : "064912d3a54714935f2ee77d55cc66816166300f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7727",
"triggerID" : "064912d3a54714935f2ee77d55cc66816166300f",
"triggerType" : "PUSH"
}, {
"hash" : "77a2b45fe13dafc51f7bebb2ba859d40291ccc10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7736",
"triggerID" : "77a2b45fe13dafc51f7bebb2ba859d40291ccc10",
"triggerType" : "PUSH"
}, {
"hash" : "e4eda0e51fee0e8de5d57ea7b38dc642f53aa95d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7758",
"triggerID" : "e4eda0e51fee0e8de5d57ea7b38dc642f53aa95d",
"triggerType" : "PUSH"
}, {
"hash" : "6cc840c51fcecde95d2187fb6fd217b50fa82806",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6cc840c51fcecde95d2187fb6fd217b50fa82806",
"triggerType" : "PUSH"
}, {
"hash" : "31dab8ce9abbe4fc80e6cd81a0e433a039ca3e1d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7760",
"triggerID" : "31dab8ce9abbe4fc80e6cd81a0e433a039ca3e1d",
"triggerType" : "PUSH"
}, {
"hash" : "c04fd95dcc94146a9146c347f4f204656f3d8308",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7763",
"triggerID" : "c04fd95dcc94146a9146c347f4f204656f3d8308",
"triggerType" : "PUSH"
}, {
"hash" : "324147913d47ba8f2555e1991ce59905e5dc0b3f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "324147913d47ba8f2555e1991ce59905e5dc0b3f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 50022dcc45b0f511dd69706aeaf6464a51be49e2 UNKNOWN
* 6cc840c51fcecde95d2187fb6fd217b50fa82806 UNKNOWN
* c04fd95dcc94146a9146c347f4f204656f3d8308 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7763)
* 324147913d47ba8f2555e1991ce59905e5dc0b3f UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13699:
URL: https://github.com/apache/hudi/pull/13699#issuecomment-3175597633
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7661",
"triggerID" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"triggerType" : "PUSH"
}, {
"hash" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7666",
"triggerID" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"triggerType" : "PUSH"
}, {
"hash" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7667",
"triggerID" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"triggerType" : "PUSH"
}, {
"hash" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7672",
"triggerID" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"triggerType" : "PUSH"
}, {
"hash" : "965136302f1188e5123de37734c927dc55801b24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7691",
"triggerID" : "965136302f1188e5123de37734c927dc55801b24",
"triggerType" : "PUSH"
}, {
"hash" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7693",
"triggerID" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"triggerType" : "PUSH"
}, {
"hash" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7dbbe8212c828dc3cf2405a0634d4a0c643580e0 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7693)
* 22721bae05acf8aa296bc5732af5a90d2afd3515 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13699:
URL: https://github.com/apache/hudi/pull/13699#issuecomment-3173048911
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7661",
"triggerID" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"triggerType" : "PUSH"
}, {
"hash" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7666",
"triggerID" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"triggerType" : "PUSH"
}, {
"hash" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7667",
"triggerID" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"triggerType" : "PUSH"
}, {
"hash" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7672",
"triggerID" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"triggerType" : "PUSH"
}, {
"hash" : "965136302f1188e5123de37734c927dc55801b24",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7691",
"triggerID" : "965136302f1188e5123de37734c927dc55801b24",
"triggerType" : "PUSH"
}, {
"hash" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7693",
"triggerID" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 965136302f1188e5123de37734c927dc55801b24 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7691)
* 7dbbe8212c828dc3cf2405a0634d4a0c643580e0 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7693)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13699:
URL: https://github.com/apache/hudi/pull/13699#issuecomment-3172882175
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7661",
"triggerID" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"triggerType" : "PUSH"
}, {
"hash" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7666",
"triggerID" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"triggerType" : "PUSH"
}, {
"hash" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7667",
"triggerID" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"triggerType" : "PUSH"
}, {
"hash" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7672",
"triggerID" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"triggerType" : "PUSH"
}, {
"hash" : "965136302f1188e5123de37734c927dc55801b24",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "965136302f1188e5123de37734c927dc55801b24",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5507dfb0f3e741a539dbed0669d24dac9f1fc2b3 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7672)
* 965136302f1188e5123de37734c927dc55801b24 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13699:
URL: https://github.com/apache/hudi/pull/13699#issuecomment-3182072788
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7661",
"triggerID" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"triggerType" : "PUSH"
}, {
"hash" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7666",
"triggerID" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"triggerType" : "PUSH"
}, {
"hash" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7667",
"triggerID" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"triggerType" : "PUSH"
}, {
"hash" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7672",
"triggerID" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"triggerType" : "PUSH"
}, {
"hash" : "965136302f1188e5123de37734c927dc55801b24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7691",
"triggerID" : "965136302f1188e5123de37734c927dc55801b24",
"triggerType" : "PUSH"
}, {
"hash" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7693",
"triggerID" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"triggerType" : "PUSH"
}, {
"hash" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7717",
"triggerID" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"triggerType" : "PUSH"
}, {
"hash" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"triggerType" : "PUSH"
}, {
"hash" : "47f8303d6e61504275ecfddf553e356427e6c822",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7718",
"triggerID" : "47f8303d6e61504275ecfddf553e356427e6c822",
"triggerType" : "PUSH"
}, {
"hash" : "fb191ecf48d13cd81412535f7f8c9ed553d86517",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7725",
"triggerID" : "fb191ecf48d13cd81412535f7f8c9ed553d86517",
"triggerType" : "PUSH"
}, {
"hash" : "064912d3a54714935f2ee77d55cc66816166300f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7727",
"triggerID" : "064912d3a54714935f2ee77d55cc66816166300f",
"triggerType" : "PUSH"
}, {
"hash" : "77a2b45fe13dafc51f7bebb2ba859d40291ccc10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7736",
"triggerID" : "77a2b45fe13dafc51f7bebb2ba859d40291ccc10",
"triggerType" : "PUSH"
}, {
"hash" : "e4eda0e51fee0e8de5d57ea7b38dc642f53aa95d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7758",
"triggerID" : "e4eda0e51fee0e8de5d57ea7b38dc642f53aa95d",
"triggerType" : "PUSH"
}, {
"hash" : "6cc840c51fcecde95d2187fb6fd217b50fa82806",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6cc840c51fcecde95d2187fb6fd217b50fa82806",
"triggerType" : "PUSH"
}, {
"hash" : "31dab8ce9abbe4fc80e6cd81a0e433a039ca3e1d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7760",
"triggerID" : "31dab8ce9abbe4fc80e6cd81a0e433a039ca3e1d",
"triggerType" : "PUSH"
}, {
"hash" : "c04fd95dcc94146a9146c347f4f204656f3d8308",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7763",
"triggerID" : "c04fd95dcc94146a9146c347f4f204656f3d8308",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 50022dcc45b0f511dd69706aeaf6464a51be49e2 UNKNOWN
* 6cc840c51fcecde95d2187fb6fd217b50fa82806 UNKNOWN
* 31dab8ce9abbe4fc80e6cd81a0e433a039ca3e1d Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7760)
* c04fd95dcc94146a9146c347f4f204656f3d8308 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7763)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9622] Add implementation of MergeHandle backed by the HoodieFileGroupReader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13699:
URL: https://github.com/apache/hudi/pull/13699#issuecomment-3192296882
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7661",
"triggerID" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"triggerType" : "PUSH"
}, {
"hash" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7666",
"triggerID" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"triggerType" : "PUSH"
}, {
"hash" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7667",
"triggerID" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"triggerType" : "PUSH"
}, {
"hash" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7672",
"triggerID" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"triggerType" : "PUSH"
}, {
"hash" : "965136302f1188e5123de37734c927dc55801b24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7691",
"triggerID" : "965136302f1188e5123de37734c927dc55801b24",
"triggerType" : "PUSH"
}, {
"hash" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7693",
"triggerID" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"triggerType" : "PUSH"
}, {
"hash" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7717",
"triggerID" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"triggerType" : "PUSH"
}, {
"hash" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"triggerType" : "PUSH"
}, {
"hash" : "47f8303d6e61504275ecfddf553e356427e6c822",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7718",
"triggerID" : "47f8303d6e61504275ecfddf553e356427e6c822",
"triggerType" : "PUSH"
}, {
"hash" : "fb191ecf48d13cd81412535f7f8c9ed553d86517",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7725",
"triggerID" : "fb191ecf48d13cd81412535f7f8c9ed553d86517",
"triggerType" : "PUSH"
}, {
"hash" : "064912d3a54714935f2ee77d55cc66816166300f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7727",
"triggerID" : "064912d3a54714935f2ee77d55cc66816166300f",
"triggerType" : "PUSH"
}, {
"hash" : "77a2b45fe13dafc51f7bebb2ba859d40291ccc10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7736",
"triggerID" : "77a2b45fe13dafc51f7bebb2ba859d40291ccc10",
"triggerType" : "PUSH"
}, {
"hash" : "e4eda0e51fee0e8de5d57ea7b38dc642f53aa95d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7758",
"triggerID" : "e4eda0e51fee0e8de5d57ea7b38dc642f53aa95d",
"triggerType" : "PUSH"
}, {
"hash" : "6cc840c51fcecde95d2187fb6fd217b50fa82806",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6cc840c51fcecde95d2187fb6fd217b50fa82806",
"triggerType" : "PUSH"
}, {
"hash" : "31dab8ce9abbe4fc80e6cd81a0e433a039ca3e1d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7760",
"triggerID" : "31dab8ce9abbe4fc80e6cd81a0e433a039ca3e1d",
"triggerType" : "PUSH"
}, {
"hash" : "c04fd95dcc94146a9146c347f4f204656f3d8308",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7763",
"triggerID" : "c04fd95dcc94146a9146c347f4f204656f3d8308",
"triggerType" : "PUSH"
}, {
"hash" : "324147913d47ba8f2555e1991ce59905e5dc0b3f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7767",
"triggerID" : "324147913d47ba8f2555e1991ce59905e5dc0b3f",
"triggerType" : "PUSH"
}, {
"hash" : "3f177d42b831b30f4bd590b057dcab6ded78b634",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7774",
"triggerID" : "3f177d42b831b30f4bd590b057dcab6ded78b634",
"triggerType" : "PUSH"
}, {
"hash" : "8d6d2c06ec82d1072251ac12ddf21a890ddd5bda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7776",
"triggerID" : "8d6d2c06ec82d1072251ac12ddf21a890ddd5bda",
"triggerType" : "PUSH"
}, {
"hash" : "77589738d73b044385f5aa6f7ef87b421f06043b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "77589738d73b044385f5aa6f7ef87b421f06043b",
"triggerType" : "PUSH"
}, {
"hash" : "ad0d7ef0fbc525f880d3baadef5a22196ab97b32",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7781",
"triggerID" : "ad0d7ef0fbc525f880d3baadef5a22196ab97b32",
"triggerType" : "PUSH"
}, {
"hash" : "0db72caa3a6c7d0f0899f243ea4b8975de79e02c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7787",
"triggerID" : "0db72caa3a6c7d0f0899f243ea4b8975de79e02c",
"triggerType" : "PUSH"
}, {
"hash" : "a01ce748dc2d4cd1fd4858d1ecac8a486989e880",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7797",
"triggerID" : "a01ce748dc2d4cd1fd4858d1ecac8a486989e880",
"triggerType" : "PUSH"
}, {
"hash" : "3f35151e28bfa1dd50b31c6e5379d9769627afd9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7799",
"triggerID" : "3f35151e28bfa1dd50b31c6e5379d9769627afd9",
"triggerType" : "PUSH"
}, {
"hash" : "b0054c8fd329489d9a1e9535d9866edffc52fdd0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b0054c8fd329489d9a1e9535d9866edffc52fdd0",
"triggerType" : "PUSH"
}, {
"hash" : "363e54341420cc217bf5af62de1e09ff370f75d2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "363e54341420cc217bf5af62de1e09ff370f75d2",
"triggerType" : "PUSH"
}, {
"hash" : "a893da375eb9c65edabbd22965296eac9b3714f4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a893da375eb9c65edabbd22965296eac9b3714f4",
"triggerType" : "PUSH"
}, {
"hash" : "a7c2756e6a293cbba99bd4e821881e6b961c138f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7804",
"triggerID" : "a7c2756e6a293cbba99bd4e821881e6b961c138f",
"triggerType" : "PUSH"
}, {
"hash" : "4a140fffa35893a13f2e7ac8365d60d51a2d0870",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7813",
"triggerID" : "4a140fffa35893a13f2e7ac8365d60d51a2d0870",
"triggerType" : "PUSH"
}, {
"hash" : "500b4f529ad2f8a695825b5badc1815202164d0a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7830",
"triggerID" : "500b4f529ad2f8a695825b5badc1815202164d0a",
"triggerType" : "PUSH"
}, {
"hash" : "539cf8b9cdc7ab4e70488e4096684022a027f942",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "539cf8b9cdc7ab4e70488e4096684022a027f942",
"triggerType" : "PUSH"
}, {
"hash" : "85139a12b431931973c07d1b7f9256a1aae75669",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7831",
"triggerID" : "85139a12b431931973c07d1b7f9256a1aae75669",
"triggerType" : "PUSH"
}, {
"hash" : "cc430c949b539450110da96c0eccb52ab5e0998e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7832",
"triggerID" : "cc430c949b539450110da96c0eccb52ab5e0998e",
"triggerType" : "PUSH"
}, {
"hash" : "2a9cad38ac9c86ddacbb2de06950d515b3a1b0df",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7850",
"triggerID" : "2a9cad38ac9c86ddacbb2de06950d515b3a1b0df",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 50022dcc45b0f511dd69706aeaf6464a51be49e2 UNKNOWN
* 6cc840c51fcecde95d2187fb6fd217b50fa82806 UNKNOWN
* 77589738d73b044385f5aa6f7ef87b421f06043b UNKNOWN
* b0054c8fd329489d9a1e9535d9866edffc52fdd0 UNKNOWN
* 363e54341420cc217bf5af62de1e09ff370f75d2 UNKNOWN
* a893da375eb9c65edabbd22965296eac9b3714f4 UNKNOWN
* 539cf8b9cdc7ab4e70488e4096684022a027f942 UNKNOWN
* cc430c949b539450110da96c0eccb52ab5e0998e Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7832)
* 2a9cad38ac9c86ddacbb2de06950d515b3a1b0df Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7850)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9622] Add implementation of MergeHandle backed by the HoodieFileGroupReader [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13699:
URL: https://github.com/apache/hudi/pull/13699#discussion_r2279379802
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/FileGroupReaderBasedMergeHandle.java:
##########
@@ -156,40 +215,53 @@ private void init(CompactionOperation operation, String partitionPath) {
fileWriter = HoodieFileWriterFactory.getFileWriter(instantTime, newFilePath, hoodieTable.getStorage(),
config, writeSchemaWithMetaFields, taskContextSupplier, recordType);
} catch (IOException io) {
- LOG.error("Error in update task at commit {}", instantTime, io);
writeStatus.setGlobalError(io);
throw new HoodieUpsertException("Failed to initialize HoodieUpdateHandle for FileId: " + fileId + " on commit "
+ instantTime + " on path " + hoodieTable.getMetaClient().getBasePath(), io);
}
}
+ @Override
+ protected void populateIncomingRecordsMap(Iterator<HoodieRecord<T>> newRecordsItr) {
+ // no op.
+ }
+
+ /**
+ * This is only for spark, the engine context fetched from a serialized hoodie table is always local,
+ * overrides it to spark specific reader context.
+ */
+ public void setReaderContext(HoodieReaderContext<T> readerContext) {
+ this.readerContext = readerContext;
+ }
+
/**
* Reads the file slice of a compaction operation using a file group reader,
* by getting an iterator of the records; then writes the records to a new base file.
*/
@Override
public void doMerge() {
+ // For non-compaction operations, the merger needs to be initialized with the writer properties to handle cases like Merge-Into commands
+ if (operation.isEmpty()) {
+ this.readerContext.initRecordMergerForIngestion(config.getProps());
+ }
boolean usePosition = config.getBooleanOrDefault(MERGE_USE_RECORD_POSITIONS);
- Option<InternalSchema> internalSchemaOption = SerDeHelper.fromJson(config.getInternalSchema());
- TypedProperties props = TypedProperties.copy(config.getProps());
+ Option<InternalSchema> internalSchemaOption = SerDeHelper.fromJson(config.getInternalSchema())
+ .map(internalSchema -> AvroSchemaEvolutionUtils.reconcileSchema(writeSchemaWithMetaFields, internalSchema,
+ config.getBooleanOrDefault(HoodieCommonConfig.SET_NULL_FOR_MISSING_COLUMNS)));
long maxMemoryPerCompaction = IOUtils.getMaxMemoryPerCompaction(taskContextSupplier, config);
props.put(HoodieMemoryConfig.MAX_MEMORY_FOR_MERGE.key(), String.valueOf(maxMemoryPerCompaction));
- Stream<HoodieLogFile> logFiles = operation.getDeltaFileNames().stream().map(logFileName ->
+ Option<Stream<HoodieLogFile>> logFilesStreamOpt = operation.map(op -> op.getDeltaFileNames().stream().map(logFileName ->
new HoodieLogFile(new StoragePath(FSUtils.constructAbsolutePath(
- config.getBasePath(), operation.getPartitionPath()), logFileName)));
+ config.getBasePath(), op.getPartitionPath()), logFileName))));
// Initializes file group reader
- try (HoodieFileGroupReader<T> fileGroupReader = HoodieFileGroupReader.<T>newBuilder().withReaderContext(readerContext).withHoodieTableMetaClient(hoodieTable.getMetaClient())
- .withLatestCommitTime(maxInstantTime).withPartitionPath(partitionPath).withBaseFileOption(Option.ofNullable(baseFileToMerge)).withLogFiles(logFiles)
- .withDataSchema(writeSchemaWithMetaFields).withRequestedSchema(writeSchemaWithMetaFields).withInternalSchema(internalSchemaOption).withProps(props)
- .withShouldUseRecordPosition(usePosition).withSortOutput(hoodieTable.requireSortedRecords())
- .withFileGroupUpdateCallback(cdcLogger.map(logger -> new CDCCallback(logger, readerContext))).withEnableOptimizedLogBlockScan(config.enableOptimizedLogBlocksScan()).build()) {
+ try (HoodieFileGroupReader<T> fileGroupReader = getFileGroupReader(usePosition, internalSchemaOption, props, logFilesStreamOpt, incomingRecordsItr)) {
// Reads the records from the file slice
try (ClosableIterator<HoodieRecord<T>> recordIterator = fileGroupReader.getClosableHoodieRecordIterator()) {
while (recordIterator.hasNext()) {
HoodieRecord<T> record = recordIterator.next();
+ Option<Map<String, String>> recordMetadata = getRecordMetadata(record, writeSchema, props);
Review Comment:
We're not right now. Will update to check for operation
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9622] Add implementation of MergeHandle backed by the HoodieFileGroupReader [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13699:
URL: https://github.com/apache/hudi/pull/13699#discussion_r2279383405
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/buffer/FileGroupRecordBuffer.java:
##########
@@ -76,7 +72,7 @@ abstract class FileGroupRecordBuffer<T> implements HoodieFileGroupRecordBuffer<T
protected final RecordMergeMode recordMergeMode;
protected final PartialUpdateMode partialUpdateMode;
protected final Option<HoodieRecordMerger> recordMerger;
- protected final Option<String> payloadClass;
+ protected final Option<Pair<String, String>> payloadClasses;
Review Comment:
You can merge an expression payload with a plain avro record essentially based on my understanding of the code but it will need to be validated end to end in the next patch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9622] Add implementation of MergeHandle backed by the HoodieFileGroupReader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13699:
URL: https://github.com/apache/hudi/pull/13699#issuecomment-3190383725
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7661",
"triggerID" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"triggerType" : "PUSH"
}, {
"hash" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7666",
"triggerID" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"triggerType" : "PUSH"
}, {
"hash" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7667",
"triggerID" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"triggerType" : "PUSH"
}, {
"hash" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7672",
"triggerID" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"triggerType" : "PUSH"
}, {
"hash" : "965136302f1188e5123de37734c927dc55801b24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7691",
"triggerID" : "965136302f1188e5123de37734c927dc55801b24",
"triggerType" : "PUSH"
}, {
"hash" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7693",
"triggerID" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"triggerType" : "PUSH"
}, {
"hash" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7717",
"triggerID" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"triggerType" : "PUSH"
}, {
"hash" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"triggerType" : "PUSH"
}, {
"hash" : "47f8303d6e61504275ecfddf553e356427e6c822",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7718",
"triggerID" : "47f8303d6e61504275ecfddf553e356427e6c822",
"triggerType" : "PUSH"
}, {
"hash" : "fb191ecf48d13cd81412535f7f8c9ed553d86517",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7725",
"triggerID" : "fb191ecf48d13cd81412535f7f8c9ed553d86517",
"triggerType" : "PUSH"
}, {
"hash" : "064912d3a54714935f2ee77d55cc66816166300f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7727",
"triggerID" : "064912d3a54714935f2ee77d55cc66816166300f",
"triggerType" : "PUSH"
}, {
"hash" : "77a2b45fe13dafc51f7bebb2ba859d40291ccc10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7736",
"triggerID" : "77a2b45fe13dafc51f7bebb2ba859d40291ccc10",
"triggerType" : "PUSH"
}, {
"hash" : "e4eda0e51fee0e8de5d57ea7b38dc642f53aa95d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7758",
"triggerID" : "e4eda0e51fee0e8de5d57ea7b38dc642f53aa95d",
"triggerType" : "PUSH"
}, {
"hash" : "6cc840c51fcecde95d2187fb6fd217b50fa82806",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6cc840c51fcecde95d2187fb6fd217b50fa82806",
"triggerType" : "PUSH"
}, {
"hash" : "31dab8ce9abbe4fc80e6cd81a0e433a039ca3e1d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7760",
"triggerID" : "31dab8ce9abbe4fc80e6cd81a0e433a039ca3e1d",
"triggerType" : "PUSH"
}, {
"hash" : "c04fd95dcc94146a9146c347f4f204656f3d8308",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7763",
"triggerID" : "c04fd95dcc94146a9146c347f4f204656f3d8308",
"triggerType" : "PUSH"
}, {
"hash" : "324147913d47ba8f2555e1991ce59905e5dc0b3f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7767",
"triggerID" : "324147913d47ba8f2555e1991ce59905e5dc0b3f",
"triggerType" : "PUSH"
}, {
"hash" : "3f177d42b831b30f4bd590b057dcab6ded78b634",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7774",
"triggerID" : "3f177d42b831b30f4bd590b057dcab6ded78b634",
"triggerType" : "PUSH"
}, {
"hash" : "8d6d2c06ec82d1072251ac12ddf21a890ddd5bda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7776",
"triggerID" : "8d6d2c06ec82d1072251ac12ddf21a890ddd5bda",
"triggerType" : "PUSH"
}, {
"hash" : "77589738d73b044385f5aa6f7ef87b421f06043b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "77589738d73b044385f5aa6f7ef87b421f06043b",
"triggerType" : "PUSH"
}, {
"hash" : "ad0d7ef0fbc525f880d3baadef5a22196ab97b32",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7781",
"triggerID" : "ad0d7ef0fbc525f880d3baadef5a22196ab97b32",
"triggerType" : "PUSH"
}, {
"hash" : "0db72caa3a6c7d0f0899f243ea4b8975de79e02c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7787",
"triggerID" : "0db72caa3a6c7d0f0899f243ea4b8975de79e02c",
"triggerType" : "PUSH"
}, {
"hash" : "a01ce748dc2d4cd1fd4858d1ecac8a486989e880",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7797",
"triggerID" : "a01ce748dc2d4cd1fd4858d1ecac8a486989e880",
"triggerType" : "PUSH"
}, {
"hash" : "3f35151e28bfa1dd50b31c6e5379d9769627afd9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7799",
"triggerID" : "3f35151e28bfa1dd50b31c6e5379d9769627afd9",
"triggerType" : "PUSH"
}, {
"hash" : "b0054c8fd329489d9a1e9535d9866edffc52fdd0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b0054c8fd329489d9a1e9535d9866edffc52fdd0",
"triggerType" : "PUSH"
}, {
"hash" : "363e54341420cc217bf5af62de1e09ff370f75d2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "363e54341420cc217bf5af62de1e09ff370f75d2",
"triggerType" : "PUSH"
}, {
"hash" : "a893da375eb9c65edabbd22965296eac9b3714f4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a893da375eb9c65edabbd22965296eac9b3714f4",
"triggerType" : "PUSH"
}, {
"hash" : "a7c2756e6a293cbba99bd4e821881e6b961c138f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7804",
"triggerID" : "a7c2756e6a293cbba99bd4e821881e6b961c138f",
"triggerType" : "PUSH"
}, {
"hash" : "4a140fffa35893a13f2e7ac8365d60d51a2d0870",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7813",
"triggerID" : "4a140fffa35893a13f2e7ac8365d60d51a2d0870",
"triggerType" : "PUSH"
}, {
"hash" : "500b4f529ad2f8a695825b5badc1815202164d0a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7830",
"triggerID" : "500b4f529ad2f8a695825b5badc1815202164d0a",
"triggerType" : "PUSH"
}, {
"hash" : "539cf8b9cdc7ab4e70488e4096684022a027f942",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "539cf8b9cdc7ab4e70488e4096684022a027f942",
"triggerType" : "PUSH"
}, {
"hash" : "85139a12b431931973c07d1b7f9256a1aae75669",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "85139a12b431931973c07d1b7f9256a1aae75669",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 50022dcc45b0f511dd69706aeaf6464a51be49e2 UNKNOWN
* 6cc840c51fcecde95d2187fb6fd217b50fa82806 UNKNOWN
* 77589738d73b044385f5aa6f7ef87b421f06043b UNKNOWN
* b0054c8fd329489d9a1e9535d9866edffc52fdd0 UNKNOWN
* 363e54341420cc217bf5af62de1e09ff370f75d2 UNKNOWN
* a893da375eb9c65edabbd22965296eac9b3714f4 UNKNOWN
* 4a140fffa35893a13f2e7ac8365d60d51a2d0870 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7813)
* 500b4f529ad2f8a695825b5badc1815202164d0a Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7830)
* 539cf8b9cdc7ab4e70488e4096684022a027f942 UNKNOWN
* 85139a12b431931973c07d1b7f9256a1aae75669 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13699:
URL: https://github.com/apache/hudi/pull/13699#discussion_r2265614815
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/UpdateProcessor.java:
##########
@@ -80,17 +99,58 @@ public BufferedRecord<T> processUpdate(String recordKey, BufferedRecord<T> previ
}
return null;
} else {
- T prevRow = previousRecord != null ? previousRecord.getRecord() : null;
- T mergedRow = mergedRecord.getRecord();
- if (prevRow != null && prevRow != mergedRow) {
- mergedRecord.setHoodieOperation(HoodieOperation.UPDATE_AFTER);
- readStats.incrementNumUpdates();
- } else if (prevRow == null) {
- mergedRecord.setHoodieOperation(HoodieOperation.INSERT);
- readStats.incrementNumInserts();
+ return handleNonDeletes(previousRecord, mergedRecord);
+ }
+ }
+
+ protected BufferedRecord<T> handleNonDeletes(BufferedRecord<T> previousRecord, BufferedRecord<T> mergedRecord) {
+ T prevRow = previousRecord != null ? previousRecord.getRecord() : null;
+ T mergedRow = mergedRecord.getRecord();
+ if (prevRow != null && prevRow != mergedRow) {
+ mergedRecord.setHoodieOperation(HoodieOperation.UPDATE_AFTER);
+ readStats.incrementNumUpdates();
+ } else if (prevRow == null) {
+ mergedRecord.setHoodieOperation(HoodieOperation.INSERT);
+ readStats.incrementNumInserts();
+ }
+ return mergedRecord.seal(readerContext);
+ }
+ }
+
+ class PayloadUpdateProcessor<T> extends StandardUpdateProcessor<T> {
+ private final String payloadClass;
+ private final Properties properties;
+
+ public PayloadUpdateProcessor(HoodieReadStats readStats, HoodieReaderContext<T> readerContext, boolean emitDeletes,
+ Properties properties, String payloadClass) {
+ super(readStats, readerContext, emitDeletes);
+ this.payloadClass = payloadClass;
+ this.properties = properties;
+ }
+
+ @Override
+ protected BufferedRecord<T> handleNonDeletes(BufferedRecord<T> previousRecord, BufferedRecord<T> mergedRecord) {
+ if (previousRecord == null) {
+ // special case for payloads when there is no previous record
+ Schema recordSchema = readerContext.getRecordContext().decodeAvroSchema(mergedRecord.getSchemaId());
+ GenericRecord record = readerContext.getRecordContext().convertToAvroRecord(mergedRecord.getRecord(), recordSchema);
+ HoodieAvroRecord hoodieRecord = new HoodieAvroRecord<>(null, HoodieRecordUtils.loadPayload(payloadClass, record, mergedRecord.getOrderingValue()));
+ try {
+ if (hoodieRecord.shouldIgnore(recordSchema, properties)) {
+ return null;
+ } else {
+ Schema readerSchema = readerContext.getSchemaHandler().getRequestedSchema();
+ // If the record schema is different from the reader schema, rewrite the record using the payload methods to ensure consistency with legacy writer paths
+ if (!readerSchema.equals(recordSchema)) {
Review Comment:
This could be super costly. can it be simplified by checking the fields number?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13699:
URL: https://github.com/apache/hudi/pull/13699#discussion_r2265612035
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/FileGroupReaderBasedMergeHandle.java:
##########
@@ -267,26 +393,141 @@ private static class CDCCallback<T> implements BaseFileUpdateCallback<T> {
}
@Override
- public void onUpdate(String recordKey, T previousRecord, T mergedRecord) {
+ public void onUpdate(String recordKey, BufferedRecord<T> previousRecord, BufferedRecord<T> mergedRecord) {
cdcLogger.put(recordKey, convertOutput(previousRecord), Option.of(convertOutput(mergedRecord)));
-
}
@Override
- public void onInsert(String recordKey, T newRecord) {
+ public void onInsert(String recordKey, BufferedRecord<T> newRecord) {
cdcLogger.put(recordKey, null, Option.of(convertOutput(newRecord)));
-
}
@Override
- public void onDelete(String recordKey, T previousRecord) {
+ public void onDelete(String recordKey, BufferedRecord<T> previousRecord, HoodieOperation hoodieOperation) {
cdcLogger.put(recordKey, convertOutput(previousRecord), Option.empty());
-
}
- private GenericRecord convertOutput(T record) {
- T convertedRecord = outputConverter.get().map(converter -> record == null ? null : converter.apply(record)).orElse(record);
+ private GenericRecord convertOutput(BufferedRecord<T> record) {
+ T convertedRecord = outputConverter.get().map(converter -> record == null ? null : converter.apply(record.getRecord())).orElse(record.getRecord());
return convertedRecord == null ? null : readerContext.getRecordContext().convertToAvroRecord(convertedRecord, requestedSchema.get());
}
}
+
+ private static class RecordLevelIndexCallback<T> implements BaseFileUpdateCallback<T> {
+ private final WriteStatus writeStatus;
+ private final HoodieRecordLocation fileRecordLocation;
+ private final String partitionPath;
+
+ public RecordLevelIndexCallback(WriteStatus writeStatus, HoodieRecordLocation fileRecordLocation, String partitionPath) {
+ this.writeStatus = writeStatus;
+ this.fileRecordLocation = fileRecordLocation;
+ this.partitionPath = partitionPath;
+ }
+
+ @Override
+ public void onUpdate(String recordKey, BufferedRecord<T> previousRecord, BufferedRecord<T> mergedRecord) {
+ writeStatus.addRecordDelegate(HoodieRecordDelegate.create(recordKey, partitionPath, fileRecordLocation, fileRecordLocation, mergedRecord.getHoodieOperation() == HoodieOperation.UPDATE_BEFORE));
Review Comment:
do we even need to add the delete when `mergedRecord.getHoodieOperation() == HoodieOperation.UPDATE_BEFORE` is true ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13699:
URL: https://github.com/apache/hudi/pull/13699#discussion_r2265052470
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/IOUtils.java:
##########
@@ -119,6 +119,9 @@ public static Iterator<List<WriteStatus>> runMerge(HoodieMergeHandle<?, ?, ?, ?>
"Error in finding the old file path at commit " + instantTime + " for fileId: " + fileId);
} else {
mergeHandle.doMerge();
+ if (mergeHandle instanceof FileGroupReaderBasedMergeHandle) {
+ mergeHandle.close();
Review Comment:
Open question: Is there any reason to avoid calling close on the other merge handles?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13699:
URL: https://github.com/apache/hudi/pull/13699#discussion_r2268391105
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/FileGroupReaderBasedMergeHandle.java:
##########
@@ -267,26 +393,141 @@ private static class CDCCallback<T> implements BaseFileUpdateCallback<T> {
}
@Override
- public void onUpdate(String recordKey, T previousRecord, T mergedRecord) {
+ public void onUpdate(String recordKey, BufferedRecord<T> previousRecord, BufferedRecord<T> mergedRecord) {
cdcLogger.put(recordKey, convertOutput(previousRecord), Option.of(convertOutput(mergedRecord)));
-
}
@Override
- public void onInsert(String recordKey, T newRecord) {
+ public void onInsert(String recordKey, BufferedRecord<T> newRecord) {
cdcLogger.put(recordKey, null, Option.of(convertOutput(newRecord)));
-
}
@Override
- public void onDelete(String recordKey, T previousRecord) {
+ public void onDelete(String recordKey, BufferedRecord<T> previousRecord, HoodieOperation hoodieOperation) {
cdcLogger.put(recordKey, convertOutput(previousRecord), Option.empty());
-
}
- private GenericRecord convertOutput(T record) {
- T convertedRecord = outputConverter.get().map(converter -> record == null ? null : converter.apply(record)).orElse(record);
+ private GenericRecord convertOutput(BufferedRecord<T> record) {
+ T convertedRecord = outputConverter.get().map(converter -> record == null ? null : converter.apply(record.getRecord())).orElse(record.getRecord());
return convertedRecord == null ? null : readerContext.getRecordContext().convertToAvroRecord(convertedRecord, requestedSchema.get());
}
}
+
+ private static class RecordLevelIndexCallback<T> implements BaseFileUpdateCallback<T> {
+ private final WriteStatus writeStatus;
+ private final HoodieRecordLocation fileRecordLocation;
+ private final String partitionPath;
+
+ public RecordLevelIndexCallback(WriteStatus writeStatus, HoodieRecordLocation fileRecordLocation, String partitionPath) {
+ this.writeStatus = writeStatus;
+ this.fileRecordLocation = fileRecordLocation;
+ this.partitionPath = partitionPath;
+ }
+
+ @Override
+ public void onUpdate(String recordKey, BufferedRecord<T> previousRecord, BufferedRecord<T> mergedRecord) {
+ writeStatus.addRecordDelegate(HoodieRecordDelegate.create(recordKey, partitionPath, fileRecordLocation, fileRecordLocation, mergedRecord.getHoodieOperation() == HoodieOperation.UPDATE_BEFORE));
Review Comment:
The flag is used when all the delegates are collected into the driver and been utilized to calcurate the RLI index items for MDT, the delegate with flag `ignoreIndexUpdate` as true are just been dropped directly, so there is no need to even generate and collect them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13699:
URL: https://github.com/apache/hudi/pull/13699#discussion_r2268388626
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/FileGroupReaderBasedMergeHandle.java:
##########
@@ -267,26 +393,141 @@ private static class CDCCallback<T> implements BaseFileUpdateCallback<T> {
}
@Override
- public void onUpdate(String recordKey, T previousRecord, T mergedRecord) {
+ public void onUpdate(String recordKey, BufferedRecord<T> previousRecord, BufferedRecord<T> mergedRecord) {
cdcLogger.put(recordKey, convertOutput(previousRecord), Option.of(convertOutput(mergedRecord)));
-
}
@Override
- public void onInsert(String recordKey, T newRecord) {
+ public void onInsert(String recordKey, BufferedRecord<T> newRecord) {
cdcLogger.put(recordKey, null, Option.of(convertOutput(newRecord)));
-
}
@Override
- public void onDelete(String recordKey, T previousRecord) {
+ public void onDelete(String recordKey, BufferedRecord<T> previousRecord, HoodieOperation hoodieOperation) {
cdcLogger.put(recordKey, convertOutput(previousRecord), Option.empty());
-
}
- private GenericRecord convertOutput(T record) {
- T convertedRecord = outputConverter.get().map(converter -> record == null ? null : converter.apply(record)).orElse(record);
+ private GenericRecord convertOutput(BufferedRecord<T> record) {
+ T convertedRecord = outputConverter.get().map(converter -> record == null ? null : converter.apply(record.getRecord())).orElse(record.getRecord());
return convertedRecord == null ? null : readerContext.getRecordContext().convertToAvroRecord(convertedRecord, requestedSchema.get());
}
}
+
+ private static class RecordLevelIndexCallback<T> implements BaseFileUpdateCallback<T> {
+ private final WriteStatus writeStatus;
+ private final HoodieRecordLocation fileRecordLocation;
+ private final String partitionPath;
+
+ public RecordLevelIndexCallback(WriteStatus writeStatus, HoodieRecordLocation fileRecordLocation, String partitionPath) {
+ this.writeStatus = writeStatus;
+ this.fileRecordLocation = fileRecordLocation;
+ this.partitionPath = partitionPath;
+ }
+
+ @Override
+ public void onUpdate(String recordKey, BufferedRecord<T> previousRecord, BufferedRecord<T> mergedRecord) {
+ writeStatus.addRecordDelegate(HoodieRecordDelegate.create(recordKey, partitionPath, fileRecordLocation, fileRecordLocation, mergedRecord.getHoodieOperation() == HoodieOperation.UPDATE_BEFORE));
+ }
+
+ @Override
+ public void onInsert(String recordKey, BufferedRecord<T> newRecord) {
+ writeStatus.addRecordDelegate(HoodieRecordDelegate.create(recordKey, partitionPath, null, fileRecordLocation, newRecord.getHoodieOperation() == HoodieOperation.UPDATE_BEFORE));
Review Comment:
> but do we want to keep this in case there is some future case
I don't think so, the hoodie operation is designed to be force set up there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13699:
URL: https://github.com/apache/hudi/pull/13699#discussion_r2273349834
##########
hudi-common/src/main/java/org/apache/hudi/common/engine/RecordContext.java:
##########
@@ -340,7 +356,7 @@ public boolean supportsParquetRowIndex() {
}
private SerializableBiFunction<T, Schema, String> metadataKeyExtractor() {
- return (record, schema) -> getValue(record, schema, RECORD_KEY_METADATA_FIELD).toString();
+ return (record, schema) -> typeConverter.castToString(getValue(record, schema, RECORD_KEY_METADATA_FIELD));
Review Comment:
This may not be needed anymore. There was a bug where someone updated the BufferedRecord creation from HoodieRecord to read the values from the data instead of from the HoodieKey so I was getting some NPEs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13699:
URL: https://github.com/apache/hudi/pull/13699#issuecomment-3184016789
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7661",
"triggerID" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"triggerType" : "PUSH"
}, {
"hash" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7666",
"triggerID" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"triggerType" : "PUSH"
}, {
"hash" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7667",
"triggerID" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"triggerType" : "PUSH"
}, {
"hash" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7672",
"triggerID" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"triggerType" : "PUSH"
}, {
"hash" : "965136302f1188e5123de37734c927dc55801b24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7691",
"triggerID" : "965136302f1188e5123de37734c927dc55801b24",
"triggerType" : "PUSH"
}, {
"hash" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7693",
"triggerID" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"triggerType" : "PUSH"
}, {
"hash" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7717",
"triggerID" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"triggerType" : "PUSH"
}, {
"hash" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"triggerType" : "PUSH"
}, {
"hash" : "47f8303d6e61504275ecfddf553e356427e6c822",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7718",
"triggerID" : "47f8303d6e61504275ecfddf553e356427e6c822",
"triggerType" : "PUSH"
}, {
"hash" : "fb191ecf48d13cd81412535f7f8c9ed553d86517",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7725",
"triggerID" : "fb191ecf48d13cd81412535f7f8c9ed553d86517",
"triggerType" : "PUSH"
}, {
"hash" : "064912d3a54714935f2ee77d55cc66816166300f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7727",
"triggerID" : "064912d3a54714935f2ee77d55cc66816166300f",
"triggerType" : "PUSH"
}, {
"hash" : "77a2b45fe13dafc51f7bebb2ba859d40291ccc10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7736",
"triggerID" : "77a2b45fe13dafc51f7bebb2ba859d40291ccc10",
"triggerType" : "PUSH"
}, {
"hash" : "e4eda0e51fee0e8de5d57ea7b38dc642f53aa95d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7758",
"triggerID" : "e4eda0e51fee0e8de5d57ea7b38dc642f53aa95d",
"triggerType" : "PUSH"
}, {
"hash" : "6cc840c51fcecde95d2187fb6fd217b50fa82806",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6cc840c51fcecde95d2187fb6fd217b50fa82806",
"triggerType" : "PUSH"
}, {
"hash" : "31dab8ce9abbe4fc80e6cd81a0e433a039ca3e1d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7760",
"triggerID" : "31dab8ce9abbe4fc80e6cd81a0e433a039ca3e1d",
"triggerType" : "PUSH"
}, {
"hash" : "c04fd95dcc94146a9146c347f4f204656f3d8308",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7763",
"triggerID" : "c04fd95dcc94146a9146c347f4f204656f3d8308",
"triggerType" : "PUSH"
}, {
"hash" : "324147913d47ba8f2555e1991ce59905e5dc0b3f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7767",
"triggerID" : "324147913d47ba8f2555e1991ce59905e5dc0b3f",
"triggerType" : "PUSH"
}, {
"hash" : "3f177d42b831b30f4bd590b057dcab6ded78b634",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3f177d42b831b30f4bd590b057dcab6ded78b634",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 50022dcc45b0f511dd69706aeaf6464a51be49e2 UNKNOWN
* 6cc840c51fcecde95d2187fb6fd217b50fa82806 UNKNOWN
* 324147913d47ba8f2555e1991ce59905e5dc0b3f Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7767)
* 3f177d42b831b30f4bd590b057dcab6ded78b634 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13699:
URL: https://github.com/apache/hudi/pull/13699#issuecomment-3181922375
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7661",
"triggerID" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"triggerType" : "PUSH"
}, {
"hash" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7666",
"triggerID" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"triggerType" : "PUSH"
}, {
"hash" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7667",
"triggerID" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"triggerType" : "PUSH"
}, {
"hash" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7672",
"triggerID" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"triggerType" : "PUSH"
}, {
"hash" : "965136302f1188e5123de37734c927dc55801b24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7691",
"triggerID" : "965136302f1188e5123de37734c927dc55801b24",
"triggerType" : "PUSH"
}, {
"hash" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7693",
"triggerID" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"triggerType" : "PUSH"
}, {
"hash" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7717",
"triggerID" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"triggerType" : "PUSH"
}, {
"hash" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"triggerType" : "PUSH"
}, {
"hash" : "47f8303d6e61504275ecfddf553e356427e6c822",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7718",
"triggerID" : "47f8303d6e61504275ecfddf553e356427e6c822",
"triggerType" : "PUSH"
}, {
"hash" : "fb191ecf48d13cd81412535f7f8c9ed553d86517",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7725",
"triggerID" : "fb191ecf48d13cd81412535f7f8c9ed553d86517",
"triggerType" : "PUSH"
}, {
"hash" : "064912d3a54714935f2ee77d55cc66816166300f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7727",
"triggerID" : "064912d3a54714935f2ee77d55cc66816166300f",
"triggerType" : "PUSH"
}, {
"hash" : "77a2b45fe13dafc51f7bebb2ba859d40291ccc10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7736",
"triggerID" : "77a2b45fe13dafc51f7bebb2ba859d40291ccc10",
"triggerType" : "PUSH"
}, {
"hash" : "e4eda0e51fee0e8de5d57ea7b38dc642f53aa95d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7758",
"triggerID" : "e4eda0e51fee0e8de5d57ea7b38dc642f53aa95d",
"triggerType" : "PUSH"
}, {
"hash" : "6cc840c51fcecde95d2187fb6fd217b50fa82806",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6cc840c51fcecde95d2187fb6fd217b50fa82806",
"triggerType" : "PUSH"
}, {
"hash" : "31dab8ce9abbe4fc80e6cd81a0e433a039ca3e1d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7760",
"triggerID" : "31dab8ce9abbe4fc80e6cd81a0e433a039ca3e1d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 50022dcc45b0f511dd69706aeaf6464a51be49e2 UNKNOWN
* 6cc840c51fcecde95d2187fb6fd217b50fa82806 UNKNOWN
* 31dab8ce9abbe4fc80e6cd81a0e433a039ca3e1d Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7760)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13699:
URL: https://github.com/apache/hudi/pull/13699#issuecomment-3176859135
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7661",
"triggerID" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"triggerType" : "PUSH"
}, {
"hash" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7666",
"triggerID" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"triggerType" : "PUSH"
}, {
"hash" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7667",
"triggerID" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"triggerType" : "PUSH"
}, {
"hash" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7672",
"triggerID" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"triggerType" : "PUSH"
}, {
"hash" : "965136302f1188e5123de37734c927dc55801b24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7691",
"triggerID" : "965136302f1188e5123de37734c927dc55801b24",
"triggerType" : "PUSH"
}, {
"hash" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7693",
"triggerID" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"triggerType" : "PUSH"
}, {
"hash" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7717",
"triggerID" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"triggerType" : "PUSH"
}, {
"hash" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"triggerType" : "PUSH"
}, {
"hash" : "47f8303d6e61504275ecfddf553e356427e6c822",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7718",
"triggerID" : "47f8303d6e61504275ecfddf553e356427e6c822",
"triggerType" : "PUSH"
}, {
"hash" : "fb191ecf48d13cd81412535f7f8c9ed553d86517",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7725",
"triggerID" : "fb191ecf48d13cd81412535f7f8c9ed553d86517",
"triggerType" : "PUSH"
}, {
"hash" : "064912d3a54714935f2ee77d55cc66816166300f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "064912d3a54714935f2ee77d55cc66816166300f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 50022dcc45b0f511dd69706aeaf6464a51be49e2 UNKNOWN
* 47f8303d6e61504275ecfddf553e356427e6c822 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7718)
* fb191ecf48d13cd81412535f7f8c9ed553d86517 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7725)
* 064912d3a54714935f2ee77d55cc66816166300f UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13699:
URL: https://github.com/apache/hudi/pull/13699#discussion_r2275033897
##########
hudi-common/src/main/java/org/apache/hudi/internal/schema/utils/InternalSchemaUtils.java:
##########
@@ -57,15 +62,16 @@ public static InternalSchema pruneInternalSchema(InternalSchema schema, List<Str
List<Integer> prunedIds = names.stream().map(name -> {
int id = schema.findIdByName(name);
if (id == -1) {
- throw new IllegalArgumentException(String.format("cannot prune col: %s which does not exist in hudi table", name));
+ LOG.debug("Cannot find field name {} in schema {}, skipping.", name, schema);
+ return null;
}
return id;
- }).collect(Collectors.toList());
+ }).filter(Objects::nonNull).collect(Collectors.toList());
Review Comment:
> cannot prune col: %s which does not exist in hudi table
The original logic makes more sense, what is the use case that to prune the schema with a non existing field?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13699:
URL: https://github.com/apache/hudi/pull/13699#issuecomment-3186526317
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7661",
"triggerID" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"triggerType" : "PUSH"
}, {
"hash" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7666",
"triggerID" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"triggerType" : "PUSH"
}, {
"hash" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7667",
"triggerID" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"triggerType" : "PUSH"
}, {
"hash" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7672",
"triggerID" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"triggerType" : "PUSH"
}, {
"hash" : "965136302f1188e5123de37734c927dc55801b24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7691",
"triggerID" : "965136302f1188e5123de37734c927dc55801b24",
"triggerType" : "PUSH"
}, {
"hash" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7693",
"triggerID" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"triggerType" : "PUSH"
}, {
"hash" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7717",
"triggerID" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"triggerType" : "PUSH"
}, {
"hash" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"triggerType" : "PUSH"
}, {
"hash" : "47f8303d6e61504275ecfddf553e356427e6c822",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7718",
"triggerID" : "47f8303d6e61504275ecfddf553e356427e6c822",
"triggerType" : "PUSH"
}, {
"hash" : "fb191ecf48d13cd81412535f7f8c9ed553d86517",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7725",
"triggerID" : "fb191ecf48d13cd81412535f7f8c9ed553d86517",
"triggerType" : "PUSH"
}, {
"hash" : "064912d3a54714935f2ee77d55cc66816166300f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7727",
"triggerID" : "064912d3a54714935f2ee77d55cc66816166300f",
"triggerType" : "PUSH"
}, {
"hash" : "77a2b45fe13dafc51f7bebb2ba859d40291ccc10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7736",
"triggerID" : "77a2b45fe13dafc51f7bebb2ba859d40291ccc10",
"triggerType" : "PUSH"
}, {
"hash" : "e4eda0e51fee0e8de5d57ea7b38dc642f53aa95d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7758",
"triggerID" : "e4eda0e51fee0e8de5d57ea7b38dc642f53aa95d",
"triggerType" : "PUSH"
}, {
"hash" : "6cc840c51fcecde95d2187fb6fd217b50fa82806",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6cc840c51fcecde95d2187fb6fd217b50fa82806",
"triggerType" : "PUSH"
}, {
"hash" : "31dab8ce9abbe4fc80e6cd81a0e433a039ca3e1d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7760",
"triggerID" : "31dab8ce9abbe4fc80e6cd81a0e433a039ca3e1d",
"triggerType" : "PUSH"
}, {
"hash" : "c04fd95dcc94146a9146c347f4f204656f3d8308",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7763",
"triggerID" : "c04fd95dcc94146a9146c347f4f204656f3d8308",
"triggerType" : "PUSH"
}, {
"hash" : "324147913d47ba8f2555e1991ce59905e5dc0b3f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7767",
"triggerID" : "324147913d47ba8f2555e1991ce59905e5dc0b3f",
"triggerType" : "PUSH"
}, {
"hash" : "3f177d42b831b30f4bd590b057dcab6ded78b634",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7774",
"triggerID" : "3f177d42b831b30f4bd590b057dcab6ded78b634",
"triggerType" : "PUSH"
}, {
"hash" : "8d6d2c06ec82d1072251ac12ddf21a890ddd5bda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7776",
"triggerID" : "8d6d2c06ec82d1072251ac12ddf21a890ddd5bda",
"triggerType" : "PUSH"
}, {
"hash" : "77589738d73b044385f5aa6f7ef87b421f06043b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "77589738d73b044385f5aa6f7ef87b421f06043b",
"triggerType" : "PUSH"
}, {
"hash" : "ad0d7ef0fbc525f880d3baadef5a22196ab97b32",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7781",
"triggerID" : "ad0d7ef0fbc525f880d3baadef5a22196ab97b32",
"triggerType" : "PUSH"
}, {
"hash" : "0db72caa3a6c7d0f0899f243ea4b8975de79e02c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7787",
"triggerID" : "0db72caa3a6c7d0f0899f243ea4b8975de79e02c",
"triggerType" : "PUSH"
}, {
"hash" : "a01ce748dc2d4cd1fd4858d1ecac8a486989e880",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7797",
"triggerID" : "a01ce748dc2d4cd1fd4858d1ecac8a486989e880",
"triggerType" : "PUSH"
}, {
"hash" : "3f35151e28bfa1dd50b31c6e5379d9769627afd9",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7799",
"triggerID" : "3f35151e28bfa1dd50b31c6e5379d9769627afd9",
"triggerType" : "PUSH"
}, {
"hash" : "b0054c8fd329489d9a1e9535d9866edffc52fdd0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b0054c8fd329489d9a1e9535d9866edffc52fdd0",
"triggerType" : "PUSH"
}, {
"hash" : "363e54341420cc217bf5af62de1e09ff370f75d2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "363e54341420cc217bf5af62de1e09ff370f75d2",
"triggerType" : "PUSH"
}, {
"hash" : "a893da375eb9c65edabbd22965296eac9b3714f4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a893da375eb9c65edabbd22965296eac9b3714f4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 50022dcc45b0f511dd69706aeaf6464a51be49e2 UNKNOWN
* 6cc840c51fcecde95d2187fb6fd217b50fa82806 UNKNOWN
* 77589738d73b044385f5aa6f7ef87b421f06043b UNKNOWN
* 3f35151e28bfa1dd50b31c6e5379d9769627afd9 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7799)
* b0054c8fd329489d9a1e9535d9866edffc52fdd0 UNKNOWN
* 363e54341420cc217bf5af62de1e09ff370f75d2 UNKNOWN
* a893da375eb9c65edabbd22965296eac9b3714f4 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13699:
URL: https://github.com/apache/hudi/pull/13699#issuecomment-3186764296
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7661",
"triggerID" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"triggerType" : "PUSH"
}, {
"hash" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7666",
"triggerID" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"triggerType" : "PUSH"
}, {
"hash" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7667",
"triggerID" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"triggerType" : "PUSH"
}, {
"hash" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7672",
"triggerID" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"triggerType" : "PUSH"
}, {
"hash" : "965136302f1188e5123de37734c927dc55801b24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7691",
"triggerID" : "965136302f1188e5123de37734c927dc55801b24",
"triggerType" : "PUSH"
}, {
"hash" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7693",
"triggerID" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"triggerType" : "PUSH"
}, {
"hash" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7717",
"triggerID" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"triggerType" : "PUSH"
}, {
"hash" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"triggerType" : "PUSH"
}, {
"hash" : "47f8303d6e61504275ecfddf553e356427e6c822",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7718",
"triggerID" : "47f8303d6e61504275ecfddf553e356427e6c822",
"triggerType" : "PUSH"
}, {
"hash" : "fb191ecf48d13cd81412535f7f8c9ed553d86517",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7725",
"triggerID" : "fb191ecf48d13cd81412535f7f8c9ed553d86517",
"triggerType" : "PUSH"
}, {
"hash" : "064912d3a54714935f2ee77d55cc66816166300f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7727",
"triggerID" : "064912d3a54714935f2ee77d55cc66816166300f",
"triggerType" : "PUSH"
}, {
"hash" : "77a2b45fe13dafc51f7bebb2ba859d40291ccc10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7736",
"triggerID" : "77a2b45fe13dafc51f7bebb2ba859d40291ccc10",
"triggerType" : "PUSH"
}, {
"hash" : "e4eda0e51fee0e8de5d57ea7b38dc642f53aa95d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7758",
"triggerID" : "e4eda0e51fee0e8de5d57ea7b38dc642f53aa95d",
"triggerType" : "PUSH"
}, {
"hash" : "6cc840c51fcecde95d2187fb6fd217b50fa82806",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6cc840c51fcecde95d2187fb6fd217b50fa82806",
"triggerType" : "PUSH"
}, {
"hash" : "31dab8ce9abbe4fc80e6cd81a0e433a039ca3e1d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7760",
"triggerID" : "31dab8ce9abbe4fc80e6cd81a0e433a039ca3e1d",
"triggerType" : "PUSH"
}, {
"hash" : "c04fd95dcc94146a9146c347f4f204656f3d8308",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7763",
"triggerID" : "c04fd95dcc94146a9146c347f4f204656f3d8308",
"triggerType" : "PUSH"
}, {
"hash" : "324147913d47ba8f2555e1991ce59905e5dc0b3f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7767",
"triggerID" : "324147913d47ba8f2555e1991ce59905e5dc0b3f",
"triggerType" : "PUSH"
}, {
"hash" : "3f177d42b831b30f4bd590b057dcab6ded78b634",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7774",
"triggerID" : "3f177d42b831b30f4bd590b057dcab6ded78b634",
"triggerType" : "PUSH"
}, {
"hash" : "8d6d2c06ec82d1072251ac12ddf21a890ddd5bda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7776",
"triggerID" : "8d6d2c06ec82d1072251ac12ddf21a890ddd5bda",
"triggerType" : "PUSH"
}, {
"hash" : "77589738d73b044385f5aa6f7ef87b421f06043b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "77589738d73b044385f5aa6f7ef87b421f06043b",
"triggerType" : "PUSH"
}, {
"hash" : "ad0d7ef0fbc525f880d3baadef5a22196ab97b32",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7781",
"triggerID" : "ad0d7ef0fbc525f880d3baadef5a22196ab97b32",
"triggerType" : "PUSH"
}, {
"hash" : "0db72caa3a6c7d0f0899f243ea4b8975de79e02c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7787",
"triggerID" : "0db72caa3a6c7d0f0899f243ea4b8975de79e02c",
"triggerType" : "PUSH"
}, {
"hash" : "a01ce748dc2d4cd1fd4858d1ecac8a486989e880",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7797",
"triggerID" : "a01ce748dc2d4cd1fd4858d1ecac8a486989e880",
"triggerType" : "PUSH"
}, {
"hash" : "3f35151e28bfa1dd50b31c6e5379d9769627afd9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7799",
"triggerID" : "3f35151e28bfa1dd50b31c6e5379d9769627afd9",
"triggerType" : "PUSH"
}, {
"hash" : "b0054c8fd329489d9a1e9535d9866edffc52fdd0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b0054c8fd329489d9a1e9535d9866edffc52fdd0",
"triggerType" : "PUSH"
}, {
"hash" : "363e54341420cc217bf5af62de1e09ff370f75d2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "363e54341420cc217bf5af62de1e09ff370f75d2",
"triggerType" : "PUSH"
}, {
"hash" : "a893da375eb9c65edabbd22965296eac9b3714f4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a893da375eb9c65edabbd22965296eac9b3714f4",
"triggerType" : "PUSH"
}, {
"hash" : "a7c2756e6a293cbba99bd4e821881e6b961c138f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7804",
"triggerID" : "a7c2756e6a293cbba99bd4e821881e6b961c138f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 50022dcc45b0f511dd69706aeaf6464a51be49e2 UNKNOWN
* 6cc840c51fcecde95d2187fb6fd217b50fa82806 UNKNOWN
* 77589738d73b044385f5aa6f7ef87b421f06043b UNKNOWN
* b0054c8fd329489d9a1e9535d9866edffc52fdd0 UNKNOWN
* 363e54341420cc217bf5af62de1e09ff370f75d2 UNKNOWN
* a893da375eb9c65edabbd22965296eac9b3714f4 UNKNOWN
* a7c2756e6a293cbba99bd4e821881e6b961c138f Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7804)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13699:
URL: https://github.com/apache/hudi/pull/13699#issuecomment-3188534110
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7661",
"triggerID" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"triggerType" : "PUSH"
}, {
"hash" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7666",
"triggerID" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"triggerType" : "PUSH"
}, {
"hash" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7667",
"triggerID" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"triggerType" : "PUSH"
}, {
"hash" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7672",
"triggerID" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"triggerType" : "PUSH"
}, {
"hash" : "965136302f1188e5123de37734c927dc55801b24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7691",
"triggerID" : "965136302f1188e5123de37734c927dc55801b24",
"triggerType" : "PUSH"
}, {
"hash" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7693",
"triggerID" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"triggerType" : "PUSH"
}, {
"hash" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7717",
"triggerID" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"triggerType" : "PUSH"
}, {
"hash" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"triggerType" : "PUSH"
}, {
"hash" : "47f8303d6e61504275ecfddf553e356427e6c822",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7718",
"triggerID" : "47f8303d6e61504275ecfddf553e356427e6c822",
"triggerType" : "PUSH"
}, {
"hash" : "fb191ecf48d13cd81412535f7f8c9ed553d86517",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7725",
"triggerID" : "fb191ecf48d13cd81412535f7f8c9ed553d86517",
"triggerType" : "PUSH"
}, {
"hash" : "064912d3a54714935f2ee77d55cc66816166300f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7727",
"triggerID" : "064912d3a54714935f2ee77d55cc66816166300f",
"triggerType" : "PUSH"
}, {
"hash" : "77a2b45fe13dafc51f7bebb2ba859d40291ccc10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7736",
"triggerID" : "77a2b45fe13dafc51f7bebb2ba859d40291ccc10",
"triggerType" : "PUSH"
}, {
"hash" : "e4eda0e51fee0e8de5d57ea7b38dc642f53aa95d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7758",
"triggerID" : "e4eda0e51fee0e8de5d57ea7b38dc642f53aa95d",
"triggerType" : "PUSH"
}, {
"hash" : "6cc840c51fcecde95d2187fb6fd217b50fa82806",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6cc840c51fcecde95d2187fb6fd217b50fa82806",
"triggerType" : "PUSH"
}, {
"hash" : "31dab8ce9abbe4fc80e6cd81a0e433a039ca3e1d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7760",
"triggerID" : "31dab8ce9abbe4fc80e6cd81a0e433a039ca3e1d",
"triggerType" : "PUSH"
}, {
"hash" : "c04fd95dcc94146a9146c347f4f204656f3d8308",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7763",
"triggerID" : "c04fd95dcc94146a9146c347f4f204656f3d8308",
"triggerType" : "PUSH"
}, {
"hash" : "324147913d47ba8f2555e1991ce59905e5dc0b3f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7767",
"triggerID" : "324147913d47ba8f2555e1991ce59905e5dc0b3f",
"triggerType" : "PUSH"
}, {
"hash" : "3f177d42b831b30f4bd590b057dcab6ded78b634",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7774",
"triggerID" : "3f177d42b831b30f4bd590b057dcab6ded78b634",
"triggerType" : "PUSH"
}, {
"hash" : "8d6d2c06ec82d1072251ac12ddf21a890ddd5bda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7776",
"triggerID" : "8d6d2c06ec82d1072251ac12ddf21a890ddd5bda",
"triggerType" : "PUSH"
}, {
"hash" : "77589738d73b044385f5aa6f7ef87b421f06043b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "77589738d73b044385f5aa6f7ef87b421f06043b",
"triggerType" : "PUSH"
}, {
"hash" : "ad0d7ef0fbc525f880d3baadef5a22196ab97b32",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7781",
"triggerID" : "ad0d7ef0fbc525f880d3baadef5a22196ab97b32",
"triggerType" : "PUSH"
}, {
"hash" : "0db72caa3a6c7d0f0899f243ea4b8975de79e02c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7787",
"triggerID" : "0db72caa3a6c7d0f0899f243ea4b8975de79e02c",
"triggerType" : "PUSH"
}, {
"hash" : "a01ce748dc2d4cd1fd4858d1ecac8a486989e880",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7797",
"triggerID" : "a01ce748dc2d4cd1fd4858d1ecac8a486989e880",
"triggerType" : "PUSH"
}, {
"hash" : "3f35151e28bfa1dd50b31c6e5379d9769627afd9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7799",
"triggerID" : "3f35151e28bfa1dd50b31c6e5379d9769627afd9",
"triggerType" : "PUSH"
}, {
"hash" : "b0054c8fd329489d9a1e9535d9866edffc52fdd0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b0054c8fd329489d9a1e9535d9866edffc52fdd0",
"triggerType" : "PUSH"
}, {
"hash" : "363e54341420cc217bf5af62de1e09ff370f75d2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "363e54341420cc217bf5af62de1e09ff370f75d2",
"triggerType" : "PUSH"
}, {
"hash" : "a893da375eb9c65edabbd22965296eac9b3714f4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a893da375eb9c65edabbd22965296eac9b3714f4",
"triggerType" : "PUSH"
}, {
"hash" : "a7c2756e6a293cbba99bd4e821881e6b961c138f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7804",
"triggerID" : "a7c2756e6a293cbba99bd4e821881e6b961c138f",
"triggerType" : "PUSH"
}, {
"hash" : "4a140fffa35893a13f2e7ac8365d60d51a2d0870",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4a140fffa35893a13f2e7ac8365d60d51a2d0870",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 50022dcc45b0f511dd69706aeaf6464a51be49e2 UNKNOWN
* 6cc840c51fcecde95d2187fb6fd217b50fa82806 UNKNOWN
* 77589738d73b044385f5aa6f7ef87b421f06043b UNKNOWN
* b0054c8fd329489d9a1e9535d9866edffc52fdd0 UNKNOWN
* 363e54341420cc217bf5af62de1e09ff370f75d2 UNKNOWN
* a893da375eb9c65edabbd22965296eac9b3714f4 UNKNOWN
* a7c2756e6a293cbba99bd4e821881e6b961c138f Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7804)
* 4a140fffa35893a13f2e7ac8365d60d51a2d0870 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9622] Add implementation of MergeHandle backed by the HoodieFileGroupReader [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13699:
URL: https://github.com/apache/hudi/pull/13699#discussion_r2277901642
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/index/HoodieIndexUtils.java:
##########
@@ -526,14 +524,14 @@ public static <R> HoodieData<HoodieRecord<R>> mergeForPartitionUpdatesIfNeeded(
.distinct(updatedConfig.getGlobalIndexReconcileParallelism());
// define the buffered record merger.
ReaderContextFactory<R> readerContextFactory = (ReaderContextFactory<R>) hoodieTable.getContext()
- .getReaderContextFactoryForWrite(hoodieTable.getMetaClient(), config.getRecordMerger().getRecordType(), config.getProps());
+ .getReaderContextFactoryForWrite(hoodieTable.getMetaClient(), config.getRecordMerger().getRecordType(), config.getProps(), true);
Review Comment:
Right now we don't have any logic around when the payload is required or not. That will be handled in a future change for deprecating the payload classes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9622] Add implementation of MergeHandle backed by the HoodieFileGroupReader [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13699:
URL: https://github.com/apache/hudi/pull/13699#discussion_r2277908876
##########
hudi-client/hudi-client-common/src/test/java/org/apache/hudi/utils/HoodieWriterClientTestHarness.java:
##########
@@ -551,7 +565,7 @@ protected List<HoodieRecord<RawTripTestPayload>> dedupForCopyOnWriteStorage(Hood
int dedupParallelism = records.getNumPartitions() + additionalParallelism;
BaseHoodieWriteClient writeClient = getHoodieWriteClient(writeConfig);
HoodieReaderContext readerContext = writeClient.getEngineContext()
- .getReaderContextFactoryForWrite(metaClient, HoodieRecord.HoodieRecordType.AVRO, writeConfig.getProps()).getContext();
+ .getReaderContextFactoryForWrite(metaClient, HoodieRecord.HoodieRecordType.AVRO, writeConfig.getProps(), true).getContext();
Review Comment:
Yes, the decision will be based on the payload class name set and handled internally in the context
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9622] Add implementation of MergeHandle backed by the HoodieFileGroupReader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13699:
URL: https://github.com/apache/hudi/pull/13699#issuecomment-3190360422
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7661",
"triggerID" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"triggerType" : "PUSH"
}, {
"hash" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7666",
"triggerID" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"triggerType" : "PUSH"
}, {
"hash" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7667",
"triggerID" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"triggerType" : "PUSH"
}, {
"hash" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7672",
"triggerID" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"triggerType" : "PUSH"
}, {
"hash" : "965136302f1188e5123de37734c927dc55801b24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7691",
"triggerID" : "965136302f1188e5123de37734c927dc55801b24",
"triggerType" : "PUSH"
}, {
"hash" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7693",
"triggerID" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"triggerType" : "PUSH"
}, {
"hash" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7717",
"triggerID" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"triggerType" : "PUSH"
}, {
"hash" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"triggerType" : "PUSH"
}, {
"hash" : "47f8303d6e61504275ecfddf553e356427e6c822",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7718",
"triggerID" : "47f8303d6e61504275ecfddf553e356427e6c822",
"triggerType" : "PUSH"
}, {
"hash" : "fb191ecf48d13cd81412535f7f8c9ed553d86517",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7725",
"triggerID" : "fb191ecf48d13cd81412535f7f8c9ed553d86517",
"triggerType" : "PUSH"
}, {
"hash" : "064912d3a54714935f2ee77d55cc66816166300f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7727",
"triggerID" : "064912d3a54714935f2ee77d55cc66816166300f",
"triggerType" : "PUSH"
}, {
"hash" : "77a2b45fe13dafc51f7bebb2ba859d40291ccc10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7736",
"triggerID" : "77a2b45fe13dafc51f7bebb2ba859d40291ccc10",
"triggerType" : "PUSH"
}, {
"hash" : "e4eda0e51fee0e8de5d57ea7b38dc642f53aa95d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7758",
"triggerID" : "e4eda0e51fee0e8de5d57ea7b38dc642f53aa95d",
"triggerType" : "PUSH"
}, {
"hash" : "6cc840c51fcecde95d2187fb6fd217b50fa82806",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6cc840c51fcecde95d2187fb6fd217b50fa82806",
"triggerType" : "PUSH"
}, {
"hash" : "31dab8ce9abbe4fc80e6cd81a0e433a039ca3e1d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7760",
"triggerID" : "31dab8ce9abbe4fc80e6cd81a0e433a039ca3e1d",
"triggerType" : "PUSH"
}, {
"hash" : "c04fd95dcc94146a9146c347f4f204656f3d8308",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7763",
"triggerID" : "c04fd95dcc94146a9146c347f4f204656f3d8308",
"triggerType" : "PUSH"
}, {
"hash" : "324147913d47ba8f2555e1991ce59905e5dc0b3f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7767",
"triggerID" : "324147913d47ba8f2555e1991ce59905e5dc0b3f",
"triggerType" : "PUSH"
}, {
"hash" : "3f177d42b831b30f4bd590b057dcab6ded78b634",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7774",
"triggerID" : "3f177d42b831b30f4bd590b057dcab6ded78b634",
"triggerType" : "PUSH"
}, {
"hash" : "8d6d2c06ec82d1072251ac12ddf21a890ddd5bda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7776",
"triggerID" : "8d6d2c06ec82d1072251ac12ddf21a890ddd5bda",
"triggerType" : "PUSH"
}, {
"hash" : "77589738d73b044385f5aa6f7ef87b421f06043b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "77589738d73b044385f5aa6f7ef87b421f06043b",
"triggerType" : "PUSH"
}, {
"hash" : "ad0d7ef0fbc525f880d3baadef5a22196ab97b32",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7781",
"triggerID" : "ad0d7ef0fbc525f880d3baadef5a22196ab97b32",
"triggerType" : "PUSH"
}, {
"hash" : "0db72caa3a6c7d0f0899f243ea4b8975de79e02c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7787",
"triggerID" : "0db72caa3a6c7d0f0899f243ea4b8975de79e02c",
"triggerType" : "PUSH"
}, {
"hash" : "a01ce748dc2d4cd1fd4858d1ecac8a486989e880",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7797",
"triggerID" : "a01ce748dc2d4cd1fd4858d1ecac8a486989e880",
"triggerType" : "PUSH"
}, {
"hash" : "3f35151e28bfa1dd50b31c6e5379d9769627afd9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7799",
"triggerID" : "3f35151e28bfa1dd50b31c6e5379d9769627afd9",
"triggerType" : "PUSH"
}, {
"hash" : "b0054c8fd329489d9a1e9535d9866edffc52fdd0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b0054c8fd329489d9a1e9535d9866edffc52fdd0",
"triggerType" : "PUSH"
}, {
"hash" : "363e54341420cc217bf5af62de1e09ff370f75d2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "363e54341420cc217bf5af62de1e09ff370f75d2",
"triggerType" : "PUSH"
}, {
"hash" : "a893da375eb9c65edabbd22965296eac9b3714f4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a893da375eb9c65edabbd22965296eac9b3714f4",
"triggerType" : "PUSH"
}, {
"hash" : "a7c2756e6a293cbba99bd4e821881e6b961c138f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7804",
"triggerID" : "a7c2756e6a293cbba99bd4e821881e6b961c138f",
"triggerType" : "PUSH"
}, {
"hash" : "4a140fffa35893a13f2e7ac8365d60d51a2d0870",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7813",
"triggerID" : "4a140fffa35893a13f2e7ac8365d60d51a2d0870",
"triggerType" : "PUSH"
}, {
"hash" : "500b4f529ad2f8a695825b5badc1815202164d0a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "500b4f529ad2f8a695825b5badc1815202164d0a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 50022dcc45b0f511dd69706aeaf6464a51be49e2 UNKNOWN
* 6cc840c51fcecde95d2187fb6fd217b50fa82806 UNKNOWN
* 77589738d73b044385f5aa6f7ef87b421f06043b UNKNOWN
* b0054c8fd329489d9a1e9535d9866edffc52fdd0 UNKNOWN
* 363e54341420cc217bf5af62de1e09ff370f75d2 UNKNOWN
* a893da375eb9c65edabbd22965296eac9b3714f4 UNKNOWN
* 4a140fffa35893a13f2e7ac8365d60d51a2d0870 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7813)
* 500b4f529ad2f8a695825b5badc1815202164d0a UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9622] Add implementation of MergeHandle backed by the HoodieFileGroupReader [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13699:
URL: https://github.com/apache/hudi/pull/13699#discussion_r2278061588
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/dml/others/TestMergeIntoTable2.scala:
##########
@@ -838,72 +839,72 @@ class TestMergeIntoTable2 extends HoodieSparkSqlTestBase {
}
}
- test("Test only insert for source table in dup key without preCombineField") {
- spark.sql(s"set ${MERGE_SMALL_FILE_GROUP_CANDIDATES_LIMIT.key} = ${MERGE_SMALL_FILE_GROUP_CANDIDATES_LIMIT.defaultValue()}")
- Seq("cow", "mor").foreach {
- tableType => {
- withTempDir { tmp =>
- val tableName = generateTableName
- spark.sql(
- s"""
- | create table $tableName (
- | id int,
- | name string,
- | price double,
- | ts int,
- | dt string
- | ) using hudi
- | tblproperties (
- | type = '$tableType',
- | primaryKey = 'id'
- | )
- | partitioned by(dt)
- | location '${tmp.getCanonicalPath}'
- """.stripMargin)
- // append records to small file is use update bucket, set this conf use concat handler
- spark.sql("set hoodie.merge.allow.duplicate.on.inserts = true")
-
- // Insert data without matched condition
- spark.sql(
- s"""
- | merge into $tableName as t0
- | using (
- | select 1 as id, 'a1' as name, 10.1 as price, 1000 as ts, '2021-03-21' as dt
- | union all
- | select 1 as id, 'a2' as name, 10.2 as price, 1002 as ts, '2021-03-21' as dt
- | ) as s0
- | on t0.id = s0.id
- | when not matched then insert *
- """.stripMargin
- )
- checkAnswer(s"select id, name, price, ts, dt from $tableName")(
- Seq(1, "a1", 10.1, 1000, "2021-03-21"),
- Seq(1, "a2", 10.2, 1002, "2021-03-21")
- )
-
- // Insert data with matched condition
- spark.sql(
- s"""
- | merge into $tableName as t0
- | using (
- | select 3 as id, 'a3' as name, 10.3 as price, 1003 as ts, '2021-03-21' as dt
- | union all
- | select 1 as id, 'a2' as name, 10.4 as price, 1004 as ts, '2021-03-21' as dt
- | ) as s0
- | on t0.id = s0.id
- | when matched then update set *
- | when not matched then insert *
- """.stripMargin
- )
- checkAnswer(s"select id, name, price, ts, dt from $tableName")(
- Seq(1, "a2", 10.4, 1004, "2021-03-21"),
- Seq(1, "a2", 10.4, 1004, "2021-03-21"),
- Seq(3, "a3", 10.3, 1003, "2021-03-21")
- )
- }
- }
- }
- }
+// test("Test only insert for source table in dup key without preCombineField") {
Review Comment:
yes, just change the `test(...` to `ignore(...`, we can also comment the JIRA task ID that will renable this test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9622] Add implementation of MergeHandle backed by the HoodieFileGroupReader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13699:
URL: https://github.com/apache/hudi/pull/13699#issuecomment-3192265408
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7661",
"triggerID" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"triggerType" : "PUSH"
}, {
"hash" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7666",
"triggerID" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"triggerType" : "PUSH"
}, {
"hash" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7667",
"triggerID" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"triggerType" : "PUSH"
}, {
"hash" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7672",
"triggerID" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"triggerType" : "PUSH"
}, {
"hash" : "965136302f1188e5123de37734c927dc55801b24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7691",
"triggerID" : "965136302f1188e5123de37734c927dc55801b24",
"triggerType" : "PUSH"
}, {
"hash" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7693",
"triggerID" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"triggerType" : "PUSH"
}, {
"hash" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7717",
"triggerID" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"triggerType" : "PUSH"
}, {
"hash" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"triggerType" : "PUSH"
}, {
"hash" : "47f8303d6e61504275ecfddf553e356427e6c822",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7718",
"triggerID" : "47f8303d6e61504275ecfddf553e356427e6c822",
"triggerType" : "PUSH"
}, {
"hash" : "fb191ecf48d13cd81412535f7f8c9ed553d86517",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7725",
"triggerID" : "fb191ecf48d13cd81412535f7f8c9ed553d86517",
"triggerType" : "PUSH"
}, {
"hash" : "064912d3a54714935f2ee77d55cc66816166300f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7727",
"triggerID" : "064912d3a54714935f2ee77d55cc66816166300f",
"triggerType" : "PUSH"
}, {
"hash" : "77a2b45fe13dafc51f7bebb2ba859d40291ccc10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7736",
"triggerID" : "77a2b45fe13dafc51f7bebb2ba859d40291ccc10",
"triggerType" : "PUSH"
}, {
"hash" : "e4eda0e51fee0e8de5d57ea7b38dc642f53aa95d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7758",
"triggerID" : "e4eda0e51fee0e8de5d57ea7b38dc642f53aa95d",
"triggerType" : "PUSH"
}, {
"hash" : "6cc840c51fcecde95d2187fb6fd217b50fa82806",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6cc840c51fcecde95d2187fb6fd217b50fa82806",
"triggerType" : "PUSH"
}, {
"hash" : "31dab8ce9abbe4fc80e6cd81a0e433a039ca3e1d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7760",
"triggerID" : "31dab8ce9abbe4fc80e6cd81a0e433a039ca3e1d",
"triggerType" : "PUSH"
}, {
"hash" : "c04fd95dcc94146a9146c347f4f204656f3d8308",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7763",
"triggerID" : "c04fd95dcc94146a9146c347f4f204656f3d8308",
"triggerType" : "PUSH"
}, {
"hash" : "324147913d47ba8f2555e1991ce59905e5dc0b3f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7767",
"triggerID" : "324147913d47ba8f2555e1991ce59905e5dc0b3f",
"triggerType" : "PUSH"
}, {
"hash" : "3f177d42b831b30f4bd590b057dcab6ded78b634",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7774",
"triggerID" : "3f177d42b831b30f4bd590b057dcab6ded78b634",
"triggerType" : "PUSH"
}, {
"hash" : "8d6d2c06ec82d1072251ac12ddf21a890ddd5bda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7776",
"triggerID" : "8d6d2c06ec82d1072251ac12ddf21a890ddd5bda",
"triggerType" : "PUSH"
}, {
"hash" : "77589738d73b044385f5aa6f7ef87b421f06043b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "77589738d73b044385f5aa6f7ef87b421f06043b",
"triggerType" : "PUSH"
}, {
"hash" : "ad0d7ef0fbc525f880d3baadef5a22196ab97b32",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7781",
"triggerID" : "ad0d7ef0fbc525f880d3baadef5a22196ab97b32",
"triggerType" : "PUSH"
}, {
"hash" : "0db72caa3a6c7d0f0899f243ea4b8975de79e02c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7787",
"triggerID" : "0db72caa3a6c7d0f0899f243ea4b8975de79e02c",
"triggerType" : "PUSH"
}, {
"hash" : "a01ce748dc2d4cd1fd4858d1ecac8a486989e880",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7797",
"triggerID" : "a01ce748dc2d4cd1fd4858d1ecac8a486989e880",
"triggerType" : "PUSH"
}, {
"hash" : "3f35151e28bfa1dd50b31c6e5379d9769627afd9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7799",
"triggerID" : "3f35151e28bfa1dd50b31c6e5379d9769627afd9",
"triggerType" : "PUSH"
}, {
"hash" : "b0054c8fd329489d9a1e9535d9866edffc52fdd0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b0054c8fd329489d9a1e9535d9866edffc52fdd0",
"triggerType" : "PUSH"
}, {
"hash" : "363e54341420cc217bf5af62de1e09ff370f75d2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "363e54341420cc217bf5af62de1e09ff370f75d2",
"triggerType" : "PUSH"
}, {
"hash" : "a893da375eb9c65edabbd22965296eac9b3714f4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a893da375eb9c65edabbd22965296eac9b3714f4",
"triggerType" : "PUSH"
}, {
"hash" : "a7c2756e6a293cbba99bd4e821881e6b961c138f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7804",
"triggerID" : "a7c2756e6a293cbba99bd4e821881e6b961c138f",
"triggerType" : "PUSH"
}, {
"hash" : "4a140fffa35893a13f2e7ac8365d60d51a2d0870",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7813",
"triggerID" : "4a140fffa35893a13f2e7ac8365d60d51a2d0870",
"triggerType" : "PUSH"
}, {
"hash" : "500b4f529ad2f8a695825b5badc1815202164d0a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7830",
"triggerID" : "500b4f529ad2f8a695825b5badc1815202164d0a",
"triggerType" : "PUSH"
}, {
"hash" : "539cf8b9cdc7ab4e70488e4096684022a027f942",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "539cf8b9cdc7ab4e70488e4096684022a027f942",
"triggerType" : "PUSH"
}, {
"hash" : "85139a12b431931973c07d1b7f9256a1aae75669",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7831",
"triggerID" : "85139a12b431931973c07d1b7f9256a1aae75669",
"triggerType" : "PUSH"
}, {
"hash" : "cc430c949b539450110da96c0eccb52ab5e0998e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7832",
"triggerID" : "cc430c949b539450110da96c0eccb52ab5e0998e",
"triggerType" : "PUSH"
}, {
"hash" : "2a9cad38ac9c86ddacbb2de06950d515b3a1b0df",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2a9cad38ac9c86ddacbb2de06950d515b3a1b0df",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 50022dcc45b0f511dd69706aeaf6464a51be49e2 UNKNOWN
* 6cc840c51fcecde95d2187fb6fd217b50fa82806 UNKNOWN
* 77589738d73b044385f5aa6f7ef87b421f06043b UNKNOWN
* b0054c8fd329489d9a1e9535d9866edffc52fdd0 UNKNOWN
* 363e54341420cc217bf5af62de1e09ff370f75d2 UNKNOWN
* a893da375eb9c65edabbd22965296eac9b3714f4 UNKNOWN
* 539cf8b9cdc7ab4e70488e4096684022a027f942 UNKNOWN
* cc430c949b539450110da96c0eccb52ab5e0998e Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7832)
* 2a9cad38ac9c86ddacbb2de06950d515b3a1b0df UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13699:
URL: https://github.com/apache/hudi/pull/13699#issuecomment-3186250772
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7661",
"triggerID" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"triggerType" : "PUSH"
}, {
"hash" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7666",
"triggerID" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"triggerType" : "PUSH"
}, {
"hash" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7667",
"triggerID" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"triggerType" : "PUSH"
}, {
"hash" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7672",
"triggerID" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"triggerType" : "PUSH"
}, {
"hash" : "965136302f1188e5123de37734c927dc55801b24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7691",
"triggerID" : "965136302f1188e5123de37734c927dc55801b24",
"triggerType" : "PUSH"
}, {
"hash" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7693",
"triggerID" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"triggerType" : "PUSH"
}, {
"hash" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7717",
"triggerID" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"triggerType" : "PUSH"
}, {
"hash" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"triggerType" : "PUSH"
}, {
"hash" : "47f8303d6e61504275ecfddf553e356427e6c822",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7718",
"triggerID" : "47f8303d6e61504275ecfddf553e356427e6c822",
"triggerType" : "PUSH"
}, {
"hash" : "fb191ecf48d13cd81412535f7f8c9ed553d86517",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7725",
"triggerID" : "fb191ecf48d13cd81412535f7f8c9ed553d86517",
"triggerType" : "PUSH"
}, {
"hash" : "064912d3a54714935f2ee77d55cc66816166300f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7727",
"triggerID" : "064912d3a54714935f2ee77d55cc66816166300f",
"triggerType" : "PUSH"
}, {
"hash" : "77a2b45fe13dafc51f7bebb2ba859d40291ccc10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7736",
"triggerID" : "77a2b45fe13dafc51f7bebb2ba859d40291ccc10",
"triggerType" : "PUSH"
}, {
"hash" : "e4eda0e51fee0e8de5d57ea7b38dc642f53aa95d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7758",
"triggerID" : "e4eda0e51fee0e8de5d57ea7b38dc642f53aa95d",
"triggerType" : "PUSH"
}, {
"hash" : "6cc840c51fcecde95d2187fb6fd217b50fa82806",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6cc840c51fcecde95d2187fb6fd217b50fa82806",
"triggerType" : "PUSH"
}, {
"hash" : "31dab8ce9abbe4fc80e6cd81a0e433a039ca3e1d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7760",
"triggerID" : "31dab8ce9abbe4fc80e6cd81a0e433a039ca3e1d",
"triggerType" : "PUSH"
}, {
"hash" : "c04fd95dcc94146a9146c347f4f204656f3d8308",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7763",
"triggerID" : "c04fd95dcc94146a9146c347f4f204656f3d8308",
"triggerType" : "PUSH"
}, {
"hash" : "324147913d47ba8f2555e1991ce59905e5dc0b3f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7767",
"triggerID" : "324147913d47ba8f2555e1991ce59905e5dc0b3f",
"triggerType" : "PUSH"
}, {
"hash" : "3f177d42b831b30f4bd590b057dcab6ded78b634",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7774",
"triggerID" : "3f177d42b831b30f4bd590b057dcab6ded78b634",
"triggerType" : "PUSH"
}, {
"hash" : "8d6d2c06ec82d1072251ac12ddf21a890ddd5bda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7776",
"triggerID" : "8d6d2c06ec82d1072251ac12ddf21a890ddd5bda",
"triggerType" : "PUSH"
}, {
"hash" : "77589738d73b044385f5aa6f7ef87b421f06043b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "77589738d73b044385f5aa6f7ef87b421f06043b",
"triggerType" : "PUSH"
}, {
"hash" : "ad0d7ef0fbc525f880d3baadef5a22196ab97b32",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7781",
"triggerID" : "ad0d7ef0fbc525f880d3baadef5a22196ab97b32",
"triggerType" : "PUSH"
}, {
"hash" : "0db72caa3a6c7d0f0899f243ea4b8975de79e02c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7787",
"triggerID" : "0db72caa3a6c7d0f0899f243ea4b8975de79e02c",
"triggerType" : "PUSH"
}, {
"hash" : "a01ce748dc2d4cd1fd4858d1ecac8a486989e880",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7797",
"triggerID" : "a01ce748dc2d4cd1fd4858d1ecac8a486989e880",
"triggerType" : "PUSH"
}, {
"hash" : "3f35151e28bfa1dd50b31c6e5379d9769627afd9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3f35151e28bfa1dd50b31c6e5379d9769627afd9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 50022dcc45b0f511dd69706aeaf6464a51be49e2 UNKNOWN
* 6cc840c51fcecde95d2187fb6fd217b50fa82806 UNKNOWN
* 77589738d73b044385f5aa6f7ef87b421f06043b UNKNOWN
* a01ce748dc2d4cd1fd4858d1ecac8a486989e880 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7797)
* 3f35151e28bfa1dd50b31c6e5379d9769627afd9 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9622] Add implementation of MergeHandle backed by the HoodieFileGroupReader [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13699:
URL: https://github.com/apache/hudi/pull/13699#discussion_r2278089309
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/FileGroupReaderBasedMergeHandle.java:
##########
@@ -198,8 +270,10 @@ public void doMerge() {
}
// Writes the record
try {
- writeToFile(record.getKey(), record, writeSchemaWithMetaFields,
- config.getPayloadConfig().getProps(), preserveMetadata);
+ // if the record is not being updated and is not a new insert for the file group, we must preserve the existing record metadata.
+ boolean shouldPreserveRecordMetadata = preserveMetadata || record.getOperation() == null;
Review Comment:
looks great, thanks for the effort.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9622] Add implementation of MergeHandle backed by the HoodieFileGroupReader [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13699:
URL: https://github.com/apache/hudi/pull/13699#discussion_r2279581770
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/dml/others/TestMergeIntoTable2.scala:
##########
@@ -838,72 +839,72 @@ class TestMergeIntoTable2 extends HoodieSparkSqlTestBase {
}
}
- test("Test only insert for source table in dup key without preCombineField") {
- spark.sql(s"set ${MERGE_SMALL_FILE_GROUP_CANDIDATES_LIMIT.key} = ${MERGE_SMALL_FILE_GROUP_CANDIDATES_LIMIT.defaultValue()}")
- Seq("cow", "mor").foreach {
- tableType => {
- withTempDir { tmp =>
- val tableName = generateTableName
- spark.sql(
- s"""
- | create table $tableName (
- | id int,
- | name string,
- | price double,
- | ts int,
- | dt string
- | ) using hudi
- | tblproperties (
- | type = '$tableType',
- | primaryKey = 'id'
- | )
- | partitioned by(dt)
- | location '${tmp.getCanonicalPath}'
- """.stripMargin)
- // append records to small file is use update bucket, set this conf use concat handler
- spark.sql("set hoodie.merge.allow.duplicate.on.inserts = true")
-
- // Insert data without matched condition
- spark.sql(
- s"""
- | merge into $tableName as t0
- | using (
- | select 1 as id, 'a1' as name, 10.1 as price, 1000 as ts, '2021-03-21' as dt
- | union all
- | select 1 as id, 'a2' as name, 10.2 as price, 1002 as ts, '2021-03-21' as dt
- | ) as s0
- | on t0.id = s0.id
- | when not matched then insert *
- """.stripMargin
- )
- checkAnswer(s"select id, name, price, ts, dt from $tableName")(
- Seq(1, "a1", 10.1, 1000, "2021-03-21"),
- Seq(1, "a2", 10.2, 1002, "2021-03-21")
- )
-
- // Insert data with matched condition
- spark.sql(
- s"""
- | merge into $tableName as t0
- | using (
- | select 3 as id, 'a3' as name, 10.3 as price, 1003 as ts, '2021-03-21' as dt
- | union all
- | select 1 as id, 'a2' as name, 10.4 as price, 1004 as ts, '2021-03-21' as dt
- | ) as s0
- | on t0.id = s0.id
- | when matched then update set *
- | when not matched then insert *
- """.stripMargin
- )
- checkAnswer(s"select id, name, price, ts, dt from $tableName")(
- Seq(1, "a2", 10.4, 1004, "2021-03-21"),
- Seq(1, "a2", 10.4, 1004, "2021-03-21"),
- Seq(3, "a3", 10.3, 1003, "2021-03-21")
- )
- }
- }
- }
- }
+// test("Test only insert for source table in dup key without preCombineField") {
Review Comment:
No, that is not quite accurate. The duplicates are still carried over. Previously the duplicates would all receive the update but now only one version of the record will receive the update.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13699:
URL: https://github.com/apache/hudi/pull/13699#issuecomment-3186233952
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7661",
"triggerID" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"triggerType" : "PUSH"
}, {
"hash" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7666",
"triggerID" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"triggerType" : "PUSH"
}, {
"hash" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7667",
"triggerID" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"triggerType" : "PUSH"
}, {
"hash" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7672",
"triggerID" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"triggerType" : "PUSH"
}, {
"hash" : "965136302f1188e5123de37734c927dc55801b24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7691",
"triggerID" : "965136302f1188e5123de37734c927dc55801b24",
"triggerType" : "PUSH"
}, {
"hash" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7693",
"triggerID" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"triggerType" : "PUSH"
}, {
"hash" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7717",
"triggerID" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"triggerType" : "PUSH"
}, {
"hash" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"triggerType" : "PUSH"
}, {
"hash" : "47f8303d6e61504275ecfddf553e356427e6c822",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7718",
"triggerID" : "47f8303d6e61504275ecfddf553e356427e6c822",
"triggerType" : "PUSH"
}, {
"hash" : "fb191ecf48d13cd81412535f7f8c9ed553d86517",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7725",
"triggerID" : "fb191ecf48d13cd81412535f7f8c9ed553d86517",
"triggerType" : "PUSH"
}, {
"hash" : "064912d3a54714935f2ee77d55cc66816166300f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7727",
"triggerID" : "064912d3a54714935f2ee77d55cc66816166300f",
"triggerType" : "PUSH"
}, {
"hash" : "77a2b45fe13dafc51f7bebb2ba859d40291ccc10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7736",
"triggerID" : "77a2b45fe13dafc51f7bebb2ba859d40291ccc10",
"triggerType" : "PUSH"
}, {
"hash" : "e4eda0e51fee0e8de5d57ea7b38dc642f53aa95d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7758",
"triggerID" : "e4eda0e51fee0e8de5d57ea7b38dc642f53aa95d",
"triggerType" : "PUSH"
}, {
"hash" : "6cc840c51fcecde95d2187fb6fd217b50fa82806",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6cc840c51fcecde95d2187fb6fd217b50fa82806",
"triggerType" : "PUSH"
}, {
"hash" : "31dab8ce9abbe4fc80e6cd81a0e433a039ca3e1d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7760",
"triggerID" : "31dab8ce9abbe4fc80e6cd81a0e433a039ca3e1d",
"triggerType" : "PUSH"
}, {
"hash" : "c04fd95dcc94146a9146c347f4f204656f3d8308",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7763",
"triggerID" : "c04fd95dcc94146a9146c347f4f204656f3d8308",
"triggerType" : "PUSH"
}, {
"hash" : "324147913d47ba8f2555e1991ce59905e5dc0b3f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7767",
"triggerID" : "324147913d47ba8f2555e1991ce59905e5dc0b3f",
"triggerType" : "PUSH"
}, {
"hash" : "3f177d42b831b30f4bd590b057dcab6ded78b634",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7774",
"triggerID" : "3f177d42b831b30f4bd590b057dcab6ded78b634",
"triggerType" : "PUSH"
}, {
"hash" : "8d6d2c06ec82d1072251ac12ddf21a890ddd5bda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7776",
"triggerID" : "8d6d2c06ec82d1072251ac12ddf21a890ddd5bda",
"triggerType" : "PUSH"
}, {
"hash" : "77589738d73b044385f5aa6f7ef87b421f06043b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "77589738d73b044385f5aa6f7ef87b421f06043b",
"triggerType" : "PUSH"
}, {
"hash" : "ad0d7ef0fbc525f880d3baadef5a22196ab97b32",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7781",
"triggerID" : "ad0d7ef0fbc525f880d3baadef5a22196ab97b32",
"triggerType" : "PUSH"
}, {
"hash" : "0db72caa3a6c7d0f0899f243ea4b8975de79e02c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7787",
"triggerID" : "0db72caa3a6c7d0f0899f243ea4b8975de79e02c",
"triggerType" : "PUSH"
}, {
"hash" : "a01ce748dc2d4cd1fd4858d1ecac8a486989e880",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7797",
"triggerID" : "a01ce748dc2d4cd1fd4858d1ecac8a486989e880",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 50022dcc45b0f511dd69706aeaf6464a51be49e2 UNKNOWN
* 6cc840c51fcecde95d2187fb6fd217b50fa82806 UNKNOWN
* 77589738d73b044385f5aa6f7ef87b421f06043b UNKNOWN
* a01ce748dc2d4cd1fd4858d1ecac8a486989e880 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7797)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13699:
URL: https://github.com/apache/hudi/pull/13699#issuecomment-3186392367
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7661",
"triggerID" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"triggerType" : "PUSH"
}, {
"hash" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7666",
"triggerID" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"triggerType" : "PUSH"
}, {
"hash" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7667",
"triggerID" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"triggerType" : "PUSH"
}, {
"hash" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7672",
"triggerID" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"triggerType" : "PUSH"
}, {
"hash" : "965136302f1188e5123de37734c927dc55801b24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7691",
"triggerID" : "965136302f1188e5123de37734c927dc55801b24",
"triggerType" : "PUSH"
}, {
"hash" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7693",
"triggerID" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"triggerType" : "PUSH"
}, {
"hash" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7717",
"triggerID" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"triggerType" : "PUSH"
}, {
"hash" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"triggerType" : "PUSH"
}, {
"hash" : "47f8303d6e61504275ecfddf553e356427e6c822",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7718",
"triggerID" : "47f8303d6e61504275ecfddf553e356427e6c822",
"triggerType" : "PUSH"
}, {
"hash" : "fb191ecf48d13cd81412535f7f8c9ed553d86517",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7725",
"triggerID" : "fb191ecf48d13cd81412535f7f8c9ed553d86517",
"triggerType" : "PUSH"
}, {
"hash" : "064912d3a54714935f2ee77d55cc66816166300f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7727",
"triggerID" : "064912d3a54714935f2ee77d55cc66816166300f",
"triggerType" : "PUSH"
}, {
"hash" : "77a2b45fe13dafc51f7bebb2ba859d40291ccc10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7736",
"triggerID" : "77a2b45fe13dafc51f7bebb2ba859d40291ccc10",
"triggerType" : "PUSH"
}, {
"hash" : "e4eda0e51fee0e8de5d57ea7b38dc642f53aa95d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7758",
"triggerID" : "e4eda0e51fee0e8de5d57ea7b38dc642f53aa95d",
"triggerType" : "PUSH"
}, {
"hash" : "6cc840c51fcecde95d2187fb6fd217b50fa82806",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6cc840c51fcecde95d2187fb6fd217b50fa82806",
"triggerType" : "PUSH"
}, {
"hash" : "31dab8ce9abbe4fc80e6cd81a0e433a039ca3e1d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7760",
"triggerID" : "31dab8ce9abbe4fc80e6cd81a0e433a039ca3e1d",
"triggerType" : "PUSH"
}, {
"hash" : "c04fd95dcc94146a9146c347f4f204656f3d8308",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7763",
"triggerID" : "c04fd95dcc94146a9146c347f4f204656f3d8308",
"triggerType" : "PUSH"
}, {
"hash" : "324147913d47ba8f2555e1991ce59905e5dc0b3f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7767",
"triggerID" : "324147913d47ba8f2555e1991ce59905e5dc0b3f",
"triggerType" : "PUSH"
}, {
"hash" : "3f177d42b831b30f4bd590b057dcab6ded78b634",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7774",
"triggerID" : "3f177d42b831b30f4bd590b057dcab6ded78b634",
"triggerType" : "PUSH"
}, {
"hash" : "8d6d2c06ec82d1072251ac12ddf21a890ddd5bda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7776",
"triggerID" : "8d6d2c06ec82d1072251ac12ddf21a890ddd5bda",
"triggerType" : "PUSH"
}, {
"hash" : "77589738d73b044385f5aa6f7ef87b421f06043b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "77589738d73b044385f5aa6f7ef87b421f06043b",
"triggerType" : "PUSH"
}, {
"hash" : "ad0d7ef0fbc525f880d3baadef5a22196ab97b32",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7781",
"triggerID" : "ad0d7ef0fbc525f880d3baadef5a22196ab97b32",
"triggerType" : "PUSH"
}, {
"hash" : "0db72caa3a6c7d0f0899f243ea4b8975de79e02c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7787",
"triggerID" : "0db72caa3a6c7d0f0899f243ea4b8975de79e02c",
"triggerType" : "PUSH"
}, {
"hash" : "a01ce748dc2d4cd1fd4858d1ecac8a486989e880",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7797",
"triggerID" : "a01ce748dc2d4cd1fd4858d1ecac8a486989e880",
"triggerType" : "PUSH"
}, {
"hash" : "3f35151e28bfa1dd50b31c6e5379d9769627afd9",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7799",
"triggerID" : "3f35151e28bfa1dd50b31c6e5379d9769627afd9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 50022dcc45b0f511dd69706aeaf6464a51be49e2 UNKNOWN
* 6cc840c51fcecde95d2187fb6fd217b50fa82806 UNKNOWN
* 77589738d73b044385f5aa6f7ef87b421f06043b UNKNOWN
* 3f35151e28bfa1dd50b31c6e5379d9769627afd9 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7799)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13699:
URL: https://github.com/apache/hudi/pull/13699#issuecomment-3185055928
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7661",
"triggerID" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"triggerType" : "PUSH"
}, {
"hash" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7666",
"triggerID" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"triggerType" : "PUSH"
}, {
"hash" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7667",
"triggerID" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"triggerType" : "PUSH"
}, {
"hash" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7672",
"triggerID" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"triggerType" : "PUSH"
}, {
"hash" : "965136302f1188e5123de37734c927dc55801b24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7691",
"triggerID" : "965136302f1188e5123de37734c927dc55801b24",
"triggerType" : "PUSH"
}, {
"hash" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7693",
"triggerID" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"triggerType" : "PUSH"
}, {
"hash" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7717",
"triggerID" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"triggerType" : "PUSH"
}, {
"hash" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"triggerType" : "PUSH"
}, {
"hash" : "47f8303d6e61504275ecfddf553e356427e6c822",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7718",
"triggerID" : "47f8303d6e61504275ecfddf553e356427e6c822",
"triggerType" : "PUSH"
}, {
"hash" : "fb191ecf48d13cd81412535f7f8c9ed553d86517",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7725",
"triggerID" : "fb191ecf48d13cd81412535f7f8c9ed553d86517",
"triggerType" : "PUSH"
}, {
"hash" : "064912d3a54714935f2ee77d55cc66816166300f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7727",
"triggerID" : "064912d3a54714935f2ee77d55cc66816166300f",
"triggerType" : "PUSH"
}, {
"hash" : "77a2b45fe13dafc51f7bebb2ba859d40291ccc10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7736",
"triggerID" : "77a2b45fe13dafc51f7bebb2ba859d40291ccc10",
"triggerType" : "PUSH"
}, {
"hash" : "e4eda0e51fee0e8de5d57ea7b38dc642f53aa95d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7758",
"triggerID" : "e4eda0e51fee0e8de5d57ea7b38dc642f53aa95d",
"triggerType" : "PUSH"
}, {
"hash" : "6cc840c51fcecde95d2187fb6fd217b50fa82806",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6cc840c51fcecde95d2187fb6fd217b50fa82806",
"triggerType" : "PUSH"
}, {
"hash" : "31dab8ce9abbe4fc80e6cd81a0e433a039ca3e1d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7760",
"triggerID" : "31dab8ce9abbe4fc80e6cd81a0e433a039ca3e1d",
"triggerType" : "PUSH"
}, {
"hash" : "c04fd95dcc94146a9146c347f4f204656f3d8308",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7763",
"triggerID" : "c04fd95dcc94146a9146c347f4f204656f3d8308",
"triggerType" : "PUSH"
}, {
"hash" : "324147913d47ba8f2555e1991ce59905e5dc0b3f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7767",
"triggerID" : "324147913d47ba8f2555e1991ce59905e5dc0b3f",
"triggerType" : "PUSH"
}, {
"hash" : "3f177d42b831b30f4bd590b057dcab6ded78b634",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7774",
"triggerID" : "3f177d42b831b30f4bd590b057dcab6ded78b634",
"triggerType" : "PUSH"
}, {
"hash" : "8d6d2c06ec82d1072251ac12ddf21a890ddd5bda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7776",
"triggerID" : "8d6d2c06ec82d1072251ac12ddf21a890ddd5bda",
"triggerType" : "PUSH"
}, {
"hash" : "77589738d73b044385f5aa6f7ef87b421f06043b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "77589738d73b044385f5aa6f7ef87b421f06043b",
"triggerType" : "PUSH"
}, {
"hash" : "ad0d7ef0fbc525f880d3baadef5a22196ab97b32",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7781",
"triggerID" : "ad0d7ef0fbc525f880d3baadef5a22196ab97b32",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 50022dcc45b0f511dd69706aeaf6464a51be49e2 UNKNOWN
* 6cc840c51fcecde95d2187fb6fd217b50fa82806 UNKNOWN
* 77589738d73b044385f5aa6f7ef87b421f06043b UNKNOWN
* ad0d7ef0fbc525f880d3baadef5a22196ab97b32 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7781)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9622] Add implementation of MergeHandle backed by the HoodieFileGroupReader [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13699:
URL: https://github.com/apache/hudi/pull/13699#discussion_r2280066470
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/dml/others/TestMergeIntoTable2.scala:
##########
@@ -838,72 +839,72 @@ class TestMergeIntoTable2 extends HoodieSparkSqlTestBase {
}
}
- test("Test only insert for source table in dup key without preCombineField") {
- spark.sql(s"set ${MERGE_SMALL_FILE_GROUP_CANDIDATES_LIMIT.key} = ${MERGE_SMALL_FILE_GROUP_CANDIDATES_LIMIT.defaultValue()}")
- Seq("cow", "mor").foreach {
- tableType => {
- withTempDir { tmp =>
- val tableName = generateTableName
- spark.sql(
- s"""
- | create table $tableName (
- | id int,
- | name string,
- | price double,
- | ts int,
- | dt string
- | ) using hudi
- | tblproperties (
- | type = '$tableType',
- | primaryKey = 'id'
- | )
- | partitioned by(dt)
- | location '${tmp.getCanonicalPath}'
- """.stripMargin)
- // append records to small file is use update bucket, set this conf use concat handler
- spark.sql("set hoodie.merge.allow.duplicate.on.inserts = true")
-
- // Insert data without matched condition
- spark.sql(
- s"""
- | merge into $tableName as t0
- | using (
- | select 1 as id, 'a1' as name, 10.1 as price, 1000 as ts, '2021-03-21' as dt
- | union all
- | select 1 as id, 'a2' as name, 10.2 as price, 1002 as ts, '2021-03-21' as dt
- | ) as s0
- | on t0.id = s0.id
- | when not matched then insert *
- """.stripMargin
- )
- checkAnswer(s"select id, name, price, ts, dt from $tableName")(
- Seq(1, "a1", 10.1, 1000, "2021-03-21"),
- Seq(1, "a2", 10.2, 1002, "2021-03-21")
- )
-
- // Insert data with matched condition
- spark.sql(
- s"""
- | merge into $tableName as t0
- | using (
- | select 3 as id, 'a3' as name, 10.3 as price, 1003 as ts, '2021-03-21' as dt
- | union all
- | select 1 as id, 'a2' as name, 10.4 as price, 1004 as ts, '2021-03-21' as dt
- | ) as s0
- | on t0.id = s0.id
- | when matched then update set *
- | when not matched then insert *
- """.stripMargin
- )
- checkAnswer(s"select id, name, price, ts, dt from $tableName")(
- Seq(1, "a2", 10.4, 1004, "2021-03-21"),
- Seq(1, "a2", 10.4, 1004, "2021-03-21"),
- Seq(3, "a3", 10.3, 1003, "2021-03-21")
- )
- }
- }
- }
- }
+// test("Test only insert for source table in dup key without preCombineField") {
Review Comment:
> For mutable streams, I agree that duplicates should not be allowed.
The duplicates happens here is a pk table with MIT statement, which is incorrect.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9622] Add implementation of MergeHandle backed by the HoodieFileGroupReader [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #13699:
URL: https://github.com/apache/hudi/pull/13699#discussion_r2277620789
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/index/HoodieIndexUtils.java:
##########
@@ -492,10 +487,13 @@ private static <R> Option<HoodieRecord<R>> mergeIncomingWithExistingRecord(
// the record was deleted
return Option.empty();
}
- String partitionPath = inferPartitionPath(incoming, existing, writeSchemaWithMetaFields, keyGenerator, existingRecordContext, mergeResult);
+ Schema recordSchema = incomingRecordContext.getSchemaFromBufferRecord(mergeResult);
+ String partitionPath = inferPartitionPath(incoming, existing, recordSchema, keyGenerator, existingRecordContext, mergeResult);
HoodieRecord<R> result = existingRecordContext.constructHoodieRecord(mergeResult, partitionPath);
+ HoodieRecord<R> withMeta = result.prependMetaFields(recordSchema, writeSchemaWithMetaFields,
Review Comment:
prior to this patch, where are we prepending the meta fields? or are we fetching it from the hoodie meta fields?
```
HoodieRecordCompatibilityInterface.wrapIntoHoodieRecordPayloadWithParams() -> HoodieAvroUtils.createHoodieRecordFromAvro(...) -> SpillableMapUtils.convertToHoodieRecordPayload(...)
```
was this the path taken?
does that mean, the `result` already has meta fields populated before this patch?
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/index/HoodieIndexUtils.java:
##########
@@ -526,14 +524,14 @@ public static <R> HoodieData<HoodieRecord<R>> mergeForPartitionUpdatesIfNeeded(
.distinct(updatedConfig.getGlobalIndexReconcileParallelism());
// define the buffered record merger.
ReaderContextFactory<R> readerContextFactory = (ReaderContextFactory<R>) hoodieTable.getContext()
- .getReaderContextFactoryForWrite(hoodieTable.getMetaClient(), config.getRecordMerger().getRecordType(), config.getProps());
+ .getReaderContextFactoryForWrite(hoodieTable.getMetaClient(), config.getRecordMerger().getRecordType(), config.getProps(), true);
Review Comment:
why setting last arg (outputsCustomPayloads) to true in all cases? not all cases require payload based HoodieRecord is what I was assuming. may be I am missing something.
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/FileGroupReaderBasedMergeHandle.java:
##########
@@ -156,40 +215,53 @@ private void init(CompactionOperation operation, String partitionPath) {
fileWriter = HoodieFileWriterFactory.getFileWriter(instantTime, newFilePath, hoodieTable.getStorage(),
config, writeSchemaWithMetaFields, taskContextSupplier, recordType);
} catch (IOException io) {
- LOG.error("Error in update task at commit {}", instantTime, io);
writeStatus.setGlobalError(io);
throw new HoodieUpsertException("Failed to initialize HoodieUpdateHandle for FileId: " + fileId + " on commit "
+ instantTime + " on path " + hoodieTable.getMetaClient().getBasePath(), io);
}
}
+ @Override
+ protected void populateIncomingRecordsMap(Iterator<HoodieRecord<T>> newRecordsItr) {
+ // no op.
+ }
+
+ /**
+ * This is only for spark, the engine context fetched from a serialized hoodie table is always local,
+ * overrides it to spark specific reader context.
+ */
+ public void setReaderContext(HoodieReaderContext<T> readerContext) {
+ this.readerContext = readerContext;
+ }
+
/**
* Reads the file slice of a compaction operation using a file group reader,
* by getting an iterator of the records; then writes the records to a new base file.
*/
@Override
public void doMerge() {
+ // For non-compaction operations, the merger needs to be initialized with the writer properties to handle cases like Merge-Into commands
+ if (operation.isEmpty()) {
+ this.readerContext.initRecordMergerForIngestion(config.getProps());
+ }
boolean usePosition = config.getBooleanOrDefault(MERGE_USE_RECORD_POSITIONS);
- Option<InternalSchema> internalSchemaOption = SerDeHelper.fromJson(config.getInternalSchema());
- TypedProperties props = TypedProperties.copy(config.getProps());
+ Option<InternalSchema> internalSchemaOption = SerDeHelper.fromJson(config.getInternalSchema())
+ .map(internalSchema -> AvroSchemaEvolutionUtils.reconcileSchema(writeSchemaWithMetaFields, internalSchema,
+ config.getBooleanOrDefault(HoodieCommonConfig.SET_NULL_FOR_MISSING_COLUMNS)));
long maxMemoryPerCompaction = IOUtils.getMaxMemoryPerCompaction(taskContextSupplier, config);
props.put(HoodieMemoryConfig.MAX_MEMORY_FOR_MERGE.key(), String.valueOf(maxMemoryPerCompaction));
- Stream<HoodieLogFile> logFiles = operation.getDeltaFileNames().stream().map(logFileName ->
+ Option<Stream<HoodieLogFile>> logFilesStreamOpt = operation.map(op -> op.getDeltaFileNames().stream().map(logFileName ->
new HoodieLogFile(new StoragePath(FSUtils.constructAbsolutePath(
- config.getBasePath(), operation.getPartitionPath()), logFileName)));
+ config.getBasePath(), op.getPartitionPath()), logFileName))));
// Initializes file group reader
- try (HoodieFileGroupReader<T> fileGroupReader = HoodieFileGroupReader.<T>newBuilder().withReaderContext(readerContext).withHoodieTableMetaClient(hoodieTable.getMetaClient())
- .withLatestCommitTime(maxInstantTime).withPartitionPath(partitionPath).withBaseFileOption(Option.ofNullable(baseFileToMerge)).withLogFiles(logFiles)
- .withDataSchema(writeSchemaWithMetaFields).withRequestedSchema(writeSchemaWithMetaFields).withInternalSchema(internalSchemaOption).withProps(props)
- .withShouldUseRecordPosition(usePosition).withSortOutput(hoodieTable.requireSortedRecords())
- .withFileGroupUpdateCallback(cdcLogger.map(logger -> new CDCCallback(logger, readerContext))).withEnableOptimizedLogBlockScan(config.enableOptimizedLogBlocksScan()).build()) {
+ try (HoodieFileGroupReader<T> fileGroupReader = getFileGroupReader(usePosition, internalSchemaOption, props, logFilesStreamOpt, incomingRecordsItr)) {
// Reads the records from the file slice
try (ClosableIterator<HoodieRecord<T>> recordIterator = fileGroupReader.getClosableHoodieRecordIterator()) {
while (recordIterator.hasNext()) {
HoodieRecord<T> record = recordIterator.next();
+ Option<Map<String, String>> recordMetadata = getRecordMetadata(record, writeSchema, props);
Review Comment:
Note to reviewer: This fixes the event time metadata tracking for FG reader based merge handle
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/table/action/commit/BaseSparkCommitActionExecutor.java:
##########
@@ -397,6 +401,9 @@ protected HoodieMergeHandle getUpdateHandle(String partitionPath, String fileId,
mergeHandle.setPartitionFields(partitionFields);
mergeHandle.setPartitionValues(partitionValues);
}
+ if (readerContextFactory != null && mergeHandle instanceof FileGroupReaderBasedMergeHandle) {
Review Comment:
anyways, constructor for FileGroupReaderBasedMergeHandle is diff from HoodieWriteMergeHandle.
so, do you think we can add a new arg to constructor as Option<SerializableFunc> which can assist with setting the reader context from within FileGroupReaderBasedMergeHandle class only, rather than special casing here at L405.
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/UpdateProcessor.java:
##########
@@ -80,17 +101,56 @@ public BufferedRecord<T> processUpdate(String recordKey, BufferedRecord<T> previ
}
return null;
} else {
- T prevRow = previousRecord != null ? previousRecord.getRecord() : null;
- T mergedRow = mergedRecord.getRecord();
- if (prevRow != null && prevRow != mergedRow) {
- mergedRecord.setHoodieOperation(HoodieOperation.UPDATE_AFTER);
- readStats.incrementNumUpdates();
- } else if (prevRow == null) {
- mergedRecord.setHoodieOperation(HoodieOperation.INSERT);
- readStats.incrementNumInserts();
+ return handleNonDeletes(previousRecord, mergedRecord);
+ }
+ }
+
+ protected BufferedRecord<T> handleNonDeletes(BufferedRecord<T> previousRecord, BufferedRecord<T> mergedRecord) {
Review Comment:
did we add good UTs against these new code
##########
hudi-spark-datasource/hudi-spark/src/test/java/org/apache/hudi/io/TestMergeHandle.java:
##########
@@ -124,4 +159,390 @@ public void testMergeHandleRLIAndSIStatsWithUpdatesAndDeletes() throws Exception
assertEquals(2 * numUpdates + numDeletes, writeStatus.getIndexStats().getSecondaryIndexStats().values().stream().findFirst().get().size());
validateSecondaryIndexStatsContent(writeStatus, numUpdates, numDeletes);
}
+
+ @ParameterizedTest // TODO add CUSTOM_MERGER once deletes are handled properly
+ @ValueSource(strings = {"EVENT_TIME_ORDERING", "COMMIT_TIME_ORDERING", "CUSTOM"})
Review Comment:
where are we testing CUSTOM_MERGER ? I only see 3 being tested here.
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/HoodieTable.java:
##########
@@ -1280,4 +1281,10 @@ private Set<String> getDropPartitionColNames() {
}
return new HashSet<>(Arrays.asList(partitionFields.get()));
}
+
+ public ReaderContextFactory<T> getReaderContextFactoryForWrite() {
+ // question: should we just return null when context is serialized as null? the mismatch reader context would throw anyway.
+ return (ReaderContextFactory<T>) getContext().getReaderContextFactoryForWrite(metaClient, config.getRecordMerger().getRecordType(),
Review Comment:
I feel the spark reader context usage in write path as a one off thing. and we can leave it as is. apart from this use-case, we don't forsee more use-cases in near future.
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/index/HoodieIndexUtils.java:
##########
@@ -526,14 +524,14 @@ public static <R> HoodieData<HoodieRecord<R>> mergeForPartitionUpdatesIfNeeded(
.distinct(updatedConfig.getGlobalIndexReconcileParallelism());
// define the buffered record merger.
ReaderContextFactory<R> readerContextFactory = (ReaderContextFactory<R>) hoodieTable.getContext()
- .getReaderContextFactoryForWrite(hoodieTable.getMetaClient(), config.getRecordMerger().getRecordType(), config.getProps());
+ .getReaderContextFactoryForWrite(hoodieTable.getMetaClient(), config.getRecordMerger().getRecordType(), config.getProps(), true);
HoodieReaderContext<R> readerContext = readerContextFactory.getContext();
RecordContext<R> incomingRecordContext = readerContext.getRecordContext();
readerContext.initRecordMergerForIngestion(config.getProps());
// Create a reader context for the existing records. In the case of merge-into commands, the incoming records
// can be using an expression payload so here we rely on the table's configured payload class if it is required.
ReaderContextFactory<R> readerContextFactoryForExistingRecords = (ReaderContextFactory<R>) hoodieTable.getContext()
- .getReaderContextFactoryForWrite(hoodieTable.getMetaClient(), config.getRecordMerger().getRecordType(), hoodieTable.getMetaClient().getTableConfig().getProps());
+ .getReaderContextFactoryForWrite(hoodieTable.getMetaClient(), config.getRecordMerger().getRecordType(), hoodieTable.getMetaClient().getTableConfig().getProps(), true);
Review Comment:
same q as above
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/WriteStatus.java:
##########
@@ -109,6 +109,17 @@ public void markSuccess(HoodieRecord record, Option<Map<String, String>> optiona
updateStatsForSuccess(optionalRecordMetadata);
}
+ /**
+ * Allows the writer to manually add record delegates to the index stats.
+ */
+ public void manuallyTrackSuccess() {
Review Comment:
I would have probably moved
```
this.writeStatus = (WriteStatus) ReflectionUtils.loadClass(config.getWriteStatusClassName(),
hoodieTable.shouldTrackSuccessRecords(), config.getWriteStatusFailureFraction(), hoodieTable.isMetadataTable());
```
to a protected method in HoodieWriteHandle and override it from FileGroupReaderBasedMergeHandle and override this behavior just for cow merge cases.
Just that it does not look neat when write handle initiates w/ `trackSuccessRecords` as true, and later we disable it.
not really a blocker comment. will leave it to you to address if possible.
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/FileGroupReaderBasedMergeHandle.java:
##########
@@ -251,6 +344,31 @@ public List<WriteStatus> close() {
}
}
+ private Option<BaseFileUpdateCallback<T>> createCallback() {
+ List<BaseFileUpdateCallback<T>> callbacks = new ArrayList<>();
+ // Handle CDC workflow.
+ if (cdcLogger.isPresent()) {
+ callbacks.add(new CDCCallback<>(cdcLogger.get(), readerContext));
+ }
+ // Indexes are not updated during compaction
+ if (operation.isEmpty()) {
+ // record index callback
+ if (this.writeStatus.isTrackingSuccessfulWrites()) {
+ writeStatus.manuallyTrackSuccess();
+ callbacks.add(new RecordLevelIndexCallback<>(writeStatus, newRecordLocation, partitionPath));
+ }
+ // Stream secondary index stats.
+ if (isSecondaryIndexStatsStreamingWritesEnabled) {
+ callbacks.add(new SecondaryIndexCallback<>(
Review Comment:
I thought we had a discussion to have just 1 callback for RLI and SI. did you attempt and had to revert due to some issues?
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/FileGroupReaderBasedMergeHandle.java:
##########
@@ -198,8 +270,10 @@ public void doMerge() {
}
// Writes the record
try {
- writeToFile(record.getKey(), record, writeSchemaWithMetaFields,
- config.getPayloadConfig().getProps(), preserveMetadata);
+ // if the record is not being updated and is not a new insert for the file group, we must preserve the existing record metadata.
+ boolean shouldPreserveRecordMetadata = preserveMetadata || record.getOperation() == null;
Review Comment:
can you help me understand how the record.operation is being set.
I see that we set it to HoodieOperation.UPDATE_BEFORE where we don't need to update the index . otherwise, its the operation that we set in driver when we create the HoodieRecords which is likely null right?
##########
hudi-client/hudi-client-common/src/test/java/org/apache/hudi/utils/HoodieWriterClientTestHarness.java:
##########
@@ -551,7 +565,7 @@ protected List<HoodieRecord<RawTripTestPayload>> dedupForCopyOnWriteStorage(Hood
int dedupParallelism = records.getNumPartitions() + additionalParallelism;
BaseHoodieWriteClient writeClient = getHoodieWriteClient(writeConfig);
HoodieReaderContext readerContext = writeClient.getEngineContext()
- .getReaderContextFactoryForWrite(metaClient, HoodieRecord.HoodieRecordType.AVRO, writeConfig.getProps()).getContext();
+ .getReaderContextFactoryForWrite(metaClient, HoodieRecord.HoodieRecordType.AVRO, writeConfig.getProps(), true).getContext();
Review Comment:
all these places where we are setting to true for `outputsCustomPayloads` will be fixed in your next patch where we will be looking to move away from payload based HoodieRecord ?
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/UpdateProcessor.java:
##########
@@ -39,8 +52,16 @@ public interface UpdateProcessor<T> {
BufferedRecord<T> processUpdate(String recordKey, BufferedRecord<T> previousRecord, BufferedRecord<T> mergedRecord, boolean isDelete);
static <T> UpdateProcessor<T> create(HoodieReadStats readStats, HoodieReaderContext<T> readerContext,
- boolean emitDeletes, Option<BaseFileUpdateCallback<T>> updateCallback) {
- UpdateProcessor<T> handler = new StandardUpdateProcessor<>(readStats, readerContext, emitDeletes);
+ boolean emitDeletes, Option<BaseFileUpdateCallback<T>> updateCallback,
+ TypedProperties properties) {
+ UpdateProcessor<T> handler;
+ Option<String> payloadClass = readerContext.getPayloadClasses(properties).map(Pair::getRight);
Review Comment:
`HoodeReaderContext.getPayloadClasses()` will not return payload class unless the merge strategyId is = PAYLOAD_BASED_MERGE_STRATEGY_UUID right.
so, once we land w/ table v9 table creation patch, and rebase this patch, we do not need to make any additional fixes in these lines?
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/buffer/PositionBasedFileGroupRecordBuffer.java:
##########
@@ -203,7 +203,7 @@ public void processDeleteBlock(HoodieDeleteBlock deleteBlock) throws IOException
// this delete-vector could be kept in the records cache(see the check in #fallbackToKeyBasedBuffer),
// and these keys would be deleted no matter whether there are following-up inserts/updates.
DeleteRecord deleteRecord = deleteRecords[commitTimeBasedRecordIndex++];
- BufferedRecord<T> record = BufferedRecords.fromDeleteRecord(deleteRecord, deleteRecord.getOrderingValue());
+ BufferedRecord<T> record = BufferedRecords.fromDeleteRecord(deleteRecord, readerContext.getRecordContext());
Review Comment:
can we update PR description to call out what changes we are doing wrt delete records and ordering values
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/UpdateProcessor.java:
##########
@@ -80,17 +101,56 @@ public BufferedRecord<T> processUpdate(String recordKey, BufferedRecord<T> previ
}
return null;
} else {
- T prevRow = previousRecord != null ? previousRecord.getRecord() : null;
- T mergedRow = mergedRecord.getRecord();
- if (prevRow != null && prevRow != mergedRow) {
- mergedRecord.setHoodieOperation(HoodieOperation.UPDATE_AFTER);
- readStats.incrementNumUpdates();
- } else if (prevRow == null) {
- mergedRecord.setHoodieOperation(HoodieOperation.INSERT);
- readStats.incrementNumInserts();
+ return handleNonDeletes(previousRecord, mergedRecord);
+ }
+ }
+
+ protected BufferedRecord<T> handleNonDeletes(BufferedRecord<T> previousRecord, BufferedRecord<T> mergedRecord) {
+ T prevRow = previousRecord != null ? previousRecord.getRecord() : null;
+ T mergedRow = mergedRecord.getRecord();
+ if (prevRow != null && prevRow != mergedRow) {
+ mergedRecord.setHoodieOperation(HoodieOperation.UPDATE_AFTER);
+ readStats.incrementNumUpdates();
+ } else if (prevRow == null) {
+ mergedRecord.setHoodieOperation(HoodieOperation.INSERT);
+ readStats.incrementNumInserts();
+ }
+ return mergedRecord.seal(readerContext.getRecordContext());
+ }
+ }
+
+ class PayloadUpdateProcessor<T> extends StandardUpdateProcessor<T> {
Review Comment:
same as above. did we add good UTs against these new code
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/SecondaryIndexStreamingTracker.java:
##########
@@ -226,6 +228,81 @@ static <T> void trackSecondaryIndexStats(@Nullable HoodieKey hoodieKey, Option<H
});
}
+ /**
+ * The utility function used by Merge Handle to generate secondary index stats for the corresponding record.
+ * It considers the new merged version of the record and compares it with the older version of the record to generate
+ * secondary index stats.
+ *
+ * @param hoodieKey The hoodie key
+ * @param combinedRecordOpt New record merged with the old record
+ * @param oldRecord The old record
+ * @param isDelete Whether the record is a DELETE
+ * @param writeStatus The Write status
+ * @param secondaryIndexDefns Definitions for secondary index which need to be updated
+ */
+ static <T> void trackSecondaryIndexStats(HoodieKey hoodieKey, Option<BufferedRecord<T>> combinedRecordOpt, @Nullable BufferedRecord<T> oldRecord, boolean isDelete,
Review Comment:
cool. have we added UTs for this
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/buffer/FileGroupRecordBuffer.java:
##########
@@ -76,7 +72,7 @@ abstract class FileGroupRecordBuffer<T> implements HoodieFileGroupRecordBuffer<T
protected final RecordMergeMode recordMergeMode;
protected final PartialUpdateMode partialUpdateMode;
protected final Option<HoodieRecordMerger> recordMerger;
- protected final Option<String> payloadClass;
+ protected final Option<Pair<String, String>> payloadClasses;
Review Comment:
can we add java docs as to why we need a Pair of payload classes instead of just 1 entry
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/BufferedRecordMergerFactory.java:
##########
@@ -426,6 +428,16 @@ protected Pair<HoodieRecord, HoodieRecord> getFinalMergeRecords(BufferedRecord<T
HoodieRecord newHoodieRecord = constructHoodieAvroRecord(recordContext, newerRecord, payloadClass);
return Pair.of(oldHoodieRecord, newHoodieRecord);
}
+
+ @Override
+ protected Option<Pair<HoodieRecord, Schema>> getMergedRecord(BufferedRecord<T> olderRecord, BufferedRecord<T> newerRecord, boolean isFinalMerge) throws IOException {
Review Comment:
did we add UTs for this at BufferedRecordMerger layer. essentially for the ExpressionPayloadRecordMerger
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/FileGroupReaderBasedMergeHandle.java:
##########
@@ -198,8 +270,10 @@ public void doMerge() {
}
// Writes the record
try {
- writeToFile(record.getKey(), record, writeSchemaWithMetaFields,
- config.getPayloadConfig().getProps(), preserveMetadata);
+ // if the record is not being updated and is not a new insert for the file group, we must preserve the existing record metadata.
+ boolean shouldPreserveRecordMetadata = preserveMetadata || record.getOperation() == null;
Review Comment:
I understand the java docs. but trying to tie the code to that
for compaction, its straight forward since `preserveMetadata` is true.
incase of cow merge case, we might get 3 diff kind of records.
A : records copied over from previous base file w/o any new updates.
B: records being updated.
C: new inserts which was never present in prev base file.
w/o understanding what the code is doing, I would assume we might wanna preserve metadata for case A only. for other two cases, we wanted to override the metadata.
Even within B, we could have a branching, where the updated version has lower ordering value and so the previous version of the record is chosen. but lets just keep it to A, B and C.
Can you help me understand what the patch does for these 3 cases.
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/dml/others/TestMergeIntoTable2.scala:
##########
@@ -838,72 +839,72 @@ class TestMergeIntoTable2 extends HoodieSparkSqlTestBase {
}
}
- test("Test only insert for source table in dup key without preCombineField") {
- spark.sql(s"set ${MERGE_SMALL_FILE_GROUP_CANDIDATES_LIMIT.key} = ${MERGE_SMALL_FILE_GROUP_CANDIDATES_LIMIT.defaultValue()}")
- Seq("cow", "mor").foreach {
- tableType => {
- withTempDir { tmp =>
- val tableName = generateTableName
- spark.sql(
- s"""
- | create table $tableName (
- | id int,
- | name string,
- | price double,
- | ts int,
- | dt string
- | ) using hudi
- | tblproperties (
- | type = '$tableType',
- | primaryKey = 'id'
- | )
- | partitioned by(dt)
- | location '${tmp.getCanonicalPath}'
- """.stripMargin)
- // append records to small file is use update bucket, set this conf use concat handler
- spark.sql("set hoodie.merge.allow.duplicate.on.inserts = true")
-
- // Insert data without matched condition
- spark.sql(
- s"""
- | merge into $tableName as t0
- | using (
- | select 1 as id, 'a1' as name, 10.1 as price, 1000 as ts, '2021-03-21' as dt
- | union all
- | select 1 as id, 'a2' as name, 10.2 as price, 1002 as ts, '2021-03-21' as dt
- | ) as s0
- | on t0.id = s0.id
- | when not matched then insert *
- """.stripMargin
- )
- checkAnswer(s"select id, name, price, ts, dt from $tableName")(
- Seq(1, "a1", 10.1, 1000, "2021-03-21"),
- Seq(1, "a2", 10.2, 1002, "2021-03-21")
- )
-
- // Insert data with matched condition
- spark.sql(
- s"""
- | merge into $tableName as t0
- | using (
- | select 3 as id, 'a3' as name, 10.3 as price, 1003 as ts, '2021-03-21' as dt
- | union all
- | select 1 as id, 'a2' as name, 10.4 as price, 1004 as ts, '2021-03-21' as dt
- | ) as s0
- | on t0.id = s0.id
- | when matched then update set *
- | when not matched then insert *
- """.stripMargin
- )
- checkAnswer(s"select id, name, price, ts, dt from $tableName")(
- Seq(1, "a2", 10.4, 1004, "2021-03-21"),
- Seq(1, "a2", 10.4, 1004, "2021-03-21"),
- Seq(3, "a3", 10.3, 1003, "2021-03-21")
- )
- }
- }
- }
- }
+// test("Test only insert for source table in dup key without preCombineField") {
Review Comment:
why disabled? do we plan to revisit this later and fix it.
if its not a valid use case anymore, why not remove it fully?
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/index/HoodieIndexUtils.java:
##########
@@ -492,10 +487,13 @@ private static <R> Option<HoodieRecord<R>> mergeIncomingWithExistingRecord(
// the record was deleted
return Option.empty();
}
- String partitionPath = inferPartitionPath(incoming, existing, writeSchemaWithMetaFields, keyGenerator, existingRecordContext, mergeResult);
+ Schema recordSchema = incomingRecordContext.getSchemaFromBufferRecord(mergeResult);
+ String partitionPath = inferPartitionPath(incoming, existing, recordSchema, keyGenerator, existingRecordContext, mergeResult);
HoodieRecord<R> result = existingRecordContext.constructHoodieRecord(mergeResult, partitionPath);
+ HoodieRecord<R> withMeta = result.prependMetaFields(recordSchema, writeSchemaWithMetaFields,
Review Comment:
or is it that, the `result` had meta fields before this patch (AvroRecord in payload format), where as w/ this patch, we are converting it to AvroIndexedRecord and it does not contain the meta fields?
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/table/action/commit/BaseSparkCommitActionExecutor.java:
##########
@@ -397,6 +401,9 @@ protected HoodieMergeHandle getUpdateHandle(String partitionPath, String fileId,
mergeHandle.setPartitionFields(partitionFields);
mergeHandle.setPartitionValues(partitionValues);
}
+ if (readerContextFactory != null && mergeHandle instanceof FileGroupReaderBasedMergeHandle) {
Review Comment:
not to strong on the suggestion though. Will let you decide.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9622] Add implementation of MergeHandle backed by the HoodieFileGroupReader [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13699:
URL: https://github.com/apache/hudi/pull/13699#discussion_r2278040845
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/dml/others/TestMergeIntoTable2.scala:
##########
@@ -838,72 +839,72 @@ class TestMergeIntoTable2 extends HoodieSparkSqlTestBase {
}
}
- test("Test only insert for source table in dup key without preCombineField") {
- spark.sql(s"set ${MERGE_SMALL_FILE_GROUP_CANDIDATES_LIMIT.key} = ${MERGE_SMALL_FILE_GROUP_CANDIDATES_LIMIT.defaultValue()}")
- Seq("cow", "mor").foreach {
- tableType => {
- withTempDir { tmp =>
- val tableName = generateTableName
- spark.sql(
- s"""
- | create table $tableName (
- | id int,
- | name string,
- | price double,
- | ts int,
- | dt string
- | ) using hudi
- | tblproperties (
- | type = '$tableType',
- | primaryKey = 'id'
- | )
- | partitioned by(dt)
- | location '${tmp.getCanonicalPath}'
- """.stripMargin)
- // append records to small file is use update bucket, set this conf use concat handler
- spark.sql("set hoodie.merge.allow.duplicate.on.inserts = true")
-
- // Insert data without matched condition
- spark.sql(
- s"""
- | merge into $tableName as t0
- | using (
- | select 1 as id, 'a1' as name, 10.1 as price, 1000 as ts, '2021-03-21' as dt
- | union all
- | select 1 as id, 'a2' as name, 10.2 as price, 1002 as ts, '2021-03-21' as dt
- | ) as s0
- | on t0.id = s0.id
- | when not matched then insert *
- """.stripMargin
- )
- checkAnswer(s"select id, name, price, ts, dt from $tableName")(
- Seq(1, "a1", 10.1, 1000, "2021-03-21"),
- Seq(1, "a2", 10.2, 1002, "2021-03-21")
- )
-
- // Insert data with matched condition
- spark.sql(
- s"""
- | merge into $tableName as t0
- | using (
- | select 3 as id, 'a3' as name, 10.3 as price, 1003 as ts, '2021-03-21' as dt
- | union all
- | select 1 as id, 'a2' as name, 10.4 as price, 1004 as ts, '2021-03-21' as dt
- | ) as s0
- | on t0.id = s0.id
- | when matched then update set *
- | when not matched then insert *
- """.stripMargin
- )
- checkAnswer(s"select id, name, price, ts, dt from $tableName")(
- Seq(1, "a2", 10.4, 1004, "2021-03-21"),
- Seq(1, "a2", 10.4, 1004, "2021-03-21"),
- Seq(3, "a3", 10.3, 1003, "2021-03-21")
- )
- }
- }
- }
- }
+// test("Test only insert for source table in dup key without preCombineField") {
Review Comment:
It does not make sense for a pk table to have duplicates in any case, will fix it in https://issues.apache.org/jira/browse/HUDI-9708, the MIT got a lot of issues today and would be consolidated in 1.2 release.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9622] Add implementation of MergeHandle backed by the HoodieFileGroupReader [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13699:
URL: https://github.com/apache/hudi/pull/13699#discussion_r2278043646
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/table/action/commit/BaseSparkCommitActionExecutor.java:
##########
@@ -397,6 +401,9 @@ protected HoodieMergeHandle getUpdateHandle(String partitionPath, String fileId,
mergeHandle.setPartitionFields(partitionFields);
mergeHandle.setPartitionValues(partitionValues);
}
+ if (readerContextFactory != null && mergeHandle instanceof FileGroupReaderBasedMergeHandle) {
Review Comment:
Another reason is this param is only needed for the FG merge handle now, we don't want to init the reader context for all the merge handles, also modifying the merge handle param will need to modify all the executors, which is also unnecessary, like Java and Flink can fetch the reader context directly from hoodie table.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9622] Add implementation of MergeHandle backed by the HoodieFileGroupReader [hudi]
Posted by "cshuo (via GitHub)" <gi...@apache.org>.
cshuo commented on code in PR #13699:
URL: https://github.com/apache/hudi/pull/13699#discussion_r2278085687
##########
hudi-common/src/main/java/org/apache/hudi/common/model/HoodieRecordPayload.java:
##########
@@ -217,4 +217,16 @@ static Option<String> getPayloadClassNameIfPresent(Properties props) {
// Need to transparently change to org.apache.hudi.
return Option.ofNullable(payloadClassName).map(className -> className.replace("com.uber.hoodie", "org.apache.hudi"));
}
+
+ /**
+ * Returns the writer payload override class if present in the properties.
+ * @param properties the properties to inspect
+ * @return an Option containing the writer payload override class name if present, otherwise an empty Option
+ */
+ static Option<String> getWriterPayloadOverride(Properties properties) {
+ if (properties.containsKey("hoodie.datasource.write.payload.class")) {
Review Comment:
`if (containsKey()..) {Option.of()} else {}` -> `Option.ofNullable().map()` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9622] Add implementation of MergeHandle backed by the HoodieFileGroupReader [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #13699:
URL: https://github.com/apache/hudi/pull/13699#discussion_r2279125925
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/dml/others/TestMergeIntoTable2.scala:
##########
@@ -838,72 +839,72 @@ class TestMergeIntoTable2 extends HoodieSparkSqlTestBase {
}
}
- test("Test only insert for source table in dup key without preCombineField") {
- spark.sql(s"set ${MERGE_SMALL_FILE_GROUP_CANDIDATES_LIMIT.key} = ${MERGE_SMALL_FILE_GROUP_CANDIDATES_LIMIT.defaultValue()}")
- Seq("cow", "mor").foreach {
- tableType => {
- withTempDir { tmp =>
- val tableName = generateTableName
- spark.sql(
- s"""
- | create table $tableName (
- | id int,
- | name string,
- | price double,
- | ts int,
- | dt string
- | ) using hudi
- | tblproperties (
- | type = '$tableType',
- | primaryKey = 'id'
- | )
- | partitioned by(dt)
- | location '${tmp.getCanonicalPath}'
- """.stripMargin)
- // append records to small file is use update bucket, set this conf use concat handler
- spark.sql("set hoodie.merge.allow.duplicate.on.inserts = true")
-
- // Insert data without matched condition
- spark.sql(
- s"""
- | merge into $tableName as t0
- | using (
- | select 1 as id, 'a1' as name, 10.1 as price, 1000 as ts, '2021-03-21' as dt
- | union all
- | select 1 as id, 'a2' as name, 10.2 as price, 1002 as ts, '2021-03-21' as dt
- | ) as s0
- | on t0.id = s0.id
- | when not matched then insert *
- """.stripMargin
- )
- checkAnswer(s"select id, name, price, ts, dt from $tableName")(
- Seq(1, "a1", 10.1, 1000, "2021-03-21"),
- Seq(1, "a2", 10.2, 1002, "2021-03-21")
- )
-
- // Insert data with matched condition
- spark.sql(
- s"""
- | merge into $tableName as t0
- | using (
- | select 3 as id, 'a3' as name, 10.3 as price, 1003 as ts, '2021-03-21' as dt
- | union all
- | select 1 as id, 'a2' as name, 10.4 as price, 1004 as ts, '2021-03-21' as dt
- | ) as s0
- | on t0.id = s0.id
- | when matched then update set *
- | when not matched then insert *
- """.stripMargin
- )
- checkAnswer(s"select id, name, price, ts, dt from $tableName")(
- Seq(1, "a2", 10.4, 1004, "2021-03-21"),
- Seq(1, "a2", 10.4, 1004, "2021-03-21"),
- Seq(3, "a3", 10.3, 1003, "2021-03-21")
- )
- }
- }
- }
- }
+// test("Test only insert for source table in dup key without preCombineField") {
Review Comment:
actually now I am curious about this comment in PR desc
```
Currently the HoodieWriteMergeHandle can handle applying a single update to multiple records with the same key. This functionality does not exist in the FileGroupReaderBasedMergeHandle
```
We have an external spillable map in HoodieWriteMergeHandle where we add all incoming records. So, we can only have 1 entry per record key for incoming record. How do we claim we retain duplicate records? May be I am missing something.
As called out earlier, we have concat handle to assist w/ retaining duplicates which is not touched or migrated w/ this patch. We are only migrating regular cow merge to FG reader based w/ this patch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9622] Add implementation of MergeHandle backed by the HoodieFileGroupReader [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #13699:
URL: https://github.com/apache/hudi/pull/13699#discussion_r2279073195
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/WriteStatus.java:
##########
@@ -109,6 +109,17 @@ public void markSuccess(HoodieRecord record, Option<Map<String, String>> optiona
updateStatsForSuccess(optionalRecordMetadata);
}
+ /**
+ * Allows the writer to manually add record delegates to the index stats.
+ */
+ public void manuallyTrackSuccess() {
Review Comment:
oh, this is called in the constructor of HoodieWriteHandle. so, may not be easy to override in subclasses. unless we make it a function as an argument to assist w/ instantiation. which we can override just for FG reader based merge handle constructor which is used for COW merge use-cases.
I am ok not touching this for now in this patch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13699:
URL: https://github.com/apache/hudi/pull/13699#issuecomment-3175960698
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7661",
"triggerID" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"triggerType" : "PUSH"
}, {
"hash" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7666",
"triggerID" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"triggerType" : "PUSH"
}, {
"hash" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7667",
"triggerID" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"triggerType" : "PUSH"
}, {
"hash" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7672",
"triggerID" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"triggerType" : "PUSH"
}, {
"hash" : "965136302f1188e5123de37734c927dc55801b24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7691",
"triggerID" : "965136302f1188e5123de37734c927dc55801b24",
"triggerType" : "PUSH"
}, {
"hash" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7693",
"triggerID" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"triggerType" : "PUSH"
}, {
"hash" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7717",
"triggerID" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"triggerType" : "PUSH"
}, {
"hash" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"triggerType" : "PUSH"
}, {
"hash" : "47f8303d6e61504275ecfddf553e356427e6c822",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7718",
"triggerID" : "47f8303d6e61504275ecfddf553e356427e6c822",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 22721bae05acf8aa296bc5732af5a90d2afd3515 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7717)
* 50022dcc45b0f511dd69706aeaf6464a51be49e2 UNKNOWN
* 47f8303d6e61504275ecfddf553e356427e6c822 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7718)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13699:
URL: https://github.com/apache/hudi/pull/13699#issuecomment-3176959898
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7661",
"triggerID" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"triggerType" : "PUSH"
}, {
"hash" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7666",
"triggerID" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"triggerType" : "PUSH"
}, {
"hash" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7667",
"triggerID" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"triggerType" : "PUSH"
}, {
"hash" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7672",
"triggerID" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"triggerType" : "PUSH"
}, {
"hash" : "965136302f1188e5123de37734c927dc55801b24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7691",
"triggerID" : "965136302f1188e5123de37734c927dc55801b24",
"triggerType" : "PUSH"
}, {
"hash" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7693",
"triggerID" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"triggerType" : "PUSH"
}, {
"hash" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7717",
"triggerID" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"triggerType" : "PUSH"
}, {
"hash" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"triggerType" : "PUSH"
}, {
"hash" : "47f8303d6e61504275ecfddf553e356427e6c822",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7718",
"triggerID" : "47f8303d6e61504275ecfddf553e356427e6c822",
"triggerType" : "PUSH"
}, {
"hash" : "fb191ecf48d13cd81412535f7f8c9ed553d86517",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7725",
"triggerID" : "fb191ecf48d13cd81412535f7f8c9ed553d86517",
"triggerType" : "PUSH"
}, {
"hash" : "064912d3a54714935f2ee77d55cc66816166300f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7727",
"triggerID" : "064912d3a54714935f2ee77d55cc66816166300f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 50022dcc45b0f511dd69706aeaf6464a51be49e2 UNKNOWN
* fb191ecf48d13cd81412535f7f8c9ed553d86517 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7725)
* 064912d3a54714935f2ee77d55cc66816166300f Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7727)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13699:
URL: https://github.com/apache/hudi/pull/13699#discussion_r2267993965
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/buffer/FileGroupRecordBuffer.java:
##########
@@ -56,27 +53,29 @@
import java.io.IOException;
import java.io.Serializable;
+import java.util.HashSet;
import java.util.Iterator;
import java.util.List;
import java.util.Locale;
import java.util.Map;
+import java.util.Set;
import java.util.function.Function;
import static org.apache.hudi.common.config.HoodieCommonConfig.DISK_MAP_BITCASK_COMPRESSION_ENABLED;
import static org.apache.hudi.common.config.HoodieCommonConfig.SPILLABLE_DISK_MAP_TYPE;
import static org.apache.hudi.common.config.HoodieMemoryConfig.MAX_MEMORY_FOR_MERGE;
import static org.apache.hudi.common.config.HoodieMemoryConfig.SPILLABLE_MAP_BASE_PATH;
-import static org.apache.hudi.common.model.HoodieRecordMerger.PAYLOAD_BASED_MERGE_STRATEGY_UUID;
import static org.apache.hudi.common.table.log.block.HoodieLogBlock.HeaderMetadataType.INSTANT_TIME;
abstract class FileGroupRecordBuffer<T> implements HoodieFileGroupRecordBuffer<T> {
+ protected final Set<String> usedKeys = new HashSet<>();
Review Comment:
There is a possibility of duplicate keys in a file and there is an expectation that updates are applied to both rows. See `Test only insert for source table in dup key without preCombineField` for an example. We need to figure out if there is a better way to handle this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13699:
URL: https://github.com/apache/hudi/pull/13699#issuecomment-3176868864
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7661",
"triggerID" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"triggerType" : "PUSH"
}, {
"hash" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7666",
"triggerID" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"triggerType" : "PUSH"
}, {
"hash" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7667",
"triggerID" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"triggerType" : "PUSH"
}, {
"hash" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7672",
"triggerID" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"triggerType" : "PUSH"
}, {
"hash" : "965136302f1188e5123de37734c927dc55801b24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7691",
"triggerID" : "965136302f1188e5123de37734c927dc55801b24",
"triggerType" : "PUSH"
}, {
"hash" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7693",
"triggerID" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"triggerType" : "PUSH"
}, {
"hash" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7717",
"triggerID" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"triggerType" : "PUSH"
}, {
"hash" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"triggerType" : "PUSH"
}, {
"hash" : "47f8303d6e61504275ecfddf553e356427e6c822",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7718",
"triggerID" : "47f8303d6e61504275ecfddf553e356427e6c822",
"triggerType" : "PUSH"
}, {
"hash" : "fb191ecf48d13cd81412535f7f8c9ed553d86517",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7725",
"triggerID" : "fb191ecf48d13cd81412535f7f8c9ed553d86517",
"triggerType" : "PUSH"
}, {
"hash" : "064912d3a54714935f2ee77d55cc66816166300f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "064912d3a54714935f2ee77d55cc66816166300f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 50022dcc45b0f511dd69706aeaf6464a51be49e2 UNKNOWN
* fb191ecf48d13cd81412535f7f8c9ed553d86517 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7725)
* 064912d3a54714935f2ee77d55cc66816166300f UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13699:
URL: https://github.com/apache/hudi/pull/13699#discussion_r2271573930
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/index/HoodieIndexUtils.java:
##########
@@ -430,16 +430,11 @@ private static <R> Option<HoodieRecord<R>> mergeIncomingWithExistingRecordWithEx
//the record was deleted
return Option.empty();
}
- if (mergeResult.getRecord() == null) {
- // SENTINEL case: the record did not match and merge case and should not be modified
+ if (mergeResult.getRecord() == null || mergeResult == existingBufferedRecord) {
Review Comment:
Yes, the expression payload is only used for Merge Into
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13699:
URL: https://github.com/apache/hudi/pull/13699#issuecomment-3181910292
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7661",
"triggerID" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"triggerType" : "PUSH"
}, {
"hash" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7666",
"triggerID" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"triggerType" : "PUSH"
}, {
"hash" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7667",
"triggerID" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"triggerType" : "PUSH"
}, {
"hash" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7672",
"triggerID" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"triggerType" : "PUSH"
}, {
"hash" : "965136302f1188e5123de37734c927dc55801b24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7691",
"triggerID" : "965136302f1188e5123de37734c927dc55801b24",
"triggerType" : "PUSH"
}, {
"hash" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7693",
"triggerID" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"triggerType" : "PUSH"
}, {
"hash" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7717",
"triggerID" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"triggerType" : "PUSH"
}, {
"hash" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"triggerType" : "PUSH"
}, {
"hash" : "47f8303d6e61504275ecfddf553e356427e6c822",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7718",
"triggerID" : "47f8303d6e61504275ecfddf553e356427e6c822",
"triggerType" : "PUSH"
}, {
"hash" : "fb191ecf48d13cd81412535f7f8c9ed553d86517",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7725",
"triggerID" : "fb191ecf48d13cd81412535f7f8c9ed553d86517",
"triggerType" : "PUSH"
}, {
"hash" : "064912d3a54714935f2ee77d55cc66816166300f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7727",
"triggerID" : "064912d3a54714935f2ee77d55cc66816166300f",
"triggerType" : "PUSH"
}, {
"hash" : "77a2b45fe13dafc51f7bebb2ba859d40291ccc10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7736",
"triggerID" : "77a2b45fe13dafc51f7bebb2ba859d40291ccc10",
"triggerType" : "PUSH"
}, {
"hash" : "e4eda0e51fee0e8de5d57ea7b38dc642f53aa95d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7758",
"triggerID" : "e4eda0e51fee0e8de5d57ea7b38dc642f53aa95d",
"triggerType" : "PUSH"
}, {
"hash" : "6cc840c51fcecde95d2187fb6fd217b50fa82806",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6cc840c51fcecde95d2187fb6fd217b50fa82806",
"triggerType" : "PUSH"
}, {
"hash" : "31dab8ce9abbe4fc80e6cd81a0e433a039ca3e1d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7760",
"triggerID" : "31dab8ce9abbe4fc80e6cd81a0e433a039ca3e1d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 50022dcc45b0f511dd69706aeaf6464a51be49e2 UNKNOWN
* e4eda0e51fee0e8de5d57ea7b38dc642f53aa95d Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7758)
* 6cc840c51fcecde95d2187fb6fd217b50fa82806 UNKNOWN
* 31dab8ce9abbe4fc80e6cd81a0e433a039ca3e1d Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7760)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13699:
URL: https://github.com/apache/hudi/pull/13699#issuecomment-3182596617
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7661",
"triggerID" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"triggerType" : "PUSH"
}, {
"hash" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7666",
"triggerID" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"triggerType" : "PUSH"
}, {
"hash" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7667",
"triggerID" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"triggerType" : "PUSH"
}, {
"hash" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7672",
"triggerID" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"triggerType" : "PUSH"
}, {
"hash" : "965136302f1188e5123de37734c927dc55801b24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7691",
"triggerID" : "965136302f1188e5123de37734c927dc55801b24",
"triggerType" : "PUSH"
}, {
"hash" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7693",
"triggerID" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"triggerType" : "PUSH"
}, {
"hash" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7717",
"triggerID" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"triggerType" : "PUSH"
}, {
"hash" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"triggerType" : "PUSH"
}, {
"hash" : "47f8303d6e61504275ecfddf553e356427e6c822",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7718",
"triggerID" : "47f8303d6e61504275ecfddf553e356427e6c822",
"triggerType" : "PUSH"
}, {
"hash" : "fb191ecf48d13cd81412535f7f8c9ed553d86517",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7725",
"triggerID" : "fb191ecf48d13cd81412535f7f8c9ed553d86517",
"triggerType" : "PUSH"
}, {
"hash" : "064912d3a54714935f2ee77d55cc66816166300f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7727",
"triggerID" : "064912d3a54714935f2ee77d55cc66816166300f",
"triggerType" : "PUSH"
}, {
"hash" : "77a2b45fe13dafc51f7bebb2ba859d40291ccc10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7736",
"triggerID" : "77a2b45fe13dafc51f7bebb2ba859d40291ccc10",
"triggerType" : "PUSH"
}, {
"hash" : "e4eda0e51fee0e8de5d57ea7b38dc642f53aa95d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7758",
"triggerID" : "e4eda0e51fee0e8de5d57ea7b38dc642f53aa95d",
"triggerType" : "PUSH"
}, {
"hash" : "6cc840c51fcecde95d2187fb6fd217b50fa82806",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6cc840c51fcecde95d2187fb6fd217b50fa82806",
"triggerType" : "PUSH"
}, {
"hash" : "31dab8ce9abbe4fc80e6cd81a0e433a039ca3e1d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7760",
"triggerID" : "31dab8ce9abbe4fc80e6cd81a0e433a039ca3e1d",
"triggerType" : "PUSH"
}, {
"hash" : "c04fd95dcc94146a9146c347f4f204656f3d8308",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7763",
"triggerID" : "c04fd95dcc94146a9146c347f4f204656f3d8308",
"triggerType" : "PUSH"
}, {
"hash" : "324147913d47ba8f2555e1991ce59905e5dc0b3f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7767",
"triggerID" : "324147913d47ba8f2555e1991ce59905e5dc0b3f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 50022dcc45b0f511dd69706aeaf6464a51be49e2 UNKNOWN
* 6cc840c51fcecde95d2187fb6fd217b50fa82806 UNKNOWN
* c04fd95dcc94146a9146c347f4f204656f3d8308 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7763)
* 324147913d47ba8f2555e1991ce59905e5dc0b3f Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7767)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13699:
URL: https://github.com/apache/hudi/pull/13699#discussion_r2272721268
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/UpdateProcessor.java:
##########
@@ -80,16 +106,79 @@ public BufferedRecord<T> processUpdate(String recordKey, BufferedRecord<T> previ
}
return null;
} else {
- T prevRow = previousRecord != null ? previousRecord.getRecord() : null;
- T mergedRow = mergedRecord.getRecord();
- if (prevRow != null && prevRow != mergedRow) {
- mergedRecord.setHoodieOperation(HoodieOperation.UPDATE_AFTER);
- readStats.incrementNumUpdates();
- } else if (prevRow == null) {
- mergedRecord.setHoodieOperation(HoodieOperation.INSERT);
- readStats.incrementNumInserts();
+ return handleNonDeletes(previousRecord, mergedRecord);
+ }
+ }
+
+ protected BufferedRecord<T> handleNonDeletes(BufferedRecord<T> previousRecord, BufferedRecord<T> mergedRecord) {
+ T prevRow = previousRecord != null ? previousRecord.getRecord() : null;
+ T mergedRow = mergedRecord.getRecord();
+ if (prevRow != null && prevRow != mergedRow) {
+ mergedRecord.setHoodieOperation(HoodieOperation.UPDATE_AFTER);
+ readStats.incrementNumUpdates();
+ } else if (prevRow == null) {
+ mergedRecord.setHoodieOperation(HoodieOperation.INSERT);
+ readStats.incrementNumInserts();
+ }
+ return mergedRecord.seal(readerContext);
+ }
+ }
+
+ class PayloadUpdateProcessor<T> extends StandardUpdateProcessor<T> {
+ private final String payloadClass;
+ private final Properties properties;
+
+ public PayloadUpdateProcessor(HoodieReadStats readStats, HoodieReaderContext<T> readerContext, boolean emitDeletes,
+ Properties properties, String payloadClass) {
+ super(readStats, readerContext, emitDeletes);
+ this.payloadClass = payloadClass;
+ this.properties = properties;
+ }
+
+ @Override
+ protected BufferedRecord<T> handleNonDeletes(BufferedRecord<T> previousRecord, BufferedRecord<T> mergedRecord) {
+ if (previousRecord == null) {
+ // special case for payloads when there is no previous record
+ Schema recordSchema = readerContext.getRecordContext().decodeAvroSchema(mergedRecord.getSchemaId());
+ GenericRecord record = readerContext.getRecordContext().convertToAvroRecord(mergedRecord.getRecord(), recordSchema);
+ HoodieAvroRecord hoodieRecord = new HoodieAvroRecord<>(null, HoodieRecordUtils.loadPayload(payloadClass, record, mergedRecord.getOrderingValue()));
+ try {
+ if (hoodieRecord.shouldIgnore(recordSchema, properties)) {
+ return null;
+ } else {
+ Schema readerSchema = readerContext.getSchemaHandler().getRequestedSchema();
+ // If the record schema is different from the reader schema, rewrite the record using the payload methods to ensure consistency with legacy writer paths
+ hoodieRecord.rewriteRecordWithNewSchema(recordSchema, properties, readerSchema).toIndexedRecord(readerSchema, properties)
+ .ifPresent(rewrittenRecord -> mergedRecord.replaceRecord(readerContext.getRecordContext().convertAvroRecord(rewrittenRecord.getData())));
+ }
+ } catch (IOException e) {
+ throw new HoodieIOException("Error processing record with payload class: " + payloadClass, e);
+ }
+ }
+ return super.handleNonDeletes(previousRecord, mergedRecord);
+ }
+ }
+
+ class CustomMergerUpdateProcessor<T> extends StandardUpdateProcessor<T> {
+ private final HoodieRecordMerger merger;
+ private final TypedProperties properties;
+
+ CustomMergerUpdateProcessor(HoodieReadStats readStats, HoodieReaderContext<T> readerContext, boolean emitDeletes,
+ TypedProperties properties) {
+ super(readStats, readerContext, emitDeletes);
+ this.merger = readerContext.getRecordMerger().get();
+ this.properties = properties;
+ }
+
+ @Override
+ protected BufferedRecord<T> handleNonDeletes(BufferedRecord<T> previousRecord, BufferedRecord<T> mergedRecord) {
+ try {
+ if (merger.shouldFlush(readerContext.getRecordContext().constructHoodieRecord(mergedRecord), readerContext.getRecordContext().getSchemaFromBufferRecord(mergedRecord), properties)) {
Review Comment:
JIRA created to trace the following-up fixes: https://issues.apache.org/jira/browse/HUDI-9709
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13699:
URL: https://github.com/apache/hudi/pull/13699#discussion_r2268395740
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/FileGroupReaderBasedMergeHandle.java:
##########
@@ -75,13 +85,60 @@
public class FileGroupReaderBasedMergeHandle<T, I, K, O> extends HoodieWriteMergeHandle<T, I, K, O> {
private static final Logger LOG = LoggerFactory.getLogger(FileGroupReaderBasedMergeHandle.class);
- private final HoodieReaderContext<T> readerContext;
- private final CompactionOperation operation;
+ private final Option<CompactionOperation> operation;
private final String maxInstantTime;
+ private HoodieReaderContext<T> readerContext;
private HoodieReadStats readStats;
- private final HoodieRecord.HoodieRecordType recordType;
- private final Option<HoodieCDCLogger> cdcLogger;
+ private HoodieRecord.HoodieRecordType recordType;
+ private Option<HoodieCDCLogger> cdcLogger;
+ private Option<RecordLevelIndexCallback> recordIndexCallbackOpt;
+ private Option<SecondaryIndexCallback> secondaryIndexCallbackOpt;
+ private final boolean isCompaction;
+ private final TypedProperties props;
+ private Iterator<HoodieRecord> incomingRecordsItr;
+ /**
+ * Constructor for Copy-On-Write (COW) merge path.
+ * Takes in a base path and an iterator of records to be merged with that file.
+ * @param config instance of {@link HoodieWriteConfig} to use.
+ * @param instantTime instant time of the current commit.
+ * @param hoodieTable instance of {@link HoodieTable} being updated.
+ * @param recordItr iterator of records to be merged with the file.
+ * @param partitionPath partition path of the base file.
+ * @param fileId file ID of the base file.
+ * @param taskContextSupplier instance of {@link TaskContextSupplier} to use.
+ * @param keyGeneratorOpt optional instance of {@link BaseKeyGenerator} to use for extracting keys from records.
+ * @param readerContext instance of {@link HoodieReaderContext} to use while merging for accessing fields and transforming records.
+ */
+ public FileGroupReaderBasedMergeHandle(HoodieWriteConfig config, String instantTime, HoodieTable<T, I, K, O> hoodieTable,
+ Iterator<HoodieRecord<T>> recordItr, String partitionPath, String fileId,
+ TaskContextSupplier taskContextSupplier, Option<BaseKeyGenerator> keyGeneratorOpt, HoodieReaderContext<T> readerContext) {
Review Comment:
> always return a local engine context instead of a spark engine context when required
Can we fix that, like `hoodieTable.getContextForWrite` or something.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9622] Add implementation of MergeHandle backed by the HoodieFileGroupReader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13699:
URL: https://github.com/apache/hudi/pull/13699#issuecomment-3190421862
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7661",
"triggerID" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"triggerType" : "PUSH"
}, {
"hash" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7666",
"triggerID" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"triggerType" : "PUSH"
}, {
"hash" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7667",
"triggerID" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"triggerType" : "PUSH"
}, {
"hash" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7672",
"triggerID" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"triggerType" : "PUSH"
}, {
"hash" : "965136302f1188e5123de37734c927dc55801b24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7691",
"triggerID" : "965136302f1188e5123de37734c927dc55801b24",
"triggerType" : "PUSH"
}, {
"hash" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7693",
"triggerID" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"triggerType" : "PUSH"
}, {
"hash" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7717",
"triggerID" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"triggerType" : "PUSH"
}, {
"hash" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"triggerType" : "PUSH"
}, {
"hash" : "47f8303d6e61504275ecfddf553e356427e6c822",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7718",
"triggerID" : "47f8303d6e61504275ecfddf553e356427e6c822",
"triggerType" : "PUSH"
}, {
"hash" : "fb191ecf48d13cd81412535f7f8c9ed553d86517",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7725",
"triggerID" : "fb191ecf48d13cd81412535f7f8c9ed553d86517",
"triggerType" : "PUSH"
}, {
"hash" : "064912d3a54714935f2ee77d55cc66816166300f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7727",
"triggerID" : "064912d3a54714935f2ee77d55cc66816166300f",
"triggerType" : "PUSH"
}, {
"hash" : "77a2b45fe13dafc51f7bebb2ba859d40291ccc10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7736",
"triggerID" : "77a2b45fe13dafc51f7bebb2ba859d40291ccc10",
"triggerType" : "PUSH"
}, {
"hash" : "e4eda0e51fee0e8de5d57ea7b38dc642f53aa95d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7758",
"triggerID" : "e4eda0e51fee0e8de5d57ea7b38dc642f53aa95d",
"triggerType" : "PUSH"
}, {
"hash" : "6cc840c51fcecde95d2187fb6fd217b50fa82806",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6cc840c51fcecde95d2187fb6fd217b50fa82806",
"triggerType" : "PUSH"
}, {
"hash" : "31dab8ce9abbe4fc80e6cd81a0e433a039ca3e1d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7760",
"triggerID" : "31dab8ce9abbe4fc80e6cd81a0e433a039ca3e1d",
"triggerType" : "PUSH"
}, {
"hash" : "c04fd95dcc94146a9146c347f4f204656f3d8308",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7763",
"triggerID" : "c04fd95dcc94146a9146c347f4f204656f3d8308",
"triggerType" : "PUSH"
}, {
"hash" : "324147913d47ba8f2555e1991ce59905e5dc0b3f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7767",
"triggerID" : "324147913d47ba8f2555e1991ce59905e5dc0b3f",
"triggerType" : "PUSH"
}, {
"hash" : "3f177d42b831b30f4bd590b057dcab6ded78b634",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7774",
"triggerID" : "3f177d42b831b30f4bd590b057dcab6ded78b634",
"triggerType" : "PUSH"
}, {
"hash" : "8d6d2c06ec82d1072251ac12ddf21a890ddd5bda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7776",
"triggerID" : "8d6d2c06ec82d1072251ac12ddf21a890ddd5bda",
"triggerType" : "PUSH"
}, {
"hash" : "77589738d73b044385f5aa6f7ef87b421f06043b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "77589738d73b044385f5aa6f7ef87b421f06043b",
"triggerType" : "PUSH"
}, {
"hash" : "ad0d7ef0fbc525f880d3baadef5a22196ab97b32",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7781",
"triggerID" : "ad0d7ef0fbc525f880d3baadef5a22196ab97b32",
"triggerType" : "PUSH"
}, {
"hash" : "0db72caa3a6c7d0f0899f243ea4b8975de79e02c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7787",
"triggerID" : "0db72caa3a6c7d0f0899f243ea4b8975de79e02c",
"triggerType" : "PUSH"
}, {
"hash" : "a01ce748dc2d4cd1fd4858d1ecac8a486989e880",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7797",
"triggerID" : "a01ce748dc2d4cd1fd4858d1ecac8a486989e880",
"triggerType" : "PUSH"
}, {
"hash" : "3f35151e28bfa1dd50b31c6e5379d9769627afd9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7799",
"triggerID" : "3f35151e28bfa1dd50b31c6e5379d9769627afd9",
"triggerType" : "PUSH"
}, {
"hash" : "b0054c8fd329489d9a1e9535d9866edffc52fdd0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b0054c8fd329489d9a1e9535d9866edffc52fdd0",
"triggerType" : "PUSH"
}, {
"hash" : "363e54341420cc217bf5af62de1e09ff370f75d2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "363e54341420cc217bf5af62de1e09ff370f75d2",
"triggerType" : "PUSH"
}, {
"hash" : "a893da375eb9c65edabbd22965296eac9b3714f4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a893da375eb9c65edabbd22965296eac9b3714f4",
"triggerType" : "PUSH"
}, {
"hash" : "a7c2756e6a293cbba99bd4e821881e6b961c138f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7804",
"triggerID" : "a7c2756e6a293cbba99bd4e821881e6b961c138f",
"triggerType" : "PUSH"
}, {
"hash" : "4a140fffa35893a13f2e7ac8365d60d51a2d0870",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7813",
"triggerID" : "4a140fffa35893a13f2e7ac8365d60d51a2d0870",
"triggerType" : "PUSH"
}, {
"hash" : "500b4f529ad2f8a695825b5badc1815202164d0a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7830",
"triggerID" : "500b4f529ad2f8a695825b5badc1815202164d0a",
"triggerType" : "PUSH"
}, {
"hash" : "539cf8b9cdc7ab4e70488e4096684022a027f942",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "539cf8b9cdc7ab4e70488e4096684022a027f942",
"triggerType" : "PUSH"
}, {
"hash" : "85139a12b431931973c07d1b7f9256a1aae75669",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "85139a12b431931973c07d1b7f9256a1aae75669",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 50022dcc45b0f511dd69706aeaf6464a51be49e2 UNKNOWN
* 6cc840c51fcecde95d2187fb6fd217b50fa82806 UNKNOWN
* 77589738d73b044385f5aa6f7ef87b421f06043b UNKNOWN
* b0054c8fd329489d9a1e9535d9866edffc52fdd0 UNKNOWN
* 363e54341420cc217bf5af62de1e09ff370f75d2 UNKNOWN
* a893da375eb9c65edabbd22965296eac9b3714f4 UNKNOWN
* 500b4f529ad2f8a695825b5badc1815202164d0a Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7830)
* 539cf8b9cdc7ab4e70488e4096684022a027f942 UNKNOWN
* 85139a12b431931973c07d1b7f9256a1aae75669 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13699:
URL: https://github.com/apache/hudi/pull/13699#discussion_r2273653510
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/HoodieTable.java:
##########
@@ -150,6 +151,7 @@ public abstract class HoodieTable<T, I, K, O> implements Serializable {
private transient FileSystemViewManager viewManager;
protected final transient HoodieEngineContext context;
+ private final ReaderContextFactory<T> readerContextFactoryForWrite;
Review Comment:
we can remove the reader context factory in hoodie table, and instantiate a reader context factory in `SparkUpsertCommitActionExecutor`, in `SparkUpsertCommitActionExecutor`, override the method `BaseSparkCommitActionExecutor#getUpdateHandle` to set up the reader context explicitly, may also need to add a set method in `FileGroupReaderBasedMergeHandle`, only this handle needs this currently.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13699:
URL: https://github.com/apache/hudi/pull/13699#discussion_r2271941785
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/buffer/FileGroupRecordBuffer.java:
##########
@@ -56,27 +53,29 @@
import java.io.IOException;
import java.io.Serializable;
+import java.util.HashSet;
import java.util.Iterator;
import java.util.List;
import java.util.Locale;
import java.util.Map;
+import java.util.Set;
import java.util.function.Function;
import static org.apache.hudi.common.config.HoodieCommonConfig.DISK_MAP_BITCASK_COMPRESSION_ENABLED;
import static org.apache.hudi.common.config.HoodieCommonConfig.SPILLABLE_DISK_MAP_TYPE;
import static org.apache.hudi.common.config.HoodieMemoryConfig.MAX_MEMORY_FOR_MERGE;
import static org.apache.hudi.common.config.HoodieMemoryConfig.SPILLABLE_MAP_BASE_PATH;
-import static org.apache.hudi.common.model.HoodieRecordMerger.PAYLOAD_BASED_MERGE_STRATEGY_UUID;
import static org.apache.hudi.common.table.log.block.HoodieLogBlock.HeaderMetadataType.INSTANT_TIME;
abstract class FileGroupRecordBuffer<T> implements HoodieFileGroupRecordBuffer<T> {
+ protected final Set<String> usedKeys = new HashSet<>();
Review Comment:
The fix is not right, the discrepencies come from the log comparison, for old write merge handle, we do not remove the log record when a base record key matches it, and bookeep a set of log record keys that have been matched with base(used for skipping for those non matched log records).
While the FG reader current remove the log record directly, which makes it impossbie to match the same log records with duplicates in base multiple times.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9622] Add implementation of MergeHandle backed by the HoodieFileGroupReader [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13699:
URL: https://github.com/apache/hudi/pull/13699#discussion_r2278046171
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/FileGroupReaderBasedMergeHandle.java:
##########
@@ -198,8 +270,10 @@ public void doMerge() {
}
// Writes the record
try {
- writeToFile(record.getKey(), record, writeSchemaWithMetaFields,
- config.getPayloadConfig().getProps(), preserveMetadata);
+ // if the record is not being updated and is not a new insert for the file group, we must preserve the existing record metadata.
+ boolean shouldPreserveRecordMetadata = preserveMetadata || record.getOperation() == null;
Review Comment:
maybe we doc it well for the intricate details so it is more maintanable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13699:
URL: https://github.com/apache/hudi/pull/13699#issuecomment-3181898001
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7661",
"triggerID" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"triggerType" : "PUSH"
}, {
"hash" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7666",
"triggerID" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"triggerType" : "PUSH"
}, {
"hash" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7667",
"triggerID" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"triggerType" : "PUSH"
}, {
"hash" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7672",
"triggerID" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"triggerType" : "PUSH"
}, {
"hash" : "965136302f1188e5123de37734c927dc55801b24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7691",
"triggerID" : "965136302f1188e5123de37734c927dc55801b24",
"triggerType" : "PUSH"
}, {
"hash" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7693",
"triggerID" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"triggerType" : "PUSH"
}, {
"hash" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7717",
"triggerID" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"triggerType" : "PUSH"
}, {
"hash" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"triggerType" : "PUSH"
}, {
"hash" : "47f8303d6e61504275ecfddf553e356427e6c822",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7718",
"triggerID" : "47f8303d6e61504275ecfddf553e356427e6c822",
"triggerType" : "PUSH"
}, {
"hash" : "fb191ecf48d13cd81412535f7f8c9ed553d86517",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7725",
"triggerID" : "fb191ecf48d13cd81412535f7f8c9ed553d86517",
"triggerType" : "PUSH"
}, {
"hash" : "064912d3a54714935f2ee77d55cc66816166300f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7727",
"triggerID" : "064912d3a54714935f2ee77d55cc66816166300f",
"triggerType" : "PUSH"
}, {
"hash" : "77a2b45fe13dafc51f7bebb2ba859d40291ccc10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7736",
"triggerID" : "77a2b45fe13dafc51f7bebb2ba859d40291ccc10",
"triggerType" : "PUSH"
}, {
"hash" : "e4eda0e51fee0e8de5d57ea7b38dc642f53aa95d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7758",
"triggerID" : "e4eda0e51fee0e8de5d57ea7b38dc642f53aa95d",
"triggerType" : "PUSH"
}, {
"hash" : "6cc840c51fcecde95d2187fb6fd217b50fa82806",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6cc840c51fcecde95d2187fb6fd217b50fa82806",
"triggerType" : "PUSH"
}, {
"hash" : "31dab8ce9abbe4fc80e6cd81a0e433a039ca3e1d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "31dab8ce9abbe4fc80e6cd81a0e433a039ca3e1d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 50022dcc45b0f511dd69706aeaf6464a51be49e2 UNKNOWN
* e4eda0e51fee0e8de5d57ea7b38dc642f53aa95d Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7758)
* 6cc840c51fcecde95d2187fb6fd217b50fa82806 UNKNOWN
* 31dab8ce9abbe4fc80e6cd81a0e433a039ca3e1d UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13699:
URL: https://github.com/apache/hudi/pull/13699#issuecomment-3184271739
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7661",
"triggerID" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"triggerType" : "PUSH"
}, {
"hash" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7666",
"triggerID" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"triggerType" : "PUSH"
}, {
"hash" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7667",
"triggerID" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"triggerType" : "PUSH"
}, {
"hash" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7672",
"triggerID" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"triggerType" : "PUSH"
}, {
"hash" : "965136302f1188e5123de37734c927dc55801b24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7691",
"triggerID" : "965136302f1188e5123de37734c927dc55801b24",
"triggerType" : "PUSH"
}, {
"hash" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7693",
"triggerID" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"triggerType" : "PUSH"
}, {
"hash" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7717",
"triggerID" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"triggerType" : "PUSH"
}, {
"hash" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"triggerType" : "PUSH"
}, {
"hash" : "47f8303d6e61504275ecfddf553e356427e6c822",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7718",
"triggerID" : "47f8303d6e61504275ecfddf553e356427e6c822",
"triggerType" : "PUSH"
}, {
"hash" : "fb191ecf48d13cd81412535f7f8c9ed553d86517",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7725",
"triggerID" : "fb191ecf48d13cd81412535f7f8c9ed553d86517",
"triggerType" : "PUSH"
}, {
"hash" : "064912d3a54714935f2ee77d55cc66816166300f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7727",
"triggerID" : "064912d3a54714935f2ee77d55cc66816166300f",
"triggerType" : "PUSH"
}, {
"hash" : "77a2b45fe13dafc51f7bebb2ba859d40291ccc10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7736",
"triggerID" : "77a2b45fe13dafc51f7bebb2ba859d40291ccc10",
"triggerType" : "PUSH"
}, {
"hash" : "e4eda0e51fee0e8de5d57ea7b38dc642f53aa95d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7758",
"triggerID" : "e4eda0e51fee0e8de5d57ea7b38dc642f53aa95d",
"triggerType" : "PUSH"
}, {
"hash" : "6cc840c51fcecde95d2187fb6fd217b50fa82806",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6cc840c51fcecde95d2187fb6fd217b50fa82806",
"triggerType" : "PUSH"
}, {
"hash" : "31dab8ce9abbe4fc80e6cd81a0e433a039ca3e1d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7760",
"triggerID" : "31dab8ce9abbe4fc80e6cd81a0e433a039ca3e1d",
"triggerType" : "PUSH"
}, {
"hash" : "c04fd95dcc94146a9146c347f4f204656f3d8308",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7763",
"triggerID" : "c04fd95dcc94146a9146c347f4f204656f3d8308",
"triggerType" : "PUSH"
}, {
"hash" : "324147913d47ba8f2555e1991ce59905e5dc0b3f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7767",
"triggerID" : "324147913d47ba8f2555e1991ce59905e5dc0b3f",
"triggerType" : "PUSH"
}, {
"hash" : "3f177d42b831b30f4bd590b057dcab6ded78b634",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7774",
"triggerID" : "3f177d42b831b30f4bd590b057dcab6ded78b634",
"triggerType" : "PUSH"
}, {
"hash" : "8d6d2c06ec82d1072251ac12ddf21a890ddd5bda",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7776",
"triggerID" : "8d6d2c06ec82d1072251ac12ddf21a890ddd5bda",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 50022dcc45b0f511dd69706aeaf6464a51be49e2 UNKNOWN
* 6cc840c51fcecde95d2187fb6fd217b50fa82806 UNKNOWN
* 3f177d42b831b30f4bd590b057dcab6ded78b634 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7774)
* 8d6d2c06ec82d1072251ac12ddf21a890ddd5bda Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7776)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13699:
URL: https://github.com/apache/hudi/pull/13699#issuecomment-3186053309
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7661",
"triggerID" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"triggerType" : "PUSH"
}, {
"hash" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7666",
"triggerID" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"triggerType" : "PUSH"
}, {
"hash" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7667",
"triggerID" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"triggerType" : "PUSH"
}, {
"hash" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7672",
"triggerID" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"triggerType" : "PUSH"
}, {
"hash" : "965136302f1188e5123de37734c927dc55801b24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7691",
"triggerID" : "965136302f1188e5123de37734c927dc55801b24",
"triggerType" : "PUSH"
}, {
"hash" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7693",
"triggerID" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"triggerType" : "PUSH"
}, {
"hash" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7717",
"triggerID" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"triggerType" : "PUSH"
}, {
"hash" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"triggerType" : "PUSH"
}, {
"hash" : "47f8303d6e61504275ecfddf553e356427e6c822",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7718",
"triggerID" : "47f8303d6e61504275ecfddf553e356427e6c822",
"triggerType" : "PUSH"
}, {
"hash" : "fb191ecf48d13cd81412535f7f8c9ed553d86517",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7725",
"triggerID" : "fb191ecf48d13cd81412535f7f8c9ed553d86517",
"triggerType" : "PUSH"
}, {
"hash" : "064912d3a54714935f2ee77d55cc66816166300f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7727",
"triggerID" : "064912d3a54714935f2ee77d55cc66816166300f",
"triggerType" : "PUSH"
}, {
"hash" : "77a2b45fe13dafc51f7bebb2ba859d40291ccc10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7736",
"triggerID" : "77a2b45fe13dafc51f7bebb2ba859d40291ccc10",
"triggerType" : "PUSH"
}, {
"hash" : "e4eda0e51fee0e8de5d57ea7b38dc642f53aa95d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7758",
"triggerID" : "e4eda0e51fee0e8de5d57ea7b38dc642f53aa95d",
"triggerType" : "PUSH"
}, {
"hash" : "6cc840c51fcecde95d2187fb6fd217b50fa82806",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6cc840c51fcecde95d2187fb6fd217b50fa82806",
"triggerType" : "PUSH"
}, {
"hash" : "31dab8ce9abbe4fc80e6cd81a0e433a039ca3e1d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7760",
"triggerID" : "31dab8ce9abbe4fc80e6cd81a0e433a039ca3e1d",
"triggerType" : "PUSH"
}, {
"hash" : "c04fd95dcc94146a9146c347f4f204656f3d8308",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7763",
"triggerID" : "c04fd95dcc94146a9146c347f4f204656f3d8308",
"triggerType" : "PUSH"
}, {
"hash" : "324147913d47ba8f2555e1991ce59905e5dc0b3f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7767",
"triggerID" : "324147913d47ba8f2555e1991ce59905e5dc0b3f",
"triggerType" : "PUSH"
}, {
"hash" : "3f177d42b831b30f4bd590b057dcab6ded78b634",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7774",
"triggerID" : "3f177d42b831b30f4bd590b057dcab6ded78b634",
"triggerType" : "PUSH"
}, {
"hash" : "8d6d2c06ec82d1072251ac12ddf21a890ddd5bda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7776",
"triggerID" : "8d6d2c06ec82d1072251ac12ddf21a890ddd5bda",
"triggerType" : "PUSH"
}, {
"hash" : "77589738d73b044385f5aa6f7ef87b421f06043b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "77589738d73b044385f5aa6f7ef87b421f06043b",
"triggerType" : "PUSH"
}, {
"hash" : "ad0d7ef0fbc525f880d3baadef5a22196ab97b32",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7781",
"triggerID" : "ad0d7ef0fbc525f880d3baadef5a22196ab97b32",
"triggerType" : "PUSH"
}, {
"hash" : "0db72caa3a6c7d0f0899f243ea4b8975de79e02c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7787",
"triggerID" : "0db72caa3a6c7d0f0899f243ea4b8975de79e02c",
"triggerType" : "PUSH"
}, {
"hash" : "a01ce748dc2d4cd1fd4858d1ecac8a486989e880",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a01ce748dc2d4cd1fd4858d1ecac8a486989e880",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 50022dcc45b0f511dd69706aeaf6464a51be49e2 UNKNOWN
* 6cc840c51fcecde95d2187fb6fd217b50fa82806 UNKNOWN
* 77589738d73b044385f5aa6f7ef87b421f06043b UNKNOWN
* 0db72caa3a6c7d0f0899f243ea4b8975de79e02c Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7787)
* a01ce748dc2d4cd1fd4858d1ecac8a486989e880 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13699:
URL: https://github.com/apache/hudi/pull/13699#discussion_r2273598475
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/HoodieTable.java:
##########
@@ -150,6 +151,7 @@ public abstract class HoodieTable<T, I, K, O> implements Serializable {
private transient FileSystemViewManager viewManager;
protected final transient HoodieEngineContext context;
+ private final ReaderContextFactory<T> readerContextFactoryForWrite;
Review Comment:
The hoodie table holds all the info required there: the engine context, the meta client, and the write config. It looks not right we pass around a "reader" context factory for all the write path, even for write path that is not COW table merging scenarios.
There are also discrepancies in write path, only spark needs this factory, Flink and Java can get the factory directly from the engine context in the hoodie table.
The broadcast is already there for all the write executors before my change, if we have some manner to resolve the serialization issue of the engine context or reader context itself, that would be the best. Or if we can limit the factory only been initialized for COW table updates in write handles, that would be great.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9622] Add implementation of MergeHandle backed by the HoodieFileGroupReader [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13699:
URL: https://github.com/apache/hudi/pull/13699#discussion_r2278050450
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/dml/others/TestMergeIntoTable2.scala:
##########
@@ -838,72 +839,72 @@ class TestMergeIntoTable2 extends HoodieSparkSqlTestBase {
}
}
- test("Test only insert for source table in dup key without preCombineField") {
- spark.sql(s"set ${MERGE_SMALL_FILE_GROUP_CANDIDATES_LIMIT.key} = ${MERGE_SMALL_FILE_GROUP_CANDIDATES_LIMIT.defaultValue()}")
- Seq("cow", "mor").foreach {
- tableType => {
- withTempDir { tmp =>
- val tableName = generateTableName
- spark.sql(
- s"""
- | create table $tableName (
- | id int,
- | name string,
- | price double,
- | ts int,
- | dt string
- | ) using hudi
- | tblproperties (
- | type = '$tableType',
- | primaryKey = 'id'
- | )
- | partitioned by(dt)
- | location '${tmp.getCanonicalPath}'
- """.stripMargin)
- // append records to small file is use update bucket, set this conf use concat handler
- spark.sql("set hoodie.merge.allow.duplicate.on.inserts = true")
-
- // Insert data without matched condition
- spark.sql(
- s"""
- | merge into $tableName as t0
- | using (
- | select 1 as id, 'a1' as name, 10.1 as price, 1000 as ts, '2021-03-21' as dt
- | union all
- | select 1 as id, 'a2' as name, 10.2 as price, 1002 as ts, '2021-03-21' as dt
- | ) as s0
- | on t0.id = s0.id
- | when not matched then insert *
- """.stripMargin
- )
- checkAnswer(s"select id, name, price, ts, dt from $tableName")(
- Seq(1, "a1", 10.1, 1000, "2021-03-21"),
- Seq(1, "a2", 10.2, 1002, "2021-03-21")
- )
-
- // Insert data with matched condition
- spark.sql(
- s"""
- | merge into $tableName as t0
- | using (
- | select 3 as id, 'a3' as name, 10.3 as price, 1003 as ts, '2021-03-21' as dt
- | union all
- | select 1 as id, 'a2' as name, 10.4 as price, 1004 as ts, '2021-03-21' as dt
- | ) as s0
- | on t0.id = s0.id
- | when matched then update set *
- | when not matched then insert *
- """.stripMargin
- )
- checkAnswer(s"select id, name, price, ts, dt from $tableName")(
- Seq(1, "a2", 10.4, 1004, "2021-03-21"),
- Seq(1, "a2", 10.4, 1004, "2021-03-21"),
- Seq(3, "a3", 10.3, 1003, "2021-03-21")
- )
- }
- }
- }
- }
+// test("Test only insert for source table in dup key without preCombineField") {
Review Comment:
Is there a way to mark a scala test as disabled? We can also set a property so this test uses the legacy merge handle and it will pass
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9622] Add implementation of MergeHandle backed by the HoodieFileGroupReader [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13699:
URL: https://github.com/apache/hudi/pull/13699#discussion_r2278043646
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/table/action/commit/BaseSparkCommitActionExecutor.java:
##########
@@ -397,6 +401,9 @@ protected HoodieMergeHandle getUpdateHandle(String partitionPath, String fileId,
mergeHandle.setPartitionFields(partitionFields);
mergeHandle.setPartitionValues(partitionValues);
}
+ if (readerContextFactory != null && mergeHandle instanceof FileGroupReaderBasedMergeHandle) {
Review Comment:
We do it for these reasons:
1. the param is only needed for the FG merge handle now, we don't want to init the reader context for all the merge handles
2. also modifying the merge handle param will need to modify all the executors, which is also unnecessary, like Java and Flink can fetch the reader context directly from hoodie table.
3. thus, we want to constraint the explicit reader contex for writer path to just "upsert" executors of Spark so that all the other executors are not affected
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13699:
URL: https://github.com/apache/hudi/pull/13699#discussion_r2272262934
##########
hudi-common/src/main/java/org/apache/hudi/common/engine/RecordContext.java:
##########
@@ -340,7 +356,7 @@ public boolean supportsParquetRowIndex() {
}
private SerializableBiFunction<T, Schema, String> metadataKeyExtractor() {
- return (record, schema) -> getValue(record, schema, RECORD_KEY_METADATA_FIELD).toString();
+ return (record, schema) -> typeConverter.castToString(getValue(record, schema, RECORD_KEY_METADATA_FIELD));
Review Comment:
wondering why this does not error out before the patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13699:
URL: https://github.com/apache/hudi/pull/13699#discussion_r2275003462
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/buffer/FileGroupRecordBuffer.java:
##########
@@ -56,27 +53,29 @@
import java.io.IOException;
import java.io.Serializable;
+import java.util.HashSet;
import java.util.Iterator;
import java.util.List;
import java.util.Locale;
import java.util.Map;
+import java.util.Set;
import java.util.function.Function;
import static org.apache.hudi.common.config.HoodieCommonConfig.DISK_MAP_BITCASK_COMPRESSION_ENABLED;
import static org.apache.hudi.common.config.HoodieCommonConfig.SPILLABLE_DISK_MAP_TYPE;
import static org.apache.hudi.common.config.HoodieMemoryConfig.MAX_MEMORY_FOR_MERGE;
import static org.apache.hudi.common.config.HoodieMemoryConfig.SPILLABLE_MAP_BASE_PATH;
-import static org.apache.hudi.common.model.HoodieRecordMerger.PAYLOAD_BASED_MERGE_STRATEGY_UUID;
import static org.apache.hudi.common.table.log.block.HoodieLogBlock.HeaderMetadataType.INSTANT_TIME;
abstract class FileGroupRecordBuffer<T> implements HoodieFileGroupRecordBuffer<T> {
+ protected final Set<String> usedKeys = new HashSet<>();
Review Comment:
we can skip the whole tests first and fix the MIT in HUDI-9708, the original test means to test pk table I think.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9622] Add implementation of MergeHandle backed by the HoodieFileGroupReader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13699:
URL: https://github.com/apache/hudi/pull/13699#issuecomment-3190492893
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7661",
"triggerID" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"triggerType" : "PUSH"
}, {
"hash" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7666",
"triggerID" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"triggerType" : "PUSH"
}, {
"hash" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7667",
"triggerID" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"triggerType" : "PUSH"
}, {
"hash" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7672",
"triggerID" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"triggerType" : "PUSH"
}, {
"hash" : "965136302f1188e5123de37734c927dc55801b24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7691",
"triggerID" : "965136302f1188e5123de37734c927dc55801b24",
"triggerType" : "PUSH"
}, {
"hash" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7693",
"triggerID" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"triggerType" : "PUSH"
}, {
"hash" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7717",
"triggerID" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"triggerType" : "PUSH"
}, {
"hash" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"triggerType" : "PUSH"
}, {
"hash" : "47f8303d6e61504275ecfddf553e356427e6c822",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7718",
"triggerID" : "47f8303d6e61504275ecfddf553e356427e6c822",
"triggerType" : "PUSH"
}, {
"hash" : "fb191ecf48d13cd81412535f7f8c9ed553d86517",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7725",
"triggerID" : "fb191ecf48d13cd81412535f7f8c9ed553d86517",
"triggerType" : "PUSH"
}, {
"hash" : "064912d3a54714935f2ee77d55cc66816166300f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7727",
"triggerID" : "064912d3a54714935f2ee77d55cc66816166300f",
"triggerType" : "PUSH"
}, {
"hash" : "77a2b45fe13dafc51f7bebb2ba859d40291ccc10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7736",
"triggerID" : "77a2b45fe13dafc51f7bebb2ba859d40291ccc10",
"triggerType" : "PUSH"
}, {
"hash" : "e4eda0e51fee0e8de5d57ea7b38dc642f53aa95d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7758",
"triggerID" : "e4eda0e51fee0e8de5d57ea7b38dc642f53aa95d",
"triggerType" : "PUSH"
}, {
"hash" : "6cc840c51fcecde95d2187fb6fd217b50fa82806",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6cc840c51fcecde95d2187fb6fd217b50fa82806",
"triggerType" : "PUSH"
}, {
"hash" : "31dab8ce9abbe4fc80e6cd81a0e433a039ca3e1d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7760",
"triggerID" : "31dab8ce9abbe4fc80e6cd81a0e433a039ca3e1d",
"triggerType" : "PUSH"
}, {
"hash" : "c04fd95dcc94146a9146c347f4f204656f3d8308",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7763",
"triggerID" : "c04fd95dcc94146a9146c347f4f204656f3d8308",
"triggerType" : "PUSH"
}, {
"hash" : "324147913d47ba8f2555e1991ce59905e5dc0b3f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7767",
"triggerID" : "324147913d47ba8f2555e1991ce59905e5dc0b3f",
"triggerType" : "PUSH"
}, {
"hash" : "3f177d42b831b30f4bd590b057dcab6ded78b634",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7774",
"triggerID" : "3f177d42b831b30f4bd590b057dcab6ded78b634",
"triggerType" : "PUSH"
}, {
"hash" : "8d6d2c06ec82d1072251ac12ddf21a890ddd5bda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7776",
"triggerID" : "8d6d2c06ec82d1072251ac12ddf21a890ddd5bda",
"triggerType" : "PUSH"
}, {
"hash" : "77589738d73b044385f5aa6f7ef87b421f06043b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "77589738d73b044385f5aa6f7ef87b421f06043b",
"triggerType" : "PUSH"
}, {
"hash" : "ad0d7ef0fbc525f880d3baadef5a22196ab97b32",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7781",
"triggerID" : "ad0d7ef0fbc525f880d3baadef5a22196ab97b32",
"triggerType" : "PUSH"
}, {
"hash" : "0db72caa3a6c7d0f0899f243ea4b8975de79e02c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7787",
"triggerID" : "0db72caa3a6c7d0f0899f243ea4b8975de79e02c",
"triggerType" : "PUSH"
}, {
"hash" : "a01ce748dc2d4cd1fd4858d1ecac8a486989e880",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7797",
"triggerID" : "a01ce748dc2d4cd1fd4858d1ecac8a486989e880",
"triggerType" : "PUSH"
}, {
"hash" : "3f35151e28bfa1dd50b31c6e5379d9769627afd9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7799",
"triggerID" : "3f35151e28bfa1dd50b31c6e5379d9769627afd9",
"triggerType" : "PUSH"
}, {
"hash" : "b0054c8fd329489d9a1e9535d9866edffc52fdd0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b0054c8fd329489d9a1e9535d9866edffc52fdd0",
"triggerType" : "PUSH"
}, {
"hash" : "363e54341420cc217bf5af62de1e09ff370f75d2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "363e54341420cc217bf5af62de1e09ff370f75d2",
"triggerType" : "PUSH"
}, {
"hash" : "a893da375eb9c65edabbd22965296eac9b3714f4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a893da375eb9c65edabbd22965296eac9b3714f4",
"triggerType" : "PUSH"
}, {
"hash" : "a7c2756e6a293cbba99bd4e821881e6b961c138f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7804",
"triggerID" : "a7c2756e6a293cbba99bd4e821881e6b961c138f",
"triggerType" : "PUSH"
}, {
"hash" : "4a140fffa35893a13f2e7ac8365d60d51a2d0870",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7813",
"triggerID" : "4a140fffa35893a13f2e7ac8365d60d51a2d0870",
"triggerType" : "PUSH"
}, {
"hash" : "500b4f529ad2f8a695825b5badc1815202164d0a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7830",
"triggerID" : "500b4f529ad2f8a695825b5badc1815202164d0a",
"triggerType" : "PUSH"
}, {
"hash" : "539cf8b9cdc7ab4e70488e4096684022a027f942",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "539cf8b9cdc7ab4e70488e4096684022a027f942",
"triggerType" : "PUSH"
}, {
"hash" : "85139a12b431931973c07d1b7f9256a1aae75669",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7831",
"triggerID" : "85139a12b431931973c07d1b7f9256a1aae75669",
"triggerType" : "PUSH"
}, {
"hash" : "cc430c949b539450110da96c0eccb52ab5e0998e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "cc430c949b539450110da96c0eccb52ab5e0998e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 50022dcc45b0f511dd69706aeaf6464a51be49e2 UNKNOWN
* 6cc840c51fcecde95d2187fb6fd217b50fa82806 UNKNOWN
* 77589738d73b044385f5aa6f7ef87b421f06043b UNKNOWN
* b0054c8fd329489d9a1e9535d9866edffc52fdd0 UNKNOWN
* 363e54341420cc217bf5af62de1e09ff370f75d2 UNKNOWN
* a893da375eb9c65edabbd22965296eac9b3714f4 UNKNOWN
* 539cf8b9cdc7ab4e70488e4096684022a027f942 UNKNOWN
* 85139a12b431931973c07d1b7f9256a1aae75669 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7831)
* cc430c949b539450110da96c0eccb52ab5e0998e UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9622] Add implementation of MergeHandle backed by the HoodieFileGroupReader [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #13699:
URL: https://github.com/apache/hudi/pull/13699#discussion_r2279133779
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/SecondaryIndexStreamingTracker.java:
##########
@@ -226,6 +228,81 @@ static <T> void trackSecondaryIndexStats(@Nullable HoodieKey hoodieKey, Option<H
});
}
+ /**
+ * The utility function used by Merge Handle to generate secondary index stats for the corresponding record.
+ * It considers the new merged version of the record and compares it with the older version of the record to generate
+ * secondary index stats.
+ *
+ * @param hoodieKey The hoodie key
+ * @param combinedRecordOpt New record merged with the old record
+ * @param oldRecord The old record
+ * @param isDelete Whether the record is a DELETE
+ * @param writeStatus The Write status
+ * @param secondaryIndexDefns Definitions for secondary index which need to be updated
+ */
+ static <T> void trackSecondaryIndexStats(HoodieKey hoodieKey, Option<BufferedRecord<T>> combinedRecordOpt, @Nullable BufferedRecord<T> oldRecord, boolean isDelete,
Review Comment:
if we have not added UTs for this, can you track it in a follow up jira.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9622] Add implementation of MergeHandle backed by the HoodieFileGroupReader [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #13699:
URL: https://github.com/apache/hudi/pull/13699#discussion_r2279112626
##########
hudi-spark-datasource/hudi-spark/src/test/java/org/apache/hudi/io/TestMergeHandle.java:
##########
@@ -124,4 +159,390 @@ public void testMergeHandleRLIAndSIStatsWithUpdatesAndDeletes() throws Exception
assertEquals(2 * numUpdates + numDeletes, writeStatus.getIndexStats().getSecondaryIndexStats().values().stream().findFirst().get().size());
validateSecondaryIndexStatsContent(writeStatus, numUpdates, numDeletes);
}
+
+ @ParameterizedTest // TODO add CUSTOM_MERGER once deletes are handled properly
+ @ValueSource(strings = {"EVENT_TIME_ORDERING", "COMMIT_TIME_ORDERING", "CUSTOM"})
Review Comment:
gotcha.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9622] Add implementation of MergeHandle backed by the HoodieFileGroupReader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13699:
URL: https://github.com/apache/hudi/pull/13699#issuecomment-3192406637
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7661",
"triggerID" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"triggerType" : "PUSH"
}, {
"hash" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7666",
"triggerID" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"triggerType" : "PUSH"
}, {
"hash" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7667",
"triggerID" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"triggerType" : "PUSH"
}, {
"hash" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7672",
"triggerID" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"triggerType" : "PUSH"
}, {
"hash" : "965136302f1188e5123de37734c927dc55801b24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7691",
"triggerID" : "965136302f1188e5123de37734c927dc55801b24",
"triggerType" : "PUSH"
}, {
"hash" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7693",
"triggerID" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"triggerType" : "PUSH"
}, {
"hash" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7717",
"triggerID" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"triggerType" : "PUSH"
}, {
"hash" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"triggerType" : "PUSH"
}, {
"hash" : "47f8303d6e61504275ecfddf553e356427e6c822",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7718",
"triggerID" : "47f8303d6e61504275ecfddf553e356427e6c822",
"triggerType" : "PUSH"
}, {
"hash" : "fb191ecf48d13cd81412535f7f8c9ed553d86517",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7725",
"triggerID" : "fb191ecf48d13cd81412535f7f8c9ed553d86517",
"triggerType" : "PUSH"
}, {
"hash" : "064912d3a54714935f2ee77d55cc66816166300f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7727",
"triggerID" : "064912d3a54714935f2ee77d55cc66816166300f",
"triggerType" : "PUSH"
}, {
"hash" : "77a2b45fe13dafc51f7bebb2ba859d40291ccc10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7736",
"triggerID" : "77a2b45fe13dafc51f7bebb2ba859d40291ccc10",
"triggerType" : "PUSH"
}, {
"hash" : "e4eda0e51fee0e8de5d57ea7b38dc642f53aa95d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7758",
"triggerID" : "e4eda0e51fee0e8de5d57ea7b38dc642f53aa95d",
"triggerType" : "PUSH"
}, {
"hash" : "6cc840c51fcecde95d2187fb6fd217b50fa82806",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6cc840c51fcecde95d2187fb6fd217b50fa82806",
"triggerType" : "PUSH"
}, {
"hash" : "31dab8ce9abbe4fc80e6cd81a0e433a039ca3e1d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7760",
"triggerID" : "31dab8ce9abbe4fc80e6cd81a0e433a039ca3e1d",
"triggerType" : "PUSH"
}, {
"hash" : "c04fd95dcc94146a9146c347f4f204656f3d8308",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7763",
"triggerID" : "c04fd95dcc94146a9146c347f4f204656f3d8308",
"triggerType" : "PUSH"
}, {
"hash" : "324147913d47ba8f2555e1991ce59905e5dc0b3f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7767",
"triggerID" : "324147913d47ba8f2555e1991ce59905e5dc0b3f",
"triggerType" : "PUSH"
}, {
"hash" : "3f177d42b831b30f4bd590b057dcab6ded78b634",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7774",
"triggerID" : "3f177d42b831b30f4bd590b057dcab6ded78b634",
"triggerType" : "PUSH"
}, {
"hash" : "8d6d2c06ec82d1072251ac12ddf21a890ddd5bda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7776",
"triggerID" : "8d6d2c06ec82d1072251ac12ddf21a890ddd5bda",
"triggerType" : "PUSH"
}, {
"hash" : "77589738d73b044385f5aa6f7ef87b421f06043b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "77589738d73b044385f5aa6f7ef87b421f06043b",
"triggerType" : "PUSH"
}, {
"hash" : "ad0d7ef0fbc525f880d3baadef5a22196ab97b32",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7781",
"triggerID" : "ad0d7ef0fbc525f880d3baadef5a22196ab97b32",
"triggerType" : "PUSH"
}, {
"hash" : "0db72caa3a6c7d0f0899f243ea4b8975de79e02c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7787",
"triggerID" : "0db72caa3a6c7d0f0899f243ea4b8975de79e02c",
"triggerType" : "PUSH"
}, {
"hash" : "a01ce748dc2d4cd1fd4858d1ecac8a486989e880",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7797",
"triggerID" : "a01ce748dc2d4cd1fd4858d1ecac8a486989e880",
"triggerType" : "PUSH"
}, {
"hash" : "3f35151e28bfa1dd50b31c6e5379d9769627afd9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7799",
"triggerID" : "3f35151e28bfa1dd50b31c6e5379d9769627afd9",
"triggerType" : "PUSH"
}, {
"hash" : "b0054c8fd329489d9a1e9535d9866edffc52fdd0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b0054c8fd329489d9a1e9535d9866edffc52fdd0",
"triggerType" : "PUSH"
}, {
"hash" : "363e54341420cc217bf5af62de1e09ff370f75d2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "363e54341420cc217bf5af62de1e09ff370f75d2",
"triggerType" : "PUSH"
}, {
"hash" : "a893da375eb9c65edabbd22965296eac9b3714f4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a893da375eb9c65edabbd22965296eac9b3714f4",
"triggerType" : "PUSH"
}, {
"hash" : "a7c2756e6a293cbba99bd4e821881e6b961c138f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7804",
"triggerID" : "a7c2756e6a293cbba99bd4e821881e6b961c138f",
"triggerType" : "PUSH"
}, {
"hash" : "4a140fffa35893a13f2e7ac8365d60d51a2d0870",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7813",
"triggerID" : "4a140fffa35893a13f2e7ac8365d60d51a2d0870",
"triggerType" : "PUSH"
}, {
"hash" : "500b4f529ad2f8a695825b5badc1815202164d0a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7830",
"triggerID" : "500b4f529ad2f8a695825b5badc1815202164d0a",
"triggerType" : "PUSH"
}, {
"hash" : "539cf8b9cdc7ab4e70488e4096684022a027f942",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "539cf8b9cdc7ab4e70488e4096684022a027f942",
"triggerType" : "PUSH"
}, {
"hash" : "85139a12b431931973c07d1b7f9256a1aae75669",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7831",
"triggerID" : "85139a12b431931973c07d1b7f9256a1aae75669",
"triggerType" : "PUSH"
}, {
"hash" : "cc430c949b539450110da96c0eccb52ab5e0998e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7832",
"triggerID" : "cc430c949b539450110da96c0eccb52ab5e0998e",
"triggerType" : "PUSH"
}, {
"hash" : "2a9cad38ac9c86ddacbb2de06950d515b3a1b0df",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7850",
"triggerID" : "2a9cad38ac9c86ddacbb2de06950d515b3a1b0df",
"triggerType" : "PUSH"
}, {
"hash" : "64d3575067a762fabfede759147da73ef3df0d1a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "64d3575067a762fabfede759147da73ef3df0d1a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 50022dcc45b0f511dd69706aeaf6464a51be49e2 UNKNOWN
* 6cc840c51fcecde95d2187fb6fd217b50fa82806 UNKNOWN
* 77589738d73b044385f5aa6f7ef87b421f06043b UNKNOWN
* b0054c8fd329489d9a1e9535d9866edffc52fdd0 UNKNOWN
* 363e54341420cc217bf5af62de1e09ff370f75d2 UNKNOWN
* a893da375eb9c65edabbd22965296eac9b3714f4 UNKNOWN
* 539cf8b9cdc7ab4e70488e4096684022a027f942 UNKNOWN
* 2a9cad38ac9c86ddacbb2de06950d515b3a1b0df Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7850)
* 64d3575067a762fabfede759147da73ef3df0d1a UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13699:
URL: https://github.com/apache/hudi/pull/13699#issuecomment-3176694696
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7661",
"triggerID" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"triggerType" : "PUSH"
}, {
"hash" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7666",
"triggerID" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"triggerType" : "PUSH"
}, {
"hash" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7667",
"triggerID" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"triggerType" : "PUSH"
}, {
"hash" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7672",
"triggerID" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"triggerType" : "PUSH"
}, {
"hash" : "965136302f1188e5123de37734c927dc55801b24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7691",
"triggerID" : "965136302f1188e5123de37734c927dc55801b24",
"triggerType" : "PUSH"
}, {
"hash" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7693",
"triggerID" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"triggerType" : "PUSH"
}, {
"hash" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7717",
"triggerID" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"triggerType" : "PUSH"
}, {
"hash" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"triggerType" : "PUSH"
}, {
"hash" : "47f8303d6e61504275ecfddf553e356427e6c822",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7718",
"triggerID" : "47f8303d6e61504275ecfddf553e356427e6c822",
"triggerType" : "PUSH"
}, {
"hash" : "fb191ecf48d13cd81412535f7f8c9ed553d86517",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "fb191ecf48d13cd81412535f7f8c9ed553d86517",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 50022dcc45b0f511dd69706aeaf6464a51be49e2 UNKNOWN
* 47f8303d6e61504275ecfddf553e356427e6c822 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7718)
* fb191ecf48d13cd81412535f7f8c9ed553d86517 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13699:
URL: https://github.com/apache/hudi/pull/13699#issuecomment-3172215957
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7661",
"triggerID" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"triggerType" : "PUSH"
}, {
"hash" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7666",
"triggerID" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"triggerType" : "PUSH"
}, {
"hash" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7667",
"triggerID" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"triggerType" : "PUSH"
}, {
"hash" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* af88f9e4a548191ff0cba41db6ad3e60ba9ea43b Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7667)
* 5507dfb0f3e741a539dbed0669d24dac9f1fc2b3 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13699:
URL: https://github.com/apache/hudi/pull/13699#discussion_r2265607160
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/FileGroupReaderBasedMergeHandle.java:
##########
@@ -251,6 +353,30 @@ public List<WriteStatus> close() {
}
}
+ private Option<BaseFileUpdateCallback<T>> createCallback() {
+ List<BaseFileUpdateCallback<T>> callbacks = new ArrayList<>();
+ // Handle CDC workflow.
+ if (cdcLogger.isPresent()) {
+ callbacks.add(new CDCCallback<>(cdcLogger.get(), readerContext));
+ }
+ // record index callback
+ recordIndexCallbackOpt.ifPresent(callbacks::add);
+ // Stream secondary index stats.
+ if (isSecondaryIndexStatsStreamingWritesEnabled) {
+ this.secondaryIndexCallbackOpt = Option.of(new SecondaryIndexCallback<>(
+ partitionPath,
+ writeSchemaWithMetaFields,
+ readerContext,
+ writeStatus,
+ secondaryIndexDefns
+ ));
+ } else {
+ this.secondaryIndexCallbackOpt = Option.empty();
+ }
+ secondaryIndexCallbackOpt.ifPresent(callbacks::add);
+ return callbacks.isEmpty() ? Option.empty() : Option.of(CompositeCallback.of(callbacks));
Review Comment:
all the callbacks can be initialzied as local variables, there is no need to use `CompositeCallback` when the callbacks is single.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13699:
URL: https://github.com/apache/hudi/pull/13699#discussion_r2265602705
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/FileGroupReaderBasedMergeHandle.java:
##########
@@ -106,7 +171,15 @@ public FileGroupReaderBasedMergeHandle(HoodieWriteConfig config, String instantT
} else {
this.cdcLogger = Option.empty();
}
- init(operation, this.partitionPath);
+ }
+
+ private void initRecordIndexCallback() {
+ if (this.writeStatus.isTrackingSuccessfulWrites()) {
Review Comment:
The `isTrackingSuccessfulWrites` flag in write status comes from `hoodieTable.shouldTrackSuccessRecords()`, which is true when RLI or partitioned RLI is enabled, we should skip the location tracing for compaction which is redundant.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13699:
URL: https://github.com/apache/hudi/pull/13699#discussion_r2273299236
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/UpdateProcessor.java:
##########
@@ -80,16 +106,79 @@ public BufferedRecord<T> processUpdate(String recordKey, BufferedRecord<T> previ
}
return null;
} else {
- T prevRow = previousRecord != null ? previousRecord.getRecord() : null;
- T mergedRow = mergedRecord.getRecord();
- if (prevRow != null && prevRow != mergedRow) {
- mergedRecord.setHoodieOperation(HoodieOperation.UPDATE_AFTER);
- readStats.incrementNumUpdates();
- } else if (prevRow == null) {
- mergedRecord.setHoodieOperation(HoodieOperation.INSERT);
- readStats.incrementNumInserts();
+ return handleNonDeletes(previousRecord, mergedRecord);
+ }
+ }
+
+ protected BufferedRecord<T> handleNonDeletes(BufferedRecord<T> previousRecord, BufferedRecord<T> mergedRecord) {
+ T prevRow = previousRecord != null ? previousRecord.getRecord() : null;
+ T mergedRow = mergedRecord.getRecord();
+ if (prevRow != null && prevRow != mergedRow) {
+ mergedRecord.setHoodieOperation(HoodieOperation.UPDATE_AFTER);
+ readStats.incrementNumUpdates();
+ } else if (prevRow == null) {
+ mergedRecord.setHoodieOperation(HoodieOperation.INSERT);
+ readStats.incrementNumInserts();
+ }
+ return mergedRecord.seal(readerContext);
+ }
+ }
+
+ class PayloadUpdateProcessor<T> extends StandardUpdateProcessor<T> {
+ private final String payloadClass;
+ private final Properties properties;
+
+ public PayloadUpdateProcessor(HoodieReadStats readStats, HoodieReaderContext<T> readerContext, boolean emitDeletes,
+ Properties properties, String payloadClass) {
+ super(readStats, readerContext, emitDeletes);
+ this.payloadClass = payloadClass;
+ this.properties = properties;
+ }
+
+ @Override
+ protected BufferedRecord<T> handleNonDeletes(BufferedRecord<T> previousRecord, BufferedRecord<T> mergedRecord) {
+ if (previousRecord == null) {
+ // special case for payloads when there is no previous record
+ Schema recordSchema = readerContext.getRecordContext().decodeAvroSchema(mergedRecord.getSchemaId());
+ GenericRecord record = readerContext.getRecordContext().convertToAvroRecord(mergedRecord.getRecord(), recordSchema);
+ HoodieAvroRecord hoodieRecord = new HoodieAvroRecord<>(null, HoodieRecordUtils.loadPayload(payloadClass, record, mergedRecord.getOrderingValue()));
+ try {
+ if (hoodieRecord.shouldIgnore(recordSchema, properties)) {
+ return null;
+ } else {
+ Schema readerSchema = readerContext.getSchemaHandler().getRequestedSchema();
+ // If the record schema is different from the reader schema, rewrite the record using the payload methods to ensure consistency with legacy writer paths
+ hoodieRecord.rewriteRecordWithNewSchema(recordSchema, properties, readerSchema).toIndexedRecord(readerSchema, properties)
+ .ifPresent(rewrittenRecord -> mergedRecord.replaceRecord(readerContext.getRecordContext().convertAvroRecord(rewrittenRecord.getData())));
+ }
+ } catch (IOException e) {
+ throw new HoodieIOException("Error processing record with payload class: " + payloadClass, e);
+ }
+ }
+ return super.handleNonDeletes(previousRecord, mergedRecord);
+ }
+ }
+
+ class CustomMergerUpdateProcessor<T> extends StandardUpdateProcessor<T> {
+ private final HoodieRecordMerger merger;
+ private final TypedProperties properties;
+
+ CustomMergerUpdateProcessor(HoodieReadStats readStats, HoodieReaderContext<T> readerContext, boolean emitDeletes,
+ TypedProperties properties) {
+ super(readStats, readerContext, emitDeletes);
+ this.merger = readerContext.getRecordMerger().get();
+ this.properties = properties;
+ }
+
+ @Override
+ protected BufferedRecord<T> handleNonDeletes(BufferedRecord<T> previousRecord, BufferedRecord<T> mergedRecord) {
+ try {
+ if (merger.shouldFlush(readerContext.getRecordContext().constructHoodieRecord(mergedRecord), readerContext.getRecordContext().getSchemaFromBufferRecord(mergedRecord), properties)) {
Review Comment:
What should we do about the failing tests? Override the write handle to use the old class?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13699:
URL: https://github.com/apache/hudi/pull/13699#issuecomment-3184191617
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7661",
"triggerID" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"triggerType" : "PUSH"
}, {
"hash" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7666",
"triggerID" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"triggerType" : "PUSH"
}, {
"hash" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7667",
"triggerID" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"triggerType" : "PUSH"
}, {
"hash" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7672",
"triggerID" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"triggerType" : "PUSH"
}, {
"hash" : "965136302f1188e5123de37734c927dc55801b24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7691",
"triggerID" : "965136302f1188e5123de37734c927dc55801b24",
"triggerType" : "PUSH"
}, {
"hash" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7693",
"triggerID" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"triggerType" : "PUSH"
}, {
"hash" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7717",
"triggerID" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"triggerType" : "PUSH"
}, {
"hash" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"triggerType" : "PUSH"
}, {
"hash" : "47f8303d6e61504275ecfddf553e356427e6c822",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7718",
"triggerID" : "47f8303d6e61504275ecfddf553e356427e6c822",
"triggerType" : "PUSH"
}, {
"hash" : "fb191ecf48d13cd81412535f7f8c9ed553d86517",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7725",
"triggerID" : "fb191ecf48d13cd81412535f7f8c9ed553d86517",
"triggerType" : "PUSH"
}, {
"hash" : "064912d3a54714935f2ee77d55cc66816166300f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7727",
"triggerID" : "064912d3a54714935f2ee77d55cc66816166300f",
"triggerType" : "PUSH"
}, {
"hash" : "77a2b45fe13dafc51f7bebb2ba859d40291ccc10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7736",
"triggerID" : "77a2b45fe13dafc51f7bebb2ba859d40291ccc10",
"triggerType" : "PUSH"
}, {
"hash" : "e4eda0e51fee0e8de5d57ea7b38dc642f53aa95d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7758",
"triggerID" : "e4eda0e51fee0e8de5d57ea7b38dc642f53aa95d",
"triggerType" : "PUSH"
}, {
"hash" : "6cc840c51fcecde95d2187fb6fd217b50fa82806",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6cc840c51fcecde95d2187fb6fd217b50fa82806",
"triggerType" : "PUSH"
}, {
"hash" : "31dab8ce9abbe4fc80e6cd81a0e433a039ca3e1d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7760",
"triggerID" : "31dab8ce9abbe4fc80e6cd81a0e433a039ca3e1d",
"triggerType" : "PUSH"
}, {
"hash" : "c04fd95dcc94146a9146c347f4f204656f3d8308",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7763",
"triggerID" : "c04fd95dcc94146a9146c347f4f204656f3d8308",
"triggerType" : "PUSH"
}, {
"hash" : "324147913d47ba8f2555e1991ce59905e5dc0b3f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7767",
"triggerID" : "324147913d47ba8f2555e1991ce59905e5dc0b3f",
"triggerType" : "PUSH"
}, {
"hash" : "3f177d42b831b30f4bd590b057dcab6ded78b634",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7774",
"triggerID" : "3f177d42b831b30f4bd590b057dcab6ded78b634",
"triggerType" : "PUSH"
}, {
"hash" : "8d6d2c06ec82d1072251ac12ddf21a890ddd5bda",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8d6d2c06ec82d1072251ac12ddf21a890ddd5bda",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 50022dcc45b0f511dd69706aeaf6464a51be49e2 UNKNOWN
* 6cc840c51fcecde95d2187fb6fd217b50fa82806 UNKNOWN
* 3f177d42b831b30f4bd590b057dcab6ded78b634 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7774)
* 8d6d2c06ec82d1072251ac12ddf21a890ddd5bda UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13699:
URL: https://github.com/apache/hudi/pull/13699#issuecomment-3186335448
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7661",
"triggerID" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"triggerType" : "PUSH"
}, {
"hash" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7666",
"triggerID" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"triggerType" : "PUSH"
}, {
"hash" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7667",
"triggerID" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"triggerType" : "PUSH"
}, {
"hash" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7672",
"triggerID" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"triggerType" : "PUSH"
}, {
"hash" : "965136302f1188e5123de37734c927dc55801b24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7691",
"triggerID" : "965136302f1188e5123de37734c927dc55801b24",
"triggerType" : "PUSH"
}, {
"hash" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7693",
"triggerID" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"triggerType" : "PUSH"
}, {
"hash" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7717",
"triggerID" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"triggerType" : "PUSH"
}, {
"hash" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"triggerType" : "PUSH"
}, {
"hash" : "47f8303d6e61504275ecfddf553e356427e6c822",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7718",
"triggerID" : "47f8303d6e61504275ecfddf553e356427e6c822",
"triggerType" : "PUSH"
}, {
"hash" : "fb191ecf48d13cd81412535f7f8c9ed553d86517",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7725",
"triggerID" : "fb191ecf48d13cd81412535f7f8c9ed553d86517",
"triggerType" : "PUSH"
}, {
"hash" : "064912d3a54714935f2ee77d55cc66816166300f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7727",
"triggerID" : "064912d3a54714935f2ee77d55cc66816166300f",
"triggerType" : "PUSH"
}, {
"hash" : "77a2b45fe13dafc51f7bebb2ba859d40291ccc10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7736",
"triggerID" : "77a2b45fe13dafc51f7bebb2ba859d40291ccc10",
"triggerType" : "PUSH"
}, {
"hash" : "e4eda0e51fee0e8de5d57ea7b38dc642f53aa95d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7758",
"triggerID" : "e4eda0e51fee0e8de5d57ea7b38dc642f53aa95d",
"triggerType" : "PUSH"
}, {
"hash" : "6cc840c51fcecde95d2187fb6fd217b50fa82806",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6cc840c51fcecde95d2187fb6fd217b50fa82806",
"triggerType" : "PUSH"
}, {
"hash" : "31dab8ce9abbe4fc80e6cd81a0e433a039ca3e1d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7760",
"triggerID" : "31dab8ce9abbe4fc80e6cd81a0e433a039ca3e1d",
"triggerType" : "PUSH"
}, {
"hash" : "c04fd95dcc94146a9146c347f4f204656f3d8308",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7763",
"triggerID" : "c04fd95dcc94146a9146c347f4f204656f3d8308",
"triggerType" : "PUSH"
}, {
"hash" : "324147913d47ba8f2555e1991ce59905e5dc0b3f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7767",
"triggerID" : "324147913d47ba8f2555e1991ce59905e5dc0b3f",
"triggerType" : "PUSH"
}, {
"hash" : "3f177d42b831b30f4bd590b057dcab6ded78b634",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7774",
"triggerID" : "3f177d42b831b30f4bd590b057dcab6ded78b634",
"triggerType" : "PUSH"
}, {
"hash" : "8d6d2c06ec82d1072251ac12ddf21a890ddd5bda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7776",
"triggerID" : "8d6d2c06ec82d1072251ac12ddf21a890ddd5bda",
"triggerType" : "PUSH"
}, {
"hash" : "77589738d73b044385f5aa6f7ef87b421f06043b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "77589738d73b044385f5aa6f7ef87b421f06043b",
"triggerType" : "PUSH"
}, {
"hash" : "ad0d7ef0fbc525f880d3baadef5a22196ab97b32",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7781",
"triggerID" : "ad0d7ef0fbc525f880d3baadef5a22196ab97b32",
"triggerType" : "PUSH"
}, {
"hash" : "0db72caa3a6c7d0f0899f243ea4b8975de79e02c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7787",
"triggerID" : "0db72caa3a6c7d0f0899f243ea4b8975de79e02c",
"triggerType" : "PUSH"
}, {
"hash" : "a01ce748dc2d4cd1fd4858d1ecac8a486989e880",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7797",
"triggerID" : "a01ce748dc2d4cd1fd4858d1ecac8a486989e880",
"triggerType" : "PUSH"
}, {
"hash" : "3f35151e28bfa1dd50b31c6e5379d9769627afd9",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7799",
"triggerID" : "3f35151e28bfa1dd50b31c6e5379d9769627afd9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 50022dcc45b0f511dd69706aeaf6464a51be49e2 UNKNOWN
* 6cc840c51fcecde95d2187fb6fd217b50fa82806 UNKNOWN
* 77589738d73b044385f5aa6f7ef87b421f06043b UNKNOWN
* a01ce748dc2d4cd1fd4858d1ecac8a486989e880 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7797)
* 3f35151e28bfa1dd50b31c6e5379d9769627afd9 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7799)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13699:
URL: https://github.com/apache/hudi/pull/13699#issuecomment-3181861490
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7661",
"triggerID" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"triggerType" : "PUSH"
}, {
"hash" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7666",
"triggerID" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"triggerType" : "PUSH"
}, {
"hash" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7667",
"triggerID" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"triggerType" : "PUSH"
}, {
"hash" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7672",
"triggerID" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"triggerType" : "PUSH"
}, {
"hash" : "965136302f1188e5123de37734c927dc55801b24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7691",
"triggerID" : "965136302f1188e5123de37734c927dc55801b24",
"triggerType" : "PUSH"
}, {
"hash" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7693",
"triggerID" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"triggerType" : "PUSH"
}, {
"hash" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7717",
"triggerID" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"triggerType" : "PUSH"
}, {
"hash" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"triggerType" : "PUSH"
}, {
"hash" : "47f8303d6e61504275ecfddf553e356427e6c822",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7718",
"triggerID" : "47f8303d6e61504275ecfddf553e356427e6c822",
"triggerType" : "PUSH"
}, {
"hash" : "fb191ecf48d13cd81412535f7f8c9ed553d86517",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7725",
"triggerID" : "fb191ecf48d13cd81412535f7f8c9ed553d86517",
"triggerType" : "PUSH"
}, {
"hash" : "064912d3a54714935f2ee77d55cc66816166300f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7727",
"triggerID" : "064912d3a54714935f2ee77d55cc66816166300f",
"triggerType" : "PUSH"
}, {
"hash" : "77a2b45fe13dafc51f7bebb2ba859d40291ccc10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7736",
"triggerID" : "77a2b45fe13dafc51f7bebb2ba859d40291ccc10",
"triggerType" : "PUSH"
}, {
"hash" : "e4eda0e51fee0e8de5d57ea7b38dc642f53aa95d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7758",
"triggerID" : "e4eda0e51fee0e8de5d57ea7b38dc642f53aa95d",
"triggerType" : "PUSH"
}, {
"hash" : "6cc840c51fcecde95d2187fb6fd217b50fa82806",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6cc840c51fcecde95d2187fb6fd217b50fa82806",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 50022dcc45b0f511dd69706aeaf6464a51be49e2 UNKNOWN
* e4eda0e51fee0e8de5d57ea7b38dc642f53aa95d Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7758)
* 6cc840c51fcecde95d2187fb6fd217b50fa82806 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13699:
URL: https://github.com/apache/hudi/pull/13699#issuecomment-3177461811
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7661",
"triggerID" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"triggerType" : "PUSH"
}, {
"hash" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7666",
"triggerID" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"triggerType" : "PUSH"
}, {
"hash" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7667",
"triggerID" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"triggerType" : "PUSH"
}, {
"hash" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7672",
"triggerID" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"triggerType" : "PUSH"
}, {
"hash" : "965136302f1188e5123de37734c927dc55801b24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7691",
"triggerID" : "965136302f1188e5123de37734c927dc55801b24",
"triggerType" : "PUSH"
}, {
"hash" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7693",
"triggerID" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"triggerType" : "PUSH"
}, {
"hash" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7717",
"triggerID" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"triggerType" : "PUSH"
}, {
"hash" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"triggerType" : "PUSH"
}, {
"hash" : "47f8303d6e61504275ecfddf553e356427e6c822",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7718",
"triggerID" : "47f8303d6e61504275ecfddf553e356427e6c822",
"triggerType" : "PUSH"
}, {
"hash" : "fb191ecf48d13cd81412535f7f8c9ed553d86517",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7725",
"triggerID" : "fb191ecf48d13cd81412535f7f8c9ed553d86517",
"triggerType" : "PUSH"
}, {
"hash" : "064912d3a54714935f2ee77d55cc66816166300f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7727",
"triggerID" : "064912d3a54714935f2ee77d55cc66816166300f",
"triggerType" : "PUSH"
}, {
"hash" : "77a2b45fe13dafc51f7bebb2ba859d40291ccc10",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "77a2b45fe13dafc51f7bebb2ba859d40291ccc10",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 50022dcc45b0f511dd69706aeaf6464a51be49e2 UNKNOWN
* 064912d3a54714935f2ee77d55cc66816166300f Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7727)
* 77a2b45fe13dafc51f7bebb2ba859d40291ccc10 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13699:
URL: https://github.com/apache/hudi/pull/13699#discussion_r2271909577
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/IOUtils.java:
##########
@@ -119,6 +119,9 @@ public static Iterator<List<WriteStatus>> runMerge(HoodieMergeHandle<?, ?, ?, ?>
"Error in finding the old file path at commit " + instantTime + " for fileId: " + fileId);
} else {
mergeHandle.doMerge();
+ if (mergeHandle instanceof FileGroupReaderBasedMergeHandle) {
+ mergeHandle.close();
Review Comment:
It seems we shoud just return `mergeHandle.close` in line 132 instead of closing here for all the merge handles.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13699:
URL: https://github.com/apache/hudi/pull/13699#discussion_r2272587794
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/buffer/FileGroupRecordBuffer.java:
##########
@@ -56,27 +53,29 @@
import java.io.IOException;
import java.io.Serializable;
+import java.util.HashSet;
import java.util.Iterator;
import java.util.List;
import java.util.Locale;
import java.util.Map;
+import java.util.Set;
import java.util.function.Function;
import static org.apache.hudi.common.config.HoodieCommonConfig.DISK_MAP_BITCASK_COMPRESSION_ENABLED;
import static org.apache.hudi.common.config.HoodieCommonConfig.SPILLABLE_DISK_MAP_TYPE;
import static org.apache.hudi.common.config.HoodieMemoryConfig.MAX_MEMORY_FOR_MERGE;
import static org.apache.hudi.common.config.HoodieMemoryConfig.SPILLABLE_MAP_BASE_PATH;
-import static org.apache.hudi.common.model.HoodieRecordMerger.PAYLOAD_BASED_MERGE_STRATEGY_UUID;
import static org.apache.hudi.common.table.log.block.HoodieLogBlock.HeaderMetadataType.INSTANT_TIME;
abstract class FileGroupRecordBuffer<T> implements HoodieFileGroupRecordBuffer<T> {
+ protected final Set<String> usedKeys = new HashSet<>();
Review Comment:
The behavior is incorrect, fire a JIRA to trace it: https://issues.apache.org/jira/browse/HUDI-9708, let's abadon the support for duplicates in base file and fixe it in a separate PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13699:
URL: https://github.com/apache/hudi/pull/13699#discussion_r2265614815
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/UpdateProcessor.java:
##########
@@ -80,17 +99,58 @@ public BufferedRecord<T> processUpdate(String recordKey, BufferedRecord<T> previ
}
return null;
} else {
- T prevRow = previousRecord != null ? previousRecord.getRecord() : null;
- T mergedRow = mergedRecord.getRecord();
- if (prevRow != null && prevRow != mergedRow) {
- mergedRecord.setHoodieOperation(HoodieOperation.UPDATE_AFTER);
- readStats.incrementNumUpdates();
- } else if (prevRow == null) {
- mergedRecord.setHoodieOperation(HoodieOperation.INSERT);
- readStats.incrementNumInserts();
+ return handleNonDeletes(previousRecord, mergedRecord);
+ }
+ }
+
+ protected BufferedRecord<T> handleNonDeletes(BufferedRecord<T> previousRecord, BufferedRecord<T> mergedRecord) {
+ T prevRow = previousRecord != null ? previousRecord.getRecord() : null;
+ T mergedRow = mergedRecord.getRecord();
+ if (prevRow != null && prevRow != mergedRow) {
+ mergedRecord.setHoodieOperation(HoodieOperation.UPDATE_AFTER);
+ readStats.incrementNumUpdates();
+ } else if (prevRow == null) {
+ mergedRecord.setHoodieOperation(HoodieOperation.INSERT);
+ readStats.incrementNumInserts();
+ }
+ return mergedRecord.seal(readerContext);
+ }
+ }
+
+ class PayloadUpdateProcessor<T> extends StandardUpdateProcessor<T> {
+ private final String payloadClass;
+ private final Properties properties;
+
+ public PayloadUpdateProcessor(HoodieReadStats readStats, HoodieReaderContext<T> readerContext, boolean emitDeletes,
+ Properties properties, String payloadClass) {
+ super(readStats, readerContext, emitDeletes);
+ this.payloadClass = payloadClass;
+ this.properties = properties;
+ }
+
+ @Override
+ protected BufferedRecord<T> handleNonDeletes(BufferedRecord<T> previousRecord, BufferedRecord<T> mergedRecord) {
+ if (previousRecord == null) {
+ // special case for payloads when there is no previous record
+ Schema recordSchema = readerContext.getRecordContext().decodeAvroSchema(mergedRecord.getSchemaId());
+ GenericRecord record = readerContext.getRecordContext().convertToAvroRecord(mergedRecord.getRecord(), recordSchema);
+ HoodieAvroRecord hoodieRecord = new HoodieAvroRecord<>(null, HoodieRecordUtils.loadPayload(payloadClass, record, mergedRecord.getOrderingValue()));
+ try {
+ if (hoodieRecord.shouldIgnore(recordSchema, properties)) {
+ return null;
+ } else {
+ Schema readerSchema = readerContext.getSchemaHandler().getRequestedSchema();
+ // If the record schema is different from the reader schema, rewrite the record using the payload methods to ensure consistency with legacy writer paths
+ if (!readerSchema.equals(recordSchema)) {
Review Comment:
This could be super costly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13699:
URL: https://github.com/apache/hudi/pull/13699#issuecomment-3175725067
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7661",
"triggerID" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"triggerType" : "PUSH"
}, {
"hash" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7666",
"triggerID" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"triggerType" : "PUSH"
}, {
"hash" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7667",
"triggerID" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"triggerType" : "PUSH"
}, {
"hash" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7672",
"triggerID" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"triggerType" : "PUSH"
}, {
"hash" : "965136302f1188e5123de37734c927dc55801b24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7691",
"triggerID" : "965136302f1188e5123de37734c927dc55801b24",
"triggerType" : "PUSH"
}, {
"hash" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7693",
"triggerID" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"triggerType" : "PUSH"
}, {
"hash" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7717",
"triggerID" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"triggerType" : "PUSH"
}, {
"hash" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"triggerType" : "PUSH"
}, {
"hash" : "47f8303d6e61504275ecfddf553e356427e6c822",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "47f8303d6e61504275ecfddf553e356427e6c822",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 22721bae05acf8aa296bc5732af5a90d2afd3515 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7717)
* 50022dcc45b0f511dd69706aeaf6464a51be49e2 UNKNOWN
* 47f8303d6e61504275ecfddf553e356427e6c822 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13699:
URL: https://github.com/apache/hudi/pull/13699#issuecomment-3175707731
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7661",
"triggerID" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"triggerType" : "PUSH"
}, {
"hash" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7666",
"triggerID" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"triggerType" : "PUSH"
}, {
"hash" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7667",
"triggerID" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"triggerType" : "PUSH"
}, {
"hash" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7672",
"triggerID" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"triggerType" : "PUSH"
}, {
"hash" : "965136302f1188e5123de37734c927dc55801b24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7691",
"triggerID" : "965136302f1188e5123de37734c927dc55801b24",
"triggerType" : "PUSH"
}, {
"hash" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7693",
"triggerID" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"triggerType" : "PUSH"
}, {
"hash" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7717",
"triggerID" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"triggerType" : "PUSH"
}, {
"hash" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 22721bae05acf8aa296bc5732af5a90d2afd3515 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7717)
* 50022dcc45b0f511dd69706aeaf6464a51be49e2 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13699:
URL: https://github.com/apache/hudi/pull/13699#discussion_r2271842003
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/FileGroupReaderBasedMergeHandle.java:
##########
@@ -251,6 +353,30 @@ public List<WriteStatus> close() {
}
}
+ private Option<BaseFileUpdateCallback<T>> createCallback() {
+ List<BaseFileUpdateCallback<T>> callbacks = new ArrayList<>();
+ // Handle CDC workflow.
+ if (cdcLogger.isPresent()) {
+ callbacks.add(new CDCCallback<>(cdcLogger.get(), readerContext));
+ }
+ // record index callback
+ recordIndexCallbackOpt.ifPresent(callbacks::add);
+ // Stream secondary index stats.
+ if (isSecondaryIndexStatsStreamingWritesEnabled) {
+ this.secondaryIndexCallbackOpt = Option.of(new SecondaryIndexCallback<>(
+ partitionPath,
+ writeSchemaWithMetaFields,
+ readerContext,
+ writeStatus,
+ secondaryIndexDefns
+ ));
+ } else {
+ this.secondaryIndexCallbackOpt = Option.empty();
+ }
+ secondaryIndexCallbackOpt.ifPresent(callbacks::add);
+ return callbacks.isEmpty() ? Option.empty() : Option.of(CompositeCallback.of(callbacks));
Review Comment:
The static constructor will just return the first callback if there is a single element
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13699:
URL: https://github.com/apache/hudi/pull/13699#issuecomment-3176799028
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7661",
"triggerID" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"triggerType" : "PUSH"
}, {
"hash" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7666",
"triggerID" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"triggerType" : "PUSH"
}, {
"hash" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7667",
"triggerID" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"triggerType" : "PUSH"
}, {
"hash" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7672",
"triggerID" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"triggerType" : "PUSH"
}, {
"hash" : "965136302f1188e5123de37734c927dc55801b24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7691",
"triggerID" : "965136302f1188e5123de37734c927dc55801b24",
"triggerType" : "PUSH"
}, {
"hash" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7693",
"triggerID" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"triggerType" : "PUSH"
}, {
"hash" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7717",
"triggerID" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"triggerType" : "PUSH"
}, {
"hash" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"triggerType" : "PUSH"
}, {
"hash" : "47f8303d6e61504275ecfddf553e356427e6c822",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7718",
"triggerID" : "47f8303d6e61504275ecfddf553e356427e6c822",
"triggerType" : "PUSH"
}, {
"hash" : "fb191ecf48d13cd81412535f7f8c9ed553d86517",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7725",
"triggerID" : "fb191ecf48d13cd81412535f7f8c9ed553d86517",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 50022dcc45b0f511dd69706aeaf6464a51be49e2 UNKNOWN
* 47f8303d6e61504275ecfddf553e356427e6c822 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7718)
* fb191ecf48d13cd81412535f7f8c9ed553d86517 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7725)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9622] Add implementation of MergeHandle backed by the HoodieFileGroupReader [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13699:
URL: https://github.com/apache/hudi/pull/13699#discussion_r2279043962
##########
hudi-common/src/main/java/org/apache/hudi/common/model/HoodieRecordPayload.java:
##########
@@ -217,4 +217,16 @@ static Option<String> getPayloadClassNameIfPresent(Properties props) {
// Need to transparently change to org.apache.hudi.
return Option.ofNullable(payloadClassName).map(className -> className.replace("com.uber.hoodie", "org.apache.hudi"));
}
+
+ /**
+ * Returns the writer payload override class if present in the properties.
+ * @param properties the properties to inspect
+ * @return an Option containing the writer payload override class name if present, otherwise an empty Option
+ */
+ static Option<String> getWriterPayloadOverride(Properties properties) {
+ if (properties.containsKey("hoodie.datasource.write.payload.class")) {
Review Comment:
Thanks! I'm realizing I can also just do `Option.ofNullable` on the result of the `get`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9622] Add implementation of MergeHandle backed by the HoodieFileGroupReader [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13699:
URL: https://github.com/apache/hudi/pull/13699#discussion_r2277896506
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/UpdateProcessor.java:
##########
@@ -80,17 +101,56 @@ public BufferedRecord<T> processUpdate(String recordKey, BufferedRecord<T> previ
}
return null;
} else {
- T prevRow = previousRecord != null ? previousRecord.getRecord() : null;
- T mergedRow = mergedRecord.getRecord();
- if (prevRow != null && prevRow != mergedRow) {
- mergedRecord.setHoodieOperation(HoodieOperation.UPDATE_AFTER);
- readStats.incrementNumUpdates();
- } else if (prevRow == null) {
- mergedRecord.setHoodieOperation(HoodieOperation.INSERT);
- readStats.incrementNumInserts();
+ return handleNonDeletes(previousRecord, mergedRecord);
+ }
+ }
+
+ protected BufferedRecord<T> handleNonDeletes(BufferedRecord<T> previousRecord, BufferedRecord<T> mergedRecord) {
+ T prevRow = previousRecord != null ? previousRecord.getRecord() : null;
+ T mergedRow = mergedRecord.getRecord();
+ if (prevRow != null && prevRow != mergedRow) {
+ mergedRecord.setHoodieOperation(HoodieOperation.UPDATE_AFTER);
+ readStats.incrementNumUpdates();
+ } else if (prevRow == null) {
+ mergedRecord.setHoodieOperation(HoodieOperation.INSERT);
+ readStats.incrementNumInserts();
+ }
+ return mergedRecord.seal(readerContext.getRecordContext());
+ }
+ }
+
+ class PayloadUpdateProcessor<T> extends StandardUpdateProcessor<T> {
Review Comment:
There are existing test cases that will fail without these processor changes. We can add some UTs as well though
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9622] Add implementation of MergeHandle backed by the HoodieFileGroupReader [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13699:
URL: https://github.com/apache/hudi/pull/13699#discussion_r2277909608
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/table/action/commit/BaseSparkCommitActionExecutor.java:
##########
@@ -397,6 +401,9 @@ protected HoodieMergeHandle getUpdateHandle(String partitionPath, String fileId,
mergeHandle.setPartitionFields(partitionFields);
mergeHandle.setPartitionValues(partitionValues);
}
+ if (readerContextFactory != null && mergeHandle instanceof FileGroupReaderBasedMergeHandle) {
Review Comment:
I did this before but @danny0405 reverted it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9622] Add implementation of MergeHandle backed by the HoodieFileGroupReader [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13699:
URL: https://github.com/apache/hudi/pull/13699#discussion_r2278057054
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/index/HoodieIndexUtils.java:
##########
@@ -492,10 +487,13 @@ private static <R> Option<HoodieRecord<R>> mergeIncomingWithExistingRecord(
// the record was deleted
return Option.empty();
}
- String partitionPath = inferPartitionPath(incoming, existing, writeSchemaWithMetaFields, keyGenerator, existingRecordContext, mergeResult);
+ Schema recordSchema = incomingRecordContext.getSchemaFromBufferRecord(mergeResult);
+ String partitionPath = inferPartitionPath(incoming, existing, recordSchema, keyGenerator, existingRecordContext, mergeResult);
HoodieRecord<R> result = existingRecordContext.constructHoodieRecord(mergeResult, partitionPath);
+ HoodieRecord<R> withMeta = result.prependMetaFields(recordSchema, writeSchemaWithMetaFields,
Review Comment:
I've removed this now, let's see if anything fails. I ran the RLI and secondary index testing locally and they were passing. May have been due to some other error at the time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13699:
URL: https://github.com/apache/hudi/pull/13699#issuecomment-3186506630
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7661",
"triggerID" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"triggerType" : "PUSH"
}, {
"hash" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7666",
"triggerID" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"triggerType" : "PUSH"
}, {
"hash" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7667",
"triggerID" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"triggerType" : "PUSH"
}, {
"hash" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7672",
"triggerID" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"triggerType" : "PUSH"
}, {
"hash" : "965136302f1188e5123de37734c927dc55801b24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7691",
"triggerID" : "965136302f1188e5123de37734c927dc55801b24",
"triggerType" : "PUSH"
}, {
"hash" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7693",
"triggerID" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"triggerType" : "PUSH"
}, {
"hash" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7717",
"triggerID" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"triggerType" : "PUSH"
}, {
"hash" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"triggerType" : "PUSH"
}, {
"hash" : "47f8303d6e61504275ecfddf553e356427e6c822",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7718",
"triggerID" : "47f8303d6e61504275ecfddf553e356427e6c822",
"triggerType" : "PUSH"
}, {
"hash" : "fb191ecf48d13cd81412535f7f8c9ed553d86517",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7725",
"triggerID" : "fb191ecf48d13cd81412535f7f8c9ed553d86517",
"triggerType" : "PUSH"
}, {
"hash" : "064912d3a54714935f2ee77d55cc66816166300f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7727",
"triggerID" : "064912d3a54714935f2ee77d55cc66816166300f",
"triggerType" : "PUSH"
}, {
"hash" : "77a2b45fe13dafc51f7bebb2ba859d40291ccc10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7736",
"triggerID" : "77a2b45fe13dafc51f7bebb2ba859d40291ccc10",
"triggerType" : "PUSH"
}, {
"hash" : "e4eda0e51fee0e8de5d57ea7b38dc642f53aa95d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7758",
"triggerID" : "e4eda0e51fee0e8de5d57ea7b38dc642f53aa95d",
"triggerType" : "PUSH"
}, {
"hash" : "6cc840c51fcecde95d2187fb6fd217b50fa82806",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6cc840c51fcecde95d2187fb6fd217b50fa82806",
"triggerType" : "PUSH"
}, {
"hash" : "31dab8ce9abbe4fc80e6cd81a0e433a039ca3e1d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7760",
"triggerID" : "31dab8ce9abbe4fc80e6cd81a0e433a039ca3e1d",
"triggerType" : "PUSH"
}, {
"hash" : "c04fd95dcc94146a9146c347f4f204656f3d8308",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7763",
"triggerID" : "c04fd95dcc94146a9146c347f4f204656f3d8308",
"triggerType" : "PUSH"
}, {
"hash" : "324147913d47ba8f2555e1991ce59905e5dc0b3f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7767",
"triggerID" : "324147913d47ba8f2555e1991ce59905e5dc0b3f",
"triggerType" : "PUSH"
}, {
"hash" : "3f177d42b831b30f4bd590b057dcab6ded78b634",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7774",
"triggerID" : "3f177d42b831b30f4bd590b057dcab6ded78b634",
"triggerType" : "PUSH"
}, {
"hash" : "8d6d2c06ec82d1072251ac12ddf21a890ddd5bda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7776",
"triggerID" : "8d6d2c06ec82d1072251ac12ddf21a890ddd5bda",
"triggerType" : "PUSH"
}, {
"hash" : "77589738d73b044385f5aa6f7ef87b421f06043b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "77589738d73b044385f5aa6f7ef87b421f06043b",
"triggerType" : "PUSH"
}, {
"hash" : "ad0d7ef0fbc525f880d3baadef5a22196ab97b32",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7781",
"triggerID" : "ad0d7ef0fbc525f880d3baadef5a22196ab97b32",
"triggerType" : "PUSH"
}, {
"hash" : "0db72caa3a6c7d0f0899f243ea4b8975de79e02c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7787",
"triggerID" : "0db72caa3a6c7d0f0899f243ea4b8975de79e02c",
"triggerType" : "PUSH"
}, {
"hash" : "a01ce748dc2d4cd1fd4858d1ecac8a486989e880",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7797",
"triggerID" : "a01ce748dc2d4cd1fd4858d1ecac8a486989e880",
"triggerType" : "PUSH"
}, {
"hash" : "3f35151e28bfa1dd50b31c6e5379d9769627afd9",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7799",
"triggerID" : "3f35151e28bfa1dd50b31c6e5379d9769627afd9",
"triggerType" : "PUSH"
}, {
"hash" : "b0054c8fd329489d9a1e9535d9866edffc52fdd0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b0054c8fd329489d9a1e9535d9866edffc52fdd0",
"triggerType" : "PUSH"
}, {
"hash" : "363e54341420cc217bf5af62de1e09ff370f75d2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "363e54341420cc217bf5af62de1e09ff370f75d2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 50022dcc45b0f511dd69706aeaf6464a51be49e2 UNKNOWN
* 6cc840c51fcecde95d2187fb6fd217b50fa82806 UNKNOWN
* 77589738d73b044385f5aa6f7ef87b421f06043b UNKNOWN
* 3f35151e28bfa1dd50b31c6e5379d9769627afd9 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7799)
* b0054c8fd329489d9a1e9535d9866edffc52fdd0 UNKNOWN
* 363e54341420cc217bf5af62de1e09ff370f75d2 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9622] Add implementation of MergeHandle backed by the HoodieFileGroupReader [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #13699:
URL: https://github.com/apache/hudi/pull/13699#discussion_r2279576959
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/dml/others/TestMergeIntoTable2.scala:
##########
@@ -838,72 +839,72 @@ class TestMergeIntoTable2 extends HoodieSparkSqlTestBase {
}
}
- test("Test only insert for source table in dup key without preCombineField") {
- spark.sql(s"set ${MERGE_SMALL_FILE_GROUP_CANDIDATES_LIMIT.key} = ${MERGE_SMALL_FILE_GROUP_CANDIDATES_LIMIT.defaultValue()}")
- Seq("cow", "mor").foreach {
- tableType => {
- withTempDir { tmp =>
- val tableName = generateTableName
- spark.sql(
- s"""
- | create table $tableName (
- | id int,
- | name string,
- | price double,
- | ts int,
- | dt string
- | ) using hudi
- | tblproperties (
- | type = '$tableType',
- | primaryKey = 'id'
- | )
- | partitioned by(dt)
- | location '${tmp.getCanonicalPath}'
- """.stripMargin)
- // append records to small file is use update bucket, set this conf use concat handler
- spark.sql("set hoodie.merge.allow.duplicate.on.inserts = true")
-
- // Insert data without matched condition
- spark.sql(
- s"""
- | merge into $tableName as t0
- | using (
- | select 1 as id, 'a1' as name, 10.1 as price, 1000 as ts, '2021-03-21' as dt
- | union all
- | select 1 as id, 'a2' as name, 10.2 as price, 1002 as ts, '2021-03-21' as dt
- | ) as s0
- | on t0.id = s0.id
- | when not matched then insert *
- """.stripMargin
- )
- checkAnswer(s"select id, name, price, ts, dt from $tableName")(
- Seq(1, "a1", 10.1, 1000, "2021-03-21"),
- Seq(1, "a2", 10.2, 1002, "2021-03-21")
- )
-
- // Insert data with matched condition
- spark.sql(
- s"""
- | merge into $tableName as t0
- | using (
- | select 3 as id, 'a3' as name, 10.3 as price, 1003 as ts, '2021-03-21' as dt
- | union all
- | select 1 as id, 'a2' as name, 10.4 as price, 1004 as ts, '2021-03-21' as dt
- | ) as s0
- | on t0.id = s0.id
- | when matched then update set *
- | when not matched then insert *
- """.stripMargin
- )
- checkAnswer(s"select id, name, price, ts, dt from $tableName")(
- Seq(1, "a2", 10.4, 1004, "2021-03-21"),
- Seq(1, "a2", 10.4, 1004, "2021-03-21"),
- Seq(3, "a3", 10.3, 1003, "2021-03-21")
- )
- }
- }
- }
- }
+// test("Test only insert for source table in dup key without preCombineField") {
Review Comment:
gotcha. So if there were duplicates in a given file group, eventually the dups will get cleared automatically when new file slice is created after this migration to FG reader. Prior to this, the dups might still get carried over and over unless we intervene w/ some manual repairs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9622] Add implementation of MergeHandle backed by the HoodieFileGroupReader [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13699:
URL: https://github.com/apache/hudi/pull/13699#discussion_r2277907392
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/FileGroupReaderBasedMergeHandle.java:
##########
@@ -251,6 +344,31 @@ public List<WriteStatus> close() {
}
}
+ private Option<BaseFileUpdateCallback<T>> createCallback() {
+ List<BaseFileUpdateCallback<T>> callbacks = new ArrayList<>();
+ // Handle CDC workflow.
+ if (cdcLogger.isPresent()) {
+ callbacks.add(new CDCCallback<>(cdcLogger.get(), readerContext));
+ }
+ // Indexes are not updated during compaction
+ if (operation.isEmpty()) {
+ // record index callback
+ if (this.writeStatus.isTrackingSuccessfulWrites()) {
+ writeStatus.manuallyTrackSuccess();
+ callbacks.add(new RecordLevelIndexCallback<>(writeStatus, newRecordLocation, partitionPath));
+ }
+ // Stream secondary index stats.
+ if (isSecondaryIndexStatsStreamingWritesEnabled) {
+ callbacks.add(new SecondaryIndexCallback<>(
Review Comment:
No, that is not what we discussed. The discussion was to avoid conversion to HoodieRecord just to update SI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9622] Add implementation of MergeHandle backed by the HoodieFileGroupReader [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on PR #13699:
URL: https://github.com/apache/hudi/pull/13699#issuecomment-3191694586
Pending comments to resolve before I can give it a go.
https://github.com/apache/hudi/pull/13699#discussion_r2277777028
https://github.com/apache/hudi/pull/13699#discussion_r2279120259
and suggested to track few minor follow ups (UTs ) in a jira.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9622] Add implementation of MergeHandle backed by the HoodieFileGroupReader [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #13699:
URL: https://github.com/apache/hudi/pull/13699#discussion_r2279570809
##########
hudi-client/hudi-client-common/src/test/java/org/apache/hudi/testutils/MetadataMergeWriteStatus.java:
##########
@@ -60,12 +60,12 @@ private static void mergeMetadataMaps(Map<String, String> mergeFromMap, Map<Stri
if (!mergeToMap.containsKey(key)) {
mergeToMap.put(key, "0");
}
- mergeToMap.put(key, addStrsAsInt(entry.getValue(), mergeToMap.get(key)));
+ mergeToMap.put(key, addStrsAsLong(entry.getValue(), mergeToMap.get(key)));
}
}
- private static String addStrsAsInt(String a, String b) {
- return String.valueOf(Integer.parseInt(a) + Integer.parseInt(b));
+ private static String addStrsAsLong(String a, String b) {
+ return String.valueOf(Long.parseLong(a) + Long.parseLong(b));
Review Comment:
I triaged this.
looks like the event time metadata tracking could have been broken only if not for this fix. I am not sure if someone was ever using this feature only.
##########
hudi-client/hudi-client-common/src/test/java/org/apache/hudi/testutils/MetadataMergeWriteStatus.java:
##########
@@ -60,12 +60,12 @@ private static void mergeMetadataMaps(Map<String, String> mergeFromMap, Map<Stri
if (!mergeToMap.containsKey(key)) {
mergeToMap.put(key, "0");
}
- mergeToMap.put(key, addStrsAsInt(entry.getValue(), mergeToMap.get(key)));
+ mergeToMap.put(key, addStrsAsLong(entry.getValue(), mergeToMap.get(key)));
}
}
- private static String addStrsAsInt(String a, String b) {
- return String.valueOf(Integer.parseInt(a) + Integer.parseInt(b));
+ private static String addStrsAsLong(String a, String b) {
+ return String.valueOf(Long.parseLong(a) + Long.parseLong(b));
Review Comment:
whats this fix is about?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9622] Add implementation of MergeHandle backed by the HoodieFileGroupReader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13699:
URL: https://github.com/apache/hudi/pull/13699#issuecomment-3190510872
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7661",
"triggerID" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"triggerType" : "PUSH"
}, {
"hash" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7666",
"triggerID" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"triggerType" : "PUSH"
}, {
"hash" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7667",
"triggerID" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"triggerType" : "PUSH"
}, {
"hash" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7672",
"triggerID" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"triggerType" : "PUSH"
}, {
"hash" : "965136302f1188e5123de37734c927dc55801b24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7691",
"triggerID" : "965136302f1188e5123de37734c927dc55801b24",
"triggerType" : "PUSH"
}, {
"hash" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7693",
"triggerID" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"triggerType" : "PUSH"
}, {
"hash" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7717",
"triggerID" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"triggerType" : "PUSH"
}, {
"hash" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"triggerType" : "PUSH"
}, {
"hash" : "47f8303d6e61504275ecfddf553e356427e6c822",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7718",
"triggerID" : "47f8303d6e61504275ecfddf553e356427e6c822",
"triggerType" : "PUSH"
}, {
"hash" : "fb191ecf48d13cd81412535f7f8c9ed553d86517",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7725",
"triggerID" : "fb191ecf48d13cd81412535f7f8c9ed553d86517",
"triggerType" : "PUSH"
}, {
"hash" : "064912d3a54714935f2ee77d55cc66816166300f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7727",
"triggerID" : "064912d3a54714935f2ee77d55cc66816166300f",
"triggerType" : "PUSH"
}, {
"hash" : "77a2b45fe13dafc51f7bebb2ba859d40291ccc10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7736",
"triggerID" : "77a2b45fe13dafc51f7bebb2ba859d40291ccc10",
"triggerType" : "PUSH"
}, {
"hash" : "e4eda0e51fee0e8de5d57ea7b38dc642f53aa95d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7758",
"triggerID" : "e4eda0e51fee0e8de5d57ea7b38dc642f53aa95d",
"triggerType" : "PUSH"
}, {
"hash" : "6cc840c51fcecde95d2187fb6fd217b50fa82806",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6cc840c51fcecde95d2187fb6fd217b50fa82806",
"triggerType" : "PUSH"
}, {
"hash" : "31dab8ce9abbe4fc80e6cd81a0e433a039ca3e1d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7760",
"triggerID" : "31dab8ce9abbe4fc80e6cd81a0e433a039ca3e1d",
"triggerType" : "PUSH"
}, {
"hash" : "c04fd95dcc94146a9146c347f4f204656f3d8308",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7763",
"triggerID" : "c04fd95dcc94146a9146c347f4f204656f3d8308",
"triggerType" : "PUSH"
}, {
"hash" : "324147913d47ba8f2555e1991ce59905e5dc0b3f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7767",
"triggerID" : "324147913d47ba8f2555e1991ce59905e5dc0b3f",
"triggerType" : "PUSH"
}, {
"hash" : "3f177d42b831b30f4bd590b057dcab6ded78b634",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7774",
"triggerID" : "3f177d42b831b30f4bd590b057dcab6ded78b634",
"triggerType" : "PUSH"
}, {
"hash" : "8d6d2c06ec82d1072251ac12ddf21a890ddd5bda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7776",
"triggerID" : "8d6d2c06ec82d1072251ac12ddf21a890ddd5bda",
"triggerType" : "PUSH"
}, {
"hash" : "77589738d73b044385f5aa6f7ef87b421f06043b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "77589738d73b044385f5aa6f7ef87b421f06043b",
"triggerType" : "PUSH"
}, {
"hash" : "ad0d7ef0fbc525f880d3baadef5a22196ab97b32",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7781",
"triggerID" : "ad0d7ef0fbc525f880d3baadef5a22196ab97b32",
"triggerType" : "PUSH"
}, {
"hash" : "0db72caa3a6c7d0f0899f243ea4b8975de79e02c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7787",
"triggerID" : "0db72caa3a6c7d0f0899f243ea4b8975de79e02c",
"triggerType" : "PUSH"
}, {
"hash" : "a01ce748dc2d4cd1fd4858d1ecac8a486989e880",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7797",
"triggerID" : "a01ce748dc2d4cd1fd4858d1ecac8a486989e880",
"triggerType" : "PUSH"
}, {
"hash" : "3f35151e28bfa1dd50b31c6e5379d9769627afd9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7799",
"triggerID" : "3f35151e28bfa1dd50b31c6e5379d9769627afd9",
"triggerType" : "PUSH"
}, {
"hash" : "b0054c8fd329489d9a1e9535d9866edffc52fdd0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b0054c8fd329489d9a1e9535d9866edffc52fdd0",
"triggerType" : "PUSH"
}, {
"hash" : "363e54341420cc217bf5af62de1e09ff370f75d2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "363e54341420cc217bf5af62de1e09ff370f75d2",
"triggerType" : "PUSH"
}, {
"hash" : "a893da375eb9c65edabbd22965296eac9b3714f4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a893da375eb9c65edabbd22965296eac9b3714f4",
"triggerType" : "PUSH"
}, {
"hash" : "a7c2756e6a293cbba99bd4e821881e6b961c138f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7804",
"triggerID" : "a7c2756e6a293cbba99bd4e821881e6b961c138f",
"triggerType" : "PUSH"
}, {
"hash" : "4a140fffa35893a13f2e7ac8365d60d51a2d0870",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7813",
"triggerID" : "4a140fffa35893a13f2e7ac8365d60d51a2d0870",
"triggerType" : "PUSH"
}, {
"hash" : "500b4f529ad2f8a695825b5badc1815202164d0a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7830",
"triggerID" : "500b4f529ad2f8a695825b5badc1815202164d0a",
"triggerType" : "PUSH"
}, {
"hash" : "539cf8b9cdc7ab4e70488e4096684022a027f942",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "539cf8b9cdc7ab4e70488e4096684022a027f942",
"triggerType" : "PUSH"
}, {
"hash" : "85139a12b431931973c07d1b7f9256a1aae75669",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7831",
"triggerID" : "85139a12b431931973c07d1b7f9256a1aae75669",
"triggerType" : "PUSH"
}, {
"hash" : "cc430c949b539450110da96c0eccb52ab5e0998e",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7832",
"triggerID" : "cc430c949b539450110da96c0eccb52ab5e0998e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 50022dcc45b0f511dd69706aeaf6464a51be49e2 UNKNOWN
* 6cc840c51fcecde95d2187fb6fd217b50fa82806 UNKNOWN
* 77589738d73b044385f5aa6f7ef87b421f06043b UNKNOWN
* b0054c8fd329489d9a1e9535d9866edffc52fdd0 UNKNOWN
* 363e54341420cc217bf5af62de1e09ff370f75d2 UNKNOWN
* a893da375eb9c65edabbd22965296eac9b3714f4 UNKNOWN
* 539cf8b9cdc7ab4e70488e4096684022a027f942 UNKNOWN
* 85139a12b431931973c07d1b7f9256a1aae75669 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7831)
* cc430c949b539450110da96c0eccb52ab5e0998e Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7832)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9622] Add implementation of MergeHandle backed by the HoodieFileGroupReader [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13699:
URL: https://github.com/apache/hudi/pull/13699#discussion_r2280067070
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/FileGroupReaderBasedMergeHandle.java:
##########
@@ -156,40 +215,53 @@ private void init(CompactionOperation operation, String partitionPath) {
fileWriter = HoodieFileWriterFactory.getFileWriter(instantTime, newFilePath, hoodieTable.getStorage(),
config, writeSchemaWithMetaFields, taskContextSupplier, recordType);
} catch (IOException io) {
- LOG.error("Error in update task at commit {}", instantTime, io);
writeStatus.setGlobalError(io);
throw new HoodieUpsertException("Failed to initialize HoodieUpdateHandle for FileId: " + fileId + " on commit "
+ instantTime + " on path " + hoodieTable.getMetaClient().getBasePath(), io);
}
}
+ @Override
+ protected void populateIncomingRecordsMap(Iterator<HoodieRecord<T>> newRecordsItr) {
+ // no op.
+ }
+
+ /**
+ * This is only for spark, the engine context fetched from a serialized hoodie table is always local,
+ * overrides it to spark specific reader context.
+ */
+ public void setReaderContext(HoodieReaderContext<T> readerContext) {
+ this.readerContext = readerContext;
+ }
+
/**
* Reads the file slice of a compaction operation using a file group reader,
* by getting an iterator of the records; then writes the records to a new base file.
*/
@Override
public void doMerge() {
+ // For non-compaction operations, the merger needs to be initialized with the writer properties to handle cases like Merge-Into commands
+ if (operation.isEmpty()) {
+ this.readerContext.initRecordMergerForIngestion(config.getProps());
+ }
boolean usePosition = config.getBooleanOrDefault(MERGE_USE_RECORD_POSITIONS);
- Option<InternalSchema> internalSchemaOption = SerDeHelper.fromJson(config.getInternalSchema());
- TypedProperties props = TypedProperties.copy(config.getProps());
+ Option<InternalSchema> internalSchemaOption = SerDeHelper.fromJson(config.getInternalSchema())
+ .map(internalSchema -> AvroSchemaEvolutionUtils.reconcileSchema(writeSchemaWithMetaFields, internalSchema,
+ config.getBooleanOrDefault(HoodieCommonConfig.SET_NULL_FOR_MISSING_COLUMNS)));
long maxMemoryPerCompaction = IOUtils.getMaxMemoryPerCompaction(taskContextSupplier, config);
props.put(HoodieMemoryConfig.MAX_MEMORY_FOR_MERGE.key(), String.valueOf(maxMemoryPerCompaction));
- Stream<HoodieLogFile> logFiles = operation.getDeltaFileNames().stream().map(logFileName ->
+ Option<Stream<HoodieLogFile>> logFilesStreamOpt = operation.map(op -> op.getDeltaFileNames().stream().map(logFileName ->
new HoodieLogFile(new StoragePath(FSUtils.constructAbsolutePath(
- config.getBasePath(), operation.getPartitionPath()), logFileName)));
+ config.getBasePath(), op.getPartitionPath()), logFileName))));
// Initializes file group reader
- try (HoodieFileGroupReader<T> fileGroupReader = HoodieFileGroupReader.<T>newBuilder().withReaderContext(readerContext).withHoodieTableMetaClient(hoodieTable.getMetaClient())
- .withLatestCommitTime(maxInstantTime).withPartitionPath(partitionPath).withBaseFileOption(Option.ofNullable(baseFileToMerge)).withLogFiles(logFiles)
- .withDataSchema(writeSchemaWithMetaFields).withRequestedSchema(writeSchemaWithMetaFields).withInternalSchema(internalSchemaOption).withProps(props)
- .withShouldUseRecordPosition(usePosition).withSortOutput(hoodieTable.requireSortedRecords())
- .withFileGroupUpdateCallback(cdcLogger.map(logger -> new CDCCallback(logger, readerContext))).withEnableOptimizedLogBlockScan(config.enableOptimizedLogBlocksScan()).build()) {
+ try (HoodieFileGroupReader<T> fileGroupReader = getFileGroupReader(usePosition, internalSchemaOption, props, logFilesStreamOpt, incomingRecordsItr)) {
// Reads the records from the file slice
try (ClosableIterator<HoodieRecord<T>> recordIterator = fileGroupReader.getClosableHoodieRecordIterator()) {
while (recordIterator.hasNext()) {
HoodieRecord<T> record = recordIterator.next();
+ Option<Map<String, String>> recordMetadata = getRecordMetadata(record, writeSchema, props);
Review Comment:
we should also fix `HoodieWriteHandle` to keep the behavior alligned: skip the event time tracing when `preserveMetadata` is true.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13699:
URL: https://github.com/apache/hudi/pull/13699#issuecomment-3188622335
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7661",
"triggerID" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"triggerType" : "PUSH"
}, {
"hash" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7666",
"triggerID" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"triggerType" : "PUSH"
}, {
"hash" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7667",
"triggerID" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"triggerType" : "PUSH"
}, {
"hash" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7672",
"triggerID" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"triggerType" : "PUSH"
}, {
"hash" : "965136302f1188e5123de37734c927dc55801b24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7691",
"triggerID" : "965136302f1188e5123de37734c927dc55801b24",
"triggerType" : "PUSH"
}, {
"hash" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7693",
"triggerID" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"triggerType" : "PUSH"
}, {
"hash" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7717",
"triggerID" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"triggerType" : "PUSH"
}, {
"hash" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"triggerType" : "PUSH"
}, {
"hash" : "47f8303d6e61504275ecfddf553e356427e6c822",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7718",
"triggerID" : "47f8303d6e61504275ecfddf553e356427e6c822",
"triggerType" : "PUSH"
}, {
"hash" : "fb191ecf48d13cd81412535f7f8c9ed553d86517",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7725",
"triggerID" : "fb191ecf48d13cd81412535f7f8c9ed553d86517",
"triggerType" : "PUSH"
}, {
"hash" : "064912d3a54714935f2ee77d55cc66816166300f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7727",
"triggerID" : "064912d3a54714935f2ee77d55cc66816166300f",
"triggerType" : "PUSH"
}, {
"hash" : "77a2b45fe13dafc51f7bebb2ba859d40291ccc10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7736",
"triggerID" : "77a2b45fe13dafc51f7bebb2ba859d40291ccc10",
"triggerType" : "PUSH"
}, {
"hash" : "e4eda0e51fee0e8de5d57ea7b38dc642f53aa95d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7758",
"triggerID" : "e4eda0e51fee0e8de5d57ea7b38dc642f53aa95d",
"triggerType" : "PUSH"
}, {
"hash" : "6cc840c51fcecde95d2187fb6fd217b50fa82806",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6cc840c51fcecde95d2187fb6fd217b50fa82806",
"triggerType" : "PUSH"
}, {
"hash" : "31dab8ce9abbe4fc80e6cd81a0e433a039ca3e1d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7760",
"triggerID" : "31dab8ce9abbe4fc80e6cd81a0e433a039ca3e1d",
"triggerType" : "PUSH"
}, {
"hash" : "c04fd95dcc94146a9146c347f4f204656f3d8308",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7763",
"triggerID" : "c04fd95dcc94146a9146c347f4f204656f3d8308",
"triggerType" : "PUSH"
}, {
"hash" : "324147913d47ba8f2555e1991ce59905e5dc0b3f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7767",
"triggerID" : "324147913d47ba8f2555e1991ce59905e5dc0b3f",
"triggerType" : "PUSH"
}, {
"hash" : "3f177d42b831b30f4bd590b057dcab6ded78b634",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7774",
"triggerID" : "3f177d42b831b30f4bd590b057dcab6ded78b634",
"triggerType" : "PUSH"
}, {
"hash" : "8d6d2c06ec82d1072251ac12ddf21a890ddd5bda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7776",
"triggerID" : "8d6d2c06ec82d1072251ac12ddf21a890ddd5bda",
"triggerType" : "PUSH"
}, {
"hash" : "77589738d73b044385f5aa6f7ef87b421f06043b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "77589738d73b044385f5aa6f7ef87b421f06043b",
"triggerType" : "PUSH"
}, {
"hash" : "ad0d7ef0fbc525f880d3baadef5a22196ab97b32",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7781",
"triggerID" : "ad0d7ef0fbc525f880d3baadef5a22196ab97b32",
"triggerType" : "PUSH"
}, {
"hash" : "0db72caa3a6c7d0f0899f243ea4b8975de79e02c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7787",
"triggerID" : "0db72caa3a6c7d0f0899f243ea4b8975de79e02c",
"triggerType" : "PUSH"
}, {
"hash" : "a01ce748dc2d4cd1fd4858d1ecac8a486989e880",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7797",
"triggerID" : "a01ce748dc2d4cd1fd4858d1ecac8a486989e880",
"triggerType" : "PUSH"
}, {
"hash" : "3f35151e28bfa1dd50b31c6e5379d9769627afd9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7799",
"triggerID" : "3f35151e28bfa1dd50b31c6e5379d9769627afd9",
"triggerType" : "PUSH"
}, {
"hash" : "b0054c8fd329489d9a1e9535d9866edffc52fdd0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b0054c8fd329489d9a1e9535d9866edffc52fdd0",
"triggerType" : "PUSH"
}, {
"hash" : "363e54341420cc217bf5af62de1e09ff370f75d2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "363e54341420cc217bf5af62de1e09ff370f75d2",
"triggerType" : "PUSH"
}, {
"hash" : "a893da375eb9c65edabbd22965296eac9b3714f4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a893da375eb9c65edabbd22965296eac9b3714f4",
"triggerType" : "PUSH"
}, {
"hash" : "a7c2756e6a293cbba99bd4e821881e6b961c138f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7804",
"triggerID" : "a7c2756e6a293cbba99bd4e821881e6b961c138f",
"triggerType" : "PUSH"
}, {
"hash" : "4a140fffa35893a13f2e7ac8365d60d51a2d0870",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7813",
"triggerID" : "4a140fffa35893a13f2e7ac8365d60d51a2d0870",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 50022dcc45b0f511dd69706aeaf6464a51be49e2 UNKNOWN
* 6cc840c51fcecde95d2187fb6fd217b50fa82806 UNKNOWN
* 77589738d73b044385f5aa6f7ef87b421f06043b UNKNOWN
* b0054c8fd329489d9a1e9535d9866edffc52fdd0 UNKNOWN
* 363e54341420cc217bf5af62de1e09ff370f75d2 UNKNOWN
* a893da375eb9c65edabbd22965296eac9b3714f4 UNKNOWN
* a7c2756e6a293cbba99bd4e821881e6b961c138f Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7804)
* 4a140fffa35893a13f2e7ac8365d60d51a2d0870 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7813)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9622] Add implementation of MergeHandle backed by the HoodieFileGroupReader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13699:
URL: https://github.com/apache/hudi/pull/13699#issuecomment-3192364397
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7661",
"triggerID" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"triggerType" : "PUSH"
}, {
"hash" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7666",
"triggerID" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"triggerType" : "PUSH"
}, {
"hash" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7667",
"triggerID" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"triggerType" : "PUSH"
}, {
"hash" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7672",
"triggerID" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"triggerType" : "PUSH"
}, {
"hash" : "965136302f1188e5123de37734c927dc55801b24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7691",
"triggerID" : "965136302f1188e5123de37734c927dc55801b24",
"triggerType" : "PUSH"
}, {
"hash" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7693",
"triggerID" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"triggerType" : "PUSH"
}, {
"hash" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7717",
"triggerID" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"triggerType" : "PUSH"
}, {
"hash" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"triggerType" : "PUSH"
}, {
"hash" : "47f8303d6e61504275ecfddf553e356427e6c822",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7718",
"triggerID" : "47f8303d6e61504275ecfddf553e356427e6c822",
"triggerType" : "PUSH"
}, {
"hash" : "fb191ecf48d13cd81412535f7f8c9ed553d86517",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7725",
"triggerID" : "fb191ecf48d13cd81412535f7f8c9ed553d86517",
"triggerType" : "PUSH"
}, {
"hash" : "064912d3a54714935f2ee77d55cc66816166300f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7727",
"triggerID" : "064912d3a54714935f2ee77d55cc66816166300f",
"triggerType" : "PUSH"
}, {
"hash" : "77a2b45fe13dafc51f7bebb2ba859d40291ccc10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7736",
"triggerID" : "77a2b45fe13dafc51f7bebb2ba859d40291ccc10",
"triggerType" : "PUSH"
}, {
"hash" : "e4eda0e51fee0e8de5d57ea7b38dc642f53aa95d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7758",
"triggerID" : "e4eda0e51fee0e8de5d57ea7b38dc642f53aa95d",
"triggerType" : "PUSH"
}, {
"hash" : "6cc840c51fcecde95d2187fb6fd217b50fa82806",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6cc840c51fcecde95d2187fb6fd217b50fa82806",
"triggerType" : "PUSH"
}, {
"hash" : "31dab8ce9abbe4fc80e6cd81a0e433a039ca3e1d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7760",
"triggerID" : "31dab8ce9abbe4fc80e6cd81a0e433a039ca3e1d",
"triggerType" : "PUSH"
}, {
"hash" : "c04fd95dcc94146a9146c347f4f204656f3d8308",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7763",
"triggerID" : "c04fd95dcc94146a9146c347f4f204656f3d8308",
"triggerType" : "PUSH"
}, {
"hash" : "324147913d47ba8f2555e1991ce59905e5dc0b3f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7767",
"triggerID" : "324147913d47ba8f2555e1991ce59905e5dc0b3f",
"triggerType" : "PUSH"
}, {
"hash" : "3f177d42b831b30f4bd590b057dcab6ded78b634",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7774",
"triggerID" : "3f177d42b831b30f4bd590b057dcab6ded78b634",
"triggerType" : "PUSH"
}, {
"hash" : "8d6d2c06ec82d1072251ac12ddf21a890ddd5bda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7776",
"triggerID" : "8d6d2c06ec82d1072251ac12ddf21a890ddd5bda",
"triggerType" : "PUSH"
}, {
"hash" : "77589738d73b044385f5aa6f7ef87b421f06043b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "77589738d73b044385f5aa6f7ef87b421f06043b",
"triggerType" : "PUSH"
}, {
"hash" : "ad0d7ef0fbc525f880d3baadef5a22196ab97b32",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7781",
"triggerID" : "ad0d7ef0fbc525f880d3baadef5a22196ab97b32",
"triggerType" : "PUSH"
}, {
"hash" : "0db72caa3a6c7d0f0899f243ea4b8975de79e02c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7787",
"triggerID" : "0db72caa3a6c7d0f0899f243ea4b8975de79e02c",
"triggerType" : "PUSH"
}, {
"hash" : "a01ce748dc2d4cd1fd4858d1ecac8a486989e880",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7797",
"triggerID" : "a01ce748dc2d4cd1fd4858d1ecac8a486989e880",
"triggerType" : "PUSH"
}, {
"hash" : "3f35151e28bfa1dd50b31c6e5379d9769627afd9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7799",
"triggerID" : "3f35151e28bfa1dd50b31c6e5379d9769627afd9",
"triggerType" : "PUSH"
}, {
"hash" : "b0054c8fd329489d9a1e9535d9866edffc52fdd0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b0054c8fd329489d9a1e9535d9866edffc52fdd0",
"triggerType" : "PUSH"
}, {
"hash" : "363e54341420cc217bf5af62de1e09ff370f75d2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "363e54341420cc217bf5af62de1e09ff370f75d2",
"triggerType" : "PUSH"
}, {
"hash" : "a893da375eb9c65edabbd22965296eac9b3714f4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a893da375eb9c65edabbd22965296eac9b3714f4",
"triggerType" : "PUSH"
}, {
"hash" : "a7c2756e6a293cbba99bd4e821881e6b961c138f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7804",
"triggerID" : "a7c2756e6a293cbba99bd4e821881e6b961c138f",
"triggerType" : "PUSH"
}, {
"hash" : "4a140fffa35893a13f2e7ac8365d60d51a2d0870",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7813",
"triggerID" : "4a140fffa35893a13f2e7ac8365d60d51a2d0870",
"triggerType" : "PUSH"
}, {
"hash" : "500b4f529ad2f8a695825b5badc1815202164d0a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7830",
"triggerID" : "500b4f529ad2f8a695825b5badc1815202164d0a",
"triggerType" : "PUSH"
}, {
"hash" : "539cf8b9cdc7ab4e70488e4096684022a027f942",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "539cf8b9cdc7ab4e70488e4096684022a027f942",
"triggerType" : "PUSH"
}, {
"hash" : "85139a12b431931973c07d1b7f9256a1aae75669",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7831",
"triggerID" : "85139a12b431931973c07d1b7f9256a1aae75669",
"triggerType" : "PUSH"
}, {
"hash" : "cc430c949b539450110da96c0eccb52ab5e0998e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7832",
"triggerID" : "cc430c949b539450110da96c0eccb52ab5e0998e",
"triggerType" : "PUSH"
}, {
"hash" : "2a9cad38ac9c86ddacbb2de06950d515b3a1b0df",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7850",
"triggerID" : "2a9cad38ac9c86ddacbb2de06950d515b3a1b0df",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 50022dcc45b0f511dd69706aeaf6464a51be49e2 UNKNOWN
* 6cc840c51fcecde95d2187fb6fd217b50fa82806 UNKNOWN
* 77589738d73b044385f5aa6f7ef87b421f06043b UNKNOWN
* b0054c8fd329489d9a1e9535d9866edffc52fdd0 UNKNOWN
* 363e54341420cc217bf5af62de1e09ff370f75d2 UNKNOWN
* a893da375eb9c65edabbd22965296eac9b3714f4 UNKNOWN
* 539cf8b9cdc7ab4e70488e4096684022a027f942 UNKNOWN
* 2a9cad38ac9c86ddacbb2de06950d515b3a1b0df Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7850)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9622] Add implementation of MergeHandle backed by the HoodieFileGroupReader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13699:
URL: https://github.com/apache/hudi/pull/13699#issuecomment-3192430012
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7661",
"triggerID" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"triggerType" : "PUSH"
}, {
"hash" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7666",
"triggerID" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"triggerType" : "PUSH"
}, {
"hash" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7667",
"triggerID" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"triggerType" : "PUSH"
}, {
"hash" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7672",
"triggerID" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"triggerType" : "PUSH"
}, {
"hash" : "965136302f1188e5123de37734c927dc55801b24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7691",
"triggerID" : "965136302f1188e5123de37734c927dc55801b24",
"triggerType" : "PUSH"
}, {
"hash" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7693",
"triggerID" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"triggerType" : "PUSH"
}, {
"hash" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7717",
"triggerID" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"triggerType" : "PUSH"
}, {
"hash" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"triggerType" : "PUSH"
}, {
"hash" : "47f8303d6e61504275ecfddf553e356427e6c822",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7718",
"triggerID" : "47f8303d6e61504275ecfddf553e356427e6c822",
"triggerType" : "PUSH"
}, {
"hash" : "fb191ecf48d13cd81412535f7f8c9ed553d86517",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7725",
"triggerID" : "fb191ecf48d13cd81412535f7f8c9ed553d86517",
"triggerType" : "PUSH"
}, {
"hash" : "064912d3a54714935f2ee77d55cc66816166300f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7727",
"triggerID" : "064912d3a54714935f2ee77d55cc66816166300f",
"triggerType" : "PUSH"
}, {
"hash" : "77a2b45fe13dafc51f7bebb2ba859d40291ccc10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7736",
"triggerID" : "77a2b45fe13dafc51f7bebb2ba859d40291ccc10",
"triggerType" : "PUSH"
}, {
"hash" : "e4eda0e51fee0e8de5d57ea7b38dc642f53aa95d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7758",
"triggerID" : "e4eda0e51fee0e8de5d57ea7b38dc642f53aa95d",
"triggerType" : "PUSH"
}, {
"hash" : "6cc840c51fcecde95d2187fb6fd217b50fa82806",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6cc840c51fcecde95d2187fb6fd217b50fa82806",
"triggerType" : "PUSH"
}, {
"hash" : "31dab8ce9abbe4fc80e6cd81a0e433a039ca3e1d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7760",
"triggerID" : "31dab8ce9abbe4fc80e6cd81a0e433a039ca3e1d",
"triggerType" : "PUSH"
}, {
"hash" : "c04fd95dcc94146a9146c347f4f204656f3d8308",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7763",
"triggerID" : "c04fd95dcc94146a9146c347f4f204656f3d8308",
"triggerType" : "PUSH"
}, {
"hash" : "324147913d47ba8f2555e1991ce59905e5dc0b3f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7767",
"triggerID" : "324147913d47ba8f2555e1991ce59905e5dc0b3f",
"triggerType" : "PUSH"
}, {
"hash" : "3f177d42b831b30f4bd590b057dcab6ded78b634",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7774",
"triggerID" : "3f177d42b831b30f4bd590b057dcab6ded78b634",
"triggerType" : "PUSH"
}, {
"hash" : "8d6d2c06ec82d1072251ac12ddf21a890ddd5bda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7776",
"triggerID" : "8d6d2c06ec82d1072251ac12ddf21a890ddd5bda",
"triggerType" : "PUSH"
}, {
"hash" : "77589738d73b044385f5aa6f7ef87b421f06043b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "77589738d73b044385f5aa6f7ef87b421f06043b",
"triggerType" : "PUSH"
}, {
"hash" : "ad0d7ef0fbc525f880d3baadef5a22196ab97b32",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7781",
"triggerID" : "ad0d7ef0fbc525f880d3baadef5a22196ab97b32",
"triggerType" : "PUSH"
}, {
"hash" : "0db72caa3a6c7d0f0899f243ea4b8975de79e02c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7787",
"triggerID" : "0db72caa3a6c7d0f0899f243ea4b8975de79e02c",
"triggerType" : "PUSH"
}, {
"hash" : "a01ce748dc2d4cd1fd4858d1ecac8a486989e880",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7797",
"triggerID" : "a01ce748dc2d4cd1fd4858d1ecac8a486989e880",
"triggerType" : "PUSH"
}, {
"hash" : "3f35151e28bfa1dd50b31c6e5379d9769627afd9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7799",
"triggerID" : "3f35151e28bfa1dd50b31c6e5379d9769627afd9",
"triggerType" : "PUSH"
}, {
"hash" : "b0054c8fd329489d9a1e9535d9866edffc52fdd0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b0054c8fd329489d9a1e9535d9866edffc52fdd0",
"triggerType" : "PUSH"
}, {
"hash" : "363e54341420cc217bf5af62de1e09ff370f75d2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "363e54341420cc217bf5af62de1e09ff370f75d2",
"triggerType" : "PUSH"
}, {
"hash" : "a893da375eb9c65edabbd22965296eac9b3714f4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a893da375eb9c65edabbd22965296eac9b3714f4",
"triggerType" : "PUSH"
}, {
"hash" : "a7c2756e6a293cbba99bd4e821881e6b961c138f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7804",
"triggerID" : "a7c2756e6a293cbba99bd4e821881e6b961c138f",
"triggerType" : "PUSH"
}, {
"hash" : "4a140fffa35893a13f2e7ac8365d60d51a2d0870",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7813",
"triggerID" : "4a140fffa35893a13f2e7ac8365d60d51a2d0870",
"triggerType" : "PUSH"
}, {
"hash" : "500b4f529ad2f8a695825b5badc1815202164d0a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7830",
"triggerID" : "500b4f529ad2f8a695825b5badc1815202164d0a",
"triggerType" : "PUSH"
}, {
"hash" : "539cf8b9cdc7ab4e70488e4096684022a027f942",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "539cf8b9cdc7ab4e70488e4096684022a027f942",
"triggerType" : "PUSH"
}, {
"hash" : "85139a12b431931973c07d1b7f9256a1aae75669",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7831",
"triggerID" : "85139a12b431931973c07d1b7f9256a1aae75669",
"triggerType" : "PUSH"
}, {
"hash" : "cc430c949b539450110da96c0eccb52ab5e0998e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7832",
"triggerID" : "cc430c949b539450110da96c0eccb52ab5e0998e",
"triggerType" : "PUSH"
}, {
"hash" : "2a9cad38ac9c86ddacbb2de06950d515b3a1b0df",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7850",
"triggerID" : "2a9cad38ac9c86ddacbb2de06950d515b3a1b0df",
"triggerType" : "PUSH"
}, {
"hash" : "64d3575067a762fabfede759147da73ef3df0d1a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7852",
"triggerID" : "64d3575067a762fabfede759147da73ef3df0d1a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 50022dcc45b0f511dd69706aeaf6464a51be49e2 UNKNOWN
* 6cc840c51fcecde95d2187fb6fd217b50fa82806 UNKNOWN
* 77589738d73b044385f5aa6f7ef87b421f06043b UNKNOWN
* b0054c8fd329489d9a1e9535d9866edffc52fdd0 UNKNOWN
* 363e54341420cc217bf5af62de1e09ff370f75d2 UNKNOWN
* a893da375eb9c65edabbd22965296eac9b3714f4 UNKNOWN
* 539cf8b9cdc7ab4e70488e4096684022a027f942 UNKNOWN
* 2a9cad38ac9c86ddacbb2de06950d515b3a1b0df Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7850)
* 64d3575067a762fabfede759147da73ef3df0d1a Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7852)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13699:
URL: https://github.com/apache/hudi/pull/13699#issuecomment-3177103486
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7661",
"triggerID" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"triggerType" : "PUSH"
}, {
"hash" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7666",
"triggerID" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"triggerType" : "PUSH"
}, {
"hash" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7667",
"triggerID" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"triggerType" : "PUSH"
}, {
"hash" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7672",
"triggerID" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"triggerType" : "PUSH"
}, {
"hash" : "965136302f1188e5123de37734c927dc55801b24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7691",
"triggerID" : "965136302f1188e5123de37734c927dc55801b24",
"triggerType" : "PUSH"
}, {
"hash" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7693",
"triggerID" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"triggerType" : "PUSH"
}, {
"hash" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7717",
"triggerID" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"triggerType" : "PUSH"
}, {
"hash" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"triggerType" : "PUSH"
}, {
"hash" : "47f8303d6e61504275ecfddf553e356427e6c822",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7718",
"triggerID" : "47f8303d6e61504275ecfddf553e356427e6c822",
"triggerType" : "PUSH"
}, {
"hash" : "fb191ecf48d13cd81412535f7f8c9ed553d86517",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7725",
"triggerID" : "fb191ecf48d13cd81412535f7f8c9ed553d86517",
"triggerType" : "PUSH"
}, {
"hash" : "064912d3a54714935f2ee77d55cc66816166300f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7727",
"triggerID" : "064912d3a54714935f2ee77d55cc66816166300f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 50022dcc45b0f511dd69706aeaf6464a51be49e2 UNKNOWN
* 064912d3a54714935f2ee77d55cc66816166300f Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7727)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13699:
URL: https://github.com/apache/hudi/pull/13699#issuecomment-3177475699
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7661",
"triggerID" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"triggerType" : "PUSH"
}, {
"hash" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7666",
"triggerID" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"triggerType" : "PUSH"
}, {
"hash" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7667",
"triggerID" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"triggerType" : "PUSH"
}, {
"hash" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7672",
"triggerID" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"triggerType" : "PUSH"
}, {
"hash" : "965136302f1188e5123de37734c927dc55801b24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7691",
"triggerID" : "965136302f1188e5123de37734c927dc55801b24",
"triggerType" : "PUSH"
}, {
"hash" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7693",
"triggerID" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"triggerType" : "PUSH"
}, {
"hash" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7717",
"triggerID" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"triggerType" : "PUSH"
}, {
"hash" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"triggerType" : "PUSH"
}, {
"hash" : "47f8303d6e61504275ecfddf553e356427e6c822",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7718",
"triggerID" : "47f8303d6e61504275ecfddf553e356427e6c822",
"triggerType" : "PUSH"
}, {
"hash" : "fb191ecf48d13cd81412535f7f8c9ed553d86517",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7725",
"triggerID" : "fb191ecf48d13cd81412535f7f8c9ed553d86517",
"triggerType" : "PUSH"
}, {
"hash" : "064912d3a54714935f2ee77d55cc66816166300f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7727",
"triggerID" : "064912d3a54714935f2ee77d55cc66816166300f",
"triggerType" : "PUSH"
}, {
"hash" : "77a2b45fe13dafc51f7bebb2ba859d40291ccc10",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7736",
"triggerID" : "77a2b45fe13dafc51f7bebb2ba859d40291ccc10",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 50022dcc45b0f511dd69706aeaf6464a51be49e2 UNKNOWN
* 064912d3a54714935f2ee77d55cc66816166300f Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7727)
* 77a2b45fe13dafc51f7bebb2ba859d40291ccc10 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7736)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13699:
URL: https://github.com/apache/hudi/pull/13699#issuecomment-3184712502
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7661",
"triggerID" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"triggerType" : "PUSH"
}, {
"hash" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7666",
"triggerID" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"triggerType" : "PUSH"
}, {
"hash" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7667",
"triggerID" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"triggerType" : "PUSH"
}, {
"hash" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7672",
"triggerID" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"triggerType" : "PUSH"
}, {
"hash" : "965136302f1188e5123de37734c927dc55801b24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7691",
"triggerID" : "965136302f1188e5123de37734c927dc55801b24",
"triggerType" : "PUSH"
}, {
"hash" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7693",
"triggerID" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"triggerType" : "PUSH"
}, {
"hash" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7717",
"triggerID" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"triggerType" : "PUSH"
}, {
"hash" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"triggerType" : "PUSH"
}, {
"hash" : "47f8303d6e61504275ecfddf553e356427e6c822",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7718",
"triggerID" : "47f8303d6e61504275ecfddf553e356427e6c822",
"triggerType" : "PUSH"
}, {
"hash" : "fb191ecf48d13cd81412535f7f8c9ed553d86517",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7725",
"triggerID" : "fb191ecf48d13cd81412535f7f8c9ed553d86517",
"triggerType" : "PUSH"
}, {
"hash" : "064912d3a54714935f2ee77d55cc66816166300f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7727",
"triggerID" : "064912d3a54714935f2ee77d55cc66816166300f",
"triggerType" : "PUSH"
}, {
"hash" : "77a2b45fe13dafc51f7bebb2ba859d40291ccc10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7736",
"triggerID" : "77a2b45fe13dafc51f7bebb2ba859d40291ccc10",
"triggerType" : "PUSH"
}, {
"hash" : "e4eda0e51fee0e8de5d57ea7b38dc642f53aa95d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7758",
"triggerID" : "e4eda0e51fee0e8de5d57ea7b38dc642f53aa95d",
"triggerType" : "PUSH"
}, {
"hash" : "6cc840c51fcecde95d2187fb6fd217b50fa82806",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6cc840c51fcecde95d2187fb6fd217b50fa82806",
"triggerType" : "PUSH"
}, {
"hash" : "31dab8ce9abbe4fc80e6cd81a0e433a039ca3e1d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7760",
"triggerID" : "31dab8ce9abbe4fc80e6cd81a0e433a039ca3e1d",
"triggerType" : "PUSH"
}, {
"hash" : "c04fd95dcc94146a9146c347f4f204656f3d8308",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7763",
"triggerID" : "c04fd95dcc94146a9146c347f4f204656f3d8308",
"triggerType" : "PUSH"
}, {
"hash" : "324147913d47ba8f2555e1991ce59905e5dc0b3f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7767",
"triggerID" : "324147913d47ba8f2555e1991ce59905e5dc0b3f",
"triggerType" : "PUSH"
}, {
"hash" : "3f177d42b831b30f4bd590b057dcab6ded78b634",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7774",
"triggerID" : "3f177d42b831b30f4bd590b057dcab6ded78b634",
"triggerType" : "PUSH"
}, {
"hash" : "8d6d2c06ec82d1072251ac12ddf21a890ddd5bda",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7776",
"triggerID" : "8d6d2c06ec82d1072251ac12ddf21a890ddd5bda",
"triggerType" : "PUSH"
}, {
"hash" : "77589738d73b044385f5aa6f7ef87b421f06043b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "77589738d73b044385f5aa6f7ef87b421f06043b",
"triggerType" : "PUSH"
}, {
"hash" : "ad0d7ef0fbc525f880d3baadef5a22196ab97b32",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7781",
"triggerID" : "ad0d7ef0fbc525f880d3baadef5a22196ab97b32",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 50022dcc45b0f511dd69706aeaf6464a51be49e2 UNKNOWN
* 6cc840c51fcecde95d2187fb6fd217b50fa82806 UNKNOWN
* 8d6d2c06ec82d1072251ac12ddf21a890ddd5bda Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7776)
* 77589738d73b044385f5aa6f7ef87b421f06043b UNKNOWN
* ad0d7ef0fbc525f880d3baadef5a22196ab97b32 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7781)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13699:
URL: https://github.com/apache/hudi/pull/13699#discussion_r2272280766
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/buffer/FileGroupRecordBuffer.java:
##########
@@ -56,27 +53,29 @@
import java.io.IOException;
import java.io.Serializable;
+import java.util.HashSet;
import java.util.Iterator;
import java.util.List;
import java.util.Locale;
import java.util.Map;
+import java.util.Set;
import java.util.function.Function;
import static org.apache.hudi.common.config.HoodieCommonConfig.DISK_MAP_BITCASK_COMPRESSION_ENABLED;
import static org.apache.hudi.common.config.HoodieCommonConfig.SPILLABLE_DISK_MAP_TYPE;
import static org.apache.hudi.common.config.HoodieMemoryConfig.MAX_MEMORY_FOR_MERGE;
import static org.apache.hudi.common.config.HoodieMemoryConfig.SPILLABLE_MAP_BASE_PATH;
-import static org.apache.hudi.common.model.HoodieRecordMerger.PAYLOAD_BASED_MERGE_STRATEGY_UUID;
import static org.apache.hudi.common.table.log.block.HoodieLogBlock.HeaderMetadataType.INSTANT_TIME;
abstract class FileGroupRecordBuffer<T> implements HoodieFileGroupRecordBuffer<T> {
+ protected final Set<String> usedKeys = new HashSet<>();
Review Comment:
Fired a discussion here: https://github.com/apache/hudi/pull/6824#issuecomment-3182443235
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13699:
URL: https://github.com/apache/hudi/pull/13699#discussion_r2271628282
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/buffer/StreamingFileGroupRecordBufferLoader.java:
##########
@@ -51,19 +53,20 @@
* @param <T> Engine native presentation of the record.
*/
public class StreamingFileGroupRecordBufferLoader<T> implements FileGroupRecordBufferLoader<T> {
- private static final StreamingFileGroupRecordBufferLoader INSTANCE = new StreamingFileGroupRecordBufferLoader<>();
+ private final Schema recordSchema;
- static <T> StreamingFileGroupRecordBufferLoader<T> getInstance() {
- return INSTANCE;
+ StreamingFileGroupRecordBufferLoader(Schema recordSchema) {
+ this.recordSchema = recordSchema;
Review Comment:
Nice, this will simplify the changeset
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13699:
URL: https://github.com/apache/hudi/pull/13699#discussion_r2271943944
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/UpdateProcessor.java:
##########
@@ -80,16 +106,79 @@ public BufferedRecord<T> processUpdate(String recordKey, BufferedRecord<T> previ
}
return null;
} else {
- T prevRow = previousRecord != null ? previousRecord.getRecord() : null;
- T mergedRow = mergedRecord.getRecord();
- if (prevRow != null && prevRow != mergedRow) {
- mergedRecord.setHoodieOperation(HoodieOperation.UPDATE_AFTER);
- readStats.incrementNumUpdates();
- } else if (prevRow == null) {
- mergedRecord.setHoodieOperation(HoodieOperation.INSERT);
- readStats.incrementNumInserts();
+ return handleNonDeletes(previousRecord, mergedRecord);
+ }
+ }
+
+ protected BufferedRecord<T> handleNonDeletes(BufferedRecord<T> previousRecord, BufferedRecord<T> mergedRecord) {
+ T prevRow = previousRecord != null ? previousRecord.getRecord() : null;
+ T mergedRow = mergedRecord.getRecord();
+ if (prevRow != null && prevRow != mergedRow) {
+ mergedRecord.setHoodieOperation(HoodieOperation.UPDATE_AFTER);
+ readStats.incrementNumUpdates();
+ } else if (prevRow == null) {
+ mergedRecord.setHoodieOperation(HoodieOperation.INSERT);
+ readStats.incrementNumInserts();
+ }
+ return mergedRecord.seal(readerContext);
+ }
+ }
+
+ class PayloadUpdateProcessor<T> extends StandardUpdateProcessor<T> {
+ private final String payloadClass;
+ private final Properties properties;
+
+ public PayloadUpdateProcessor(HoodieReadStats readStats, HoodieReaderContext<T> readerContext, boolean emitDeletes,
+ Properties properties, String payloadClass) {
+ super(readStats, readerContext, emitDeletes);
+ this.payloadClass = payloadClass;
+ this.properties = properties;
+ }
+
+ @Override
+ protected BufferedRecord<T> handleNonDeletes(BufferedRecord<T> previousRecord, BufferedRecord<T> mergedRecord) {
+ if (previousRecord == null) {
+ // special case for payloads when there is no previous record
+ Schema recordSchema = readerContext.getRecordContext().decodeAvroSchema(mergedRecord.getSchemaId());
+ GenericRecord record = readerContext.getRecordContext().convertToAvroRecord(mergedRecord.getRecord(), recordSchema);
+ HoodieAvroRecord hoodieRecord = new HoodieAvroRecord<>(null, HoodieRecordUtils.loadPayload(payloadClass, record, mergedRecord.getOrderingValue()));
+ try {
+ if (hoodieRecord.shouldIgnore(recordSchema, properties)) {
+ return null;
+ } else {
+ Schema readerSchema = readerContext.getSchemaHandler().getRequestedSchema();
+ // If the record schema is different from the reader schema, rewrite the record using the payload methods to ensure consistency with legacy writer paths
+ hoodieRecord.rewriteRecordWithNewSchema(recordSchema, properties, readerSchema).toIndexedRecord(readerSchema, properties)
+ .ifPresent(rewrittenRecord -> mergedRecord.replaceRecord(readerContext.getRecordContext().convertAvroRecord(rewrittenRecord.getData())));
+ }
+ } catch (IOException e) {
+ throw new HoodieIOException("Error processing record with payload class: " + payloadClass, e);
+ }
+ }
+ return super.handleNonDeletes(previousRecord, mergedRecord);
+ }
+ }
+
+ class CustomMergerUpdateProcessor<T> extends StandardUpdateProcessor<T> {
+ private final HoodieRecordMerger merger;
+ private final TypedProperties properties;
+
+ CustomMergerUpdateProcessor(HoodieReadStats readStats, HoodieReaderContext<T> readerContext, boolean emitDeletes,
+ TypedProperties properties) {
+ super(readStats, readerContext, emitDeletes);
+ this.merger = readerContext.getRecordMerger().get();
+ this.properties = properties;
+ }
+
+ @Override
+ protected BufferedRecord<T> handleNonDeletes(BufferedRecord<T> previousRecord, BufferedRecord<T> mergedRecord) {
+ try {
+ if (merger.shouldFlush(readerContext.getRecordContext().constructHoodieRecord(mergedRecord), readerContext.getRecordContext().getSchemaFromBufferRecord(mergedRecord), properties)) {
Review Comment:
we have discussed to drop the `shouldFlush` functionality as of now, it is not in good design and the fix in just log and base merging is not complete, let's just drop this first and fire a JIRA to trace it first.
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/UpdateProcessor.java:
##########
@@ -80,16 +106,79 @@ public BufferedRecord<T> processUpdate(String recordKey, BufferedRecord<T> previ
}
return null;
} else {
- T prevRow = previousRecord != null ? previousRecord.getRecord() : null;
- T mergedRow = mergedRecord.getRecord();
- if (prevRow != null && prevRow != mergedRow) {
- mergedRecord.setHoodieOperation(HoodieOperation.UPDATE_AFTER);
- readStats.incrementNumUpdates();
- } else if (prevRow == null) {
- mergedRecord.setHoodieOperation(HoodieOperation.INSERT);
- readStats.incrementNumInserts();
+ return handleNonDeletes(previousRecord, mergedRecord);
+ }
+ }
+
+ protected BufferedRecord<T> handleNonDeletes(BufferedRecord<T> previousRecord, BufferedRecord<T> mergedRecord) {
+ T prevRow = previousRecord != null ? previousRecord.getRecord() : null;
+ T mergedRow = mergedRecord.getRecord();
+ if (prevRow != null && prevRow != mergedRow) {
+ mergedRecord.setHoodieOperation(HoodieOperation.UPDATE_AFTER);
+ readStats.incrementNumUpdates();
+ } else if (prevRow == null) {
+ mergedRecord.setHoodieOperation(HoodieOperation.INSERT);
+ readStats.incrementNumInserts();
+ }
+ return mergedRecord.seal(readerContext);
+ }
+ }
+
+ class PayloadUpdateProcessor<T> extends StandardUpdateProcessor<T> {
+ private final String payloadClass;
+ private final Properties properties;
+
+ public PayloadUpdateProcessor(HoodieReadStats readStats, HoodieReaderContext<T> readerContext, boolean emitDeletes,
+ Properties properties, String payloadClass) {
+ super(readStats, readerContext, emitDeletes);
+ this.payloadClass = payloadClass;
+ this.properties = properties;
+ }
+
+ @Override
+ protected BufferedRecord<T> handleNonDeletes(BufferedRecord<T> previousRecord, BufferedRecord<T> mergedRecord) {
+ if (previousRecord == null) {
+ // special case for payloads when there is no previous record
+ Schema recordSchema = readerContext.getRecordContext().decodeAvroSchema(mergedRecord.getSchemaId());
+ GenericRecord record = readerContext.getRecordContext().convertToAvroRecord(mergedRecord.getRecord(), recordSchema);
+ HoodieAvroRecord hoodieRecord = new HoodieAvroRecord<>(null, HoodieRecordUtils.loadPayload(payloadClass, record, mergedRecord.getOrderingValue()));
+ try {
+ if (hoodieRecord.shouldIgnore(recordSchema, properties)) {
+ return null;
+ } else {
+ Schema readerSchema = readerContext.getSchemaHandler().getRequestedSchema();
+ // If the record schema is different from the reader schema, rewrite the record using the payload methods to ensure consistency with legacy writer paths
+ hoodieRecord.rewriteRecordWithNewSchema(recordSchema, properties, readerSchema).toIndexedRecord(readerSchema, properties)
+ .ifPresent(rewrittenRecord -> mergedRecord.replaceRecord(readerContext.getRecordContext().convertAvroRecord(rewrittenRecord.getData())));
+ }
+ } catch (IOException e) {
+ throw new HoodieIOException("Error processing record with payload class: " + payloadClass, e);
+ }
+ }
+ return super.handleNonDeletes(previousRecord, mergedRecord);
+ }
+ }
+
+ class CustomMergerUpdateProcessor<T> extends StandardUpdateProcessor<T> {
+ private final HoodieRecordMerger merger;
+ private final TypedProperties properties;
+
+ CustomMergerUpdateProcessor(HoodieReadStats readStats, HoodieReaderContext<T> readerContext, boolean emitDeletes,
+ TypedProperties properties) {
+ super(readStats, readerContext, emitDeletes);
+ this.merger = readerContext.getRecordMerger().get();
+ this.properties = properties;
+ }
+
+ @Override
+ protected BufferedRecord<T> handleNonDeletes(BufferedRecord<T> previousRecord, BufferedRecord<T> mergedRecord) {
+ try {
+ if (merger.shouldFlush(readerContext.getRecordContext().constructHoodieRecord(mergedRecord), readerContext.getRecordContext().getSchemaFromBufferRecord(mergedRecord), properties)) {
Review Comment:
we have discussed to drop the `shouldFlush` functionality as of now, it is not in good design and the fix in just log and base merging is not complete, let's just drop this first and fire a JIRA to trace it instead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13699:
URL: https://github.com/apache/hudi/pull/13699#issuecomment-3186092657
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7661",
"triggerID" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"triggerType" : "PUSH"
}, {
"hash" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7666",
"triggerID" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"triggerType" : "PUSH"
}, {
"hash" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7667",
"triggerID" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"triggerType" : "PUSH"
}, {
"hash" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7672",
"triggerID" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"triggerType" : "PUSH"
}, {
"hash" : "965136302f1188e5123de37734c927dc55801b24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7691",
"triggerID" : "965136302f1188e5123de37734c927dc55801b24",
"triggerType" : "PUSH"
}, {
"hash" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7693",
"triggerID" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"triggerType" : "PUSH"
}, {
"hash" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7717",
"triggerID" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"triggerType" : "PUSH"
}, {
"hash" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"triggerType" : "PUSH"
}, {
"hash" : "47f8303d6e61504275ecfddf553e356427e6c822",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7718",
"triggerID" : "47f8303d6e61504275ecfddf553e356427e6c822",
"triggerType" : "PUSH"
}, {
"hash" : "fb191ecf48d13cd81412535f7f8c9ed553d86517",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7725",
"triggerID" : "fb191ecf48d13cd81412535f7f8c9ed553d86517",
"triggerType" : "PUSH"
}, {
"hash" : "064912d3a54714935f2ee77d55cc66816166300f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7727",
"triggerID" : "064912d3a54714935f2ee77d55cc66816166300f",
"triggerType" : "PUSH"
}, {
"hash" : "77a2b45fe13dafc51f7bebb2ba859d40291ccc10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7736",
"triggerID" : "77a2b45fe13dafc51f7bebb2ba859d40291ccc10",
"triggerType" : "PUSH"
}, {
"hash" : "e4eda0e51fee0e8de5d57ea7b38dc642f53aa95d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7758",
"triggerID" : "e4eda0e51fee0e8de5d57ea7b38dc642f53aa95d",
"triggerType" : "PUSH"
}, {
"hash" : "6cc840c51fcecde95d2187fb6fd217b50fa82806",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6cc840c51fcecde95d2187fb6fd217b50fa82806",
"triggerType" : "PUSH"
}, {
"hash" : "31dab8ce9abbe4fc80e6cd81a0e433a039ca3e1d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7760",
"triggerID" : "31dab8ce9abbe4fc80e6cd81a0e433a039ca3e1d",
"triggerType" : "PUSH"
}, {
"hash" : "c04fd95dcc94146a9146c347f4f204656f3d8308",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7763",
"triggerID" : "c04fd95dcc94146a9146c347f4f204656f3d8308",
"triggerType" : "PUSH"
}, {
"hash" : "324147913d47ba8f2555e1991ce59905e5dc0b3f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7767",
"triggerID" : "324147913d47ba8f2555e1991ce59905e5dc0b3f",
"triggerType" : "PUSH"
}, {
"hash" : "3f177d42b831b30f4bd590b057dcab6ded78b634",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7774",
"triggerID" : "3f177d42b831b30f4bd590b057dcab6ded78b634",
"triggerType" : "PUSH"
}, {
"hash" : "8d6d2c06ec82d1072251ac12ddf21a890ddd5bda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7776",
"triggerID" : "8d6d2c06ec82d1072251ac12ddf21a890ddd5bda",
"triggerType" : "PUSH"
}, {
"hash" : "77589738d73b044385f5aa6f7ef87b421f06043b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "77589738d73b044385f5aa6f7ef87b421f06043b",
"triggerType" : "PUSH"
}, {
"hash" : "ad0d7ef0fbc525f880d3baadef5a22196ab97b32",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7781",
"triggerID" : "ad0d7ef0fbc525f880d3baadef5a22196ab97b32",
"triggerType" : "PUSH"
}, {
"hash" : "0db72caa3a6c7d0f0899f243ea4b8975de79e02c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7787",
"triggerID" : "0db72caa3a6c7d0f0899f243ea4b8975de79e02c",
"triggerType" : "PUSH"
}, {
"hash" : "a01ce748dc2d4cd1fd4858d1ecac8a486989e880",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7797",
"triggerID" : "a01ce748dc2d4cd1fd4858d1ecac8a486989e880",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 50022dcc45b0f511dd69706aeaf6464a51be49e2 UNKNOWN
* 6cc840c51fcecde95d2187fb6fd217b50fa82806 UNKNOWN
* 77589738d73b044385f5aa6f7ef87b421f06043b UNKNOWN
* 0db72caa3a6c7d0f0899f243ea4b8975de79e02c Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7787)
* a01ce748dc2d4cd1fd4858d1ecac8a486989e880 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7797)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13699:
URL: https://github.com/apache/hudi/pull/13699#issuecomment-3186534123
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7661",
"triggerID" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"triggerType" : "PUSH"
}, {
"hash" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7666",
"triggerID" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"triggerType" : "PUSH"
}, {
"hash" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7667",
"triggerID" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"triggerType" : "PUSH"
}, {
"hash" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7672",
"triggerID" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"triggerType" : "PUSH"
}, {
"hash" : "965136302f1188e5123de37734c927dc55801b24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7691",
"triggerID" : "965136302f1188e5123de37734c927dc55801b24",
"triggerType" : "PUSH"
}, {
"hash" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7693",
"triggerID" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"triggerType" : "PUSH"
}, {
"hash" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7717",
"triggerID" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"triggerType" : "PUSH"
}, {
"hash" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"triggerType" : "PUSH"
}, {
"hash" : "47f8303d6e61504275ecfddf553e356427e6c822",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7718",
"triggerID" : "47f8303d6e61504275ecfddf553e356427e6c822",
"triggerType" : "PUSH"
}, {
"hash" : "fb191ecf48d13cd81412535f7f8c9ed553d86517",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7725",
"triggerID" : "fb191ecf48d13cd81412535f7f8c9ed553d86517",
"triggerType" : "PUSH"
}, {
"hash" : "064912d3a54714935f2ee77d55cc66816166300f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7727",
"triggerID" : "064912d3a54714935f2ee77d55cc66816166300f",
"triggerType" : "PUSH"
}, {
"hash" : "77a2b45fe13dafc51f7bebb2ba859d40291ccc10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7736",
"triggerID" : "77a2b45fe13dafc51f7bebb2ba859d40291ccc10",
"triggerType" : "PUSH"
}, {
"hash" : "e4eda0e51fee0e8de5d57ea7b38dc642f53aa95d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7758",
"triggerID" : "e4eda0e51fee0e8de5d57ea7b38dc642f53aa95d",
"triggerType" : "PUSH"
}, {
"hash" : "6cc840c51fcecde95d2187fb6fd217b50fa82806",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6cc840c51fcecde95d2187fb6fd217b50fa82806",
"triggerType" : "PUSH"
}, {
"hash" : "31dab8ce9abbe4fc80e6cd81a0e433a039ca3e1d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7760",
"triggerID" : "31dab8ce9abbe4fc80e6cd81a0e433a039ca3e1d",
"triggerType" : "PUSH"
}, {
"hash" : "c04fd95dcc94146a9146c347f4f204656f3d8308",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7763",
"triggerID" : "c04fd95dcc94146a9146c347f4f204656f3d8308",
"triggerType" : "PUSH"
}, {
"hash" : "324147913d47ba8f2555e1991ce59905e5dc0b3f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7767",
"triggerID" : "324147913d47ba8f2555e1991ce59905e5dc0b3f",
"triggerType" : "PUSH"
}, {
"hash" : "3f177d42b831b30f4bd590b057dcab6ded78b634",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7774",
"triggerID" : "3f177d42b831b30f4bd590b057dcab6ded78b634",
"triggerType" : "PUSH"
}, {
"hash" : "8d6d2c06ec82d1072251ac12ddf21a890ddd5bda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7776",
"triggerID" : "8d6d2c06ec82d1072251ac12ddf21a890ddd5bda",
"triggerType" : "PUSH"
}, {
"hash" : "77589738d73b044385f5aa6f7ef87b421f06043b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "77589738d73b044385f5aa6f7ef87b421f06043b",
"triggerType" : "PUSH"
}, {
"hash" : "ad0d7ef0fbc525f880d3baadef5a22196ab97b32",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7781",
"triggerID" : "ad0d7ef0fbc525f880d3baadef5a22196ab97b32",
"triggerType" : "PUSH"
}, {
"hash" : "0db72caa3a6c7d0f0899f243ea4b8975de79e02c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7787",
"triggerID" : "0db72caa3a6c7d0f0899f243ea4b8975de79e02c",
"triggerType" : "PUSH"
}, {
"hash" : "a01ce748dc2d4cd1fd4858d1ecac8a486989e880",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7797",
"triggerID" : "a01ce748dc2d4cd1fd4858d1ecac8a486989e880",
"triggerType" : "PUSH"
}, {
"hash" : "3f35151e28bfa1dd50b31c6e5379d9769627afd9",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7799",
"triggerID" : "3f35151e28bfa1dd50b31c6e5379d9769627afd9",
"triggerType" : "PUSH"
}, {
"hash" : "b0054c8fd329489d9a1e9535d9866edffc52fdd0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b0054c8fd329489d9a1e9535d9866edffc52fdd0",
"triggerType" : "PUSH"
}, {
"hash" : "363e54341420cc217bf5af62de1e09ff370f75d2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "363e54341420cc217bf5af62de1e09ff370f75d2",
"triggerType" : "PUSH"
}, {
"hash" : "a893da375eb9c65edabbd22965296eac9b3714f4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a893da375eb9c65edabbd22965296eac9b3714f4",
"triggerType" : "PUSH"
}, {
"hash" : "a7c2756e6a293cbba99bd4e821881e6b961c138f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a7c2756e6a293cbba99bd4e821881e6b961c138f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 50022dcc45b0f511dd69706aeaf6464a51be49e2 UNKNOWN
* 6cc840c51fcecde95d2187fb6fd217b50fa82806 UNKNOWN
* 77589738d73b044385f5aa6f7ef87b421f06043b UNKNOWN
* 3f35151e28bfa1dd50b31c6e5379d9769627afd9 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7799)
* b0054c8fd329489d9a1e9535d9866edffc52fdd0 UNKNOWN
* 363e54341420cc217bf5af62de1e09ff370f75d2 UNKNOWN
* a893da375eb9c65edabbd22965296eac9b3714f4 UNKNOWN
* a7c2756e6a293cbba99bd4e821881e6b961c138f UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13699:
URL: https://github.com/apache/hudi/pull/13699#discussion_r2271868106
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/FileGroupReaderBasedMergeHandle.java:
##########
@@ -75,13 +85,60 @@
public class FileGroupReaderBasedMergeHandle<T, I, K, O> extends HoodieWriteMergeHandle<T, I, K, O> {
private static final Logger LOG = LoggerFactory.getLogger(FileGroupReaderBasedMergeHandle.class);
- private final HoodieReaderContext<T> readerContext;
- private final CompactionOperation operation;
+ private final Option<CompactionOperation> operation;
private final String maxInstantTime;
+ private HoodieReaderContext<T> readerContext;
private HoodieReadStats readStats;
- private final HoodieRecord.HoodieRecordType recordType;
- private final Option<HoodieCDCLogger> cdcLogger;
+ private HoodieRecord.HoodieRecordType recordType;
+ private Option<HoodieCDCLogger> cdcLogger;
+ private Option<RecordLevelIndexCallback> recordIndexCallbackOpt;
+ private Option<SecondaryIndexCallback> secondaryIndexCallbackOpt;
+ private final boolean isCompaction;
+ private final TypedProperties props;
+ private Iterator<HoodieRecord> incomingRecordsItr;
+ /**
+ * Constructor for Copy-On-Write (COW) merge path.
+ * Takes in a base path and an iterator of records to be merged with that file.
+ * @param config instance of {@link HoodieWriteConfig} to use.
+ * @param instantTime instant time of the current commit.
+ * @param hoodieTable instance of {@link HoodieTable} being updated.
+ * @param recordItr iterator of records to be merged with the file.
+ * @param partitionPath partition path of the base file.
+ * @param fileId file ID of the base file.
+ * @param taskContextSupplier instance of {@link TaskContextSupplier} to use.
+ * @param keyGeneratorOpt optional instance of {@link BaseKeyGenerator} to use for extracting keys from records.
+ * @param readerContext instance of {@link HoodieReaderContext} to use while merging for accessing fields and transforming records.
+ */
+ public FileGroupReaderBasedMergeHandle(HoodieWriteConfig config, String instantTime, HoodieTable<T, I, K, O> hoodieTable,
+ Iterator<HoodieRecord<T>> recordItr, String partitionPath, String fileId,
+ TaskContextSupplier taskContextSupplier, Option<BaseKeyGenerator> keyGeneratorOpt, HoodieReaderContext<T> readerContext) {
Review Comment:
yeah, not sure if it is feasible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13699:
URL: https://github.com/apache/hudi/pull/13699#issuecomment-3182613872
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7661",
"triggerID" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"triggerType" : "PUSH"
}, {
"hash" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7666",
"triggerID" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"triggerType" : "PUSH"
}, {
"hash" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7667",
"triggerID" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"triggerType" : "PUSH"
}, {
"hash" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7672",
"triggerID" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"triggerType" : "PUSH"
}, {
"hash" : "965136302f1188e5123de37734c927dc55801b24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7691",
"triggerID" : "965136302f1188e5123de37734c927dc55801b24",
"triggerType" : "PUSH"
}, {
"hash" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7693",
"triggerID" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"triggerType" : "PUSH"
}, {
"hash" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7717",
"triggerID" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"triggerType" : "PUSH"
}, {
"hash" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"triggerType" : "PUSH"
}, {
"hash" : "47f8303d6e61504275ecfddf553e356427e6c822",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7718",
"triggerID" : "47f8303d6e61504275ecfddf553e356427e6c822",
"triggerType" : "PUSH"
}, {
"hash" : "fb191ecf48d13cd81412535f7f8c9ed553d86517",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7725",
"triggerID" : "fb191ecf48d13cd81412535f7f8c9ed553d86517",
"triggerType" : "PUSH"
}, {
"hash" : "064912d3a54714935f2ee77d55cc66816166300f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7727",
"triggerID" : "064912d3a54714935f2ee77d55cc66816166300f",
"triggerType" : "PUSH"
}, {
"hash" : "77a2b45fe13dafc51f7bebb2ba859d40291ccc10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7736",
"triggerID" : "77a2b45fe13dafc51f7bebb2ba859d40291ccc10",
"triggerType" : "PUSH"
}, {
"hash" : "e4eda0e51fee0e8de5d57ea7b38dc642f53aa95d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7758",
"triggerID" : "e4eda0e51fee0e8de5d57ea7b38dc642f53aa95d",
"triggerType" : "PUSH"
}, {
"hash" : "6cc840c51fcecde95d2187fb6fd217b50fa82806",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6cc840c51fcecde95d2187fb6fd217b50fa82806",
"triggerType" : "PUSH"
}, {
"hash" : "31dab8ce9abbe4fc80e6cd81a0e433a039ca3e1d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7760",
"triggerID" : "31dab8ce9abbe4fc80e6cd81a0e433a039ca3e1d",
"triggerType" : "PUSH"
}, {
"hash" : "c04fd95dcc94146a9146c347f4f204656f3d8308",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7763",
"triggerID" : "c04fd95dcc94146a9146c347f4f204656f3d8308",
"triggerType" : "PUSH"
}, {
"hash" : "324147913d47ba8f2555e1991ce59905e5dc0b3f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7767",
"triggerID" : "324147913d47ba8f2555e1991ce59905e5dc0b3f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 50022dcc45b0f511dd69706aeaf6464a51be49e2 UNKNOWN
* 6cc840c51fcecde95d2187fb6fd217b50fa82806 UNKNOWN
* 324147913d47ba8f2555e1991ce59905e5dc0b3f Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7767)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13699:
URL: https://github.com/apache/hudi/pull/13699#issuecomment-3175691792
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7661",
"triggerID" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"triggerType" : "PUSH"
}, {
"hash" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7666",
"triggerID" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"triggerType" : "PUSH"
}, {
"hash" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7667",
"triggerID" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"triggerType" : "PUSH"
}, {
"hash" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7672",
"triggerID" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"triggerType" : "PUSH"
}, {
"hash" : "965136302f1188e5123de37734c927dc55801b24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7691",
"triggerID" : "965136302f1188e5123de37734c927dc55801b24",
"triggerType" : "PUSH"
}, {
"hash" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7693",
"triggerID" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"triggerType" : "PUSH"
}, {
"hash" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7717",
"triggerID" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"triggerType" : "PUSH"
}, {
"hash" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7dbbe8212c828dc3cf2405a0634d4a0c643580e0 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7693)
* 22721bae05acf8aa296bc5732af5a90d2afd3515 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7717)
* 50022dcc45b0f511dd69706aeaf6464a51be49e2 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13699:
URL: https://github.com/apache/hudi/pull/13699#discussion_r2271842585
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/FileGroupReaderBasedMergeHandle.java:
##########
@@ -267,26 +393,141 @@ private static class CDCCallback<T> implements BaseFileUpdateCallback<T> {
}
@Override
- public void onUpdate(String recordKey, T previousRecord, T mergedRecord) {
+ public void onUpdate(String recordKey, BufferedRecord<T> previousRecord, BufferedRecord<T> mergedRecord) {
cdcLogger.put(recordKey, convertOutput(previousRecord), Option.of(convertOutput(mergedRecord)));
-
}
@Override
- public void onInsert(String recordKey, T newRecord) {
+ public void onInsert(String recordKey, BufferedRecord<T> newRecord) {
cdcLogger.put(recordKey, null, Option.of(convertOutput(newRecord)));
-
}
@Override
- public void onDelete(String recordKey, T previousRecord) {
+ public void onDelete(String recordKey, BufferedRecord<T> previousRecord, HoodieOperation hoodieOperation) {
cdcLogger.put(recordKey, convertOutput(previousRecord), Option.empty());
-
}
- private GenericRecord convertOutput(T record) {
- T convertedRecord = outputConverter.get().map(converter -> record == null ? null : converter.apply(record)).orElse(record);
+ private GenericRecord convertOutput(BufferedRecord<T> record) {
+ T convertedRecord = outputConverter.get().map(converter -> record == null ? null : converter.apply(record.getRecord())).orElse(record.getRecord());
return convertedRecord == null ? null : readerContext.getRecordContext().convertToAvroRecord(convertedRecord, requestedSchema.get());
}
}
+
+ private static class RecordLevelIndexCallback<T> implements BaseFileUpdateCallback<T> {
+ private final WriteStatus writeStatus;
+ private final HoodieRecordLocation fileRecordLocation;
+ private final String partitionPath;
+
+ public RecordLevelIndexCallback(WriteStatus writeStatus, HoodieRecordLocation fileRecordLocation, String partitionPath) {
+ this.writeStatus = writeStatus;
+ this.fileRecordLocation = fileRecordLocation;
+ this.partitionPath = partitionPath;
+ }
+
+ @Override
+ public void onUpdate(String recordKey, BufferedRecord<T> previousRecord, BufferedRecord<T> mergedRecord) {
+ writeStatus.addRecordDelegate(HoodieRecordDelegate.create(recordKey, partitionPath, fileRecordLocation, fileRecordLocation, mergedRecord.getHoodieOperation() == HoodieOperation.UPDATE_BEFORE));
+ }
+
+ @Override
+ public void onInsert(String recordKey, BufferedRecord<T> newRecord) {
+ writeStatus.addRecordDelegate(HoodieRecordDelegate.create(recordKey, partitionPath, null, fileRecordLocation, newRecord.getHoodieOperation() == HoodieOperation.UPDATE_BEFORE));
+ }
+
+ @Override
+ public void onDelete(String recordKey, BufferedRecord<T> previousRecord, HoodieOperation hoodieOperation) {
+ // The update before operation is used when a deletion is being sent to the old File Group in a different partition.
+ // In this case, we do not want to delete the record metadata from the index.
+ writeStatus.addRecordDelegate(HoodieRecordDelegate.create(recordKey, partitionPath, fileRecordLocation, null, hoodieOperation == HoodieOperation.UPDATE_BEFORE));
Review Comment:
I simplified this in a recent commit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13699:
URL: https://github.com/apache/hudi/pull/13699#issuecomment-3186601372
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7661",
"triggerID" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"triggerType" : "PUSH"
}, {
"hash" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7666",
"triggerID" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"triggerType" : "PUSH"
}, {
"hash" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7667",
"triggerID" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"triggerType" : "PUSH"
}, {
"hash" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7672",
"triggerID" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"triggerType" : "PUSH"
}, {
"hash" : "965136302f1188e5123de37734c927dc55801b24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7691",
"triggerID" : "965136302f1188e5123de37734c927dc55801b24",
"triggerType" : "PUSH"
}, {
"hash" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7693",
"triggerID" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"triggerType" : "PUSH"
}, {
"hash" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7717",
"triggerID" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"triggerType" : "PUSH"
}, {
"hash" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"triggerType" : "PUSH"
}, {
"hash" : "47f8303d6e61504275ecfddf553e356427e6c822",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7718",
"triggerID" : "47f8303d6e61504275ecfddf553e356427e6c822",
"triggerType" : "PUSH"
}, {
"hash" : "fb191ecf48d13cd81412535f7f8c9ed553d86517",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7725",
"triggerID" : "fb191ecf48d13cd81412535f7f8c9ed553d86517",
"triggerType" : "PUSH"
}, {
"hash" : "064912d3a54714935f2ee77d55cc66816166300f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7727",
"triggerID" : "064912d3a54714935f2ee77d55cc66816166300f",
"triggerType" : "PUSH"
}, {
"hash" : "77a2b45fe13dafc51f7bebb2ba859d40291ccc10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7736",
"triggerID" : "77a2b45fe13dafc51f7bebb2ba859d40291ccc10",
"triggerType" : "PUSH"
}, {
"hash" : "e4eda0e51fee0e8de5d57ea7b38dc642f53aa95d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7758",
"triggerID" : "e4eda0e51fee0e8de5d57ea7b38dc642f53aa95d",
"triggerType" : "PUSH"
}, {
"hash" : "6cc840c51fcecde95d2187fb6fd217b50fa82806",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6cc840c51fcecde95d2187fb6fd217b50fa82806",
"triggerType" : "PUSH"
}, {
"hash" : "31dab8ce9abbe4fc80e6cd81a0e433a039ca3e1d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7760",
"triggerID" : "31dab8ce9abbe4fc80e6cd81a0e433a039ca3e1d",
"triggerType" : "PUSH"
}, {
"hash" : "c04fd95dcc94146a9146c347f4f204656f3d8308",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7763",
"triggerID" : "c04fd95dcc94146a9146c347f4f204656f3d8308",
"triggerType" : "PUSH"
}, {
"hash" : "324147913d47ba8f2555e1991ce59905e5dc0b3f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7767",
"triggerID" : "324147913d47ba8f2555e1991ce59905e5dc0b3f",
"triggerType" : "PUSH"
}, {
"hash" : "3f177d42b831b30f4bd590b057dcab6ded78b634",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7774",
"triggerID" : "3f177d42b831b30f4bd590b057dcab6ded78b634",
"triggerType" : "PUSH"
}, {
"hash" : "8d6d2c06ec82d1072251ac12ddf21a890ddd5bda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7776",
"triggerID" : "8d6d2c06ec82d1072251ac12ddf21a890ddd5bda",
"triggerType" : "PUSH"
}, {
"hash" : "77589738d73b044385f5aa6f7ef87b421f06043b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "77589738d73b044385f5aa6f7ef87b421f06043b",
"triggerType" : "PUSH"
}, {
"hash" : "ad0d7ef0fbc525f880d3baadef5a22196ab97b32",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7781",
"triggerID" : "ad0d7ef0fbc525f880d3baadef5a22196ab97b32",
"triggerType" : "PUSH"
}, {
"hash" : "0db72caa3a6c7d0f0899f243ea4b8975de79e02c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7787",
"triggerID" : "0db72caa3a6c7d0f0899f243ea4b8975de79e02c",
"triggerType" : "PUSH"
}, {
"hash" : "a01ce748dc2d4cd1fd4858d1ecac8a486989e880",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7797",
"triggerID" : "a01ce748dc2d4cd1fd4858d1ecac8a486989e880",
"triggerType" : "PUSH"
}, {
"hash" : "3f35151e28bfa1dd50b31c6e5379d9769627afd9",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7799",
"triggerID" : "3f35151e28bfa1dd50b31c6e5379d9769627afd9",
"triggerType" : "PUSH"
}, {
"hash" : "b0054c8fd329489d9a1e9535d9866edffc52fdd0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b0054c8fd329489d9a1e9535d9866edffc52fdd0",
"triggerType" : "PUSH"
}, {
"hash" : "363e54341420cc217bf5af62de1e09ff370f75d2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "363e54341420cc217bf5af62de1e09ff370f75d2",
"triggerType" : "PUSH"
}, {
"hash" : "a893da375eb9c65edabbd22965296eac9b3714f4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a893da375eb9c65edabbd22965296eac9b3714f4",
"triggerType" : "PUSH"
}, {
"hash" : "a7c2756e6a293cbba99bd4e821881e6b961c138f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7804",
"triggerID" : "a7c2756e6a293cbba99bd4e821881e6b961c138f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 50022dcc45b0f511dd69706aeaf6464a51be49e2 UNKNOWN
* 6cc840c51fcecde95d2187fb6fd217b50fa82806 UNKNOWN
* 77589738d73b044385f5aa6f7ef87b421f06043b UNKNOWN
* 3f35151e28bfa1dd50b31c6e5379d9769627afd9 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7799)
* b0054c8fd329489d9a1e9535d9866edffc52fdd0 UNKNOWN
* 363e54341420cc217bf5af62de1e09ff370f75d2 UNKNOWN
* a893da375eb9c65edabbd22965296eac9b3714f4 UNKNOWN
* a7c2756e6a293cbba99bd4e821881e6b961c138f Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7804)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13699:
URL: https://github.com/apache/hudi/pull/13699#discussion_r2266753079
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/FileGroupReaderBasedMergeHandle.java:
##########
@@ -75,13 +85,60 @@
public class FileGroupReaderBasedMergeHandle<T, I, K, O> extends HoodieWriteMergeHandle<T, I, K, O> {
private static final Logger LOG = LoggerFactory.getLogger(FileGroupReaderBasedMergeHandle.class);
- private final HoodieReaderContext<T> readerContext;
- private final CompactionOperation operation;
+ private final Option<CompactionOperation> operation;
private final String maxInstantTime;
+ private HoodieReaderContext<T> readerContext;
private HoodieReadStats readStats;
- private final HoodieRecord.HoodieRecordType recordType;
- private final Option<HoodieCDCLogger> cdcLogger;
+ private HoodieRecord.HoodieRecordType recordType;
+ private Option<HoodieCDCLogger> cdcLogger;
+ private Option<RecordLevelIndexCallback> recordIndexCallbackOpt;
+ private Option<SecondaryIndexCallback> secondaryIndexCallbackOpt;
+ private final boolean isCompaction;
+ private final TypedProperties props;
+ private Iterator<HoodieRecord> incomingRecordsItr;
+ /**
+ * Constructor for Copy-On-Write (COW) merge path.
+ * Takes in a base path and an iterator of records to be merged with that file.
+ * @param config instance of {@link HoodieWriteConfig} to use.
+ * @param instantTime instant time of the current commit.
+ * @param hoodieTable instance of {@link HoodieTable} being updated.
+ * @param recordItr iterator of records to be merged with the file.
+ * @param partitionPath partition path of the base file.
+ * @param fileId file ID of the base file.
+ * @param taskContextSupplier instance of {@link TaskContextSupplier} to use.
+ * @param keyGeneratorOpt optional instance of {@link BaseKeyGenerator} to use for extracting keys from records.
+ * @param readerContext instance of {@link HoodieReaderContext} to use while merging for accessing fields and transforming records.
+ */
+ public FileGroupReaderBasedMergeHandle(HoodieWriteConfig config, String instantTime, HoodieTable<T, I, K, O> hoodieTable,
+ Iterator<HoodieRecord<T>> recordItr, String partitionPath, String fileId,
+ TaskContextSupplier taskContextSupplier, Option<BaseKeyGenerator> keyGeneratorOpt, HoodieReaderContext<T> readerContext) {
Review Comment:
The issue is that the merge handles are created on the executors in spark so the `hoodieTable.getContext` will always return a local engine context instead of a spark engine context when required.
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/FileGroupReaderBasedMergeHandle.java:
##########
@@ -267,26 +393,141 @@ private static class CDCCallback<T> implements BaseFileUpdateCallback<T> {
}
@Override
- public void onUpdate(String recordKey, T previousRecord, T mergedRecord) {
+ public void onUpdate(String recordKey, BufferedRecord<T> previousRecord, BufferedRecord<T> mergedRecord) {
cdcLogger.put(recordKey, convertOutput(previousRecord), Option.of(convertOutput(mergedRecord)));
-
}
@Override
- public void onInsert(String recordKey, T newRecord) {
+ public void onInsert(String recordKey, BufferedRecord<T> newRecord) {
cdcLogger.put(recordKey, null, Option.of(convertOutput(newRecord)));
-
}
@Override
- public void onDelete(String recordKey, T previousRecord) {
+ public void onDelete(String recordKey, BufferedRecord<T> previousRecord, HoodieOperation hoodieOperation) {
cdcLogger.put(recordKey, convertOutput(previousRecord), Option.empty());
-
}
- private GenericRecord convertOutput(T record) {
- T convertedRecord = outputConverter.get().map(converter -> record == null ? null : converter.apply(record)).orElse(record);
+ private GenericRecord convertOutput(BufferedRecord<T> record) {
+ T convertedRecord = outputConverter.get().map(converter -> record == null ? null : converter.apply(record.getRecord())).orElse(record.getRecord());
return convertedRecord == null ? null : readerContext.getRecordContext().convertToAvroRecord(convertedRecord, requestedSchema.get());
}
}
+
+ private static class RecordLevelIndexCallback<T> implements BaseFileUpdateCallback<T> {
+ private final WriteStatus writeStatus;
+ private final HoodieRecordLocation fileRecordLocation;
+ private final String partitionPath;
+
+ public RecordLevelIndexCallback(WriteStatus writeStatus, HoodieRecordLocation fileRecordLocation, String partitionPath) {
+ this.writeStatus = writeStatus;
+ this.fileRecordLocation = fileRecordLocation;
+ this.partitionPath = partitionPath;
+ }
+
+ @Override
+ public void onUpdate(String recordKey, BufferedRecord<T> previousRecord, BufferedRecord<T> mergedRecord) {
+ writeStatus.addRecordDelegate(HoodieRecordDelegate.create(recordKey, partitionPath, fileRecordLocation, fileRecordLocation, mergedRecord.getHoodieOperation() == HoodieOperation.UPDATE_BEFORE));
+ }
+
+ @Override
+ public void onInsert(String recordKey, BufferedRecord<T> newRecord) {
+ writeStatus.addRecordDelegate(HoodieRecordDelegate.create(recordKey, partitionPath, null, fileRecordLocation, newRecord.getHoodieOperation() == HoodieOperation.UPDATE_BEFORE));
Review Comment:
It's always false today but do we want to keep this in case there is some future case where it may not be the case?
##########
hudi-common/src/main/java/org/apache/hudi/common/engine/HoodieEngineContext.java:
##########
@@ -139,15 +139,16 @@ public abstract <I, K, V> List<V> reduceByKey(
/**
* Returns reader context factory for write operations in the table.
*
- * @param metaClient Table meta client
- * @param recordType Record type
- * @param properties Typed properties
+ * @param metaClient Table meta client
+ * @param recordType Record type
+ * @param properties Typed properties
+ * @param outputsCustomPayloads Whether the reader context factory should output custom payloads. Final merging of records before writes does not require custom payloads.
*/
public ReaderContextFactory<?> getReaderContextFactoryForWrite(HoodieTableMetaClient metaClient, HoodieRecord.HoodieRecordType recordType,
- TypedProperties properties) {
+ TypedProperties properties, boolean outputsCustomPayloads) {
Review Comment:
I didn't find a good way right now. This flag is really representing two different stages of the writer path, the dedupe/indexing stages and the final write. In the final write, we don't want to ever use the payload based records since we just want the final indexed record representation of the record.
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/SecondaryIndexStreamingTracker.java:
##########
@@ -226,6 +228,81 @@ static <T> void trackSecondaryIndexStats(@Nullable HoodieKey hoodieKey, Option<H
});
}
+ /**
+ * The utility function used by Merge Handle to generate secondary index stats for the corresponding record.
+ * It considers the new merged version of the record and compares it with the older version of the record to generate
+ * secondary index stats.
+ *
+ * @param hoodieKey The hoodie key
+ * @param combinedRecordOpt New record merged with the old record
+ * @param oldRecord The old record
+ * @param isDelete Whether the record is a DELETE
+ * @param writeStatus The Write status
+ * @param secondaryIndexDefns Definitions for secondary index which need to be updated
+ */
+ static <T> void trackSecondaryIndexStats(HoodieKey hoodieKey, Option<BufferedRecord<T>> combinedRecordOpt, @Nullable BufferedRecord<T> oldRecord, boolean isDelete,
Review Comment:
Method mirrors the one above it but operates directly on BufferedRecord instead of converting to HoodieRecord
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/FileGroupReaderBasedMergeHandle.java:
##########
@@ -267,26 +393,141 @@ private static class CDCCallback<T> implements BaseFileUpdateCallback<T> {
}
@Override
- public void onUpdate(String recordKey, T previousRecord, T mergedRecord) {
+ public void onUpdate(String recordKey, BufferedRecord<T> previousRecord, BufferedRecord<T> mergedRecord) {
cdcLogger.put(recordKey, convertOutput(previousRecord), Option.of(convertOutput(mergedRecord)));
-
}
@Override
- public void onInsert(String recordKey, T newRecord) {
+ public void onInsert(String recordKey, BufferedRecord<T> newRecord) {
cdcLogger.put(recordKey, null, Option.of(convertOutput(newRecord)));
-
}
@Override
- public void onDelete(String recordKey, T previousRecord) {
+ public void onDelete(String recordKey, BufferedRecord<T> previousRecord, HoodieOperation hoodieOperation) {
cdcLogger.put(recordKey, convertOutput(previousRecord), Option.empty());
-
}
- private GenericRecord convertOutput(T record) {
- T convertedRecord = outputConverter.get().map(converter -> record == null ? null : converter.apply(record)).orElse(record);
+ private GenericRecord convertOutput(BufferedRecord<T> record) {
+ T convertedRecord = outputConverter.get().map(converter -> record == null ? null : converter.apply(record.getRecord())).orElse(record.getRecord());
return convertedRecord == null ? null : readerContext.getRecordContext().convertToAvroRecord(convertedRecord, requestedSchema.get());
}
}
+
+ private static class RecordLevelIndexCallback<T> implements BaseFileUpdateCallback<T> {
+ private final WriteStatus writeStatus;
+ private final HoodieRecordLocation fileRecordLocation;
+ private final String partitionPath;
+
+ public RecordLevelIndexCallback(WriteStatus writeStatus, HoodieRecordLocation fileRecordLocation, String partitionPath) {
+ this.writeStatus = writeStatus;
+ this.fileRecordLocation = fileRecordLocation;
+ this.partitionPath = partitionPath;
+ }
+
+ @Override
+ public void onUpdate(String recordKey, BufferedRecord<T> previousRecord, BufferedRecord<T> mergedRecord) {
+ writeStatus.addRecordDelegate(HoodieRecordDelegate.create(recordKey, partitionPath, fileRecordLocation, fileRecordLocation, mergedRecord.getHoodieOperation() == HoodieOperation.UPDATE_BEFORE));
Review Comment:
The write status will still be updated in the current code with this record delegate even though `ignoreIndexUpdate` is true. This is keeping parity with the old system but I am not sure of the context for this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13699:
URL: https://github.com/apache/hudi/pull/13699#issuecomment-3169617267
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ca08e20d8bacc2355f15ae37fab3c1ef61a052af UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13699:
URL: https://github.com/apache/hudi/pull/13699#issuecomment-3171281478
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7661",
"triggerID" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"triggerType" : "PUSH"
}, {
"hash" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7666",
"triggerID" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"triggerType" : "PUSH"
}, {
"hash" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7666)
* af88f9e4a548191ff0cba41db6ad3e60ba9ea43b UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13699:
URL: https://github.com/apache/hudi/pull/13699#issuecomment-3171060419
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7661",
"triggerID" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"triggerType" : "PUSH"
}, {
"hash" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ca08e20d8bacc2355f15ae37fab3c1ef61a052af Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7661)
* 7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13699:
URL: https://github.com/apache/hudi/pull/13699#discussion_r2271952726
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/buffer/FileGroupRecordBuffer.java:
##########
@@ -56,27 +53,29 @@
import java.io.IOException;
import java.io.Serializable;
+import java.util.HashSet;
import java.util.Iterator;
import java.util.List;
import java.util.Locale;
import java.util.Map;
+import java.util.Set;
import java.util.function.Function;
import static org.apache.hudi.common.config.HoodieCommonConfig.DISK_MAP_BITCASK_COMPRESSION_ENABLED;
import static org.apache.hudi.common.config.HoodieCommonConfig.SPILLABLE_DISK_MAP_TYPE;
import static org.apache.hudi.common.config.HoodieMemoryConfig.MAX_MEMORY_FOR_MERGE;
import static org.apache.hudi.common.config.HoodieMemoryConfig.SPILLABLE_MAP_BASE_PATH;
-import static org.apache.hudi.common.model.HoodieRecordMerger.PAYLOAD_BASED_MERGE_STRATEGY_UUID;
import static org.apache.hudi.common.table.log.block.HoodieLogBlock.HeaderMetadataType.INSTANT_TIME;
abstract class FileGroupRecordBuffer<T> implements HoodieFileGroupRecordBuffer<T> {
+ protected final Set<String> usedKeys = new HashSet<>();
Review Comment:
I'm wondering if this is a valid case, because the duplicates in base only occurs in pkless table, while in the test case, the table has a primary key but it sets up to allow duplicates for the first commit intentionally, seems incorrect. We should limit the "allow duplicates" for incoming records to just `INSERT` operation for pkless table, just like the doc says in `HoodieConcatHandle`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13699:
URL: https://github.com/apache/hudi/pull/13699#issuecomment-3182070388
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7661",
"triggerID" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"triggerType" : "PUSH"
}, {
"hash" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7666",
"triggerID" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"triggerType" : "PUSH"
}, {
"hash" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7667",
"triggerID" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"triggerType" : "PUSH"
}, {
"hash" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7672",
"triggerID" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"triggerType" : "PUSH"
}, {
"hash" : "965136302f1188e5123de37734c927dc55801b24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7691",
"triggerID" : "965136302f1188e5123de37734c927dc55801b24",
"triggerType" : "PUSH"
}, {
"hash" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7693",
"triggerID" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"triggerType" : "PUSH"
}, {
"hash" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7717",
"triggerID" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"triggerType" : "PUSH"
}, {
"hash" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"triggerType" : "PUSH"
}, {
"hash" : "47f8303d6e61504275ecfddf553e356427e6c822",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7718",
"triggerID" : "47f8303d6e61504275ecfddf553e356427e6c822",
"triggerType" : "PUSH"
}, {
"hash" : "fb191ecf48d13cd81412535f7f8c9ed553d86517",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7725",
"triggerID" : "fb191ecf48d13cd81412535f7f8c9ed553d86517",
"triggerType" : "PUSH"
}, {
"hash" : "064912d3a54714935f2ee77d55cc66816166300f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7727",
"triggerID" : "064912d3a54714935f2ee77d55cc66816166300f",
"triggerType" : "PUSH"
}, {
"hash" : "77a2b45fe13dafc51f7bebb2ba859d40291ccc10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7736",
"triggerID" : "77a2b45fe13dafc51f7bebb2ba859d40291ccc10",
"triggerType" : "PUSH"
}, {
"hash" : "e4eda0e51fee0e8de5d57ea7b38dc642f53aa95d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7758",
"triggerID" : "e4eda0e51fee0e8de5d57ea7b38dc642f53aa95d",
"triggerType" : "PUSH"
}, {
"hash" : "6cc840c51fcecde95d2187fb6fd217b50fa82806",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6cc840c51fcecde95d2187fb6fd217b50fa82806",
"triggerType" : "PUSH"
}, {
"hash" : "31dab8ce9abbe4fc80e6cd81a0e433a039ca3e1d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7760",
"triggerID" : "31dab8ce9abbe4fc80e6cd81a0e433a039ca3e1d",
"triggerType" : "PUSH"
}, {
"hash" : "c04fd95dcc94146a9146c347f4f204656f3d8308",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c04fd95dcc94146a9146c347f4f204656f3d8308",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 50022dcc45b0f511dd69706aeaf6464a51be49e2 UNKNOWN
* 6cc840c51fcecde95d2187fb6fd217b50fa82806 UNKNOWN
* 31dab8ce9abbe4fc80e6cd81a0e433a039ca3e1d Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7760)
* c04fd95dcc94146a9146c347f4f204656f3d8308 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13699:
URL: https://github.com/apache/hudi/pull/13699#discussion_r2271898808
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieMergeHandleFactory.java:
##########
@@ -60,21 +60,35 @@ public static <T, I, K, O> HoodieMergeHandle<T, I, K, O> create(
String partitionPath,
String fileId,
TaskContextSupplier taskContextSupplier,
- Option<BaseKeyGenerator> keyGeneratorOpt) {
+ Option<BaseKeyGenerator> keyGeneratorOpt,
+ HoodieReaderContext<T> readerContext) {
boolean isFallbackEnabled = writeConfig.isMergeHandleFallbackEnabled();
Pair<String, String> mergeHandleClasses = getMergeHandleClassesWrite(operationType, writeConfig, table);
String logContext = String.format("for fileId %s and partition path %s at commit %s", fileId, partitionPath, instantTime);
LOG.info("Create HoodieMergeHandle implementation {} {}", mergeHandleClasses.getLeft(), logContext);
Class<?>[] constructorParamTypes = new Class<?>[] {
+ HoodieWriteConfig.class, String.class, HoodieTable.class, Iterator.class,
+ String.class, String.class, TaskContextSupplier.class, Option.class, HoodieReaderContext.class
+ };
+
+ Class<?>[] legacyConstructorParamTypes = new Class<?>[] {
HoodieWriteConfig.class, String.class, HoodieTable.class, Iterator.class,
String.class, String.class, TaskContextSupplier.class, Option.class
};
- return instantiateMergeHandle(
- isFallbackEnabled, mergeHandleClasses.getLeft(), mergeHandleClasses.getRight(), logContext, constructorParamTypes,
- writeConfig, instantTime, table, recordItr, partitionPath, fileId, taskContextSupplier, keyGeneratorOpt);
+ try {
+ return instantiateMergeHandle(
+ isFallbackEnabled, mergeHandleClasses.getLeft(), mergeHandleClasses.getRight(), logContext, constructorParamTypes,
+ writeConfig, instantTime, table, recordItr, partitionPath, fileId, taskContextSupplier, keyGeneratorOpt, readerContext);
+ } catch (Exception e) {
+ // Fallback to legacy constructor if the new one fails
+ LOG.warn("Failed to instantiate HoodieMergeHandle with new constructor, falling back to legacy constructor: {}", e.getMessage());
Review Comment:
the catch exception fallback is hacky, can we match the class name `FileGroupReaderBasedMergeHandle.class.getName` to invoke different constructors.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13699:
URL: https://github.com/apache/hudi/pull/13699#discussion_r2271941785
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/buffer/FileGroupRecordBuffer.java:
##########
@@ -56,27 +53,29 @@
import java.io.IOException;
import java.io.Serializable;
+import java.util.HashSet;
import java.util.Iterator;
import java.util.List;
import java.util.Locale;
import java.util.Map;
+import java.util.Set;
import java.util.function.Function;
import static org.apache.hudi.common.config.HoodieCommonConfig.DISK_MAP_BITCASK_COMPRESSION_ENABLED;
import static org.apache.hudi.common.config.HoodieCommonConfig.SPILLABLE_DISK_MAP_TYPE;
import static org.apache.hudi.common.config.HoodieMemoryConfig.MAX_MEMORY_FOR_MERGE;
import static org.apache.hudi.common.config.HoodieMemoryConfig.SPILLABLE_MAP_BASE_PATH;
-import static org.apache.hudi.common.model.HoodieRecordMerger.PAYLOAD_BASED_MERGE_STRATEGY_UUID;
import static org.apache.hudi.common.table.log.block.HoodieLogBlock.HeaderMetadataType.INSTANT_TIME;
abstract class FileGroupRecordBuffer<T> implements HoodieFileGroupRecordBuffer<T> {
+ protected final Set<String> usedKeys = new HashSet<>();
Review Comment:
The fix is not right, the discrepencies come from the log comparison, for old write merge handle, we do not remove the log record when a base record key matches it, and bookeep a set of log record keys that have been matched with base(used for skipping for those non matched log records).
While the FG reader current remove the log record directly, which makes it impossbie to match the same log records with duplicates in base multiple times.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13699:
URL: https://github.com/apache/hudi/pull/13699#issuecomment-3184505757
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7661",
"triggerID" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"triggerType" : "PUSH"
}, {
"hash" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7666",
"triggerID" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"triggerType" : "PUSH"
}, {
"hash" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7667",
"triggerID" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"triggerType" : "PUSH"
}, {
"hash" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7672",
"triggerID" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"triggerType" : "PUSH"
}, {
"hash" : "965136302f1188e5123de37734c927dc55801b24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7691",
"triggerID" : "965136302f1188e5123de37734c927dc55801b24",
"triggerType" : "PUSH"
}, {
"hash" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7693",
"triggerID" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"triggerType" : "PUSH"
}, {
"hash" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7717",
"triggerID" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"triggerType" : "PUSH"
}, {
"hash" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"triggerType" : "PUSH"
}, {
"hash" : "47f8303d6e61504275ecfddf553e356427e6c822",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7718",
"triggerID" : "47f8303d6e61504275ecfddf553e356427e6c822",
"triggerType" : "PUSH"
}, {
"hash" : "fb191ecf48d13cd81412535f7f8c9ed553d86517",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7725",
"triggerID" : "fb191ecf48d13cd81412535f7f8c9ed553d86517",
"triggerType" : "PUSH"
}, {
"hash" : "064912d3a54714935f2ee77d55cc66816166300f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7727",
"triggerID" : "064912d3a54714935f2ee77d55cc66816166300f",
"triggerType" : "PUSH"
}, {
"hash" : "77a2b45fe13dafc51f7bebb2ba859d40291ccc10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7736",
"triggerID" : "77a2b45fe13dafc51f7bebb2ba859d40291ccc10",
"triggerType" : "PUSH"
}, {
"hash" : "e4eda0e51fee0e8de5d57ea7b38dc642f53aa95d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7758",
"triggerID" : "e4eda0e51fee0e8de5d57ea7b38dc642f53aa95d",
"triggerType" : "PUSH"
}, {
"hash" : "6cc840c51fcecde95d2187fb6fd217b50fa82806",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6cc840c51fcecde95d2187fb6fd217b50fa82806",
"triggerType" : "PUSH"
}, {
"hash" : "31dab8ce9abbe4fc80e6cd81a0e433a039ca3e1d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7760",
"triggerID" : "31dab8ce9abbe4fc80e6cd81a0e433a039ca3e1d",
"triggerType" : "PUSH"
}, {
"hash" : "c04fd95dcc94146a9146c347f4f204656f3d8308",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7763",
"triggerID" : "c04fd95dcc94146a9146c347f4f204656f3d8308",
"triggerType" : "PUSH"
}, {
"hash" : "324147913d47ba8f2555e1991ce59905e5dc0b3f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7767",
"triggerID" : "324147913d47ba8f2555e1991ce59905e5dc0b3f",
"triggerType" : "PUSH"
}, {
"hash" : "3f177d42b831b30f4bd590b057dcab6ded78b634",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7774",
"triggerID" : "3f177d42b831b30f4bd590b057dcab6ded78b634",
"triggerType" : "PUSH"
}, {
"hash" : "8d6d2c06ec82d1072251ac12ddf21a890ddd5bda",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7776",
"triggerID" : "8d6d2c06ec82d1072251ac12ddf21a890ddd5bda",
"triggerType" : "PUSH"
}, {
"hash" : "77589738d73b044385f5aa6f7ef87b421f06043b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "77589738d73b044385f5aa6f7ef87b421f06043b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 50022dcc45b0f511dd69706aeaf6464a51be49e2 UNKNOWN
* 6cc840c51fcecde95d2187fb6fd217b50fa82806 UNKNOWN
* 8d6d2c06ec82d1072251ac12ddf21a890ddd5bda Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7776)
* 77589738d73b044385f5aa6f7ef87b421f06043b UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9622] Add implementation of MergeHandle backed by the HoodieFileGroupReader [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13699:
URL: https://github.com/apache/hudi/pull/13699#discussion_r2279384576
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/dml/others/TestMergeIntoTable2.scala:
##########
@@ -838,72 +839,72 @@ class TestMergeIntoTable2 extends HoodieSparkSqlTestBase {
}
}
- test("Test only insert for source table in dup key without preCombineField") {
- spark.sql(s"set ${MERGE_SMALL_FILE_GROUP_CANDIDATES_LIMIT.key} = ${MERGE_SMALL_FILE_GROUP_CANDIDATES_LIMIT.defaultValue()}")
- Seq("cow", "mor").foreach {
- tableType => {
- withTempDir { tmp =>
- val tableName = generateTableName
- spark.sql(
- s"""
- | create table $tableName (
- | id int,
- | name string,
- | price double,
- | ts int,
- | dt string
- | ) using hudi
- | tblproperties (
- | type = '$tableType',
- | primaryKey = 'id'
- | )
- | partitioned by(dt)
- | location '${tmp.getCanonicalPath}'
- """.stripMargin)
- // append records to small file is use update bucket, set this conf use concat handler
- spark.sql("set hoodie.merge.allow.duplicate.on.inserts = true")
-
- // Insert data without matched condition
- spark.sql(
- s"""
- | merge into $tableName as t0
- | using (
- | select 1 as id, 'a1' as name, 10.1 as price, 1000 as ts, '2021-03-21' as dt
- | union all
- | select 1 as id, 'a2' as name, 10.2 as price, 1002 as ts, '2021-03-21' as dt
- | ) as s0
- | on t0.id = s0.id
- | when not matched then insert *
- """.stripMargin
- )
- checkAnswer(s"select id, name, price, ts, dt from $tableName")(
- Seq(1, "a1", 10.1, 1000, "2021-03-21"),
- Seq(1, "a2", 10.2, 1002, "2021-03-21")
- )
-
- // Insert data with matched condition
- spark.sql(
- s"""
- | merge into $tableName as t0
- | using (
- | select 3 as id, 'a3' as name, 10.3 as price, 1003 as ts, '2021-03-21' as dt
- | union all
- | select 1 as id, 'a2' as name, 10.4 as price, 1004 as ts, '2021-03-21' as dt
- | ) as s0
- | on t0.id = s0.id
- | when matched then update set *
- | when not matched then insert *
- """.stripMargin
- )
- checkAnswer(s"select id, name, price, ts, dt from $tableName")(
- Seq(1, "a2", 10.4, 1004, "2021-03-21"),
- Seq(1, "a2", 10.4, 1004, "2021-03-21"),
- Seq(3, "a3", 10.3, 1003, "2021-03-21")
- )
- }
- }
- }
- }
+// test("Test only insert for source table in dup key without preCombineField") {
Review Comment:
We retain the duplicates in the base file. The update is currently applied to both records in the base file with that key.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9622] Add implementation of MergeHandle backed by the HoodieFileGroupReader [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13699:
URL: https://github.com/apache/hudi/pull/13699#discussion_r2276850204
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/FileGroupReaderBasedMergeHandle.java:
##########
@@ -156,13 +229,28 @@ private void init(CompactionOperation operation, String partitionPath) {
fileWriter = HoodieFileWriterFactory.getFileWriter(instantTime, newFilePath, hoodieTable.getStorage(),
config, writeSchemaWithMetaFields, taskContextSupplier, recordType);
} catch (IOException io) {
- LOG.error("Error in update task at commit {}", instantTime, io);
writeStatus.setGlobalError(io);
throw new HoodieUpsertException("Failed to initialize HoodieUpdateHandle for FileId: " + fileId + " on commit "
+ instantTime + " on path " + hoodieTable.getMetaClient().getBasePath(), io);
}
}
+ @Override
+ protected void populateIncomingRecordsMap(Iterator<HoodieRecord<T>> newRecordsItr) {
+ // no op.
+ }
+
+ /**
+ * For COW merge path, lets map the incoming records to another iterator which can be routed to {@link org.apache.hudi.common.table.read.buffer.StreamingFileGroupRecordBufferLoader}.
+ * @param newRecordsItr
+ */
+ private void populateIncomingRecordsMapIterator(Iterator<HoodieRecord<T>> newRecordsItr) {
+ if (!isCompaction) {
+ // avoid populating external spillable in base {@link HoodieWriteMergeHandle)
+ this.incomingRecordsItr = new MappingIterator<>(newRecordsItr, record -> (HoodieRecord) record);
Review Comment:
This was just needed due to some type casting issues but has been fixed since.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9622] Add implementation of MergeHandle backed by the HoodieFileGroupReader [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13699:
URL: https://github.com/apache/hudi/pull/13699#discussion_r2276848029
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/FileGroupReaderBasedMergeHandle.java:
##########
@@ -90,7 +147,15 @@ public FileGroupReaderBasedMergeHandle(HoodieWriteConfig config, String instantT
this.maxInstantTime = maxInstantTime;
this.keyToNewRecords = Collections.emptyMap();
this.readerContext = readerContext;
- this.operation = operation;
+ this.operation = Option.of(operation);
+ initRecordTypeAndCdcLogger(enginRecordType);
+ init(operation, this.partitionPath);
+ this.props = TypedProperties.copy(config.getProps());
+ this.isCompaction = true;
Review Comment:
This boolean was removed in later commits
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9622] Add implementation of MergeHandle backed by the HoodieFileGroupReader [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #13699:
URL: https://github.com/apache/hudi/pull/13699#discussion_r2279120259
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/dml/others/TestMergeIntoTable2.scala:
##########
@@ -838,72 +839,72 @@ class TestMergeIntoTable2 extends HoodieSparkSqlTestBase {
}
}
- test("Test only insert for source table in dup key without preCombineField") {
- spark.sql(s"set ${MERGE_SMALL_FILE_GROUP_CANDIDATES_LIMIT.key} = ${MERGE_SMALL_FILE_GROUP_CANDIDATES_LIMIT.defaultValue()}")
- Seq("cow", "mor").foreach {
- tableType => {
- withTempDir { tmp =>
- val tableName = generateTableName
- spark.sql(
- s"""
- | create table $tableName (
- | id int,
- | name string,
- | price double,
- | ts int,
- | dt string
- | ) using hudi
- | tblproperties (
- | type = '$tableType',
- | primaryKey = 'id'
- | )
- | partitioned by(dt)
- | location '${tmp.getCanonicalPath}'
- """.stripMargin)
- // append records to small file is use update bucket, set this conf use concat handler
- spark.sql("set hoodie.merge.allow.duplicate.on.inserts = true")
-
- // Insert data without matched condition
- spark.sql(
- s"""
- | merge into $tableName as t0
- | using (
- | select 1 as id, 'a1' as name, 10.1 as price, 1000 as ts, '2021-03-21' as dt
- | union all
- | select 1 as id, 'a2' as name, 10.2 as price, 1002 as ts, '2021-03-21' as dt
- | ) as s0
- | on t0.id = s0.id
- | when not matched then insert *
- """.stripMargin
- )
- checkAnswer(s"select id, name, price, ts, dt from $tableName")(
- Seq(1, "a1", 10.1, 1000, "2021-03-21"),
- Seq(1, "a2", 10.2, 1002, "2021-03-21")
- )
-
- // Insert data with matched condition
- spark.sql(
- s"""
- | merge into $tableName as t0
- | using (
- | select 3 as id, 'a3' as name, 10.3 as price, 1003 as ts, '2021-03-21' as dt
- | union all
- | select 1 as id, 'a2' as name, 10.4 as price, 1004 as ts, '2021-03-21' as dt
- | ) as s0
- | on t0.id = s0.id
- | when matched then update set *
- | when not matched then insert *
- """.stripMargin
- )
- checkAnswer(s"select id, name, price, ts, dt from $tableName")(
- Seq(1, "a2", 10.4, 1004, "2021-03-21"),
- Seq(1, "a2", 10.4, 1004, "2021-03-21"),
- Seq(3, "a3", 10.3, 1003, "2021-03-21")
- )
- }
- }
- }
- }
+// test("Test only insert for source table in dup key without preCombineField") {
Review Comment:
For mutable streams, I agree that duplicates should not be allowed. but for `inserts` or immutable workloads, we have concat handle which is expected to retain duplicates. So, we can't blindly shut off all duplicates.
Its tough to guard this case or in other words it may not be easy to allow duplicates in this special case.
for eg, w/ small file handling, we could route updates + inserts to an existing file group, which will exercise the FileGroupReaderBasedMergeHandle after this patch. So, if the set of inserts have duplicates, we might dedup now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9622] Add implementation of MergeHandle backed by the HoodieFileGroupReader [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13699:
URL: https://github.com/apache/hudi/pull/13699#discussion_r2277894128
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/dml/others/TestMergeIntoTable2.scala:
##########
@@ -838,72 +839,72 @@ class TestMergeIntoTable2 extends HoodieSparkSqlTestBase {
}
}
- test("Test only insert for source table in dup key without preCombineField") {
- spark.sql(s"set ${MERGE_SMALL_FILE_GROUP_CANDIDATES_LIMIT.key} = ${MERGE_SMALL_FILE_GROUP_CANDIDATES_LIMIT.defaultValue()}")
- Seq("cow", "mor").foreach {
- tableType => {
- withTempDir { tmp =>
- val tableName = generateTableName
- spark.sql(
- s"""
- | create table $tableName (
- | id int,
- | name string,
- | price double,
- | ts int,
- | dt string
- | ) using hudi
- | tblproperties (
- | type = '$tableType',
- | primaryKey = 'id'
- | )
- | partitioned by(dt)
- | location '${tmp.getCanonicalPath}'
- """.stripMargin)
- // append records to small file is use update bucket, set this conf use concat handler
- spark.sql("set hoodie.merge.allow.duplicate.on.inserts = true")
-
- // Insert data without matched condition
- spark.sql(
- s"""
- | merge into $tableName as t0
- | using (
- | select 1 as id, 'a1' as name, 10.1 as price, 1000 as ts, '2021-03-21' as dt
- | union all
- | select 1 as id, 'a2' as name, 10.2 as price, 1002 as ts, '2021-03-21' as dt
- | ) as s0
- | on t0.id = s0.id
- | when not matched then insert *
- """.stripMargin
- )
- checkAnswer(s"select id, name, price, ts, dt from $tableName")(
- Seq(1, "a1", 10.1, 1000, "2021-03-21"),
- Seq(1, "a2", 10.2, 1002, "2021-03-21")
- )
-
- // Insert data with matched condition
- spark.sql(
- s"""
- | merge into $tableName as t0
- | using (
- | select 3 as id, 'a3' as name, 10.3 as price, 1003 as ts, '2021-03-21' as dt
- | union all
- | select 1 as id, 'a2' as name, 10.4 as price, 1004 as ts, '2021-03-21' as dt
- | ) as s0
- | on t0.id = s0.id
- | when matched then update set *
- | when not matched then insert *
- """.stripMargin
- )
- checkAnswer(s"select id, name, price, ts, dt from $tableName")(
- Seq(1, "a2", 10.4, 1004, "2021-03-21"),
- Seq(1, "a2", 10.4, 1004, "2021-03-21"),
- Seq(3, "a3", 10.3, 1003, "2021-03-21")
- )
- }
- }
- }
- }
+// test("Test only insert for source table in dup key without preCombineField") {
Review Comment:
This is from the conversation [here](https://github.com/apache/hudi/pull/13699#discussion_r2267993965). I am finding that there is generally support for duplicates in Hudi as of today so I was hesitant to fully remove this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9622] Add implementation of MergeHandle backed by the HoodieFileGroupReader [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13699:
URL: https://github.com/apache/hudi/pull/13699#discussion_r2277894821
##########
hudi-spark-datasource/hudi-spark/src/test/java/org/apache/hudi/io/TestMergeHandle.java:
##########
@@ -124,4 +159,390 @@ public void testMergeHandleRLIAndSIStatsWithUpdatesAndDeletes() throws Exception
assertEquals(2 * numUpdates + numDeletes, writeStatus.getIndexStats().getSecondaryIndexStats().values().stream().findFirst().get().size());
validateSecondaryIndexStatsContent(writeStatus, numUpdates, numDeletes);
}
+
+ @ParameterizedTest // TODO add CUSTOM_MERGER once deletes are handled properly
+ @ValueSource(strings = {"EVENT_TIME_ORDERING", "COMMIT_TIME_ORDERING", "CUSTOM"})
Review Comment:
See the TODO above, it will fail until we allow the custom merger to manage its own deletes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9622] Add implementation of MergeHandle backed by the HoodieFileGroupReader [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13699:
URL: https://github.com/apache/hudi/pull/13699#discussion_r2277997535
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/BufferedRecordMergerFactory.java:
##########
@@ -426,6 +428,16 @@ protected Pair<HoodieRecord, HoodieRecord> getFinalMergeRecords(BufferedRecord<T
HoodieRecord newHoodieRecord = constructHoodieAvroRecord(recordContext, newerRecord, payloadClass);
return Pair.of(oldHoodieRecord, newHoodieRecord);
}
+
+ @Override
+ protected Option<Pair<HoodieRecord, Schema>> getMergedRecord(BufferedRecord<T> olderRecord, BufferedRecord<T> newerRecord, boolean isFinalMerge) throws IOException {
Review Comment:
I'm not sure how to add UTs for the expression payload. Currently it looks like all the existing testing is done in functional tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13699:
URL: https://github.com/apache/hudi/pull/13699#issuecomment-3171681639
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7661",
"triggerID" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"triggerType" : "PUSH"
}, {
"hash" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7666",
"triggerID" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"triggerType" : "PUSH"
}, {
"hash" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7667",
"triggerID" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7666)
* af88f9e4a548191ff0cba41db6ad3e60ba9ea43b Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7667)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13699:
URL: https://github.com/apache/hudi/pull/13699#issuecomment-3172218239
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7661",
"triggerID" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"triggerType" : "PUSH"
}, {
"hash" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7666",
"triggerID" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"triggerType" : "PUSH"
}, {
"hash" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7667",
"triggerID" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"triggerType" : "PUSH"
}, {
"hash" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7672",
"triggerID" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* af88f9e4a548191ff0cba41db6ad3e60ba9ea43b Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7667)
* 5507dfb0f3e741a539dbed0669d24dac9f1fc2b3 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7672)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13699:
URL: https://github.com/apache/hudi/pull/13699#discussion_r2265603497
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/FileGroupReaderBasedMergeHandle.java:
##########
@@ -156,13 +229,28 @@ private void init(CompactionOperation operation, String partitionPath) {
fileWriter = HoodieFileWriterFactory.getFileWriter(instantTime, newFilePath, hoodieTable.getStorage(),
config, writeSchemaWithMetaFields, taskContextSupplier, recordType);
} catch (IOException io) {
- LOG.error("Error in update task at commit {}", instantTime, io);
writeStatus.setGlobalError(io);
throw new HoodieUpsertException("Failed to initialize HoodieUpdateHandle for FileId: " + fileId + " on commit "
+ instantTime + " on path " + hoodieTable.getMetaClient().getBasePath(), io);
}
}
+ @Override
+ protected void populateIncomingRecordsMap(Iterator<HoodieRecord<T>> newRecordsItr) {
+ // no op.
+ }
+
+ /**
+ * For COW merge path, lets map the incoming records to another iterator which can be routed to {@link org.apache.hudi.common.table.read.buffer.StreamingFileGroupRecordBufferLoader}.
+ * @param newRecordsItr
+ */
+ private void populateIncomingRecordsMapIterator(Iterator<HoodieRecord<T>> newRecordsItr) {
+ if (!isCompaction) {
+ // avoid populating external spillable in base {@link HoodieWriteMergeHandle)
+ this.incomingRecordsItr = new MappingIterator<>(newRecordsItr, record -> (HoodieRecord) record);
Review Comment:
is this still needed ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13699:
URL: https://github.com/apache/hudi/pull/13699#issuecomment-3173047296
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7661",
"triggerID" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"triggerType" : "PUSH"
}, {
"hash" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7666",
"triggerID" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"triggerType" : "PUSH"
}, {
"hash" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7667",
"triggerID" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"triggerType" : "PUSH"
}, {
"hash" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7672",
"triggerID" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"triggerType" : "PUSH"
}, {
"hash" : "965136302f1188e5123de37734c927dc55801b24",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7691",
"triggerID" : "965136302f1188e5123de37734c927dc55801b24",
"triggerType" : "PUSH"
}, {
"hash" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 965136302f1188e5123de37734c927dc55801b24 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7691)
* 7dbbe8212c828dc3cf2405a0634d4a0c643580e0 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13699:
URL: https://github.com/apache/hudi/pull/13699#issuecomment-3186491916
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7661",
"triggerID" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"triggerType" : "PUSH"
}, {
"hash" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7666",
"triggerID" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"triggerType" : "PUSH"
}, {
"hash" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7667",
"triggerID" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"triggerType" : "PUSH"
}, {
"hash" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7672",
"triggerID" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"triggerType" : "PUSH"
}, {
"hash" : "965136302f1188e5123de37734c927dc55801b24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7691",
"triggerID" : "965136302f1188e5123de37734c927dc55801b24",
"triggerType" : "PUSH"
}, {
"hash" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7693",
"triggerID" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"triggerType" : "PUSH"
}, {
"hash" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7717",
"triggerID" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"triggerType" : "PUSH"
}, {
"hash" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"triggerType" : "PUSH"
}, {
"hash" : "47f8303d6e61504275ecfddf553e356427e6c822",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7718",
"triggerID" : "47f8303d6e61504275ecfddf553e356427e6c822",
"triggerType" : "PUSH"
}, {
"hash" : "fb191ecf48d13cd81412535f7f8c9ed553d86517",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7725",
"triggerID" : "fb191ecf48d13cd81412535f7f8c9ed553d86517",
"triggerType" : "PUSH"
}, {
"hash" : "064912d3a54714935f2ee77d55cc66816166300f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7727",
"triggerID" : "064912d3a54714935f2ee77d55cc66816166300f",
"triggerType" : "PUSH"
}, {
"hash" : "77a2b45fe13dafc51f7bebb2ba859d40291ccc10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7736",
"triggerID" : "77a2b45fe13dafc51f7bebb2ba859d40291ccc10",
"triggerType" : "PUSH"
}, {
"hash" : "e4eda0e51fee0e8de5d57ea7b38dc642f53aa95d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7758",
"triggerID" : "e4eda0e51fee0e8de5d57ea7b38dc642f53aa95d",
"triggerType" : "PUSH"
}, {
"hash" : "6cc840c51fcecde95d2187fb6fd217b50fa82806",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6cc840c51fcecde95d2187fb6fd217b50fa82806",
"triggerType" : "PUSH"
}, {
"hash" : "31dab8ce9abbe4fc80e6cd81a0e433a039ca3e1d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7760",
"triggerID" : "31dab8ce9abbe4fc80e6cd81a0e433a039ca3e1d",
"triggerType" : "PUSH"
}, {
"hash" : "c04fd95dcc94146a9146c347f4f204656f3d8308",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7763",
"triggerID" : "c04fd95dcc94146a9146c347f4f204656f3d8308",
"triggerType" : "PUSH"
}, {
"hash" : "324147913d47ba8f2555e1991ce59905e5dc0b3f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7767",
"triggerID" : "324147913d47ba8f2555e1991ce59905e5dc0b3f",
"triggerType" : "PUSH"
}, {
"hash" : "3f177d42b831b30f4bd590b057dcab6ded78b634",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7774",
"triggerID" : "3f177d42b831b30f4bd590b057dcab6ded78b634",
"triggerType" : "PUSH"
}, {
"hash" : "8d6d2c06ec82d1072251ac12ddf21a890ddd5bda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7776",
"triggerID" : "8d6d2c06ec82d1072251ac12ddf21a890ddd5bda",
"triggerType" : "PUSH"
}, {
"hash" : "77589738d73b044385f5aa6f7ef87b421f06043b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "77589738d73b044385f5aa6f7ef87b421f06043b",
"triggerType" : "PUSH"
}, {
"hash" : "ad0d7ef0fbc525f880d3baadef5a22196ab97b32",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7781",
"triggerID" : "ad0d7ef0fbc525f880d3baadef5a22196ab97b32",
"triggerType" : "PUSH"
}, {
"hash" : "0db72caa3a6c7d0f0899f243ea4b8975de79e02c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7787",
"triggerID" : "0db72caa3a6c7d0f0899f243ea4b8975de79e02c",
"triggerType" : "PUSH"
}, {
"hash" : "a01ce748dc2d4cd1fd4858d1ecac8a486989e880",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7797",
"triggerID" : "a01ce748dc2d4cd1fd4858d1ecac8a486989e880",
"triggerType" : "PUSH"
}, {
"hash" : "3f35151e28bfa1dd50b31c6e5379d9769627afd9",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7799",
"triggerID" : "3f35151e28bfa1dd50b31c6e5379d9769627afd9",
"triggerType" : "PUSH"
}, {
"hash" : "b0054c8fd329489d9a1e9535d9866edffc52fdd0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b0054c8fd329489d9a1e9535d9866edffc52fdd0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 50022dcc45b0f511dd69706aeaf6464a51be49e2 UNKNOWN
* 6cc840c51fcecde95d2187fb6fd217b50fa82806 UNKNOWN
* 77589738d73b044385f5aa6f7ef87b421f06043b UNKNOWN
* 3f35151e28bfa1dd50b31c6e5379d9769627afd9 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7799)
* b0054c8fd329489d9a1e9535d9866edffc52fdd0 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9622] Add implementation of MergeHandle backed by the HoodieFileGroupReader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13699:
URL: https://github.com/apache/hudi/pull/13699#issuecomment-3192564005
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7661",
"triggerID" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"triggerType" : "PUSH"
}, {
"hash" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7666",
"triggerID" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"triggerType" : "PUSH"
}, {
"hash" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7667",
"triggerID" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"triggerType" : "PUSH"
}, {
"hash" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7672",
"triggerID" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"triggerType" : "PUSH"
}, {
"hash" : "965136302f1188e5123de37734c927dc55801b24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7691",
"triggerID" : "965136302f1188e5123de37734c927dc55801b24",
"triggerType" : "PUSH"
}, {
"hash" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7693",
"triggerID" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"triggerType" : "PUSH"
}, {
"hash" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7717",
"triggerID" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"triggerType" : "PUSH"
}, {
"hash" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"triggerType" : "PUSH"
}, {
"hash" : "47f8303d6e61504275ecfddf553e356427e6c822",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7718",
"triggerID" : "47f8303d6e61504275ecfddf553e356427e6c822",
"triggerType" : "PUSH"
}, {
"hash" : "fb191ecf48d13cd81412535f7f8c9ed553d86517",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7725",
"triggerID" : "fb191ecf48d13cd81412535f7f8c9ed553d86517",
"triggerType" : "PUSH"
}, {
"hash" : "064912d3a54714935f2ee77d55cc66816166300f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7727",
"triggerID" : "064912d3a54714935f2ee77d55cc66816166300f",
"triggerType" : "PUSH"
}, {
"hash" : "77a2b45fe13dafc51f7bebb2ba859d40291ccc10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7736",
"triggerID" : "77a2b45fe13dafc51f7bebb2ba859d40291ccc10",
"triggerType" : "PUSH"
}, {
"hash" : "e4eda0e51fee0e8de5d57ea7b38dc642f53aa95d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7758",
"triggerID" : "e4eda0e51fee0e8de5d57ea7b38dc642f53aa95d",
"triggerType" : "PUSH"
}, {
"hash" : "6cc840c51fcecde95d2187fb6fd217b50fa82806",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6cc840c51fcecde95d2187fb6fd217b50fa82806",
"triggerType" : "PUSH"
}, {
"hash" : "31dab8ce9abbe4fc80e6cd81a0e433a039ca3e1d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7760",
"triggerID" : "31dab8ce9abbe4fc80e6cd81a0e433a039ca3e1d",
"triggerType" : "PUSH"
}, {
"hash" : "c04fd95dcc94146a9146c347f4f204656f3d8308",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7763",
"triggerID" : "c04fd95dcc94146a9146c347f4f204656f3d8308",
"triggerType" : "PUSH"
}, {
"hash" : "324147913d47ba8f2555e1991ce59905e5dc0b3f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7767",
"triggerID" : "324147913d47ba8f2555e1991ce59905e5dc0b3f",
"triggerType" : "PUSH"
}, {
"hash" : "3f177d42b831b30f4bd590b057dcab6ded78b634",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7774",
"triggerID" : "3f177d42b831b30f4bd590b057dcab6ded78b634",
"triggerType" : "PUSH"
}, {
"hash" : "8d6d2c06ec82d1072251ac12ddf21a890ddd5bda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7776",
"triggerID" : "8d6d2c06ec82d1072251ac12ddf21a890ddd5bda",
"triggerType" : "PUSH"
}, {
"hash" : "77589738d73b044385f5aa6f7ef87b421f06043b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "77589738d73b044385f5aa6f7ef87b421f06043b",
"triggerType" : "PUSH"
}, {
"hash" : "ad0d7ef0fbc525f880d3baadef5a22196ab97b32",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7781",
"triggerID" : "ad0d7ef0fbc525f880d3baadef5a22196ab97b32",
"triggerType" : "PUSH"
}, {
"hash" : "0db72caa3a6c7d0f0899f243ea4b8975de79e02c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7787",
"triggerID" : "0db72caa3a6c7d0f0899f243ea4b8975de79e02c",
"triggerType" : "PUSH"
}, {
"hash" : "a01ce748dc2d4cd1fd4858d1ecac8a486989e880",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7797",
"triggerID" : "a01ce748dc2d4cd1fd4858d1ecac8a486989e880",
"triggerType" : "PUSH"
}, {
"hash" : "3f35151e28bfa1dd50b31c6e5379d9769627afd9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7799",
"triggerID" : "3f35151e28bfa1dd50b31c6e5379d9769627afd9",
"triggerType" : "PUSH"
}, {
"hash" : "b0054c8fd329489d9a1e9535d9866edffc52fdd0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b0054c8fd329489d9a1e9535d9866edffc52fdd0",
"triggerType" : "PUSH"
}, {
"hash" : "363e54341420cc217bf5af62de1e09ff370f75d2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "363e54341420cc217bf5af62de1e09ff370f75d2",
"triggerType" : "PUSH"
}, {
"hash" : "a893da375eb9c65edabbd22965296eac9b3714f4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a893da375eb9c65edabbd22965296eac9b3714f4",
"triggerType" : "PUSH"
}, {
"hash" : "a7c2756e6a293cbba99bd4e821881e6b961c138f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7804",
"triggerID" : "a7c2756e6a293cbba99bd4e821881e6b961c138f",
"triggerType" : "PUSH"
}, {
"hash" : "4a140fffa35893a13f2e7ac8365d60d51a2d0870",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7813",
"triggerID" : "4a140fffa35893a13f2e7ac8365d60d51a2d0870",
"triggerType" : "PUSH"
}, {
"hash" : "500b4f529ad2f8a695825b5badc1815202164d0a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7830",
"triggerID" : "500b4f529ad2f8a695825b5badc1815202164d0a",
"triggerType" : "PUSH"
}, {
"hash" : "539cf8b9cdc7ab4e70488e4096684022a027f942",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "539cf8b9cdc7ab4e70488e4096684022a027f942",
"triggerType" : "PUSH"
}, {
"hash" : "85139a12b431931973c07d1b7f9256a1aae75669",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7831",
"triggerID" : "85139a12b431931973c07d1b7f9256a1aae75669",
"triggerType" : "PUSH"
}, {
"hash" : "cc430c949b539450110da96c0eccb52ab5e0998e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7832",
"triggerID" : "cc430c949b539450110da96c0eccb52ab5e0998e",
"triggerType" : "PUSH"
}, {
"hash" : "2a9cad38ac9c86ddacbb2de06950d515b3a1b0df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7850",
"triggerID" : "2a9cad38ac9c86ddacbb2de06950d515b3a1b0df",
"triggerType" : "PUSH"
}, {
"hash" : "64d3575067a762fabfede759147da73ef3df0d1a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7852",
"triggerID" : "64d3575067a762fabfede759147da73ef3df0d1a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 50022dcc45b0f511dd69706aeaf6464a51be49e2 UNKNOWN
* 6cc840c51fcecde95d2187fb6fd217b50fa82806 UNKNOWN
* 77589738d73b044385f5aa6f7ef87b421f06043b UNKNOWN
* b0054c8fd329489d9a1e9535d9866edffc52fdd0 UNKNOWN
* 363e54341420cc217bf5af62de1e09ff370f75d2 UNKNOWN
* a893da375eb9c65edabbd22965296eac9b3714f4 UNKNOWN
* 539cf8b9cdc7ab4e70488e4096684022a027f942 UNKNOWN
* 64d3575067a762fabfede759147da73ef3df0d1a Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7852)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9622] Add implementation of MergeHandle backed by the HoodieFileGroupReader [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13699:
URL: https://github.com/apache/hudi/pull/13699#discussion_r2277909608
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/table/action/commit/BaseSparkCommitActionExecutor.java:
##########
@@ -397,6 +401,9 @@ protected HoodieMergeHandle getUpdateHandle(String partitionPath, String fileId,
mergeHandle.setPartitionFields(partitionFields);
mergeHandle.setPartitionValues(partitionValues);
}
+ if (readerContextFactory != null && mergeHandle instanceof FileGroupReaderBasedMergeHandle) {
Review Comment:
I did this before but @danny0405 reverted it in favor of keeping the args consistent since we rely on reflection.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13699:
URL: https://github.com/apache/hudi/pull/13699#issuecomment-3184570882
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7661",
"triggerID" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"triggerType" : "PUSH"
}, {
"hash" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7666",
"triggerID" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"triggerType" : "PUSH"
}, {
"hash" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7667",
"triggerID" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"triggerType" : "PUSH"
}, {
"hash" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7672",
"triggerID" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"triggerType" : "PUSH"
}, {
"hash" : "965136302f1188e5123de37734c927dc55801b24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7691",
"triggerID" : "965136302f1188e5123de37734c927dc55801b24",
"triggerType" : "PUSH"
}, {
"hash" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7693",
"triggerID" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"triggerType" : "PUSH"
}, {
"hash" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7717",
"triggerID" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"triggerType" : "PUSH"
}, {
"hash" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"triggerType" : "PUSH"
}, {
"hash" : "47f8303d6e61504275ecfddf553e356427e6c822",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7718",
"triggerID" : "47f8303d6e61504275ecfddf553e356427e6c822",
"triggerType" : "PUSH"
}, {
"hash" : "fb191ecf48d13cd81412535f7f8c9ed553d86517",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7725",
"triggerID" : "fb191ecf48d13cd81412535f7f8c9ed553d86517",
"triggerType" : "PUSH"
}, {
"hash" : "064912d3a54714935f2ee77d55cc66816166300f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7727",
"triggerID" : "064912d3a54714935f2ee77d55cc66816166300f",
"triggerType" : "PUSH"
}, {
"hash" : "77a2b45fe13dafc51f7bebb2ba859d40291ccc10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7736",
"triggerID" : "77a2b45fe13dafc51f7bebb2ba859d40291ccc10",
"triggerType" : "PUSH"
}, {
"hash" : "e4eda0e51fee0e8de5d57ea7b38dc642f53aa95d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7758",
"triggerID" : "e4eda0e51fee0e8de5d57ea7b38dc642f53aa95d",
"triggerType" : "PUSH"
}, {
"hash" : "6cc840c51fcecde95d2187fb6fd217b50fa82806",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6cc840c51fcecde95d2187fb6fd217b50fa82806",
"triggerType" : "PUSH"
}, {
"hash" : "31dab8ce9abbe4fc80e6cd81a0e433a039ca3e1d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7760",
"triggerID" : "31dab8ce9abbe4fc80e6cd81a0e433a039ca3e1d",
"triggerType" : "PUSH"
}, {
"hash" : "c04fd95dcc94146a9146c347f4f204656f3d8308",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7763",
"triggerID" : "c04fd95dcc94146a9146c347f4f204656f3d8308",
"triggerType" : "PUSH"
}, {
"hash" : "324147913d47ba8f2555e1991ce59905e5dc0b3f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7767",
"triggerID" : "324147913d47ba8f2555e1991ce59905e5dc0b3f",
"triggerType" : "PUSH"
}, {
"hash" : "3f177d42b831b30f4bd590b057dcab6ded78b634",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7774",
"triggerID" : "3f177d42b831b30f4bd590b057dcab6ded78b634",
"triggerType" : "PUSH"
}, {
"hash" : "8d6d2c06ec82d1072251ac12ddf21a890ddd5bda",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7776",
"triggerID" : "8d6d2c06ec82d1072251ac12ddf21a890ddd5bda",
"triggerType" : "PUSH"
}, {
"hash" : "77589738d73b044385f5aa6f7ef87b421f06043b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "77589738d73b044385f5aa6f7ef87b421f06043b",
"triggerType" : "PUSH"
}, {
"hash" : "ad0d7ef0fbc525f880d3baadef5a22196ab97b32",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ad0d7ef0fbc525f880d3baadef5a22196ab97b32",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 50022dcc45b0f511dd69706aeaf6464a51be49e2 UNKNOWN
* 6cc840c51fcecde95d2187fb6fd217b50fa82806 UNKNOWN
* 8d6d2c06ec82d1072251ac12ddf21a890ddd5bda Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7776)
* 77589738d73b044385f5aa6f7ef87b421f06043b UNKNOWN
* ad0d7ef0fbc525f880d3baadef5a22196ab97b32 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13699:
URL: https://github.com/apache/hudi/pull/13699#discussion_r2276432030
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/buffer/FileGroupRecordBuffer.java:
##########
@@ -56,27 +53,29 @@
import java.io.IOException;
import java.io.Serializable;
+import java.util.HashSet;
import java.util.Iterator;
import java.util.List;
import java.util.Locale;
import java.util.Map;
+import java.util.Set;
import java.util.function.Function;
import static org.apache.hudi.common.config.HoodieCommonConfig.DISK_MAP_BITCASK_COMPRESSION_ENABLED;
import static org.apache.hudi.common.config.HoodieCommonConfig.SPILLABLE_DISK_MAP_TYPE;
import static org.apache.hudi.common.config.HoodieMemoryConfig.MAX_MEMORY_FOR_MERGE;
import static org.apache.hudi.common.config.HoodieMemoryConfig.SPILLABLE_MAP_BASE_PATH;
-import static org.apache.hudi.common.model.HoodieRecordMerger.PAYLOAD_BASED_MERGE_STRATEGY_UUID;
import static org.apache.hudi.common.table.log.block.HoodieLogBlock.HeaderMetadataType.INSTANT_TIME;
abstract class FileGroupRecordBuffer<T> implements HoodieFileGroupRecordBuffer<T> {
+ protected final Set<String> usedKeys = new HashSet<>();
Review Comment:
@danny0405 there is one more case that fails without this change: `testUpsertWithoutPrecombineFieldAndCombineBeforeUpsertDisabled` - I am marking it as disabled for now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9622] Add implementation of MergeHandle backed by the HoodieFileGroupReader [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13699:
URL: https://github.com/apache/hudi/pull/13699#discussion_r2279577119
##########
hudi-client/hudi-client-common/src/test/java/org/apache/hudi/testutils/MetadataMergeWriteStatus.java:
##########
@@ -60,12 +60,12 @@ private static void mergeMetadataMaps(Map<String, String> mergeFromMap, Map<Stri
if (!mergeToMap.containsKey(key)) {
mergeToMap.put(key, "0");
}
- mergeToMap.put(key, addStrsAsInt(entry.getValue(), mergeToMap.get(key)));
+ mergeToMap.put(key, addStrsAsLong(entry.getValue(), mergeToMap.get(key)));
}
}
- private static String addStrsAsInt(String a, String b) {
- return String.valueOf(Integer.parseInt(a) + Integer.parseInt(b));
+ private static String addStrsAsLong(String a, String b) {
+ return String.valueOf(Long.parseLong(a) + Long.parseLong(b));
Review Comment:
Right, this is a test class so it may cause some test failures
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9622] Add implementation of MergeHandle backed by the HoodieFileGroupReader [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan merged PR #13699:
URL: https://github.com/apache/hudi/pull/13699
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13699:
URL: https://github.com/apache/hudi/pull/13699#discussion_r2275047264
##########
hudi-common/src/main/java/org/apache/hudi/internal/schema/utils/InternalSchemaUtils.java:
##########
@@ -57,15 +62,16 @@ public static InternalSchema pruneInternalSchema(InternalSchema schema, List<Str
List<Integer> prunedIds = names.stream().map(name -> {
int id = schema.findIdByName(name);
if (id == -1) {
- throw new IllegalArgumentException(String.format("cannot prune col: %s which does not exist in hudi table", name));
+ LOG.debug("Cannot find field name {} in schema {}, skipping.", name, schema);
+ return null;
}
return id;
- }).collect(Collectors.toList());
+ }).filter(Objects::nonNull).collect(Collectors.toList());
Review Comment:
If the internal schema has less fields than the requested schema. In the writer path, the writer schema can have new columns that are not in the in file's existing schema.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9622] Add implementation of MergeHandle backed by the HoodieFileGroupReader [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13699:
URL: https://github.com/apache/hudi/pull/13699#discussion_r2277992667
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/buffer/PositionBasedFileGroupRecordBuffer.java:
##########
@@ -203,7 +203,7 @@ public void processDeleteBlock(HoodieDeleteBlock deleteBlock) throws IOException
// this delete-vector could be kept in the records cache(see the check in #fallbackToKeyBasedBuffer),
// and these keys would be deleted no matter whether there are following-up inserts/updates.
DeleteRecord deleteRecord = deleteRecords[commitTimeBasedRecordIndex++];
- BufferedRecord<T> record = BufferedRecords.fromDeleteRecord(deleteRecord, deleteRecord.getOrderingValue());
+ BufferedRecord<T> record = BufferedRecords.fromDeleteRecord(deleteRecord, readerContext.getRecordContext());
Review Comment:
Updated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9622] Add implementation of MergeHandle backed by the HoodieFileGroupReader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13699:
URL: https://github.com/apache/hudi/pull/13699#issuecomment-3190482159
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7661",
"triggerID" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"triggerType" : "PUSH"
}, {
"hash" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7666",
"triggerID" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"triggerType" : "PUSH"
}, {
"hash" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7667",
"triggerID" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"triggerType" : "PUSH"
}, {
"hash" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7672",
"triggerID" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"triggerType" : "PUSH"
}, {
"hash" : "965136302f1188e5123de37734c927dc55801b24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7691",
"triggerID" : "965136302f1188e5123de37734c927dc55801b24",
"triggerType" : "PUSH"
}, {
"hash" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7693",
"triggerID" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"triggerType" : "PUSH"
}, {
"hash" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7717",
"triggerID" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"triggerType" : "PUSH"
}, {
"hash" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"triggerType" : "PUSH"
}, {
"hash" : "47f8303d6e61504275ecfddf553e356427e6c822",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7718",
"triggerID" : "47f8303d6e61504275ecfddf553e356427e6c822",
"triggerType" : "PUSH"
}, {
"hash" : "fb191ecf48d13cd81412535f7f8c9ed553d86517",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7725",
"triggerID" : "fb191ecf48d13cd81412535f7f8c9ed553d86517",
"triggerType" : "PUSH"
}, {
"hash" : "064912d3a54714935f2ee77d55cc66816166300f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7727",
"triggerID" : "064912d3a54714935f2ee77d55cc66816166300f",
"triggerType" : "PUSH"
}, {
"hash" : "77a2b45fe13dafc51f7bebb2ba859d40291ccc10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7736",
"triggerID" : "77a2b45fe13dafc51f7bebb2ba859d40291ccc10",
"triggerType" : "PUSH"
}, {
"hash" : "e4eda0e51fee0e8de5d57ea7b38dc642f53aa95d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7758",
"triggerID" : "e4eda0e51fee0e8de5d57ea7b38dc642f53aa95d",
"triggerType" : "PUSH"
}, {
"hash" : "6cc840c51fcecde95d2187fb6fd217b50fa82806",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6cc840c51fcecde95d2187fb6fd217b50fa82806",
"triggerType" : "PUSH"
}, {
"hash" : "31dab8ce9abbe4fc80e6cd81a0e433a039ca3e1d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7760",
"triggerID" : "31dab8ce9abbe4fc80e6cd81a0e433a039ca3e1d",
"triggerType" : "PUSH"
}, {
"hash" : "c04fd95dcc94146a9146c347f4f204656f3d8308",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7763",
"triggerID" : "c04fd95dcc94146a9146c347f4f204656f3d8308",
"triggerType" : "PUSH"
}, {
"hash" : "324147913d47ba8f2555e1991ce59905e5dc0b3f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7767",
"triggerID" : "324147913d47ba8f2555e1991ce59905e5dc0b3f",
"triggerType" : "PUSH"
}, {
"hash" : "3f177d42b831b30f4bd590b057dcab6ded78b634",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7774",
"triggerID" : "3f177d42b831b30f4bd590b057dcab6ded78b634",
"triggerType" : "PUSH"
}, {
"hash" : "8d6d2c06ec82d1072251ac12ddf21a890ddd5bda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7776",
"triggerID" : "8d6d2c06ec82d1072251ac12ddf21a890ddd5bda",
"triggerType" : "PUSH"
}, {
"hash" : "77589738d73b044385f5aa6f7ef87b421f06043b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "77589738d73b044385f5aa6f7ef87b421f06043b",
"triggerType" : "PUSH"
}, {
"hash" : "ad0d7ef0fbc525f880d3baadef5a22196ab97b32",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7781",
"triggerID" : "ad0d7ef0fbc525f880d3baadef5a22196ab97b32",
"triggerType" : "PUSH"
}, {
"hash" : "0db72caa3a6c7d0f0899f243ea4b8975de79e02c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7787",
"triggerID" : "0db72caa3a6c7d0f0899f243ea4b8975de79e02c",
"triggerType" : "PUSH"
}, {
"hash" : "a01ce748dc2d4cd1fd4858d1ecac8a486989e880",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7797",
"triggerID" : "a01ce748dc2d4cd1fd4858d1ecac8a486989e880",
"triggerType" : "PUSH"
}, {
"hash" : "3f35151e28bfa1dd50b31c6e5379d9769627afd9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7799",
"triggerID" : "3f35151e28bfa1dd50b31c6e5379d9769627afd9",
"triggerType" : "PUSH"
}, {
"hash" : "b0054c8fd329489d9a1e9535d9866edffc52fdd0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b0054c8fd329489d9a1e9535d9866edffc52fdd0",
"triggerType" : "PUSH"
}, {
"hash" : "363e54341420cc217bf5af62de1e09ff370f75d2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "363e54341420cc217bf5af62de1e09ff370f75d2",
"triggerType" : "PUSH"
}, {
"hash" : "a893da375eb9c65edabbd22965296eac9b3714f4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a893da375eb9c65edabbd22965296eac9b3714f4",
"triggerType" : "PUSH"
}, {
"hash" : "a7c2756e6a293cbba99bd4e821881e6b961c138f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7804",
"triggerID" : "a7c2756e6a293cbba99bd4e821881e6b961c138f",
"triggerType" : "PUSH"
}, {
"hash" : "4a140fffa35893a13f2e7ac8365d60d51a2d0870",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7813",
"triggerID" : "4a140fffa35893a13f2e7ac8365d60d51a2d0870",
"triggerType" : "PUSH"
}, {
"hash" : "500b4f529ad2f8a695825b5badc1815202164d0a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7830",
"triggerID" : "500b4f529ad2f8a695825b5badc1815202164d0a",
"triggerType" : "PUSH"
}, {
"hash" : "539cf8b9cdc7ab4e70488e4096684022a027f942",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "539cf8b9cdc7ab4e70488e4096684022a027f942",
"triggerType" : "PUSH"
}, {
"hash" : "85139a12b431931973c07d1b7f9256a1aae75669",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7831",
"triggerID" : "85139a12b431931973c07d1b7f9256a1aae75669",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 50022dcc45b0f511dd69706aeaf6464a51be49e2 UNKNOWN
* 6cc840c51fcecde95d2187fb6fd217b50fa82806 UNKNOWN
* 77589738d73b044385f5aa6f7ef87b421f06043b UNKNOWN
* b0054c8fd329489d9a1e9535d9866edffc52fdd0 UNKNOWN
* 363e54341420cc217bf5af62de1e09ff370f75d2 UNKNOWN
* a893da375eb9c65edabbd22965296eac9b3714f4 UNKNOWN
* 539cf8b9cdc7ab4e70488e4096684022a027f942 UNKNOWN
* 85139a12b431931973c07d1b7f9256a1aae75669 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7831)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13699:
URL: https://github.com/apache/hudi/pull/13699#discussion_r2273308281
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/buffer/FileGroupRecordBuffer.java:
##########
@@ -56,27 +53,29 @@
import java.io.IOException;
import java.io.Serializable;
+import java.util.HashSet;
import java.util.Iterator;
import java.util.List;
import java.util.Locale;
import java.util.Map;
+import java.util.Set;
import java.util.function.Function;
import static org.apache.hudi.common.config.HoodieCommonConfig.DISK_MAP_BITCASK_COMPRESSION_ENABLED;
import static org.apache.hudi.common.config.HoodieCommonConfig.SPILLABLE_DISK_MAP_TYPE;
import static org.apache.hudi.common.config.HoodieMemoryConfig.MAX_MEMORY_FOR_MERGE;
import static org.apache.hudi.common.config.HoodieMemoryConfig.SPILLABLE_MAP_BASE_PATH;
-import static org.apache.hudi.common.model.HoodieRecordMerger.PAYLOAD_BASED_MERGE_STRATEGY_UUID;
import static org.apache.hudi.common.table.log.block.HoodieLogBlock.HeaderMetadataType.INSTANT_TIME;
abstract class FileGroupRecordBuffer<T> implements HoodieFileGroupRecordBuffer<T> {
+ protected final Set<String> usedKeys = new HashSet<>();
Review Comment:
For the test case that fails without this, should we just update it to be PK-less?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13699:
URL: https://github.com/apache/hudi/pull/13699#discussion_r2273653510
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/HoodieTable.java:
##########
@@ -150,6 +151,7 @@ public abstract class HoodieTable<T, I, K, O> implements Serializable {
private transient FileSystemViewManager viewManager;
protected final transient HoodieEngineContext context;
+ private final ReaderContextFactory<T> readerContextFactoryForWrite;
Review Comment:
we can remove the reader context factory in hoodie table, and instantiate a reader context factory in `SparkUpsertCommitActionExecutor`, in `SparkUpsertCommitActionExecutor`, override the method `BaseSparkCommitActionExecutor#getUpdateHandle` to set up the reader context explicitly, may also need to add a set method in `FileGroupReaderBasedMergeHandle`, only this handle needs this currently. Just like these 2 `HoodieMergeHandle.setPartitionFields` and `HoodieMergeHandle.setPartitionValues` which are only specific for Spark COW table.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9622] Add implementation of MergeHandle backed by the HoodieFileGroupReader [hudi]
Posted by "cshuo (via GitHub)" <gi...@apache.org>.
cshuo commented on code in PR #13699:
URL: https://github.com/apache/hudi/pull/13699#discussion_r2278085687
##########
hudi-common/src/main/java/org/apache/hudi/common/model/HoodieRecordPayload.java:
##########
@@ -217,4 +217,16 @@ static Option<String> getPayloadClassNameIfPresent(Properties props) {
// Need to transparently change to org.apache.hudi.
return Option.ofNullable(payloadClassName).map(className -> className.replace("com.uber.hoodie", "org.apache.hudi"));
}
+
+ /**
+ * Returns the writer payload override class if present in the properties.
+ * @param properties the properties to inspect
+ * @return an Option containing the writer payload override class name if present, otherwise an empty Option
+ */
+ static Option<String> getWriterPayloadOverride(Properties properties) {
+ if (properties.containsKey("hoodie.datasource.write.payload.class")) {
Review Comment:
`if (containsKey()..) {Option.of()} else {}` -> `Option.ofNullable()` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9622] Add implementation of MergeHandle backed by the HoodieFileGroupReader [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13699:
URL: https://github.com/apache/hudi/pull/13699#discussion_r2280689556
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/FileGroupReaderBasedMergeHandle.java:
##########
@@ -198,8 +270,12 @@ public void doMerge() {
}
// Writes the record
try {
- writeToFile(record.getKey(), record, writeSchemaWithMetaFields,
- config.getPayloadConfig().getProps(), preserveMetadata);
+ // For Compaction operations, the preserveMetadata flag is always true as we want to preserve the existing record metadata.
+ // For other updates, we only want to preserve the metadata if the record is not being modified by this update. If the record already exists in the base file and is not updated,
+ // the operation will be null. Records that are being updated or records being added to the file group for the first time will have an operation set and must generate new metadata.
+ boolean shouldPreserveRecordMetadata = preserveMetadata || record.getOperation() == null;
+ Schema recordSchema = shouldPreserveRecordMetadata ? writeSchemaWithMetaFields : writeSchema;
+ writeToFile(record.getKey(), record, recordSchema, config.getPayloadConfig().getProps(), shouldPreserveRecordMetadata);
Review Comment:
Create this JIRA [ticket](https://issues.apache.org/jira/browse/HUDI-9724) to track and will take it up in the next day or so
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13699:
URL: https://github.com/apache/hudi/pull/13699#discussion_r2275005344
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/UpdateProcessor.java:
##########
@@ -80,16 +106,79 @@ public BufferedRecord<T> processUpdate(String recordKey, BufferedRecord<T> previ
}
return null;
} else {
- T prevRow = previousRecord != null ? previousRecord.getRecord() : null;
- T mergedRow = mergedRecord.getRecord();
- if (prevRow != null && prevRow != mergedRow) {
- mergedRecord.setHoodieOperation(HoodieOperation.UPDATE_AFTER);
- readStats.incrementNumUpdates();
- } else if (prevRow == null) {
- mergedRecord.setHoodieOperation(HoodieOperation.INSERT);
- readStats.incrementNumInserts();
+ return handleNonDeletes(previousRecord, mergedRecord);
+ }
+ }
+
+ protected BufferedRecord<T> handleNonDeletes(BufferedRecord<T> previousRecord, BufferedRecord<T> mergedRecord) {
+ T prevRow = previousRecord != null ? previousRecord.getRecord() : null;
+ T mergedRow = mergedRecord.getRecord();
+ if (prevRow != null && prevRow != mergedRow) {
+ mergedRecord.setHoodieOperation(HoodieOperation.UPDATE_AFTER);
+ readStats.incrementNumUpdates();
+ } else if (prevRow == null) {
+ mergedRecord.setHoodieOperation(HoodieOperation.INSERT);
+ readStats.incrementNumInserts();
+ }
+ return mergedRecord.seal(readerContext);
+ }
+ }
+
+ class PayloadUpdateProcessor<T> extends StandardUpdateProcessor<T> {
+ private final String payloadClass;
+ private final Properties properties;
+
+ public PayloadUpdateProcessor(HoodieReadStats readStats, HoodieReaderContext<T> readerContext, boolean emitDeletes,
+ Properties properties, String payloadClass) {
+ super(readStats, readerContext, emitDeletes);
+ this.payloadClass = payloadClass;
+ this.properties = properties;
+ }
+
+ @Override
+ protected BufferedRecord<T> handleNonDeletes(BufferedRecord<T> previousRecord, BufferedRecord<T> mergedRecord) {
+ if (previousRecord == null) {
+ // special case for payloads when there is no previous record
+ Schema recordSchema = readerContext.getRecordContext().decodeAvroSchema(mergedRecord.getSchemaId());
+ GenericRecord record = readerContext.getRecordContext().convertToAvroRecord(mergedRecord.getRecord(), recordSchema);
+ HoodieAvroRecord hoodieRecord = new HoodieAvroRecord<>(null, HoodieRecordUtils.loadPayload(payloadClass, record, mergedRecord.getOrderingValue()));
+ try {
+ if (hoodieRecord.shouldIgnore(recordSchema, properties)) {
+ return null;
+ } else {
+ Schema readerSchema = readerContext.getSchemaHandler().getRequestedSchema();
+ // If the record schema is different from the reader schema, rewrite the record using the payload methods to ensure consistency with legacy writer paths
+ hoodieRecord.rewriteRecordWithNewSchema(recordSchema, properties, readerSchema).toIndexedRecord(readerSchema, properties)
+ .ifPresent(rewrittenRecord -> mergedRecord.replaceRecord(readerContext.getRecordContext().convertAvroRecord(rewrittenRecord.getData())));
+ }
+ } catch (IOException e) {
+ throw new HoodieIOException("Error processing record with payload class: " + payloadClass, e);
+ }
+ }
+ return super.handleNonDeletes(previousRecord, mergedRecord);
+ }
+ }
+
+ class CustomMergerUpdateProcessor<T> extends StandardUpdateProcessor<T> {
+ private final HoodieRecordMerger merger;
+ private final TypedProperties properties;
+
+ CustomMergerUpdateProcessor(HoodieReadStats readStats, HoodieReaderContext<T> readerContext, boolean emitDeletes,
+ TypedProperties properties) {
+ super(readStats, readerContext, emitDeletes);
+ this.merger = readerContext.getRecordMerger().get();
+ this.properties = properties;
+ }
+
+ @Override
+ protected BufferedRecord<T> handleNonDeletes(BufferedRecord<T> previousRecord, BufferedRecord<T> mergedRecord) {
+ try {
+ if (merger.shouldFlush(readerContext.getRecordContext().constructHoodieRecord(mergedRecord), readerContext.getRecordContext().getSchemaFromBufferRecord(mergedRecord), properties)) {
Review Comment:
skip the whole tests first and fix it in HUDI-9709
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9622] Add implementation of MergeHandle backed by the HoodieFileGroupReader [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13699:
URL: https://github.com/apache/hudi/pull/13699#discussion_r2278043646
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/table/action/commit/BaseSparkCommitActionExecutor.java:
##########
@@ -397,6 +401,9 @@ protected HoodieMergeHandle getUpdateHandle(String partitionPath, String fileId,
mergeHandle.setPartitionFields(partitionFields);
mergeHandle.setPartitionValues(partitionValues);
}
+ if (readerContextFactory != null && mergeHandle instanceof FileGroupReaderBasedMergeHandle) {
Review Comment:
We do it for these reasons:
1. the param is only needed for the FG merge handle now, we don't want to init the reader context for all the merge handles
2. also modifying the merge handle param will need to modify all the executors, which is also unnecessary, like Java and Flink can fetch the reader context directly from hoodie table.
3. thus, we want to constraint the explicit reader contex for writer path to just "upsert" executors of Spark so that all the other executors are not affected
4. we already did some special handling like set partition fields and values for update handle of Spark, set reader context is another case here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9622] Add implementation of MergeHandle backed by the HoodieFileGroupReader [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13699:
URL: https://github.com/apache/hudi/pull/13699#discussion_r2280080203
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/FileGroupReaderBasedMergeHandle.java:
##########
@@ -198,8 +270,12 @@ public void doMerge() {
}
// Writes the record
try {
- writeToFile(record.getKey(), record, writeSchemaWithMetaFields,
- config.getPayloadConfig().getProps(), preserveMetadata);
+ // For Compaction operations, the preserveMetadata flag is always true as we want to preserve the existing record metadata.
+ // For other updates, we only want to preserve the metadata if the record is not being modified by this update. If the record already exists in the base file and is not updated,
+ // the operation will be null. Records that are being updated or records being added to the file group for the first time will have an operation set and must generate new metadata.
+ boolean shouldPreserveRecordMetadata = preserveMetadata || record.getOperation() == null;
+ Schema recordSchema = shouldPreserveRecordMetadata ? writeSchemaWithMetaFields : writeSchema;
+ writeToFile(record.getKey(), record, recordSchema, config.getPayloadConfig().getProps(), shouldPreserveRecordMetadata);
Review Comment:
Currently we collect SI and RLI no matter wheter the record has been written successfully, this could incur inconsistencies.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13699:
URL: https://github.com/apache/hudi/pull/13699#discussion_r2271579426
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/FileGroupReaderBasedMergeHandle.java:
##########
@@ -106,7 +171,15 @@ public FileGroupReaderBasedMergeHandle(HoodieWriteConfig config, String instantT
} else {
this.cdcLogger = Option.empty();
}
- init(operation, this.partitionPath);
+ }
+
+ private void initRecordIndexCallback() {
+ if (this.writeStatus.isTrackingSuccessfulWrites()) {
Review Comment:
Yes, this is no longer called from the compaction path
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13699:
URL: https://github.com/apache/hudi/pull/13699#issuecomment-3182084630
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7661",
"triggerID" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"triggerType" : "PUSH"
}, {
"hash" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7666",
"triggerID" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"triggerType" : "PUSH"
}, {
"hash" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7667",
"triggerID" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"triggerType" : "PUSH"
}, {
"hash" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7672",
"triggerID" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"triggerType" : "PUSH"
}, {
"hash" : "965136302f1188e5123de37734c927dc55801b24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7691",
"triggerID" : "965136302f1188e5123de37734c927dc55801b24",
"triggerType" : "PUSH"
}, {
"hash" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7693",
"triggerID" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"triggerType" : "PUSH"
}, {
"hash" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7717",
"triggerID" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"triggerType" : "PUSH"
}, {
"hash" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"triggerType" : "PUSH"
}, {
"hash" : "47f8303d6e61504275ecfddf553e356427e6c822",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7718",
"triggerID" : "47f8303d6e61504275ecfddf553e356427e6c822",
"triggerType" : "PUSH"
}, {
"hash" : "fb191ecf48d13cd81412535f7f8c9ed553d86517",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7725",
"triggerID" : "fb191ecf48d13cd81412535f7f8c9ed553d86517",
"triggerType" : "PUSH"
}, {
"hash" : "064912d3a54714935f2ee77d55cc66816166300f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7727",
"triggerID" : "064912d3a54714935f2ee77d55cc66816166300f",
"triggerType" : "PUSH"
}, {
"hash" : "77a2b45fe13dafc51f7bebb2ba859d40291ccc10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7736",
"triggerID" : "77a2b45fe13dafc51f7bebb2ba859d40291ccc10",
"triggerType" : "PUSH"
}, {
"hash" : "e4eda0e51fee0e8de5d57ea7b38dc642f53aa95d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7758",
"triggerID" : "e4eda0e51fee0e8de5d57ea7b38dc642f53aa95d",
"triggerType" : "PUSH"
}, {
"hash" : "6cc840c51fcecde95d2187fb6fd217b50fa82806",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6cc840c51fcecde95d2187fb6fd217b50fa82806",
"triggerType" : "PUSH"
}, {
"hash" : "31dab8ce9abbe4fc80e6cd81a0e433a039ca3e1d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7760",
"triggerID" : "31dab8ce9abbe4fc80e6cd81a0e433a039ca3e1d",
"triggerType" : "PUSH"
}, {
"hash" : "c04fd95dcc94146a9146c347f4f204656f3d8308",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7763",
"triggerID" : "c04fd95dcc94146a9146c347f4f204656f3d8308",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 50022dcc45b0f511dd69706aeaf6464a51be49e2 UNKNOWN
* 6cc840c51fcecde95d2187fb6fd217b50fa82806 UNKNOWN
* c04fd95dcc94146a9146c347f4f204656f3d8308 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7763)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13699:
URL: https://github.com/apache/hudi/pull/13699#issuecomment-3176261277
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7661",
"triggerID" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"triggerType" : "PUSH"
}, {
"hash" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7666",
"triggerID" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"triggerType" : "PUSH"
}, {
"hash" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7667",
"triggerID" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"triggerType" : "PUSH"
}, {
"hash" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7672",
"triggerID" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"triggerType" : "PUSH"
}, {
"hash" : "965136302f1188e5123de37734c927dc55801b24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7691",
"triggerID" : "965136302f1188e5123de37734c927dc55801b24",
"triggerType" : "PUSH"
}, {
"hash" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7693",
"triggerID" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"triggerType" : "PUSH"
}, {
"hash" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7717",
"triggerID" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"triggerType" : "PUSH"
}, {
"hash" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"triggerType" : "PUSH"
}, {
"hash" : "47f8303d6e61504275ecfddf553e356427e6c822",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7718",
"triggerID" : "47f8303d6e61504275ecfddf553e356427e6c822",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 50022dcc45b0f511dd69706aeaf6464a51be49e2 UNKNOWN
* 47f8303d6e61504275ecfddf553e356427e6c822 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7718)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13699:
URL: https://github.com/apache/hudi/pull/13699#issuecomment-3181778771
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7661",
"triggerID" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"triggerType" : "PUSH"
}, {
"hash" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7666",
"triggerID" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"triggerType" : "PUSH"
}, {
"hash" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7667",
"triggerID" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"triggerType" : "PUSH"
}, {
"hash" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7672",
"triggerID" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"triggerType" : "PUSH"
}, {
"hash" : "965136302f1188e5123de37734c927dc55801b24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7691",
"triggerID" : "965136302f1188e5123de37734c927dc55801b24",
"triggerType" : "PUSH"
}, {
"hash" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7693",
"triggerID" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"triggerType" : "PUSH"
}, {
"hash" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7717",
"triggerID" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"triggerType" : "PUSH"
}, {
"hash" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"triggerType" : "PUSH"
}, {
"hash" : "47f8303d6e61504275ecfddf553e356427e6c822",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7718",
"triggerID" : "47f8303d6e61504275ecfddf553e356427e6c822",
"triggerType" : "PUSH"
}, {
"hash" : "fb191ecf48d13cd81412535f7f8c9ed553d86517",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7725",
"triggerID" : "fb191ecf48d13cd81412535f7f8c9ed553d86517",
"triggerType" : "PUSH"
}, {
"hash" : "064912d3a54714935f2ee77d55cc66816166300f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7727",
"triggerID" : "064912d3a54714935f2ee77d55cc66816166300f",
"triggerType" : "PUSH"
}, {
"hash" : "77a2b45fe13dafc51f7bebb2ba859d40291ccc10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7736",
"triggerID" : "77a2b45fe13dafc51f7bebb2ba859d40291ccc10",
"triggerType" : "PUSH"
}, {
"hash" : "e4eda0e51fee0e8de5d57ea7b38dc642f53aa95d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7758",
"triggerID" : "e4eda0e51fee0e8de5d57ea7b38dc642f53aa95d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 50022dcc45b0f511dd69706aeaf6464a51be49e2 UNKNOWN
* e4eda0e51fee0e8de5d57ea7b38dc642f53aa95d Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7758)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13699:
URL: https://github.com/apache/hudi/pull/13699#discussion_r2265590896
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/FileGroupReaderBasedMergeHandle.java:
##########
@@ -75,13 +85,60 @@
public class FileGroupReaderBasedMergeHandle<T, I, K, O> extends HoodieWriteMergeHandle<T, I, K, O> {
private static final Logger LOG = LoggerFactory.getLogger(FileGroupReaderBasedMergeHandle.class);
- private final HoodieReaderContext<T> readerContext;
- private final CompactionOperation operation;
+ private final Option<CompactionOperation> operation;
private final String maxInstantTime;
+ private HoodieReaderContext<T> readerContext;
private HoodieReadStats readStats;
- private final HoodieRecord.HoodieRecordType recordType;
- private final Option<HoodieCDCLogger> cdcLogger;
+ private HoodieRecord.HoodieRecordType recordType;
+ private Option<HoodieCDCLogger> cdcLogger;
+ private Option<RecordLevelIndexCallback> recordIndexCallbackOpt;
+ private Option<SecondaryIndexCallback> secondaryIndexCallbackOpt;
+ private final boolean isCompaction;
+ private final TypedProperties props;
+ private Iterator<HoodieRecord> incomingRecordsItr;
+ /**
+ * Constructor for Copy-On-Write (COW) merge path.
+ * Takes in a base path and an iterator of records to be merged with that file.
+ * @param config instance of {@link HoodieWriteConfig} to use.
+ * @param instantTime instant time of the current commit.
+ * @param hoodieTable instance of {@link HoodieTable} being updated.
+ * @param recordItr iterator of records to be merged with the file.
+ * @param partitionPath partition path of the base file.
+ * @param fileId file ID of the base file.
+ * @param taskContextSupplier instance of {@link TaskContextSupplier} to use.
+ * @param keyGeneratorOpt optional instance of {@link BaseKeyGenerator} to use for extracting keys from records.
+ * @param readerContext instance of {@link HoodieReaderContext} to use while merging for accessing fields and transforming records.
+ */
+ public FileGroupReaderBasedMergeHandle(HoodieWriteConfig config, String instantTime, HoodieTable<T, I, K, O> hoodieTable,
+ Iterator<HoodieRecord<T>> recordItr, String partitionPath, String fileId,
+ TaskContextSupplier taskContextSupplier, Option<BaseKeyGenerator> keyGeneratorOpt, HoodieReaderContext<T> readerContext) {
Review Comment:
do we need to pass around the `readerContext` explicitly here? can we use `hoodieTable.getContext.getReaderContextFactoryForWrite` instead?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13699:
URL: https://github.com/apache/hudi/pull/13699#issuecomment-3169618290
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7661",
"triggerID" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ca08e20d8bacc2355f15ae37fab3c1ef61a052af Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7661)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13699:
URL: https://github.com/apache/hudi/pull/13699#discussion_r2265587234
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/WriteStatus.java:
##########
@@ -103,12 +104,20 @@ public WriteStatus() {
* @param optionalRecordMetadata optional metadata related to data contained in {@link HoodieRecord} before deflation.
*/
public void markSuccess(HoodieRecord record, Option<Map<String, String>> optionalRecordMetadata) {
- if (trackSuccessRecords) {
+ if (trackSuccessRecords && !manuallyTrackIndexUpdates) {
indexStats.addHoodieRecordDelegate(HoodieRecordDelegate.fromHoodieRecord(record));
}
updateStatsForSuccess(optionalRecordMetadata);
}
+ public void manuallyTrackSuccess() {
+ this.manuallyTrackIndexUpdates = true;
Review Comment:
Can just set up `trackSuccessRecords` as false here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13699:
URL: https://github.com/apache/hudi/pull/13699#issuecomment-3172883164
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7661",
"triggerID" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"triggerType" : "PUSH"
}, {
"hash" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7666",
"triggerID" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"triggerType" : "PUSH"
}, {
"hash" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7667",
"triggerID" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"triggerType" : "PUSH"
}, {
"hash" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7672",
"triggerID" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"triggerType" : "PUSH"
}, {
"hash" : "965136302f1188e5123de37734c927dc55801b24",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7691",
"triggerID" : "965136302f1188e5123de37734c927dc55801b24",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5507dfb0f3e741a539dbed0669d24dac9f1fc2b3 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7672)
* 965136302f1188e5123de37734c927dc55801b24 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7691)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13699:
URL: https://github.com/apache/hudi/pull/13699#issuecomment-3175614440
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7661",
"triggerID" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"triggerType" : "PUSH"
}, {
"hash" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7666",
"triggerID" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"triggerType" : "PUSH"
}, {
"hash" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7667",
"triggerID" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"triggerType" : "PUSH"
}, {
"hash" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7672",
"triggerID" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"triggerType" : "PUSH"
}, {
"hash" : "965136302f1188e5123de37734c927dc55801b24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7691",
"triggerID" : "965136302f1188e5123de37734c927dc55801b24",
"triggerType" : "PUSH"
}, {
"hash" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7693",
"triggerID" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"triggerType" : "PUSH"
}, {
"hash" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7717",
"triggerID" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7dbbe8212c828dc3cf2405a0634d4a0c643580e0 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7693)
* 22721bae05acf8aa296bc5732af5a90d2afd3515 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7717)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13699:
URL: https://github.com/apache/hudi/pull/13699#discussion_r2271597279
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/UpdateProcessor.java:
##########
@@ -80,17 +99,58 @@ public BufferedRecord<T> processUpdate(String recordKey, BufferedRecord<T> previ
}
return null;
} else {
- T prevRow = previousRecord != null ? previousRecord.getRecord() : null;
- T mergedRow = mergedRecord.getRecord();
- if (prevRow != null && prevRow != mergedRow) {
- mergedRecord.setHoodieOperation(HoodieOperation.UPDATE_AFTER);
- readStats.incrementNumUpdates();
- } else if (prevRow == null) {
- mergedRecord.setHoodieOperation(HoodieOperation.INSERT);
- readStats.incrementNumInserts();
+ return handleNonDeletes(previousRecord, mergedRecord);
+ }
+ }
+
+ protected BufferedRecord<T> handleNonDeletes(BufferedRecord<T> previousRecord, BufferedRecord<T> mergedRecord) {
+ T prevRow = previousRecord != null ? previousRecord.getRecord() : null;
+ T mergedRow = mergedRecord.getRecord();
+ if (prevRow != null && prevRow != mergedRow) {
+ mergedRecord.setHoodieOperation(HoodieOperation.UPDATE_AFTER);
+ readStats.incrementNumUpdates();
+ } else if (prevRow == null) {
+ mergedRecord.setHoodieOperation(HoodieOperation.INSERT);
+ readStats.incrementNumInserts();
+ }
+ return mergedRecord.seal(readerContext);
+ }
+ }
+
+ class PayloadUpdateProcessor<T> extends StandardUpdateProcessor<T> {
+ private final String payloadClass;
+ private final Properties properties;
+
+ public PayloadUpdateProcessor(HoodieReadStats readStats, HoodieReaderContext<T> readerContext, boolean emitDeletes,
+ Properties properties, String payloadClass) {
+ super(readStats, readerContext, emitDeletes);
+ this.payloadClass = payloadClass;
+ this.properties = properties;
+ }
+
+ @Override
+ protected BufferedRecord<T> handleNonDeletes(BufferedRecord<T> previousRecord, BufferedRecord<T> mergedRecord) {
+ if (previousRecord == null) {
+ // special case for payloads when there is no previous record
+ Schema recordSchema = readerContext.getRecordContext().decodeAvroSchema(mergedRecord.getSchemaId());
+ GenericRecord record = readerContext.getRecordContext().convertToAvroRecord(mergedRecord.getRecord(), recordSchema);
+ HoodieAvroRecord hoodieRecord = new HoodieAvroRecord<>(null, HoodieRecordUtils.loadPayload(payloadClass, record, mergedRecord.getOrderingValue()));
+ try {
+ if (hoodieRecord.shouldIgnore(recordSchema, properties)) {
+ return null;
+ } else {
+ Schema readerSchema = readerContext.getSchemaHandler().getRequestedSchema();
+ // If the record schema is different from the reader schema, rewrite the record using the payload methods to ensure consistency with legacy writer paths
+ if (!readerSchema.equals(recordSchema)) {
Review Comment:
Checking the number of fields will not be enough to guarantee safety. This case is currently limited to the Payload based mergers where there is an update in the incoming records and there is no record in the base file for that key so it should not be very common.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13699:
URL: https://github.com/apache/hudi/pull/13699#discussion_r2271575607
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/FileGroupReaderBasedMergeHandle.java:
##########
@@ -90,7 +147,15 @@ public FileGroupReaderBasedMergeHandle(HoodieWriteConfig config, String instantT
this.maxInstantTime = maxInstantTime;
this.keyToNewRecords = Collections.emptyMap();
this.readerContext = readerContext;
- this.operation = operation;
+ this.operation = Option.of(operation);
+ initRecordTypeAndCdcLogger(enginRecordType);
+ init(operation, this.partitionPath);
+ this.props = TypedProperties.copy(config.getProps());
+ this.isCompaction = true;
+ initRecordIndexCallback();
Review Comment:
Removed this, it was causing some issues with existing tests so it was good to see we have some good coverage here :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13699:
URL: https://github.com/apache/hudi/pull/13699#issuecomment-3185491750
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7661",
"triggerID" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"triggerType" : "PUSH"
}, {
"hash" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7666",
"triggerID" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"triggerType" : "PUSH"
}, {
"hash" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7667",
"triggerID" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"triggerType" : "PUSH"
}, {
"hash" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7672",
"triggerID" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"triggerType" : "PUSH"
}, {
"hash" : "965136302f1188e5123de37734c927dc55801b24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7691",
"triggerID" : "965136302f1188e5123de37734c927dc55801b24",
"triggerType" : "PUSH"
}, {
"hash" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7693",
"triggerID" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"triggerType" : "PUSH"
}, {
"hash" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7717",
"triggerID" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"triggerType" : "PUSH"
}, {
"hash" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"triggerType" : "PUSH"
}, {
"hash" : "47f8303d6e61504275ecfddf553e356427e6c822",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7718",
"triggerID" : "47f8303d6e61504275ecfddf553e356427e6c822",
"triggerType" : "PUSH"
}, {
"hash" : "fb191ecf48d13cd81412535f7f8c9ed553d86517",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7725",
"triggerID" : "fb191ecf48d13cd81412535f7f8c9ed553d86517",
"triggerType" : "PUSH"
}, {
"hash" : "064912d3a54714935f2ee77d55cc66816166300f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7727",
"triggerID" : "064912d3a54714935f2ee77d55cc66816166300f",
"triggerType" : "PUSH"
}, {
"hash" : "77a2b45fe13dafc51f7bebb2ba859d40291ccc10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7736",
"triggerID" : "77a2b45fe13dafc51f7bebb2ba859d40291ccc10",
"triggerType" : "PUSH"
}, {
"hash" : "e4eda0e51fee0e8de5d57ea7b38dc642f53aa95d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7758",
"triggerID" : "e4eda0e51fee0e8de5d57ea7b38dc642f53aa95d",
"triggerType" : "PUSH"
}, {
"hash" : "6cc840c51fcecde95d2187fb6fd217b50fa82806",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6cc840c51fcecde95d2187fb6fd217b50fa82806",
"triggerType" : "PUSH"
}, {
"hash" : "31dab8ce9abbe4fc80e6cd81a0e433a039ca3e1d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7760",
"triggerID" : "31dab8ce9abbe4fc80e6cd81a0e433a039ca3e1d",
"triggerType" : "PUSH"
}, {
"hash" : "c04fd95dcc94146a9146c347f4f204656f3d8308",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7763",
"triggerID" : "c04fd95dcc94146a9146c347f4f204656f3d8308",
"triggerType" : "PUSH"
}, {
"hash" : "324147913d47ba8f2555e1991ce59905e5dc0b3f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7767",
"triggerID" : "324147913d47ba8f2555e1991ce59905e5dc0b3f",
"triggerType" : "PUSH"
}, {
"hash" : "3f177d42b831b30f4bd590b057dcab6ded78b634",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7774",
"triggerID" : "3f177d42b831b30f4bd590b057dcab6ded78b634",
"triggerType" : "PUSH"
}, {
"hash" : "8d6d2c06ec82d1072251ac12ddf21a890ddd5bda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7776",
"triggerID" : "8d6d2c06ec82d1072251ac12ddf21a890ddd5bda",
"triggerType" : "PUSH"
}, {
"hash" : "77589738d73b044385f5aa6f7ef87b421f06043b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "77589738d73b044385f5aa6f7ef87b421f06043b",
"triggerType" : "PUSH"
}, {
"hash" : "ad0d7ef0fbc525f880d3baadef5a22196ab97b32",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7781",
"triggerID" : "ad0d7ef0fbc525f880d3baadef5a22196ab97b32",
"triggerType" : "PUSH"
}, {
"hash" : "0db72caa3a6c7d0f0899f243ea4b8975de79e02c",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7787",
"triggerID" : "0db72caa3a6c7d0f0899f243ea4b8975de79e02c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 50022dcc45b0f511dd69706aeaf6464a51be49e2 UNKNOWN
* 6cc840c51fcecde95d2187fb6fd217b50fa82806 UNKNOWN
* 77589738d73b044385f5aa6f7ef87b421f06043b UNKNOWN
* ad0d7ef0fbc525f880d3baadef5a22196ab97b32 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7781)
* 0db72caa3a6c7d0f0899f243ea4b8975de79e02c Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7787)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13699:
URL: https://github.com/apache/hudi/pull/13699#issuecomment-3184043380
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7661",
"triggerID" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"triggerType" : "PUSH"
}, {
"hash" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7666",
"triggerID" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"triggerType" : "PUSH"
}, {
"hash" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7667",
"triggerID" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"triggerType" : "PUSH"
}, {
"hash" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7672",
"triggerID" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"triggerType" : "PUSH"
}, {
"hash" : "965136302f1188e5123de37734c927dc55801b24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7691",
"triggerID" : "965136302f1188e5123de37734c927dc55801b24",
"triggerType" : "PUSH"
}, {
"hash" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7693",
"triggerID" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"triggerType" : "PUSH"
}, {
"hash" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7717",
"triggerID" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"triggerType" : "PUSH"
}, {
"hash" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"triggerType" : "PUSH"
}, {
"hash" : "47f8303d6e61504275ecfddf553e356427e6c822",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7718",
"triggerID" : "47f8303d6e61504275ecfddf553e356427e6c822",
"triggerType" : "PUSH"
}, {
"hash" : "fb191ecf48d13cd81412535f7f8c9ed553d86517",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7725",
"triggerID" : "fb191ecf48d13cd81412535f7f8c9ed553d86517",
"triggerType" : "PUSH"
}, {
"hash" : "064912d3a54714935f2ee77d55cc66816166300f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7727",
"triggerID" : "064912d3a54714935f2ee77d55cc66816166300f",
"triggerType" : "PUSH"
}, {
"hash" : "77a2b45fe13dafc51f7bebb2ba859d40291ccc10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7736",
"triggerID" : "77a2b45fe13dafc51f7bebb2ba859d40291ccc10",
"triggerType" : "PUSH"
}, {
"hash" : "e4eda0e51fee0e8de5d57ea7b38dc642f53aa95d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7758",
"triggerID" : "e4eda0e51fee0e8de5d57ea7b38dc642f53aa95d",
"triggerType" : "PUSH"
}, {
"hash" : "6cc840c51fcecde95d2187fb6fd217b50fa82806",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6cc840c51fcecde95d2187fb6fd217b50fa82806",
"triggerType" : "PUSH"
}, {
"hash" : "31dab8ce9abbe4fc80e6cd81a0e433a039ca3e1d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7760",
"triggerID" : "31dab8ce9abbe4fc80e6cd81a0e433a039ca3e1d",
"triggerType" : "PUSH"
}, {
"hash" : "c04fd95dcc94146a9146c347f4f204656f3d8308",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7763",
"triggerID" : "c04fd95dcc94146a9146c347f4f204656f3d8308",
"triggerType" : "PUSH"
}, {
"hash" : "324147913d47ba8f2555e1991ce59905e5dc0b3f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7767",
"triggerID" : "324147913d47ba8f2555e1991ce59905e5dc0b3f",
"triggerType" : "PUSH"
}, {
"hash" : "3f177d42b831b30f4bd590b057dcab6ded78b634",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7774",
"triggerID" : "3f177d42b831b30f4bd590b057dcab6ded78b634",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 50022dcc45b0f511dd69706aeaf6464a51be49e2 UNKNOWN
* 6cc840c51fcecde95d2187fb6fd217b50fa82806 UNKNOWN
* 324147913d47ba8f2555e1991ce59905e5dc0b3f Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7767)
* 3f177d42b831b30f4bd590b057dcab6ded78b634 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7774)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13699:
URL: https://github.com/apache/hudi/pull/13699#issuecomment-3184081069
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7661",
"triggerID" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"triggerType" : "PUSH"
}, {
"hash" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7666",
"triggerID" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"triggerType" : "PUSH"
}, {
"hash" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7667",
"triggerID" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"triggerType" : "PUSH"
}, {
"hash" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7672",
"triggerID" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"triggerType" : "PUSH"
}, {
"hash" : "965136302f1188e5123de37734c927dc55801b24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7691",
"triggerID" : "965136302f1188e5123de37734c927dc55801b24",
"triggerType" : "PUSH"
}, {
"hash" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7693",
"triggerID" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"triggerType" : "PUSH"
}, {
"hash" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7717",
"triggerID" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"triggerType" : "PUSH"
}, {
"hash" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"triggerType" : "PUSH"
}, {
"hash" : "47f8303d6e61504275ecfddf553e356427e6c822",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7718",
"triggerID" : "47f8303d6e61504275ecfddf553e356427e6c822",
"triggerType" : "PUSH"
}, {
"hash" : "fb191ecf48d13cd81412535f7f8c9ed553d86517",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7725",
"triggerID" : "fb191ecf48d13cd81412535f7f8c9ed553d86517",
"triggerType" : "PUSH"
}, {
"hash" : "064912d3a54714935f2ee77d55cc66816166300f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7727",
"triggerID" : "064912d3a54714935f2ee77d55cc66816166300f",
"triggerType" : "PUSH"
}, {
"hash" : "77a2b45fe13dafc51f7bebb2ba859d40291ccc10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7736",
"triggerID" : "77a2b45fe13dafc51f7bebb2ba859d40291ccc10",
"triggerType" : "PUSH"
}, {
"hash" : "e4eda0e51fee0e8de5d57ea7b38dc642f53aa95d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7758",
"triggerID" : "e4eda0e51fee0e8de5d57ea7b38dc642f53aa95d",
"triggerType" : "PUSH"
}, {
"hash" : "6cc840c51fcecde95d2187fb6fd217b50fa82806",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6cc840c51fcecde95d2187fb6fd217b50fa82806",
"triggerType" : "PUSH"
}, {
"hash" : "31dab8ce9abbe4fc80e6cd81a0e433a039ca3e1d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7760",
"triggerID" : "31dab8ce9abbe4fc80e6cd81a0e433a039ca3e1d",
"triggerType" : "PUSH"
}, {
"hash" : "c04fd95dcc94146a9146c347f4f204656f3d8308",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7763",
"triggerID" : "c04fd95dcc94146a9146c347f4f204656f3d8308",
"triggerType" : "PUSH"
}, {
"hash" : "324147913d47ba8f2555e1991ce59905e5dc0b3f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7767",
"triggerID" : "324147913d47ba8f2555e1991ce59905e5dc0b3f",
"triggerType" : "PUSH"
}, {
"hash" : "3f177d42b831b30f4bd590b057dcab6ded78b634",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7774",
"triggerID" : "3f177d42b831b30f4bd590b057dcab6ded78b634",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 50022dcc45b0f511dd69706aeaf6464a51be49e2 UNKNOWN
* 6cc840c51fcecde95d2187fb6fd217b50fa82806 UNKNOWN
* 3f177d42b831b30f4bd590b057dcab6ded78b634 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7774)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13699:
URL: https://github.com/apache/hudi/pull/13699#issuecomment-3184408767
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7661",
"triggerID" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"triggerType" : "PUSH"
}, {
"hash" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7666",
"triggerID" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"triggerType" : "PUSH"
}, {
"hash" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7667",
"triggerID" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"triggerType" : "PUSH"
}, {
"hash" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7672",
"triggerID" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"triggerType" : "PUSH"
}, {
"hash" : "965136302f1188e5123de37734c927dc55801b24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7691",
"triggerID" : "965136302f1188e5123de37734c927dc55801b24",
"triggerType" : "PUSH"
}, {
"hash" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7693",
"triggerID" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"triggerType" : "PUSH"
}, {
"hash" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7717",
"triggerID" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"triggerType" : "PUSH"
}, {
"hash" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"triggerType" : "PUSH"
}, {
"hash" : "47f8303d6e61504275ecfddf553e356427e6c822",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7718",
"triggerID" : "47f8303d6e61504275ecfddf553e356427e6c822",
"triggerType" : "PUSH"
}, {
"hash" : "fb191ecf48d13cd81412535f7f8c9ed553d86517",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7725",
"triggerID" : "fb191ecf48d13cd81412535f7f8c9ed553d86517",
"triggerType" : "PUSH"
}, {
"hash" : "064912d3a54714935f2ee77d55cc66816166300f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7727",
"triggerID" : "064912d3a54714935f2ee77d55cc66816166300f",
"triggerType" : "PUSH"
}, {
"hash" : "77a2b45fe13dafc51f7bebb2ba859d40291ccc10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7736",
"triggerID" : "77a2b45fe13dafc51f7bebb2ba859d40291ccc10",
"triggerType" : "PUSH"
}, {
"hash" : "e4eda0e51fee0e8de5d57ea7b38dc642f53aa95d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7758",
"triggerID" : "e4eda0e51fee0e8de5d57ea7b38dc642f53aa95d",
"triggerType" : "PUSH"
}, {
"hash" : "6cc840c51fcecde95d2187fb6fd217b50fa82806",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6cc840c51fcecde95d2187fb6fd217b50fa82806",
"triggerType" : "PUSH"
}, {
"hash" : "31dab8ce9abbe4fc80e6cd81a0e433a039ca3e1d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7760",
"triggerID" : "31dab8ce9abbe4fc80e6cd81a0e433a039ca3e1d",
"triggerType" : "PUSH"
}, {
"hash" : "c04fd95dcc94146a9146c347f4f204656f3d8308",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7763",
"triggerID" : "c04fd95dcc94146a9146c347f4f204656f3d8308",
"triggerType" : "PUSH"
}, {
"hash" : "324147913d47ba8f2555e1991ce59905e5dc0b3f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7767",
"triggerID" : "324147913d47ba8f2555e1991ce59905e5dc0b3f",
"triggerType" : "PUSH"
}, {
"hash" : "3f177d42b831b30f4bd590b057dcab6ded78b634",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7774",
"triggerID" : "3f177d42b831b30f4bd590b057dcab6ded78b634",
"triggerType" : "PUSH"
}, {
"hash" : "8d6d2c06ec82d1072251ac12ddf21a890ddd5bda",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7776",
"triggerID" : "8d6d2c06ec82d1072251ac12ddf21a890ddd5bda",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 50022dcc45b0f511dd69706aeaf6464a51be49e2 UNKNOWN
* 6cc840c51fcecde95d2187fb6fd217b50fa82806 UNKNOWN
* 8d6d2c06ec82d1072251ac12ddf21a890ddd5bda Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7776)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13699:
URL: https://github.com/apache/hudi/pull/13699#issuecomment-3173135649
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7661",
"triggerID" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"triggerType" : "PUSH"
}, {
"hash" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7666",
"triggerID" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"triggerType" : "PUSH"
}, {
"hash" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7667",
"triggerID" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"triggerType" : "PUSH"
}, {
"hash" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7672",
"triggerID" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"triggerType" : "PUSH"
}, {
"hash" : "965136302f1188e5123de37734c927dc55801b24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7691",
"triggerID" : "965136302f1188e5123de37734c927dc55801b24",
"triggerType" : "PUSH"
}, {
"hash" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7693",
"triggerID" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7dbbe8212c828dc3cf2405a0634d4a0c643580e0 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7693)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13699:
URL: https://github.com/apache/hudi/pull/13699#issuecomment-3169666585
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7661",
"triggerID" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ca08e20d8bacc2355f15ae37fab3c1ef61a052af Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7661)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13699:
URL: https://github.com/apache/hudi/pull/13699#issuecomment-3171078078
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7661",
"triggerID" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"triggerType" : "PUSH"
}, {
"hash" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7666",
"triggerID" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ca08e20d8bacc2355f15ae37fab3c1ef61a052af Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7661)
* 7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7666)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13699:
URL: https://github.com/apache/hudi/pull/13699#discussion_r2265620544
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/buffer/StreamingFileGroupRecordBufferLoader.java:
##########
@@ -51,19 +53,20 @@
* @param <T> Engine native presentation of the record.
*/
public class StreamingFileGroupRecordBufferLoader<T> implements FileGroupRecordBufferLoader<T> {
- private static final StreamingFileGroupRecordBufferLoader INSTANCE = new StreamingFileGroupRecordBufferLoader<>();
+ private final Schema recordSchema;
- static <T> StreamingFileGroupRecordBufferLoader<T> getInstance() {
- return INSTANCE;
+ StreamingFileGroupRecordBufferLoader(Schema recordSchema) {
+ this.recordSchema = recordSchema;
Review Comment:
there is no need to pass around the schema explicitly, it is actually the writeSchema, which equals to:
`schemaHandler.requestedSchema - metadata fields`, we already have utility method for it: `HoodieAvroUtils.removeMetadataFields`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13699:
URL: https://github.com/apache/hudi/pull/13699#issuecomment-3172925753
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7661",
"triggerID" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"triggerType" : "PUSH"
}, {
"hash" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7666",
"triggerID" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"triggerType" : "PUSH"
}, {
"hash" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7667",
"triggerID" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"triggerType" : "PUSH"
}, {
"hash" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7672",
"triggerID" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"triggerType" : "PUSH"
}, {
"hash" : "965136302f1188e5123de37734c927dc55801b24",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7691",
"triggerID" : "965136302f1188e5123de37734c927dc55801b24",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 965136302f1188e5123de37734c927dc55801b24 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7691)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13699:
URL: https://github.com/apache/hudi/pull/13699#discussion_r2265592191
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/FileGroupReaderBasedMergeHandle.java:
##########
@@ -90,7 +147,15 @@ public FileGroupReaderBasedMergeHandle(HoodieWriteConfig config, String instantT
this.maxInstantTime = maxInstantTime;
this.keyToNewRecords = Collections.emptyMap();
this.readerContext = readerContext;
- this.operation = operation;
+ this.operation = Option.of(operation);
+ initRecordTypeAndCdcLogger(enginRecordType);
+ init(operation, this.partitionPath);
+ this.props = TypedProperties.copy(config.getProps());
+ this.isCompaction = true;
+ initRecordIndexCallback();
Review Comment:
do we even need to track RLI for compactions?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13699:
URL: https://github.com/apache/hudi/pull/13699#discussion_r2265612605
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/FileGroupReaderBasedMergeHandle.java:
##########
@@ -267,26 +393,141 @@ private static class CDCCallback<T> implements BaseFileUpdateCallback<T> {
}
@Override
- public void onUpdate(String recordKey, T previousRecord, T mergedRecord) {
+ public void onUpdate(String recordKey, BufferedRecord<T> previousRecord, BufferedRecord<T> mergedRecord) {
cdcLogger.put(recordKey, convertOutput(previousRecord), Option.of(convertOutput(mergedRecord)));
-
}
@Override
- public void onInsert(String recordKey, T newRecord) {
+ public void onInsert(String recordKey, BufferedRecord<T> newRecord) {
cdcLogger.put(recordKey, null, Option.of(convertOutput(newRecord)));
-
}
@Override
- public void onDelete(String recordKey, T previousRecord) {
+ public void onDelete(String recordKey, BufferedRecord<T> previousRecord, HoodieOperation hoodieOperation) {
cdcLogger.put(recordKey, convertOutput(previousRecord), Option.empty());
-
}
- private GenericRecord convertOutput(T record) {
- T convertedRecord = outputConverter.get().map(converter -> record == null ? null : converter.apply(record)).orElse(record);
+ private GenericRecord convertOutput(BufferedRecord<T> record) {
+ T convertedRecord = outputConverter.get().map(converter -> record == null ? null : converter.apply(record.getRecord())).orElse(record.getRecord());
return convertedRecord == null ? null : readerContext.getRecordContext().convertToAvroRecord(convertedRecord, requestedSchema.get());
}
}
+
+ private static class RecordLevelIndexCallback<T> implements BaseFileUpdateCallback<T> {
+ private final WriteStatus writeStatus;
+ private final HoodieRecordLocation fileRecordLocation;
+ private final String partitionPath;
+
+ public RecordLevelIndexCallback(WriteStatus writeStatus, HoodieRecordLocation fileRecordLocation, String partitionPath) {
+ this.writeStatus = writeStatus;
+ this.fileRecordLocation = fileRecordLocation;
+ this.partitionPath = partitionPath;
+ }
+
+ @Override
+ public void onUpdate(String recordKey, BufferedRecord<T> previousRecord, BufferedRecord<T> mergedRecord) {
+ writeStatus.addRecordDelegate(HoodieRecordDelegate.create(recordKey, partitionPath, fileRecordLocation, fileRecordLocation, mergedRecord.getHoodieOperation() == HoodieOperation.UPDATE_BEFORE));
+ }
+
+ @Override
+ public void onInsert(String recordKey, BufferedRecord<T> newRecord) {
+ writeStatus.addRecordDelegate(HoodieRecordDelegate.create(recordKey, partitionPath, null, fileRecordLocation, newRecord.getHoodieOperation() == HoodieOperation.UPDATE_BEFORE));
+ }
+
+ @Override
+ public void onDelete(String recordKey, BufferedRecord<T> previousRecord, HoodieOperation hoodieOperation) {
+ // The update before operation is used when a deletion is being sent to the old File Group in a different partition.
+ // In this case, we do not want to delete the record metadata from the index.
+ writeStatus.addRecordDelegate(HoodieRecordDelegate.create(recordKey, partitionPath, fileRecordLocation, null, hoodieOperation == HoodieOperation.UPDATE_BEFORE));
Review Comment:
`hoodieOperation == HoodieOperation.UPDATE_BEFORE` is always false.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13699:
URL: https://github.com/apache/hudi/pull/13699#discussion_r2265612315
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/FileGroupReaderBasedMergeHandle.java:
##########
@@ -267,26 +393,141 @@ private static class CDCCallback<T> implements BaseFileUpdateCallback<T> {
}
@Override
- public void onUpdate(String recordKey, T previousRecord, T mergedRecord) {
+ public void onUpdate(String recordKey, BufferedRecord<T> previousRecord, BufferedRecord<T> mergedRecord) {
cdcLogger.put(recordKey, convertOutput(previousRecord), Option.of(convertOutput(mergedRecord)));
-
}
@Override
- public void onInsert(String recordKey, T newRecord) {
+ public void onInsert(String recordKey, BufferedRecord<T> newRecord) {
cdcLogger.put(recordKey, null, Option.of(convertOutput(newRecord)));
-
}
@Override
- public void onDelete(String recordKey, T previousRecord) {
+ public void onDelete(String recordKey, BufferedRecord<T> previousRecord, HoodieOperation hoodieOperation) {
cdcLogger.put(recordKey, convertOutput(previousRecord), Option.empty());
-
}
- private GenericRecord convertOutput(T record) {
- T convertedRecord = outputConverter.get().map(converter -> record == null ? null : converter.apply(record)).orElse(record);
+ private GenericRecord convertOutput(BufferedRecord<T> record) {
+ T convertedRecord = outputConverter.get().map(converter -> record == null ? null : converter.apply(record.getRecord())).orElse(record.getRecord());
return convertedRecord == null ? null : readerContext.getRecordContext().convertToAvroRecord(convertedRecord, requestedSchema.get());
}
}
+
+ private static class RecordLevelIndexCallback<T> implements BaseFileUpdateCallback<T> {
+ private final WriteStatus writeStatus;
+ private final HoodieRecordLocation fileRecordLocation;
+ private final String partitionPath;
+
+ public RecordLevelIndexCallback(WriteStatus writeStatus, HoodieRecordLocation fileRecordLocation, String partitionPath) {
+ this.writeStatus = writeStatus;
+ this.fileRecordLocation = fileRecordLocation;
+ this.partitionPath = partitionPath;
+ }
+
+ @Override
+ public void onUpdate(String recordKey, BufferedRecord<T> previousRecord, BufferedRecord<T> mergedRecord) {
+ writeStatus.addRecordDelegate(HoodieRecordDelegate.create(recordKey, partitionPath, fileRecordLocation, fileRecordLocation, mergedRecord.getHoodieOperation() == HoodieOperation.UPDATE_BEFORE));
+ }
+
+ @Override
+ public void onInsert(String recordKey, BufferedRecord<T> newRecord) {
+ writeStatus.addRecordDelegate(HoodieRecordDelegate.create(recordKey, partitionPath, null, fileRecordLocation, newRecord.getHoodieOperation() == HoodieOperation.UPDATE_BEFORE));
Review Comment:
`newRecord.getHoodieOperation() == HoodieOperation.UPDATE_BEFORE` is always false.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13699:
URL: https://github.com/apache/hudi/pull/13699#discussion_r2265594143
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/FileGroupReaderBasedMergeHandle.java:
##########
@@ -90,7 +147,15 @@ public FileGroupReaderBasedMergeHandle(HoodieWriteConfig config, String instantT
this.maxInstantTime = maxInstantTime;
this.keyToNewRecords = Collections.emptyMap();
this.readerContext = readerContext;
- this.operation = operation;
+ this.operation = Option.of(operation);
+ initRecordTypeAndCdcLogger(enginRecordType);
+ init(operation, this.partitionPath);
+ this.props = TypedProperties.copy(config.getProps());
+ this.isCompaction = true;
Review Comment:
we already have flag `preserveMetadata` to distinguish table service and regular write, can we continue to use that? some functions like SI tracing already relies on the flag `preserveMetadata`. And it seems clustering also uses this constructor.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DRAFT][DNM] CoW write handle with file group reader [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on PR #13699:
URL: https://github.com/apache/hudi/pull/13699#issuecomment-3188640201
I assume this is the patch being worked upon for COW merge handle migration. can we fix PR title, PR description as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9622] Add implementation of MergeHandle backed by the HoodieFileGroupReader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13699:
URL: https://github.com/apache/hudi/pull/13699#issuecomment-3190364297
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7661",
"triggerID" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"triggerType" : "PUSH"
}, {
"hash" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7666",
"triggerID" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"triggerType" : "PUSH"
}, {
"hash" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7667",
"triggerID" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"triggerType" : "PUSH"
}, {
"hash" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7672",
"triggerID" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"triggerType" : "PUSH"
}, {
"hash" : "965136302f1188e5123de37734c927dc55801b24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7691",
"triggerID" : "965136302f1188e5123de37734c927dc55801b24",
"triggerType" : "PUSH"
}, {
"hash" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7693",
"triggerID" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"triggerType" : "PUSH"
}, {
"hash" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7717",
"triggerID" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"triggerType" : "PUSH"
}, {
"hash" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"triggerType" : "PUSH"
}, {
"hash" : "47f8303d6e61504275ecfddf553e356427e6c822",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7718",
"triggerID" : "47f8303d6e61504275ecfddf553e356427e6c822",
"triggerType" : "PUSH"
}, {
"hash" : "fb191ecf48d13cd81412535f7f8c9ed553d86517",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7725",
"triggerID" : "fb191ecf48d13cd81412535f7f8c9ed553d86517",
"triggerType" : "PUSH"
}, {
"hash" : "064912d3a54714935f2ee77d55cc66816166300f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7727",
"triggerID" : "064912d3a54714935f2ee77d55cc66816166300f",
"triggerType" : "PUSH"
}, {
"hash" : "77a2b45fe13dafc51f7bebb2ba859d40291ccc10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7736",
"triggerID" : "77a2b45fe13dafc51f7bebb2ba859d40291ccc10",
"triggerType" : "PUSH"
}, {
"hash" : "e4eda0e51fee0e8de5d57ea7b38dc642f53aa95d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7758",
"triggerID" : "e4eda0e51fee0e8de5d57ea7b38dc642f53aa95d",
"triggerType" : "PUSH"
}, {
"hash" : "6cc840c51fcecde95d2187fb6fd217b50fa82806",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6cc840c51fcecde95d2187fb6fd217b50fa82806",
"triggerType" : "PUSH"
}, {
"hash" : "31dab8ce9abbe4fc80e6cd81a0e433a039ca3e1d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7760",
"triggerID" : "31dab8ce9abbe4fc80e6cd81a0e433a039ca3e1d",
"triggerType" : "PUSH"
}, {
"hash" : "c04fd95dcc94146a9146c347f4f204656f3d8308",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7763",
"triggerID" : "c04fd95dcc94146a9146c347f4f204656f3d8308",
"triggerType" : "PUSH"
}, {
"hash" : "324147913d47ba8f2555e1991ce59905e5dc0b3f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7767",
"triggerID" : "324147913d47ba8f2555e1991ce59905e5dc0b3f",
"triggerType" : "PUSH"
}, {
"hash" : "3f177d42b831b30f4bd590b057dcab6ded78b634",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7774",
"triggerID" : "3f177d42b831b30f4bd590b057dcab6ded78b634",
"triggerType" : "PUSH"
}, {
"hash" : "8d6d2c06ec82d1072251ac12ddf21a890ddd5bda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7776",
"triggerID" : "8d6d2c06ec82d1072251ac12ddf21a890ddd5bda",
"triggerType" : "PUSH"
}, {
"hash" : "77589738d73b044385f5aa6f7ef87b421f06043b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "77589738d73b044385f5aa6f7ef87b421f06043b",
"triggerType" : "PUSH"
}, {
"hash" : "ad0d7ef0fbc525f880d3baadef5a22196ab97b32",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7781",
"triggerID" : "ad0d7ef0fbc525f880d3baadef5a22196ab97b32",
"triggerType" : "PUSH"
}, {
"hash" : "0db72caa3a6c7d0f0899f243ea4b8975de79e02c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7787",
"triggerID" : "0db72caa3a6c7d0f0899f243ea4b8975de79e02c",
"triggerType" : "PUSH"
}, {
"hash" : "a01ce748dc2d4cd1fd4858d1ecac8a486989e880",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7797",
"triggerID" : "a01ce748dc2d4cd1fd4858d1ecac8a486989e880",
"triggerType" : "PUSH"
}, {
"hash" : "3f35151e28bfa1dd50b31c6e5379d9769627afd9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7799",
"triggerID" : "3f35151e28bfa1dd50b31c6e5379d9769627afd9",
"triggerType" : "PUSH"
}, {
"hash" : "b0054c8fd329489d9a1e9535d9866edffc52fdd0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b0054c8fd329489d9a1e9535d9866edffc52fdd0",
"triggerType" : "PUSH"
}, {
"hash" : "363e54341420cc217bf5af62de1e09ff370f75d2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "363e54341420cc217bf5af62de1e09ff370f75d2",
"triggerType" : "PUSH"
}, {
"hash" : "a893da375eb9c65edabbd22965296eac9b3714f4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a893da375eb9c65edabbd22965296eac9b3714f4",
"triggerType" : "PUSH"
}, {
"hash" : "a7c2756e6a293cbba99bd4e821881e6b961c138f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7804",
"triggerID" : "a7c2756e6a293cbba99bd4e821881e6b961c138f",
"triggerType" : "PUSH"
}, {
"hash" : "4a140fffa35893a13f2e7ac8365d60d51a2d0870",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7813",
"triggerID" : "4a140fffa35893a13f2e7ac8365d60d51a2d0870",
"triggerType" : "PUSH"
}, {
"hash" : "500b4f529ad2f8a695825b5badc1815202164d0a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7830",
"triggerID" : "500b4f529ad2f8a695825b5badc1815202164d0a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 50022dcc45b0f511dd69706aeaf6464a51be49e2 UNKNOWN
* 6cc840c51fcecde95d2187fb6fd217b50fa82806 UNKNOWN
* 77589738d73b044385f5aa6f7ef87b421f06043b UNKNOWN
* b0054c8fd329489d9a1e9535d9866edffc52fdd0 UNKNOWN
* 363e54341420cc217bf5af62de1e09ff370f75d2 UNKNOWN
* a893da375eb9c65edabbd22965296eac9b3714f4 UNKNOWN
* 4a140fffa35893a13f2e7ac8365d60d51a2d0870 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7813)
* 500b4f529ad2f8a695825b5badc1815202164d0a Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7830)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9622] Add implementation of MergeHandle backed by the HoodieFileGroupReader [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13699:
URL: https://github.com/apache/hudi/pull/13699#discussion_r2277897773
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/UpdateProcessor.java:
##########
@@ -39,8 +52,16 @@ public interface UpdateProcessor<T> {
BufferedRecord<T> processUpdate(String recordKey, BufferedRecord<T> previousRecord, BufferedRecord<T> mergedRecord, boolean isDelete);
static <T> UpdateProcessor<T> create(HoodieReadStats readStats, HoodieReaderContext<T> readerContext,
- boolean emitDeletes, Option<BaseFileUpdateCallback<T>> updateCallback) {
- UpdateProcessor<T> handler = new StandardUpdateProcessor<>(readStats, readerContext, emitDeletes);
+ boolean emitDeletes, Option<BaseFileUpdateCallback<T>> updateCallback,
+ TypedProperties properties) {
+ UpdateProcessor<T> handler;
+ Option<String> payloadClass = readerContext.getPayloadClasses(properties).map(Pair::getRight);
Review Comment:
Yes, that is the idea. This is meant to cover the legacy payload flow only
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9622] Add implementation of MergeHandle backed by the HoodieFileGroupReader [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13699:
URL: https://github.com/apache/hudi/pull/13699#discussion_r2277925479
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/index/HoodieIndexUtils.java:
##########
@@ -492,10 +487,13 @@ private static <R> Option<HoodieRecord<R>> mergeIncomingWithExistingRecord(
// the record was deleted
return Option.empty();
}
- String partitionPath = inferPartitionPath(incoming, existing, writeSchemaWithMetaFields, keyGenerator, existingRecordContext, mergeResult);
+ Schema recordSchema = incomingRecordContext.getSchemaFromBufferRecord(mergeResult);
+ String partitionPath = inferPartitionPath(incoming, existing, recordSchema, keyGenerator, existingRecordContext, mergeResult);
HoodieRecord<R> result = existingRecordContext.constructHoodieRecord(mergeResult, partitionPath);
+ HoodieRecord<R> withMeta = result.prependMetaFields(recordSchema, writeSchemaWithMetaFields,
Review Comment:
This started failing without this change. It will require some more investigation
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9622] Add implementation of MergeHandle backed by the HoodieFileGroupReader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13699:
URL: https://github.com/apache/hudi/pull/13699#issuecomment-3190370304
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7661",
"triggerID" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"triggerType" : "PUSH"
}, {
"hash" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7666",
"triggerID" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"triggerType" : "PUSH"
}, {
"hash" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7667",
"triggerID" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"triggerType" : "PUSH"
}, {
"hash" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7672",
"triggerID" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"triggerType" : "PUSH"
}, {
"hash" : "965136302f1188e5123de37734c927dc55801b24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7691",
"triggerID" : "965136302f1188e5123de37734c927dc55801b24",
"triggerType" : "PUSH"
}, {
"hash" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7693",
"triggerID" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"triggerType" : "PUSH"
}, {
"hash" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7717",
"triggerID" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"triggerType" : "PUSH"
}, {
"hash" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"triggerType" : "PUSH"
}, {
"hash" : "47f8303d6e61504275ecfddf553e356427e6c822",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7718",
"triggerID" : "47f8303d6e61504275ecfddf553e356427e6c822",
"triggerType" : "PUSH"
}, {
"hash" : "fb191ecf48d13cd81412535f7f8c9ed553d86517",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7725",
"triggerID" : "fb191ecf48d13cd81412535f7f8c9ed553d86517",
"triggerType" : "PUSH"
}, {
"hash" : "064912d3a54714935f2ee77d55cc66816166300f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7727",
"triggerID" : "064912d3a54714935f2ee77d55cc66816166300f",
"triggerType" : "PUSH"
}, {
"hash" : "77a2b45fe13dafc51f7bebb2ba859d40291ccc10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7736",
"triggerID" : "77a2b45fe13dafc51f7bebb2ba859d40291ccc10",
"triggerType" : "PUSH"
}, {
"hash" : "e4eda0e51fee0e8de5d57ea7b38dc642f53aa95d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7758",
"triggerID" : "e4eda0e51fee0e8de5d57ea7b38dc642f53aa95d",
"triggerType" : "PUSH"
}, {
"hash" : "6cc840c51fcecde95d2187fb6fd217b50fa82806",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6cc840c51fcecde95d2187fb6fd217b50fa82806",
"triggerType" : "PUSH"
}, {
"hash" : "31dab8ce9abbe4fc80e6cd81a0e433a039ca3e1d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7760",
"triggerID" : "31dab8ce9abbe4fc80e6cd81a0e433a039ca3e1d",
"triggerType" : "PUSH"
}, {
"hash" : "c04fd95dcc94146a9146c347f4f204656f3d8308",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7763",
"triggerID" : "c04fd95dcc94146a9146c347f4f204656f3d8308",
"triggerType" : "PUSH"
}, {
"hash" : "324147913d47ba8f2555e1991ce59905e5dc0b3f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7767",
"triggerID" : "324147913d47ba8f2555e1991ce59905e5dc0b3f",
"triggerType" : "PUSH"
}, {
"hash" : "3f177d42b831b30f4bd590b057dcab6ded78b634",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7774",
"triggerID" : "3f177d42b831b30f4bd590b057dcab6ded78b634",
"triggerType" : "PUSH"
}, {
"hash" : "8d6d2c06ec82d1072251ac12ddf21a890ddd5bda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7776",
"triggerID" : "8d6d2c06ec82d1072251ac12ddf21a890ddd5bda",
"triggerType" : "PUSH"
}, {
"hash" : "77589738d73b044385f5aa6f7ef87b421f06043b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "77589738d73b044385f5aa6f7ef87b421f06043b",
"triggerType" : "PUSH"
}, {
"hash" : "ad0d7ef0fbc525f880d3baadef5a22196ab97b32",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7781",
"triggerID" : "ad0d7ef0fbc525f880d3baadef5a22196ab97b32",
"triggerType" : "PUSH"
}, {
"hash" : "0db72caa3a6c7d0f0899f243ea4b8975de79e02c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7787",
"triggerID" : "0db72caa3a6c7d0f0899f243ea4b8975de79e02c",
"triggerType" : "PUSH"
}, {
"hash" : "a01ce748dc2d4cd1fd4858d1ecac8a486989e880",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7797",
"triggerID" : "a01ce748dc2d4cd1fd4858d1ecac8a486989e880",
"triggerType" : "PUSH"
}, {
"hash" : "3f35151e28bfa1dd50b31c6e5379d9769627afd9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7799",
"triggerID" : "3f35151e28bfa1dd50b31c6e5379d9769627afd9",
"triggerType" : "PUSH"
}, {
"hash" : "b0054c8fd329489d9a1e9535d9866edffc52fdd0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b0054c8fd329489d9a1e9535d9866edffc52fdd0",
"triggerType" : "PUSH"
}, {
"hash" : "363e54341420cc217bf5af62de1e09ff370f75d2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "363e54341420cc217bf5af62de1e09ff370f75d2",
"triggerType" : "PUSH"
}, {
"hash" : "a893da375eb9c65edabbd22965296eac9b3714f4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a893da375eb9c65edabbd22965296eac9b3714f4",
"triggerType" : "PUSH"
}, {
"hash" : "a7c2756e6a293cbba99bd4e821881e6b961c138f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7804",
"triggerID" : "a7c2756e6a293cbba99bd4e821881e6b961c138f",
"triggerType" : "PUSH"
}, {
"hash" : "4a140fffa35893a13f2e7ac8365d60d51a2d0870",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7813",
"triggerID" : "4a140fffa35893a13f2e7ac8365d60d51a2d0870",
"triggerType" : "PUSH"
}, {
"hash" : "500b4f529ad2f8a695825b5badc1815202164d0a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7830",
"triggerID" : "500b4f529ad2f8a695825b5badc1815202164d0a",
"triggerType" : "PUSH"
}, {
"hash" : "539cf8b9cdc7ab4e70488e4096684022a027f942",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "539cf8b9cdc7ab4e70488e4096684022a027f942",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 50022dcc45b0f511dd69706aeaf6464a51be49e2 UNKNOWN
* 6cc840c51fcecde95d2187fb6fd217b50fa82806 UNKNOWN
* 77589738d73b044385f5aa6f7ef87b421f06043b UNKNOWN
* b0054c8fd329489d9a1e9535d9866edffc52fdd0 UNKNOWN
* 363e54341420cc217bf5af62de1e09ff370f75d2 UNKNOWN
* a893da375eb9c65edabbd22965296eac9b3714f4 UNKNOWN
* 4a140fffa35893a13f2e7ac8365d60d51a2d0870 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7813)
* 500b4f529ad2f8a695825b5badc1815202164d0a Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7830)
* 539cf8b9cdc7ab4e70488e4096684022a027f942 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9622] Add implementation of MergeHandle backed by the HoodieFileGroupReader [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13699:
URL: https://github.com/apache/hudi/pull/13699#discussion_r2278056478
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/FileGroupReaderBasedMergeHandle.java:
##########
@@ -198,8 +270,10 @@ public void doMerge() {
}
// Writes the record
try {
- writeToFile(record.getKey(), record, writeSchemaWithMetaFields,
- config.getPayloadConfig().getProps(), preserveMetadata);
+ // if the record is not being updated and is not a new insert for the file group, we must preserve the existing record metadata.
+ boolean shouldPreserveRecordMetadata = preserveMetadata || record.getOperation() == null;
Review Comment:
I've updated the comment there, let me know if there are more details to add
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9622] Add implementation of MergeHandle backed by the HoodieFileGroupReader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13699:
URL: https://github.com/apache/hudi/pull/13699#issuecomment-3190438256
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7661",
"triggerID" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"triggerType" : "PUSH"
}, {
"hash" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7666",
"triggerID" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"triggerType" : "PUSH"
}, {
"hash" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7667",
"triggerID" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"triggerType" : "PUSH"
}, {
"hash" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7672",
"triggerID" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"triggerType" : "PUSH"
}, {
"hash" : "965136302f1188e5123de37734c927dc55801b24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7691",
"triggerID" : "965136302f1188e5123de37734c927dc55801b24",
"triggerType" : "PUSH"
}, {
"hash" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7693",
"triggerID" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"triggerType" : "PUSH"
}, {
"hash" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7717",
"triggerID" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"triggerType" : "PUSH"
}, {
"hash" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"triggerType" : "PUSH"
}, {
"hash" : "47f8303d6e61504275ecfddf553e356427e6c822",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7718",
"triggerID" : "47f8303d6e61504275ecfddf553e356427e6c822",
"triggerType" : "PUSH"
}, {
"hash" : "fb191ecf48d13cd81412535f7f8c9ed553d86517",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7725",
"triggerID" : "fb191ecf48d13cd81412535f7f8c9ed553d86517",
"triggerType" : "PUSH"
}, {
"hash" : "064912d3a54714935f2ee77d55cc66816166300f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7727",
"triggerID" : "064912d3a54714935f2ee77d55cc66816166300f",
"triggerType" : "PUSH"
}, {
"hash" : "77a2b45fe13dafc51f7bebb2ba859d40291ccc10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7736",
"triggerID" : "77a2b45fe13dafc51f7bebb2ba859d40291ccc10",
"triggerType" : "PUSH"
}, {
"hash" : "e4eda0e51fee0e8de5d57ea7b38dc642f53aa95d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7758",
"triggerID" : "e4eda0e51fee0e8de5d57ea7b38dc642f53aa95d",
"triggerType" : "PUSH"
}, {
"hash" : "6cc840c51fcecde95d2187fb6fd217b50fa82806",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6cc840c51fcecde95d2187fb6fd217b50fa82806",
"triggerType" : "PUSH"
}, {
"hash" : "31dab8ce9abbe4fc80e6cd81a0e433a039ca3e1d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7760",
"triggerID" : "31dab8ce9abbe4fc80e6cd81a0e433a039ca3e1d",
"triggerType" : "PUSH"
}, {
"hash" : "c04fd95dcc94146a9146c347f4f204656f3d8308",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7763",
"triggerID" : "c04fd95dcc94146a9146c347f4f204656f3d8308",
"triggerType" : "PUSH"
}, {
"hash" : "324147913d47ba8f2555e1991ce59905e5dc0b3f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7767",
"triggerID" : "324147913d47ba8f2555e1991ce59905e5dc0b3f",
"triggerType" : "PUSH"
}, {
"hash" : "3f177d42b831b30f4bd590b057dcab6ded78b634",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7774",
"triggerID" : "3f177d42b831b30f4bd590b057dcab6ded78b634",
"triggerType" : "PUSH"
}, {
"hash" : "8d6d2c06ec82d1072251ac12ddf21a890ddd5bda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7776",
"triggerID" : "8d6d2c06ec82d1072251ac12ddf21a890ddd5bda",
"triggerType" : "PUSH"
}, {
"hash" : "77589738d73b044385f5aa6f7ef87b421f06043b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "77589738d73b044385f5aa6f7ef87b421f06043b",
"triggerType" : "PUSH"
}, {
"hash" : "ad0d7ef0fbc525f880d3baadef5a22196ab97b32",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7781",
"triggerID" : "ad0d7ef0fbc525f880d3baadef5a22196ab97b32",
"triggerType" : "PUSH"
}, {
"hash" : "0db72caa3a6c7d0f0899f243ea4b8975de79e02c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7787",
"triggerID" : "0db72caa3a6c7d0f0899f243ea4b8975de79e02c",
"triggerType" : "PUSH"
}, {
"hash" : "a01ce748dc2d4cd1fd4858d1ecac8a486989e880",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7797",
"triggerID" : "a01ce748dc2d4cd1fd4858d1ecac8a486989e880",
"triggerType" : "PUSH"
}, {
"hash" : "3f35151e28bfa1dd50b31c6e5379d9769627afd9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7799",
"triggerID" : "3f35151e28bfa1dd50b31c6e5379d9769627afd9",
"triggerType" : "PUSH"
}, {
"hash" : "b0054c8fd329489d9a1e9535d9866edffc52fdd0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b0054c8fd329489d9a1e9535d9866edffc52fdd0",
"triggerType" : "PUSH"
}, {
"hash" : "363e54341420cc217bf5af62de1e09ff370f75d2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "363e54341420cc217bf5af62de1e09ff370f75d2",
"triggerType" : "PUSH"
}, {
"hash" : "a893da375eb9c65edabbd22965296eac9b3714f4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a893da375eb9c65edabbd22965296eac9b3714f4",
"triggerType" : "PUSH"
}, {
"hash" : "a7c2756e6a293cbba99bd4e821881e6b961c138f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7804",
"triggerID" : "a7c2756e6a293cbba99bd4e821881e6b961c138f",
"triggerType" : "PUSH"
}, {
"hash" : "4a140fffa35893a13f2e7ac8365d60d51a2d0870",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7813",
"triggerID" : "4a140fffa35893a13f2e7ac8365d60d51a2d0870",
"triggerType" : "PUSH"
}, {
"hash" : "500b4f529ad2f8a695825b5badc1815202164d0a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7830",
"triggerID" : "500b4f529ad2f8a695825b5badc1815202164d0a",
"triggerType" : "PUSH"
}, {
"hash" : "539cf8b9cdc7ab4e70488e4096684022a027f942",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "539cf8b9cdc7ab4e70488e4096684022a027f942",
"triggerType" : "PUSH"
}, {
"hash" : "85139a12b431931973c07d1b7f9256a1aae75669",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7831",
"triggerID" : "85139a12b431931973c07d1b7f9256a1aae75669",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 50022dcc45b0f511dd69706aeaf6464a51be49e2 UNKNOWN
* 6cc840c51fcecde95d2187fb6fd217b50fa82806 UNKNOWN
* 77589738d73b044385f5aa6f7ef87b421f06043b UNKNOWN
* b0054c8fd329489d9a1e9535d9866edffc52fdd0 UNKNOWN
* 363e54341420cc217bf5af62de1e09ff370f75d2 UNKNOWN
* a893da375eb9c65edabbd22965296eac9b3714f4 UNKNOWN
* 500b4f529ad2f8a695825b5badc1815202164d0a Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7830)
* 539cf8b9cdc7ab4e70488e4096684022a027f942 UNKNOWN
* 85139a12b431931973c07d1b7f9256a1aae75669 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7831)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9622] Add implementation of MergeHandle backed by the HoodieFileGroupReader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13699:
URL: https://github.com/apache/hudi/pull/13699#issuecomment-3190553717
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7661",
"triggerID" : "ca08e20d8bacc2355f15ae37fab3c1ef61a052af",
"triggerType" : "PUSH"
}, {
"hash" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7666",
"triggerID" : "7faf5b6e2d2d001e45356cdaa92394e54c0a2ce3",
"triggerType" : "PUSH"
}, {
"hash" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7667",
"triggerID" : "af88f9e4a548191ff0cba41db6ad3e60ba9ea43b",
"triggerType" : "PUSH"
}, {
"hash" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7672",
"triggerID" : "5507dfb0f3e741a539dbed0669d24dac9f1fc2b3",
"triggerType" : "PUSH"
}, {
"hash" : "965136302f1188e5123de37734c927dc55801b24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7691",
"triggerID" : "965136302f1188e5123de37734c927dc55801b24",
"triggerType" : "PUSH"
}, {
"hash" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7693",
"triggerID" : "7dbbe8212c828dc3cf2405a0634d4a0c643580e0",
"triggerType" : "PUSH"
}, {
"hash" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7717",
"triggerID" : "22721bae05acf8aa296bc5732af5a90d2afd3515",
"triggerType" : "PUSH"
}, {
"hash" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50022dcc45b0f511dd69706aeaf6464a51be49e2",
"triggerType" : "PUSH"
}, {
"hash" : "47f8303d6e61504275ecfddf553e356427e6c822",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7718",
"triggerID" : "47f8303d6e61504275ecfddf553e356427e6c822",
"triggerType" : "PUSH"
}, {
"hash" : "fb191ecf48d13cd81412535f7f8c9ed553d86517",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7725",
"triggerID" : "fb191ecf48d13cd81412535f7f8c9ed553d86517",
"triggerType" : "PUSH"
}, {
"hash" : "064912d3a54714935f2ee77d55cc66816166300f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7727",
"triggerID" : "064912d3a54714935f2ee77d55cc66816166300f",
"triggerType" : "PUSH"
}, {
"hash" : "77a2b45fe13dafc51f7bebb2ba859d40291ccc10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7736",
"triggerID" : "77a2b45fe13dafc51f7bebb2ba859d40291ccc10",
"triggerType" : "PUSH"
}, {
"hash" : "e4eda0e51fee0e8de5d57ea7b38dc642f53aa95d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7758",
"triggerID" : "e4eda0e51fee0e8de5d57ea7b38dc642f53aa95d",
"triggerType" : "PUSH"
}, {
"hash" : "6cc840c51fcecde95d2187fb6fd217b50fa82806",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6cc840c51fcecde95d2187fb6fd217b50fa82806",
"triggerType" : "PUSH"
}, {
"hash" : "31dab8ce9abbe4fc80e6cd81a0e433a039ca3e1d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7760",
"triggerID" : "31dab8ce9abbe4fc80e6cd81a0e433a039ca3e1d",
"triggerType" : "PUSH"
}, {
"hash" : "c04fd95dcc94146a9146c347f4f204656f3d8308",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7763",
"triggerID" : "c04fd95dcc94146a9146c347f4f204656f3d8308",
"triggerType" : "PUSH"
}, {
"hash" : "324147913d47ba8f2555e1991ce59905e5dc0b3f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7767",
"triggerID" : "324147913d47ba8f2555e1991ce59905e5dc0b3f",
"triggerType" : "PUSH"
}, {
"hash" : "3f177d42b831b30f4bd590b057dcab6ded78b634",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7774",
"triggerID" : "3f177d42b831b30f4bd590b057dcab6ded78b634",
"triggerType" : "PUSH"
}, {
"hash" : "8d6d2c06ec82d1072251ac12ddf21a890ddd5bda",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7776",
"triggerID" : "8d6d2c06ec82d1072251ac12ddf21a890ddd5bda",
"triggerType" : "PUSH"
}, {
"hash" : "77589738d73b044385f5aa6f7ef87b421f06043b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "77589738d73b044385f5aa6f7ef87b421f06043b",
"triggerType" : "PUSH"
}, {
"hash" : "ad0d7ef0fbc525f880d3baadef5a22196ab97b32",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7781",
"triggerID" : "ad0d7ef0fbc525f880d3baadef5a22196ab97b32",
"triggerType" : "PUSH"
}, {
"hash" : "0db72caa3a6c7d0f0899f243ea4b8975de79e02c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7787",
"triggerID" : "0db72caa3a6c7d0f0899f243ea4b8975de79e02c",
"triggerType" : "PUSH"
}, {
"hash" : "a01ce748dc2d4cd1fd4858d1ecac8a486989e880",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7797",
"triggerID" : "a01ce748dc2d4cd1fd4858d1ecac8a486989e880",
"triggerType" : "PUSH"
}, {
"hash" : "3f35151e28bfa1dd50b31c6e5379d9769627afd9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7799",
"triggerID" : "3f35151e28bfa1dd50b31c6e5379d9769627afd9",
"triggerType" : "PUSH"
}, {
"hash" : "b0054c8fd329489d9a1e9535d9866edffc52fdd0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b0054c8fd329489d9a1e9535d9866edffc52fdd0",
"triggerType" : "PUSH"
}, {
"hash" : "363e54341420cc217bf5af62de1e09ff370f75d2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "363e54341420cc217bf5af62de1e09ff370f75d2",
"triggerType" : "PUSH"
}, {
"hash" : "a893da375eb9c65edabbd22965296eac9b3714f4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a893da375eb9c65edabbd22965296eac9b3714f4",
"triggerType" : "PUSH"
}, {
"hash" : "a7c2756e6a293cbba99bd4e821881e6b961c138f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7804",
"triggerID" : "a7c2756e6a293cbba99bd4e821881e6b961c138f",
"triggerType" : "PUSH"
}, {
"hash" : "4a140fffa35893a13f2e7ac8365d60d51a2d0870",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7813",
"triggerID" : "4a140fffa35893a13f2e7ac8365d60d51a2d0870",
"triggerType" : "PUSH"
}, {
"hash" : "500b4f529ad2f8a695825b5badc1815202164d0a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7830",
"triggerID" : "500b4f529ad2f8a695825b5badc1815202164d0a",
"triggerType" : "PUSH"
}, {
"hash" : "539cf8b9cdc7ab4e70488e4096684022a027f942",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "539cf8b9cdc7ab4e70488e4096684022a027f942",
"triggerType" : "PUSH"
}, {
"hash" : "85139a12b431931973c07d1b7f9256a1aae75669",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7831",
"triggerID" : "85139a12b431931973c07d1b7f9256a1aae75669",
"triggerType" : "PUSH"
}, {
"hash" : "cc430c949b539450110da96c0eccb52ab5e0998e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7832",
"triggerID" : "cc430c949b539450110da96c0eccb52ab5e0998e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 50022dcc45b0f511dd69706aeaf6464a51be49e2 UNKNOWN
* 6cc840c51fcecde95d2187fb6fd217b50fa82806 UNKNOWN
* 77589738d73b044385f5aa6f7ef87b421f06043b UNKNOWN
* b0054c8fd329489d9a1e9535d9866edffc52fdd0 UNKNOWN
* 363e54341420cc217bf5af62de1e09ff370f75d2 UNKNOWN
* a893da375eb9c65edabbd22965296eac9b3714f4 UNKNOWN
* 539cf8b9cdc7ab4e70488e4096684022a027f942 UNKNOWN
* cc430c949b539450110da96c0eccb52ab5e0998e Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7832)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org